This is a simple theory, but I hope you guys like it. Firstly, let me state the facts it is based on:

Queen knows Judge.

Judge has worked with Vegapunk.

Vegapunk experimented on Kaido.

King’s uniform has the same skull symbol as Shiryu’s.

So here are my inferences:

Queen worked with Vegapunk and Judge. When World Government captured Vegapunk, Queen also got captured, but Judge managed to get away. While Vegapunk didn’t mind following orders of World Government, Queen didn’t want to, but still did because else they would either kill him or incarcerate him.

During this time, Kaido got captured by the marines, and wherever they put him, King was the warden of that place. World Government had a problem with King’s sadistic nature, but they let it slide because he was strong. King tried to torture Kaido but failed, which made him kinda respect (rival like) Kaido.

And when World Government asked Vegapunk to experiment on Kaido, Queen was with him too. Queen realized that the marines were barely managing to hold Kaido. He saw him as his ticket to freedom, and somehow sabotaged the shackles (or whatever they were using to contain Kaido).

King knew that this was the last straw and he would no longer remain a warden, so he switched sides and joined Kaido and Queen. Jack maybe wasn’t even born at this time, he joined later.

Theory by shadesrt