They literally can though. When a wizard could, at any moment, appear out of thin air and murder you there is no way to respond. They hold a monopoly on violence so complete that our modern world can't even reach it.
They can read minds, control people, teleport, be invisible, have undetectable bugs, and simply kill you from any distance.
When your masters can use fear to control and have no threat of recourse their control is complete. These people do not need to sleep, can effectively not be killed except by one of their own, and can be everywhere at all times. This is the death of politics.