How do we know the really high level theoretical physics like the big bang theory, string theory, theory of relativity, anything about black holes, etc. aren't just a bunch of made up nonsense from a secret society of Platoists that know the actual truth of the universe?

It seems like the amount of people that are actually smart enough to be able to actually do the math and prove this stuff is so small that it seems like they could all form a tight knit secret society without anything getting out. They might keep the actual truth from us brainlets because they don't think we'd be able to handle it i.e. the shadow puppets in the cave metaphor.