>>11089060I did physics. Currently doing a phd in quantum info theory.
I usually say that if I could do it all over again, I'd have gone straight for math, but I'm not sure it's that simple. Were the mandatory labs annoying to someone who knew he was going to pursue theory? Sure, but I'm glad I did it. Hell, I worked in an experimental optics lab for ~1.5 years AFTER working on a project in quantum info.
If you want to do physics, you need to know how the science - the "real" part of it, taking measurements and analyzing data - is done, even if you never actually step foot in a lab. You don't have to love labwork - I certainly didn't. But you have to appreciate its importance and come up with theories that can be practically tested, implemented, and measured.
And that's why I think that, if you want to do physics, you should do a physics degree. Yes the math is not rigorous, and that bothered me at times, but once you learn the nonrigorous stuff, it's easy enough to find a reference to learn it the "proper" way. (And then 9 times out of 10 you realize that the specifics never mattered because e.g. the rigorous version gives you the exact same answer but you just had to define a couple of extra objects that the physicists had implicitly assumed to exist anyway.) On the other hand, thinking like a physical scientist can be quite different from thinking as a mathematician, and this is the most important knowledge you can take away from an undergrad education.
tl;dr math is really elegant and beautiful, but it is not physics. If you want to do physics, even theoretical physics, you should study physics. If you want to study mathematical objects and don't care about how they describe the physical world, study math. You can always fill in gaps as you go, but you should choose which foundational "way of thinking" you want to pursue. Do you want to be a scientist, or a mathematician?