Sure at a lot of schools CS is dumbed-down, easy, trivial, whatever you call it, but from my experience, people coming from a non-CS background tend to be incapable of writing clear code, and this is well known among software developers and employers.
Physicists and mathematicians write code that is unmaintainable, ugly and horribly slow, having to work with one of you suckers is absolute hell. It would almost be better to outsource the development to someone in India whose English skills are so limited that he writes things like "With reference to my top, please see my bottom.", "Please use the backside entrance", and "I will have a fruity career in your steamy company." Sure a math graduate can prove some meme-tier useless algebraic-topology theorems, but when it comes to building real things and implementing solutions, you SUCK BALLS.
An employer will always prefer a CS graduate over a math graduate and will always pay them more and promote them faster. This is why CS is the best paid major by far -- it is easier and rewards you MORE.
All those precious hours working hard and proving meme theorems to flip burgers, while CS-chads make 6 figures while you're broke and burnt out, all that time living in your abstract world might have given you brain damage, I feel deeply sorry for you.
Sure you might find math enjoyable but is it really more enjoyable than anything else you could be doing ? Especially when in the US you have to pay thousands of dollars in tuition?