I accept 1, but 2 is wrong in two regards.
Firstly, "is the foundation" is both murky and not necessary. Clearly people can do math independent of such a strong formal foundation - Laplace dealing with partial differential equations was 200 years before Cantor.
To correct it a bit, Set theory is a formal theory in the language of logic within which you can represent essentially all well studied mathematics. In that sense it can be seen as foundational to mathematics, if you value fitting all of math into one basket. But again, you don't need this "foundation" to study or even understand math.
This also already entails my second point, namely that set theory isn't a logical language, but theory phrased within a logical language. First-order predicate logic (the syntatic framework of the language) allows you to introduce new symbols for, in this case, binary predicate obeying some rules, and that's what usually happens when setting up a set theory. That set theory isn't the language itself.
Just like writing down explicit rules of chess in English doesn't make chess or its rule a language in itself - the language is still English and it has, in this case, real world semantics of e.g. chess boards.
Now to the last point, I feel "scholars of language" in the conventional sense tend to do a lot of studying of existing language, whereas mathematics formal language quickly settled on what was given to them (the formal part was basically cooked up in 1880 with Frege, with a previous long history of course, and entirely fixed by 1920/1930 with say Neumann/Church)
Mathematicians don't study their or other languages like "scholars of language" and they aren't even historians - they are users of it.
Coming back to 1, if you get a math degree e.g. in Germany, it generally falls under "Geisteswissenshaften", as in topics of the mind, not "Naturwissenschaften" (natural sciences), and in this sense I give you that point without complaint.
