This is what I've heard over time, some of it might be off-base but it seems to be the most common explanations and theories.
For early volumes (1-3):
Things were made in Poser, which wasn't exactly what you would call high-level animation software. It was mostly just something Monty had a preference for using. It was janky as hell (for example, all the shading you see in those volumes had to be added manually because Poser couldn't really do that itself too well, the character models were actually made with other software and then ported over to Poser later, and several parts of volume 3 were too much for Poser to handle and kept crashing in rendering), and having to spend time on the problems it caused instead of actual parts of the show along with budget and time restrictions led to the show looking like it does.
For later volumes (volume 4 and onward):
The show switched to something called Autodesk Maya, and if I'm not mistaken this was actually software Monty had expressed an interest in after seeing it used in the anime Black Rock Shooter. What you see in volume 4 is the inevitable growing pains of a big change like this. You will notice problems in Volume 5 as well, which stem from an absolute disaster of a production schedule that year, it simply being the longest volume of the show to date, money and time being spent on several character shorts instead of the actual show, and suspected (but I believe never confirmed) rumors that money was siphoned from RWBY (and other RT shows) to fund the show gen;Lock instead.
The show arguably starts looking more consistent in volumes 6 and 7, but still isn't at a level I would call good. Mediocre at best would probably be more accurate.