Nefcy was passionate about her project in the first two seasons. S1A was heavily under supervisory control, but S1B through the end of the second season showed what SVTFOE was capable of. She definitely held on to the reigns tightly and didn't let things go overboard. Furthermore, Giancarlo Volpe was involved in the show. This is important because, as SVTFOE became more character- and plot-driven, his work on A:TLA and The Clone Wars would've been vital to making sure that each plot point was hit effectively and had follow-through to later episodes.
After BfM is when Volpe left, Cotugno was given more authority, and Alex Hirsch joined the party. This is where Nefcy relented a lot of her creative control to her protege and friend, and the result was immediately obvious by the tonal whiplash. Without Volpe being able to bring his skill set to the group, the writing descended into a mentality of "throw everything at the wall and see what sticks". Literally. I don't have it on me, but I'm sure someone has the picture the staff shared of the post-its where people put up show title ideas, revealing the writers had no idea what they were doing. Furthermore I don't think the writers were passionate about SVTFOE; they were just eager to have something popular to put on to their resume, especially after Hirsch signed on.
Meanwhile, fans of the series stayed devoted and were hopeful things would turn around. This change never happened. The Q&A further showed how big the divide had become between Nefcy and her fans. When random strangers are able to think of better ideas than what was officially established and never followed up on, there is something dangerously wrong with the production.
S3B and S4 are proof of that major error.