Because people have hugely different versions of what they think true AT is, and as such there's lots of times where people dropped off. Some bailed when the show started getting "pander-ey" with stuff like What Was Missing and Fionna and Cake, others when it turned into Shipping Time, then some when the lore started getting serious, others still when there began being 2deep4U episodes. To different groups these things go against the spirit of the show and are egregious enough offences to ruin the experience.
To top it all off, you have some people who only came to the show because of some of these reasons, the exact reasons others say that the show was ruined, and then they got bored and left because there wasn't enough of these things for their taste.
AT is just a show with lots of moving parts and different aspects and some people get turned off by some of them.
Also I think it's because it changed a lot really quickly. Spongebob had like 5 or 6 years of consistent funnies, Simpsons longer, Steven Universe started getting into it's deeper side much earlier than this show, but AT had several pretty sizable transitions while being only 7 years old.