To be perfectly honest, yeah. I guess that's true for me. I get off more to cuteness and innocence than the loli aspect.
For example: Take a cute loli character and draw her in the most mature and slutty way imaginable. Tattoos, piercings, heavy makeup, a cigarette, black leather and high heels, stepping on a guy's back while doing her own taxes. My interest is severely lessened. Take a 30 year old woman and draw her as a cute girl next door type in smiling brightly and wearing something loose and comfortable, add a dash of naivete and innocence, boom we're in business.
Loli body type and the idea of youth just adds to it. But that alone isn't enough. That's probably also why I don't find 3D at all interesting because it has that inherent super duper creepy vibe. Plus toons are just brighter and cuter. Same goes for incest relationships. It needs to play up that childhood friend angle.