I guess this is the kind of thing where we have to consider basic instincts/desires VS conscious moral judgements and consequences of our own actions.
As Freudanon points out, virtually everybody desires child-like traits in their partner, or wishes to "be a child" in their sexual relationship with "an adult".
That attraction to immaturity is fairly natural and universal.
However, we are aware that actually fulfilling that desire can seriously hurt the child. An actual, real relationship between an adult and a minor is almost always exploitative and manipulative, since the adult has far more power and influence. Sex is also closely tied to emotional attachment, especially in young, developing brains.
Both the immediate experience, or the regrets about it that come later in life can be very damaging to the minor's psyche.
Therefore while most people fantasize about "fucking kids", they are also responsible enough to know that they're not supposed to be doing that, and instead go for emotionally mature adults who simply act out our fantasies, or resort to fully fictional pornography.
On the flipside, many people take it too far, and rather than accept that sometimes we, as dumb apes, desire things that are harmful, they instead pretend that they definitely 100% never do any wrongthink, only have 500% objectively moral and noble tastes. And if you even remotely suggest ever having had a bad thought, you're evil and need to be destroyed.