The thing to remember is that FAN is short for FANATIC. Generally speaking if you like something to that degree you're already kind of weird.
Most people who like stuff even something with as crazy a fandom as say Sonic or MLP don't go around talking about it outside of their friends and even then only casually.
The people who just like it are probably generally cool and not weirdos. It's just that the weirdos stand out more becuase of how far they take it.