IQ might be a poor metric for intelligence, but what these people are saying is certainly plausible. I'm not going to argue about the validity of IQ as a metric because I haven't studied it, but you don't have to be a genius to realize it's hard to connect with people that can't follow your lines of thought.
Sure, you can have small talk with other people and make a couple of jokes, but that doesn't form strong relationships. Maybe you don't even find the same things funny (they laugh at 9gag memes).
The bigger gap comes when you try to discuss any topic. They dismiss your points because they don't understand your tools of analysis or because your points don't fit the rigid mantras they've been taught at school or by the media, and you get frustrated because you're failing to communicate your thoughts. Dumbing down the discourse sometimes works, but others it doesn't because you can't really dumb it down without information loss.
This is especially troublesome in work relationships (having a boss who can't appreciate your work is painful), but it can also be a problem in social relationships. A friend is not just someone you drink beers and party with. A girlfriend is not just someone you fuck. You expect to be able to have some conversations with your friends where there's some exchange of information. You want to talk about the movies you've watched, the albums you've listened to, the books you've read, etc.