I think the primary aspect of the manga that keeps its foot in the wish fulfillment camp is that, though it was hard work, he doing something he loved. Not only that, it was work at something not normally seen as cool or mainstream or even understood by the general populace. There's a reason he's a nofriends kid: he was traumatized by a childhood friend telling him everything he loved was trash and unmanly. Readers could very easily draw parallels to their own hobbies and interests that might make others call them weirdos, nerds, or losers like most posters on /a/
Maybe that very line of thinking is just a way of relating to Gojou. However, the fact remains that this positively beautiful, cheerful, and lewd girl needs the experience he has from his lifelong niche hobby. I say that this can definitely be seen as wish fulfillment, at least in part, until the story shows us something new.