Well, since I, y'know, actually live in the inner city, and am actually black, unlike you, I can tell you that it's mainly because a lot of dudes are deadbeat dads and don't give a fuck about raising their kids.
Believe it or not, not EVERY black dude who's not in his kid's life is in prison.
A LOT of them just don't want to settle down and raise a kid, they're fine just paying child support.
In fact, I'd argue that a lot of the dudes who are in prison (outside of legit crazy fuckers) are more attentive dads than the guys who AREN'T in prison and just don't give a fuck.
But, since your only experience with the black community in the inner city is TV and movies and shit, I don't expect you to actually know much about it.