how was it set up badly? Vegeta explains how pissed he was about Goku always hitting milestones before him despite being the prince, then Goku suicides vs Cell thus cutting off any chance Vegeta had in trying to one up him later in life.
When Goku came back from the afterlife, Vegeta realized this would be his final chance to make the peasant kneel before the prince. However, Goku became far more stronger while training in the afterlife due to not being restricted by a normal living body. Vegeta used Babidi to try to catch up to Goku's level.
Doesn't seem out of character to me. Vegeta clearly still had issues from being beat to near death during the saiyan arc, and then failing to avenge his race during the namek arc.