The problem was that Ultron was trying to be self contained. Where as Phase 1 had the previous 5 films set up the Avengers, Phase 2 didn't have any of the films set up AoU. Like, remember when Iron Man 3 ended, and everyone thought that Tony was going to build drones instead of using his suits? They could have had the Avengers get the staff, come home, and Tony is tinkering with Ultrons.
But instead, we had a film that had noticeable cut scenes (Like how Scarlet Witch and Quicksilver enter the church to find Ultron), with lots of teasing to future films (Ragnorok, Infinity Wars, Black Panther) and somehow the film felt overstuffed at 2 hours and 30 minutes.
What the hell happened?!