Getting some kind of significant interaction between the main Gems and the B Gems (Lapis, Peridot, arguably Bismuth) beyond the Cluster arc was something the show could have benefited from greatly, especially as the conflict with Homeworld came into clearer focus. Between the focus on the town, though, in addition to more serious issues like the sort of independent meddling that characterized Zuke, I don't think it was ever a serious possibility, which is a shame.
From season 3 onward the show was in a sort of continued/cancelled limbo, or at least if felt that way. The resolution of the mains series is testament to that, with topics the crew seem like they would have at least tried to treat with more gravity under different circumstances ending up rushed and unfulfilling. As other people have pointed out in this thread, it became too ambitious for its own good, failing to execute properly on almost all of what it set out to do, due in equal parts to the crew's inability to settle and follow through on certain themes and the limits on what they were actually allowed to do under corporate scrutiny.
At that point you have to evaluate each piece of the puzzle on its own merit. Hence, I think the farm side-side plot for Peridot and Lapis was realistically as good as it was going to get, and free of Zuke's cancer could have spent more time giving us those moments than obnoxious relationship drama.