The background lore was fine. I'm not going to pretend it was some amazing, breath-taking epic for the ages, but it was colorful and interesting in a way that complimented the story being told. The core of the show was the ongoing struggles of the title character, with the various background information informing it.
My problem with it was not that there was too much of it, but more that it felt progressively less inspired and creative following season two. The diamonds and the empire worked fine as a vague, off-screen threat that you would occasionally see glimpses of but never the full picture. They felt more threatening when it was left up to your imagination, where their misdeeds are merely alluded to and their power is represented by more immediate antagonists acting as their agents (IE, Jasper and Peridot).
The first season made me think each successive season would have a representative of the diamonds acting as the main recurring antagonist (like Jasper), but then they start fully showing the diamonds and they're instantly de-mystified. They became mundane and lost their God-like aura.