There’s also the idea of destabilizers. Disrupting their energy forces a reboot and causes the sensation of pain, even if no actual damage is dealt to their Gem or programming.
The Rejuvenator disrupted their code by forcing their life experiences into a kind of hard drive partition, which had the same effect as energy disruption; forcing a restart.
Blunt damage like being stepped on it falling onto a rock works, but only directly to the Gem itself. No Shatterings occurred this way, so it may be easier to crack than fully Shatter. Amethyst worsened over time to nearing a Shattering while Lapis was stable in her cracks, so being Overcooked may make them more brittle. The exception is Peridot who took a fucking Boulder to the Gem, and the Diamonds who’s Gem placement probably protects them to a degree.
Maybe light projections in the form of limbs and weapons are one-way so energy disruption doesn’t work? So cutting off a hand like Ruby’s destabilizing in the prison forcefield doesn’t Poof, but Pearl getting a holo- sword in the torso does. Pearl tried to stab Peridot directly in the Gem on the Era 1 ship, but maybe her spears can’t Shatter? Or maybe Peridot’s visor protects her Gem.
All that said, I’m picturing inside Peridot’s Gem being like the original Star Trek bridge with the mental Peridots just keeping power on the shields at all times.