Better than you might expect.
Gems have been shown to survive large blunt impacts, such as a high speed fall, but smaller piercing attacks, such as the training sword, can pierce them and destroy their hard light bodies. The gems themselves are also not incredibly durable; Amethyst's was cracked by falling on a rock at no less than terminal velocity.
I suspect their durability to wide impacts over more focused pressure is because gems are adapted to high-G space travel, but aren't actually adapted to war.
With this evidence, we can assume they can be harmed by conventional weaponry.
It's hard to gauge a Gem's exact combat ability compared to a human due to the fact we haven't seen an expert fight seriously, but it's notable that Pearl, the CG's dedicated fencer, was put on her toes by a 12 year old girl with only a few weeks of training, so it's not inconceivable that a well trained human or group of humans can overcome a Gem in CQC, especially if better equipment is developed.
The Gem's specialized weapons, while powerful, to almost always be melee, putting them at a severe disadvantage because modern humanity prefers ranged combat in the first place.
Moreover, Gems are just plain awful at battle strategies. They clearly rely too heavily on overpowered war machines and don't seem to have experience with combat against anything that can put up a fight. This is why we've only seen one soldier, an engineer, and half a dozen retards come in to even maybe try to apprehend the group of Gems that killed one of their world leaders. An Earth invasion would be MAYBE a few dozen assorted gems.
And while Gem tech is undeniably advanced, I'm sure it wouldn't take long for humanity to steal and reverse engineer it.
The only issue is that I'm applying this logic to real humanity; humans in SU are somehow more incompetent than the Gems themselves.