@Saarebas As durability goes I'll quote superman kinda, mundane things like a normal punch or sword stab wouldn't do much but add a little magic and he loses a lot of his durability. So like gem abilities would hurt him if it got past his shield and people stronger than he is would be able to hit with greater force than he can protect. Also fire or acid, while not highly damaging in and of themselves, are incredibly painful and would take him out of the fight for a bit.
As for the Rose Quartz and Homeworld Gem thing that's not a terrible idea, that wasn't what I was trying to imply. More like "I destroyed an important thing and/or person and really regret it." kinda deal.
[EDIT]
Upon further reading I can see how you thought that he fought Rose, I was thinking of someone else entirely. A character that I made up, sorry for any confusion.