[quote=@BlackBishop]
Aren't the Greymanes the pro-stormcloaks? The Battle-Borns would likely be tolerant of Altmer visiting Whiterun, unlike the Greymanes, unless im mixing them up.
[/quote]

Ah, I get ya. Yeah, the Greymanes are the Pro-Stormcloaks, but they're also dirt-poor. While I'm sure they can hire thugs if someone steals from them, I don't think they'd hire them to harass some random Altmer.