Look at Gerudo lore lol.
Like seriously read it. Those females give zero shits. I mean Zero. They breed with men so they survive, they just do not believe in gender role or the three goddesses. They only believe in the Sand goddess in, Thus the Spirit temple.
Now, if you look a page back, going there isn't completely out of the question.
But as my statement stands, no matter where you are from, we are all starting in the capitol. Understood?