@Joshua Tamashii First, he doesn't know those secondaries you included. He is granted the use of those through the swords but they're not his magic to command without the swords. Erza is able to use fire when she uses her Flame Empress armor but that doesn't mean she knows fire magic.
Now, the thing that's bothering me is his history. It feels a bit contrived. He cuts down his friend during
sparring match? His friend? The character just kind of suddenly has this over the top temper out of nowhere, trying to cut down people who aren't doing him any harm. And after this event he's still prideful? No remorse whatsoever? And I know later he's supposed to learn the value of life from his teacher but that should be something he already knows. Not only from killing his friend but also from his upbringing. Living in a harsh environment where surviving is your top priority already instills that kind of thing in you. Plus it's not like his tribe are ruthless people considering he was exiled because, again, he killed his friend. So, clearly the clan knows the value of life. Not to mention his near death experience. It just feels like your moving a game piece around the board, unaffected by the obstacles and events that happen along the way, but then he suddenly collects $200 when he reaches GO ie all of character development comes during his time with his teacher instead of being properly built through all of his experiences.
Here's a fix. First, I think instead of discovering his magic abilities before the sparring match with his friend, it's during and in the heat of his frustration he unconsciously summons a real blade and accidentally kills his friend. Make this the event that makes him value life above all else. With this he's banished not only because he killed his friend but also because of his use of magic which you had established was evil to them. It was weird that you made that a thing but it went nowhere. Then also from killing his friend, he starts to fear using a sword but still has a deep passion for it since it's his last connection to his clan outside of a mark of disgrace on his hand. Then, the caravan finds him and all that. This time, because the passion he still has for the sword, seeing the styles and whatnot, he wants to try picking up a sword again and enters. He loses, not just because the man is more skilled but because his fear has made his weaker as a swordsman. Seeing this the man then takes him on as his student and over the six years helps him overcome his fear of the sword as well as the use of magic and the rest is history.
Hope I didn't come off as too judgy and that this helps.