No, it wouldn't because not all skills will be learnable at level one. Some may be level three even higher at the highest element affinity, and such skills could be beyond the reach of anyone who doesn't have top tier affinity. 

But, on the flipside, there will be skills that require both elements. And if you put all of your points into one, you won't be able to use them at all either. 

This would basically make there three possible paths for each class. 

If I do just revert to the skill point idea, I will still put element point limits on some skills. So that there's upsides and downsides to your element choices. But, if I do that I will also give out some arbitrary number of element points with each "level up." 

I'd like to hear more people's thoughts/questions about these things.