why don't you make it like Index, where Espers CAN use magic, but it causes them bodily harm. That ties in with your conflicting energy thing anyway.