In prior of executing a magic with a Madou, the Madou will coat/surround the user with [Pure Energy]. In order, it goes like this:

Mage activate Madou -> Trigger Magic to execute -> [u]Madou coats/surround the user with [Pure Energy][/u] -> Magic is executed.

The third step is called channeling, in order to have the Magic to fully be under control of its user, the mage's body will have to remember the energy flowing and control it as if it's their own. It's like having an extra limb. This also why using magic wearing the user out. If this is a gun, it's like moving the bullet from magazine into the barrel before being shot.

As stated earlier, Esper emits [GPE] like a body heat. This why using Madou will harm them as two different energies will be mixed together right above their skin. 

Theoretically Esper are able to use Madou, but only if that Madou is designed for them.

-----------------------

Add:

Now that I think about it, all my attention and concern about the whole Madou and ESP concept is instead making this RP non-user friendly. I suppose inconsistency is nice once a while, letting more creative freedom runs wild.