Me and my buddy are creating a roleplay that is going to involve magic, and I was wondering if anyone could give me advice on how to keep it under control? Make sure people aren't abusing their magical abilities and are not overpowered?

There are many way to avoid this:

1. Make magic (under control of humans/elves/other sapient species) extremely rare and give it only to those which you know will not misuse it.
2. Talk with the roleplayers and set clear and non-breakable rules about magic.
3. Establish one of your characters as being the user of the most powerful magic in the setting as well as rules about magic.
4. Create a setting in which no matter how ridiculously powerful magic one has you have a character or rule which makes it completely useless.
5. Create a setting in which being magically overpowered is only important for your income (AKA a setting not concentrated on fights or even without fights at all).
And on and on...

So there are many ways to prevent or stop roleplayers from being magically overpowered trough to be true none of them guaranties 100% success rate. How prone are people to making overpowered characters and how tightly you have to tie the leash to prevent them from doing that mostly depends on what kind of roleplay and roleplayers are in question.

Sure. Magic that forces anyone's character to do anything they don't choose to do (overt mind control, blood boiling in their veins, transmuting their flesh into potassium ferrocyanide in strong acid to produce hydrogen cyanide gas, etc.,) is explicitly not allowed. Anything that causes a single character to be able to lay waste to an enemy army? Not allowed. Anything that forces you to use your powers creatively in order to maximize your potential damage output should be encouraged (ie, a single fireball cast upon the surface of an ancient, rickety, near rotten wooden staircase an enemy force of, say, forty soldiers is running up in order to get to you might do more damage than a fireball forty times the size of the original example would to half the number of soldiers on an open field) and, most importantly, if someone's abusing their powers, tell them to knock it off. It's your project, and if you feel someone's overstepping their bounds do something about it.

EDIT:

Additionally, I love your avatar. Just sayin'. <3

EDIT EDIT:

Also, asking for a detailed explanation of a the depth of a mage's powers in their character sheet can stop most problems from occurring in the first place. If you come across a character whose list of powers includes anything dubious or outright overpowered it's best not to allow them without edits, and should someone start using powers that aren't listed in their character sheet (or, at least aren't within the spirit of the character's abilities. A mage who can throw a fireball can probably light a cigarette with a smaller, more controlled flame, but if said spell is the strongest thing they've got listed it's unlikely they can rain meteors down upon their foes) it's easy enough to remind them that they themselves chose not to have such a breadth of sheer damage output.

EDIT EDIT EDIT:

Your name's wonderful, as well. Chemistry jokes need to happen more.

My mechanism for keeping magic under control for one RP idea I had was to have it so that there was very little magic in existence when the plot started, but that it would come back in an apocalyptic/major plot event.

Thank you all for the advice! I'll take what I can from this and share with my friend. And thank you Psycho, it is pretty awesome lol.

#1: Keep it simple.

#2: Output = Input. In order to toast two guys, you need to spend the time gathering the power for that spell.

#3: Force similar projectile rules to those of bows, guns, etc. Balances ranged spells very well.

#4: Newton's law: Equal and opposite. If a spell would blow someone twenty feet away, the caster also gets blown twenty feet back unless they use more magic to reduce or negate this effect. In which case, refer to #2.

#5: Have magic defense items! If they're very powerful, add magic penetration items. Arms races can be interesting to play.

Enforce mana pools and tie it to the amount of energy exerted or matter produced. Coupled with detailed explanations of the magic's workings, you will be able to easily regulate a characters power.

They throw a molten bolt of tungsten, heated to 5500 celcius, at mach 4 speeds? Okay, but producing the metal, which is pretty dense, heating and accelerating it will take a damned lot of mana to do. Casting a napalm fireball? Okay, it has less damaging potential, but napalm is lighter and needs lower temperatures to ignite, so the spell is overall cheaper.

Also, be strict with the diversity and range of spells: the more they have in their arsenal, the less proficient they must be with each spell, and vice versa. Including such a thing as "schools of magic" may also help, forcing the player to wisely choose, rather than just take everything they want and throw off the roleplay's balance.

Honestly, I would just make it clear that you don't expect magic to be too powerful. Don't accept anyone that you think is taking it too far, and if someone does something out of line give them a talking to and make them retcon their post. It's a much more surefire solution than imposing restrictions on the magic system itself, because it's almost always easier to find a loophole than to plug them.

Cadvin said
Honestly, I would just make it clear that you don't expect magic to be too powerful. Don't accept anyone that you think is taking it too far, and if someone does something out of line give them a talking to and make them retcon their post. It's a much more surefire solution than imposing restrictions on the magic system itself, because it's almost always easier to find a loophole than to plug them.

On the few times I've had magic play a part in an rp, this is pretty much what I've done. "Use common sense." If there's an issue with me feeling that the magic used is overpowered would be dealt with in a pm to the person.

I've had 2 games I've gm'd that dealt with magic.

One was meant to be an over the top exploration of a fantasy world. You played regular people thrown into a fantasy setting, essentially waking up to find that all of the sudden you're in a "video game" fantasy world. Firebomb spells that could wipe out a village? Sure, why not. It was meant to be (darkly) funny more than anything else.

The other game was more on the serious side of things. Magic was rare and not well understood. Anyone born with magic only had 1 talent (as I called it), 1 specific thing they could do, and it could be just about anything barring time-space manipulation (I just find that doesn't work well in rp'ing). With each magic ability though, came an innate repercussion to the user. One char, for example, was able to alter memories of another person. However, each time she did so, she lost a memory of her own. Sometimes big important stuff, other times something insignificant. That was the balance.

If everyone's overpowered, then no one's overpowered. (i.e. every single PC being OP is actually pretty balanced)

Add some magical laws. either those described earlier, or ones like "obliterating stuff means you get obliterated" or things like that.

Another option is to make them balanced with another opposite power, where they can negate each other. (fire/water, "earth/air", Life/death, "Healing/sickness", etc.)
Or you could add side effects. like: burn something, you get a fever. Freeze something, you get chilled. hurt something, you feel the pain. if anything, that'll limit most magi easily. of course, if that doesn't work and they still make their magic too powerful? Kill the PC.
If the RP isn't deadly serious... Make at least parts of magic a joke.