Here is a question that has been rattling around my head for a while.
Most fantasy roleplays that feature monsters also have ordinary animals. Everything from the typical farm stuff like chickens, cows and pigs to cats, dogs and snakes. Even more dangerous animals like wolves, bears and sharks may appear. But then we take a step into monster-territory, because somewhere between an ordinary wolf and the dire wolf there is an invisible line.

So I've been thinking. The line is obvious to us players and readers. Animals are the creatures that exist in the real world and monsters are the ones that doesn't. But how about the characters in the game/book/movie? Surely, to them the difference between a wolf and a dire wolf is just size, just like between a bear and a grizzly bear. So how does one define a monster in a world where both animals and monsters are natural?

It depends strongly on the scenario. Often monsters are fauna with a supernatural element, or otherwise not very well-known or widespread. Even in historical fiction settings, say Moby Dick, something like an albino whale might be considered a monster simply because it's a dangerous animal that's also unique in some way.

The culture of the setting plays into this as well. Animals to whom superstition has been attached will be considered monsters no matter how 'mundane' they are, such as, say, the Aye-aye lemur in Madagascar. It's a funny-lookin' little goblin, but despite posing no threat to anyone, it's frequently killed when spotted because it's considered a harbinger of evil.

I think generally if the distinction is actually just between animals that exist in our world and animals that don't, that'll be either a deliberate move by the author to establish the nature of the 'monster', an accident where the author didn't consider the perspective of the setting's native characters, or, most interestingly, an implication that something has changed to introduce the 'monster'- Something about these non-Earth animals is out of place in the setting.

@Antarctic Termite
Supernatural fauna - This is an interesting point of view. I do believe it to be flawed however because of how broad it is. If we would take every "animal" that has a supernatural element to it and place them in the "monster" category, we'd have practically no animals. Cows are supernatural, as are pigs, goats, cats, wolves, bears, crows, doves, swans, rats and a lot of other animals. But is it fair to put them in the same category as dragons, werewolves, minotaurs and yetis?

Cultural influence - This is of course very important, all of the animals I listed above are obviously not supernatural according to the same culture, but different ones. But it also introduces the idea of monster/animal as subjective. According to the people of Madagascar the lemur is a monster, but to a Chinese person it would simply be a monkey, an animal. So far so good, but I feel that there is still a line that separates the subjective monsters of albino whales, bigger versions of known animals (grizzly bear/dire wolf/giant bat) and freaky lemurs from the objective monsters like dragons, werewolves and zombies.

Bad planning from the author - Yes, I would say that this is at least 95% of the issue. Either the author doesn't even try to, can't or stubbornly won't justify the difference. Simply going with the "natural vs supernatural". But if monsters exist in the world they have to be natural.
I like where you're going with your last point, that something has changed to introduce the monsters. That they are somehow created or brought to the world from some other plane of existence.

I've personally never seen a dire wolf called a monster. Not saying it doesn't happen.

My first reaction to seeing a question like this is to simply look at a dictionary. According to dictionary.com...

1.
a legendary animal combining features of animal and human form or having the forms of various animals in combination, as a centaur, griffin, or sphinx.
2.
any creature so ugly or monstrous as to frighten people.
3.
any animal or human grotesquely deviating from the normal shape, behavior, or character.
4.
a person who excites horror by wickedness, cruelty, etc.
5.
any animal or thing huge in size.
6.
Biology.
an animal or plant of abnormal form or structure, as from marked malformation or the absence of certain parts or organs.
a grossly anomalous fetus or infant, especially one that is not viable.
7.
anything unnatural or monstrous.

As you can see, most if not all of Antarctic Termite's points were true. The word has been around for a long time, and has gained a lot of different meanings. The one over branching detail is that monsters have the power to scare most humans.

Here's what I came up with after thinking on it for a while:

A monster is any non-sentient creature that goes out of its way to menace and attack other creatures in and outside of its ecosystem for reasons that are not natural. Signs that you may be dealing with a monster are:

1. An inordinate number of the creatures kills are not consumed or used in any way.
2. The creature is actively destructive to its ecosystem and/or environment.
3. The creature demonstrates a willingness to go above and beyond what would be expected of a predator in pursuit of food, such as a willingness to scale fortified walls to kill those living within or attacking armed sentients when it is clearly strong enough to hunt for easier prey.
4. Persistence to the point of recklessness.
5. A willingness to incur injury in order to attack others.
6. A willingness to fight on even having sustained wounds that would cause a normal animal to abandon its hunt.
7. Any behavior that can be construed as "cruel" or "malicious" on the part of the creature where a rational explanation for the behavior can not be found.

I've personally never seen a dire wolf called a monster. Not saying it doesn't happen.
My first reaction to seeing a question like this is to simply look at a dictionary. According to dictionary.com...

1. a legendary animal combining features of animal and human form or having the forms of various animals in combination, as a centaur, griffin, or sphinx.
2. any creature so ugly or monstrous as to frighten people.
3. any animal or human grotesquely deviating from the normal shape, behavior, or character.
4. a person who excites horror by wickedness, cruelty, etc.
5. any animal or thing huge in size.
6. Biology.
an animal or plant of abnormal form or structure, as from marked malformation or the absence of certain parts or organs.
a grossly anomalous fetus or infant, especially one that is not viable.
7. anything unnatural or monstrous.

As you can see, most if not all of Antarctic Termite's points were true. The word has been around for a long time, and has gained a lot of different meanings. The one over branching detail is that monsters have the power to scare most humans.

Yes, that is the dictionary definition. It does not answer my question at all and only helps very little. It kind of seems like you didn't actually read the question at all, only the title.
I'll go through them all to show you why they does not apply to this question.
Also, first place I looked. Most famous rpg-system in the world, a whole bunch of dire animals listed as monsters. Link: d20srd.org/indexes/monsters.htm

1. A legendary animal combining features of animal and human form or having the forms of various animals in combination.

I can see this as a subcategory of monsters. There are quite a few that seems to be some kind of mix between several other animals. However, would you really say that a sphinx or chimera is the same as a platypus? I would argue that they are different.

2. Any creature so ugly or monstrous as to frighten people.

Complete nonsense. I know lots of people who are afraid of spiders and snakes. Those two are the most common phobias in the world. Does that make spiders and snakes monsters in a fantasy setting? Because that is something I have never seen.

3. Any animal or human grotesquely deviating from the normal shape, behavior, or character.
And...
4. A person who excites horror by wickedness, cruelty, etc.
And...
6. Biology. An animal or plant of abnormal form or structure, as from marked malformation or the absence of certain parts or organs.
A grossly anomalous fetus or infant, especially one that is not viable.

So a dragon is only a monster if it doesn't have wings? Because they are supposed to look like that. These definitions leaves every type of monster out except the mutants.
Also, when people are referred to as monsters one generally talks about certain behaviour. My question is about the split between animal and monster, not society's view of correct or incorrect behaviour.
Only a small part of these definitions can be seen as relevant, the behaviour of an animal, which the gentleman below has covered in more depth.

5. Any animal or thing huge in size.

This point was addressed in the first post when I claimed that a dire wolf would be seen in the same way as we see a grizzly bear.

7. Anything unnatural or monstrous.

Have to love the “thing - something like a thing” definition. This one doesn’t work either both because the definition is restating itself, but also because of what I said in the op:

Animals are the creatures that exist in the real world and monsters are the ones that doesn't. But how about the characters in the game/book/movie? Surely, to them the difference between a wolf and a dire wolf is just size, just like between a bear and a grizzly bear. So how does one define a monster in a world where both animals and monsters are natural?

Here's what I came up with after thinking on it for a while:
A monster is any non-sentient creature that goes out of its way to menace and attack other creatures in and outside of its ecosystem for reasons that are not natural. Signs that you may be dealing with a monster are:

1. An inordinate number of the creatures kills are not consumed or used in any way.
2. The creature is actively destructive to its ecosystem and/or environment.
3. The creature demonstrates a willingness to go above and beyond what would be expected of a predator in pursuit of food, such as a willingness to scale fortified walls to kill those living within or attacking armed sentients when it is clearly strong enough to hunt for easier prey.
4. Persistence to the point of recklessness.
5. A willingness to incur injury in order to attack others.
6. A willingness to fight on even having sustained wounds that would cause a normal animal to abandon its hunt.
7. Any behavior that can be construed as "cruel" or "malicious" on the part of the creature where a rational explanation for the behavior can not be found.

I really like this list you've made, there has been some thought put into this one. Just to boil it down, you're saying that the line between animal and monster lies not in its features or heritage, but behaviour. It’s a very interesting thought. The only one I could go against is really number two, could you elaborate on this? Also, would you say that monsters cannot be reasoned with?

The only one I could go against is really number two, could you elaborate on this? Also, would you say that monsters cannot be reasoned with?

Sure. The creature is a malignant force even where it lives. Maybe it's knocking down every tree in its path, or eating so much that the other creatures living in the area can't get enough food for themselves, or its got some sort of potent venom that leaves areas it moves through barren or is seeping into the water supply and making everything sick. The point is its very existence is not healthy for the environment it finds itself in.

Now, you might be wondering what differentiates that from an regular invasive species and there's really nothing. Most of the signs can be found, on an individual basis, even in cases of real world animals. That manticore haunting the woods might be perfectly adapted to its native environment and was just driven over the mountain by something or carted in with a traveling circus. In that case its still an animal. A hungry, confused, and desperate animal that should probably be put down for the good of the area, but still just an animal. That's why these are all just signs. To my mind, yes, the difference is behavioral and with monsters the behavior you're trying to identify is maliciousness. Animal act violently out many things; fear, pain, hunger, a desire to mate, a desire to protect their territory or offspring.

A monster acts violently primarily out of malice and since you can't talk to an animal and ask what it's feeling and why it's doing the things it is all you can do is look for that malice in its behavior and make your own best judgement about what it is you're dealing with. That's also why they can't be reasoned with any more than you could ask a staving tiger to please not eat you. There's no higher thought to their actions. A hungry tiger will eat you because that is the nature of a hungry tiger. A monster will kill you just to see you dead because that is the nature of a monster.

Yes, that is the dictionary definition. It does not answer my question at all and only helps very little. It kind of seems like you didn't actually read the question at all, only the title.

Your question was "how does one define a monster in a world where both animals and monsters are natural?" And my answer was that monsters are natural in this world too. But you are allowed to disagree if you'd like. I really wasn't trying to help, only explain why the word monster works for me.

Depends on the setting.

one can say stop and the other is feral

I see the nature of monsters being more varied than the nature of monsters. For example, while animals act solely off instinct with very little deviation, the term monster in my mind infers it has something beyond that. A monster might just be a malicious animal, one that knowingly acts with cruelty, but that is what sets it apart. A monster might also just be a beast or being with extraordinary capability beyond the accepted norm, but viewed rightfully or wrongly as frightening or fearsome.

A zoanthrope could be just as much a monster as it isn't. Just because it has the capacity to tear a man apart, and many chose to do so, does not mean they all are "monsters". Are they monstrous? That I would agree on by virtue of their ability, regardless if they can exercise it or not, but it again does not qualify the exceptions as monsters just by that standard.

I bring this back around by citing my example of exceptional ability beyond the norm. Would a dragon be any less a dragon if it were just a flying lizard that could breathe fire? Still very much a "dragon" even if it had nothing but animal intellect and instinct. Why? Because the qualities it expresses greatly surpass the accepted normalcy. Does that rightfully make it a monster in our example? No, not really, but the perception is different.

This goes even deeper still when we start talking about legitimately intelligent monsters; people. Regular people are perfectly able to "devolve" into monsters based upon our past, current or even future standards. Why? Because they again either give the perception of, or worse yet, carry out heinous, grievous, horrific or brutal acts. Things considered too bad for human or animal standards. They go above and beyond the expectations of regular disdainfulness.

This might be wrongly placed, even positive things could be considered "monstrous" because of the perception of the persons involved. To give an anecdote relating to roleplaying, as appropriate to our forums here, I once was playing a human character alongside others. What they did not know, in our roleplay barring the Game Master's knowledge present, was that my character was endowed with psychic powers, specifically those relating to psychometabolism and alteration. When we finally found ourselves in a bad spot otherwise beyond hope, I had my character carry them out.

Whereas most the people were just shocked, one person out of character was adamant that what happened was "evil" and "monstrous" and that my character was no longer a person. Granted this translated into their character's behavior as well, but just the thought in reality that I would do or enjoy such a thing, even in fiction, was "wrong". That the concept made the character and me, by virtue, something less than human.

Short of this, there is not much I can add to how I determine the differences. In truth, I almost explicitly play "monstrous" beings or those animalistic and savage.

I agree with Harbinger, and many others. A Monster is just as much an animal as animals are, it's intent and it's desire to act upon it's monstrous side makes it a monster. You could say a rabid dog would be a monster due to it's feral nature. But let's talk Elephants the first people who saw these animals outside of their native habitat likely thought them monsters. Since few creatures look the way an Elephant does. They're massive animals with a thick trunk and two long tusks. Anyone who's never set eyes upon them as they first charged the battlefield likely would have thought them demons, with a arm that could crush a man and spears for teeth. Now we know better but the fact remains anyone laying eyes upon a creature so odd and so massive would easily confuse it with a monster just upon looks alone.

Now there's also another detail no one's talked about. How easily can a normal man kill a monster. In many Games, Fantasy and Myth it's Heroes, Adventurers or Knights. Men or Women who are highly trained and equipped to slay monsters. And that's what makes a monster dangerous, your average Farmer cannot hope to slay a Cockatrice, a Manticore or other such creature. While they can more or less kill Wolves with some difficulty and maybe Bears if needed albeit with more danger. Monsters kill average Men, it's why Geralt of the Witchers is called upon, because he can slay beasts no other can. Why it's always been Heroes to kill the Dragon. The average man cannot hope to deal with such creatures, without specialized weapons and equipment. But most animals can be slain by the hands of a single man, be it with guns or spears.

2. The creature is actively destructive to its ecosystem and/or environment.

Why does that sound distinctly human?

@ArenaSnow, as I even referred to, people can degenerate into monsters be it knowing or unknowing.

@ArenaSnow, as I even referred to, people can degenerate into monsters be it knowing or unknowing.

I disagree. People can never be monsters. Whatever they do their actions are the actions of a person. To dismiss them as monsters is to not only dismiss a facet of human nature just because we find it uncomfortable to examine and try to understand, but also helps to absolves them of culpability for what they've done by turning them into something more on the level of an animal.

<Snipped quote by The Harbinger of Ferocity>

I disagree. People can never be monsters. Whatever they do their actions are the actions of a person. To dismiss them as monsters is to not only dismiss a facet of human nature just because we find it uncomfortable to examine and try to understand, but also helps to absolves them of culpability for what they've done by turning them into something more on the level of an animal.

I disagree on the basis that humans are simply a level of animal that have deluded themselves into civilization.

People are monsters. How much they try to excuse themselves otherwise depends on the individual, but we have all hurt and done damage without noticing or caring many a time.

But to go further into that would be to simply say, new topic.

<Snipped quote by The Harbinger of Ferocity>

I disagree. People can never be monsters. Whatever they do their actions are the actions of a person. To dismiss them as monsters is to not only dismiss a facet of human nature just because we find it uncomfortable to examine and try to understand, but also helps to absolves them of culpability for what they've done by turning them into something more on the level of an animal.

As with @ArenaSnow's proposal that human beings are just animals that have seen themselves into orderly society and think their actions within are so "excusable", I am of the firm belief that people can, will be and in some cases are monsters. Just because they are a "person" does not excuse mindsets of deplorability; if that makes something else "monstrous" in a vague, overarching concept what excuse do people have? Quite frankly, none. If anything they must be held more accountable given they have a much greater a place to fall from and how much further low they can go.

I've been planning on responding to this topic for a long while now, but (I'm mentioning this in all the threads I needed to catch up with in my backlog, with this one being the last on my list unless I find more when I go through my tabs) I haven't had the time due to a lot of problems coming up on my end and these problems haven't really gone away. I'm going to get this out of the way now, and my response is actually in two parts to an extent. I won't be going into the full detail I could by any stretch of the imagination.

In response to the discussion topic presented by @Halvtand at the beginning of this thread, I think bad writing can constitute to the inability to establish the difference between the two - as was suggested to an extent earlier - but this can often be as much a problem of defining what makes a 'monster' and what doesn't. What I mean by this is that it's difficult to establish the difference between an animal and a monster if you don't actually have an established definition for 'monster' in the first place, and how much - if any - difference there is between animal and monster can often be very reliant on this information. Failing to miss details like this is part of what makes writing bad in my eyes. People have suggested numerous concepts for what defines a monster, but none of them will apply universally to every single setting.

The literal definitions of 'monster' also need to be taken into account.

1.
A large, ugly, and frightening imaginary creature.

2.
A thing of extraordinary or daunting size.

3.
A congenitally malformed or mutant animal or plant.

1. I consider this definition the most complicated, as it's largely subjective to the universe. Any creature which meets the requirements of being a 'monster' in this setting is determined by their fictitious nature, which means that animals wouldn't be viewed as 'monsters' in this definition, nor would something considered a 'monster' in reality be considered one in this setting if they're natural to the world of that setting. With this definition, a character wouldn't call a dragon a 'monster' if dragons are commonplace in the setting, whereas an elf would be considered a 'monster' if their existence in that setting isn't widely accepted among the population. I'm aware that elves don't meet the requirement of being large or ugly, but not all things classified in fantasy settings as 'monsters' are those things. A lot of writers will also make 'monsters' some sort of creature that comes from another dimension - such as from Hell or an eldritch universe - and this usually creates a much better separation between animal and monster.

2. This definition is mostly irrelevant, because it means that any particularly large animal is a 'monster'. A whale, by its very nature, would be considered a 'monster' in this context due to its size being overwhelming in comparison to a human -- assuming that the perception of humans is the focus point of the narrative (which it usually is). In a story where ants are the main characters, a human is a 'monster' by this definition, and certain children's films have played with this idea.

3. Using this definition, a 'monster' is a creature which has been mutated through a disease or birth defect. With this in mind, in the context of the setting it's largely a matter of whether or not this creature is naturally the way it is or if its qualities are a result of some sort of birth defect or corruption. In fiction, I would imagine being malformed or mutated through supernatural or paranormal means would also make something a 'monster' in this regard. A standard dog wouldn't be considered a 'monster', but one that's been mutated into a Hellhound by demonic forces would be, though this also makes it cross into the first definition I listed.

As for the discussion about whether or not humans (or animals for the matter) can be viewed as 'monsters' in the real world - a discussion which is clearly off the main topic - I'm going to give some of my thoughts in a hider below this. Keep in mind that this discussion is actually a large part of my philosophy, and not something I'm merely expressing a random opinion about. My views are fairly controversial and I don't expect everyone to agree with me, but if anyone decides to get into an argument with me over their own insecurities I will not be humouring them. As a human, I have the power to overcome my base instincts, and that includes not feeding into flame wars or associating with people who're unable to debate like an adult. These comments aren't directed at any person in particular -- they merely present a warning to those looking for an argument. Also, since this is off topic to an extent (not that it's stopped other people, and I will indeed return full circle to the original question present by the opening post), you don't need to read it if you don't want to (not that people read my whole posts a lot of the time anyway). I simply feel the need to say it due to other people going off the topic and touching on a few nerves with me.