@shylarahI'm working on very little sleep, and will be off to rest shortly, but it's time I finish this conversation. I could end up repeating myself unconsciously in this post, so just keep that in mind since I want to get my thoughts across without any misconceptions if possible.
Also, if there ends up being some flaw in my explanation (I'll look it all over eventually, and possibly get a second opinion from my friend) due to how fatigued I'm feeling, I'll correct myself eventually.
So...literary fiction and other fiction can both make you think, but literary fiction is more deliberately encouraging discussion about it? So sort of a social commentary thing.
You're right to assume one of the differences is that 'literary fiction' is deliberately trying to discuss, but there's another difference as well -- 'literary fiction' actually gives a more compelling and mature argument, whereas 'genre fiction' tends to be very simplified and - in many cases - not even relevant or realistic. The most important way to compare them is this though, since they're ultimately about the
priority of the work.
- Literary Fiction: Works under this classification will actually have their primary focus be on the 'literary' elements, which is why they're called 'literary fiction'. They are stories just like any other, and as such they can - and often do - have genre(s); however, being engaging as a part of these genres will be less important than the discussions and themes, since the greater purpose is to provoke thought. The narrative - deep down - is driven by these themes, so whatever discussions and thoughts that the work wants to invoke in others will never be compromised just to be entertaining -- they will tell a good story like any piece of 'genre fiction', but not in a way that hinders the deeper purpose and message.
- Genre Fiction: The focus of these works is to successfully create a good story based on their 'genre', hence why they're called 'genre fiction'. In other words, while these stories could potentially try to give a discussion and have intelligent themes, they will always prioritize creating entertainment through the genre(s) they focus on over these more mature goals and topics. They'll write the narrative around the story, not the philosophy, and subsequently they will weaken the messages they want to give if they need to in order to focus on the genre.
A lot of stories attempt to give a good message and end up failing to do so in a way that's insightfully satisfying, so not everything that makes people think counts as 'literary fiction', especially when the thought provoking was not intended or was just a result of convenient storytelling. For a piece of writing to be acknowledged as 'literary fiction' it usually needs to be accepted by people in philosophy or social commentary as having merit in the fields it's attempting to discuss. You need to be able to take the piece and use it in real life adult debates, or it doesn't count at all. Simply bringing up a point about (to use your example) racism and saying it's bad isn't on its own enough -- the actual thoughts or arguments presented need to be something that a philosopher could bring to a serious debate or use as a thesis, or a psychologist could accept as being an insightful look into the human condition.
Certain pieces of writing (Romeo and Juliet, Of Mice and Men, Moby-Dick, et cetera) are widely accepted as 'literary fiction' and are used at schools or universities for study due to that. In contrast, the television series Game of Thrones (using this as an example due to it being brought up by a friend yesterday when we were discussing 'literary fiction' -- I don't watch it) doesn't count as 'literary fiction' despite it
trying to be -- it's 'genre fiction'. With that said, a piece of 'literary fiction' is - as you just said - always deliberately attempting to provoke thoughts and be philosophical, and this is generally always at
least as important - often more so - as telling a story for the person who writes these pieces.
Scifi can be hard or soft -- and actually, shows like Star Trek are able to cover moral discussion a bit more freely than ones set in the real world, because they don't have to deal with "well you're showing discrimination against X" or all the complicated censor issues that can come into play. There's one ep of the original series where there's these guys with two-color faces, and they loath the other type, and the Enterprise folks go "but you both are the same, half one color and half the other. Then the guy goes "Yeah, but he's white on the /right/, and I'm white on the /left/." Separation from reality lets you present issues in society from a less biased perspective, independently of many preconceptions.
I'm aware there's different types of sci-fi, but it only very occasionally interests me, and far more rarely when it's the only genre in the setting. I don't actually think that these sorts of stories are inherently discussing things more effectively -- separation from reality weakens the actual weight of the message - but not always - for reasons I go into a bit later in this post. Not all 'literary fiction' is set in the real world though, but this is because the discussions are presented through metaphor instead of being literal. There's nothing actually wrong with doing this - I do it a lot in my own work - but sometimes it will make the message weaker due to people not understanding it as easily or being able to relate to it, and this can cause it not to be taken as seriously due to it being too surreal -- philosophy should, at its core, by very realistic.
Like I said, you can definitely tell 'literary fiction' while including supernatural elements, since I do this myself, but it has to be treated carefully. Also, a lot of decent pieces of 'literary fiction' pretty much stick their middle finger up at things like censors or opinion consensus, which is why they're often not mainstream - and sometimes not even published under conventional means - though there's exceptions. Very frequently, for something to be 'literary fiction' is needs to be controversial. You can very easily weaken a message by making it too 'television friendly'. To give a good example of this again, Naoki Urasawa's Monster is actually not an easy show to find without the internet due to its broadcasting being limited to the best of my knowledge -- the fact that it dealt with severe issues (child abuse, discussion into the inner workings of sociopaths - as opposed to most fiction, which actually gets it wrong a lot - and realistic discussions about the roots of evil, depictions of actual atrocities that happened in society during a specific era of time and how they affected people, et cetera) meant that it wasn't able to be shown just anywhere.
With this all in mind, if you don't portray sociopaths accurately then any discussion you try to provoke about sociopaths isn't realistic - and therefore isn't 'literary fiction' since those types of works are rooted in realism - so any discussion about the human condition will actually lose a lot of merit unless it's presented in a world that at the very least fundamentally is rooted in reality (whether it's set in a more fantastic genre setting or not) -- having a discussion about how aliens taking over the planet affects people in the real world today is entirely meaningless as a piece of social commentary because we don't deal with those problems in reality -- while this could be used to portray social commentary about how humans respond to being enslaved
in general, the social commentary holds more weight if you don't detach it with escapism since a lot of people - though obviously not all - can just brush it off as 'make believe'. It's not that we can't do it, as I've been pushing as my primary point, but it can make it harder to give it weight. Escapism is the exact
opposite of social commentary -- one of them is an escape from reality while the other one is an open and heavy discussion about reality, so getting too far away from realism complicated having serious discussions.
I do have a lot of fictional elements in my own story, but I'll make myself clear on this -- all of them serve a purpose from a philosophical perspective or at least work as a metaphor, so I'll (generally speaking) never add an element into my story if I think it'll prevent me from making a compelling point or if this fictional element will compromise the integrity of my mature real world discussions and provoking of thought. Sure, I'll put a lot of magic concepts into my work to make it cool and entertaining and build a lot of lore around it, but if something's going to get in the way of the point I'm trying to make or be invasive to the philosophy then it's out the door. This is where my story being 'literary fiction' finds its roots -- every time I consider a new fictitious element to add into my project, I'll always ask myself 'will it get in the way of my philosophy' before I ask myself 'will it be an entertaining idea for the story'. If the answer to the first question is yes then I won't even humour the second question, whereas a 'genre fiction' writer will always ask those questions with reverse priority -- that's assuming they even bother asking the first question at all.
Just for the record, I've seen the entire first series of Star Trek (movies included), and a good portion of The Next Generation. I'm honestly not interested in it, and only really gave it a try on request from my mother (a Trekkie) a long time ago. I know the episode you're referring to even though it's been a very long time since I've watched it. To the best of my knowledge, Star Trek generally isn't consider 'literary fiction', even if it does have moments where it delves into it (as almost everything does). I could be mistaken on that one, but I'd need someone who studies 'literary fiction' (like my friend -- who I'll probably discuss this with in a day or so) like I do in order to confirm or deny -- best to get an extra opinion and not rely on only one. There's a possibility an odd movie or episode is considered an exception, but not the series as a whole since it prioritizes its genre and entertainment value over the discussion and themes from my experience of it. I remember thinking the show was 'okay', but not something I'd watch again and I wouldn't have watched it on my own. I could deconstruct it from a philosophical and social perspective, citing it's realism versus escapism, but I won't bother -- it's sort of off the main topic, and so is discussion about the sci-fi genre in general, which I mostly brought up before as trivia.
I'd also say that how rooted in reality something is will be more a function of believable characters and consistent rules for the world, and less a matter of magic and future tech. Yes, you'll negate certain specific situations with different possible actions (trapped underwater? Cast a spell that lets you breath!) but with literary issues they definitely can still apply.
Believable characters, yes -- consistent rules, only to the extent that those rules don't make the work surreal enough that you can't relate it to the real world. For example, it's difficult to have an intellectual discussion about whether or not the existence of aliens is significant - as in these characters need to not know that aliens exist because they're discussing aliens as a hypothetical - in a world where aliens are already proven to exist. Essentially, certain types of genre limit the ability to discuss certain ideas in a way that's relevant to reality. In a similar sense, it's difficult - though not impossible, since my story does it - to have a debate about whether or not magic or the soul exist if you do so in a setting that's proven those things are real. Basically, it can go beyond just the problem of negating certain situations and actually get in the way of the literary contemplations -- not always obviously, but this means people need to pick their genre carefully based on what they want to discuss.