As usual, your mastery of crude language and and blunt statements continue to amuse. I'll bite.
1) Why do you think newcomers overestimate their skills when they first come to the site?
As you mentioned, it's very possible that people who came from another site may have held an entirely different set of standards than what's usually expected from people here. For example, back in ye olde yonder days of when I was like 13, I first started roleplaying on a Star Fox forum that's now entirely defunct. There was one roleplaying forum, and you could basically fit most of the categories here into it with a lot of overlap. There weren't really mods who kept an eye on everything, no standards to enforce, but for the most part things were written in paragraphs and people tried to have proper spelling and grammar. Part of the reason I got out of roleplaying for quite a few years was because the people I usually roleplayed with were suddenly getting into speed posting, and I did not (still don't) have time to spend my entire evening keeping track of a thread just to stay relevant to the plot. And such, when I found RPG many moons later, I headed right for the casual section to knock off the cobwebs and see where I was at, and if I needed to improve. Not long after, I ended up in the Imprisoning War that I'm sad to see died off, and you know the rest. (Sorry about the friction from time to time, by the way. Last year was a rough year for a lot of it, so I've had quite a few off days.) So in my case, I went to the section I felt was most appropriate for my writing ability.
What I imagine is the case for a lot of the people you're describing is there's a perceived stigma about Free being full of idiots and kids, which really isn't the truth, and people want to make a good first impression. Perhaps they're wanting to try to improve their writing by aiming high but aren't quite there yet, and just need to be shown the ropes by getting into a game that's a bit more structured, but still within their comfort range with people who've been around a while. I can't really fault people for trying, but as you said, an honest effort has to be made. A player who isn't matching the expectations for that particular game may make things awkward for the other players, and I know I for one really have a hard time when there's an overlap between people who should be writing in Free doing Casual-level games (I've seen lots of ones going with one liners and speed posting in Casual the past month or so). Not because I look down on those people, but the expectations are different. It could also be that they're not used to doing character sheets, and just jump right into games wherever they came from. I'm sure there's a dozen unique explanations that differ from person to person.
Another thing I think is a common occurrence is that the sections are sometimes ignored and people just see a fandom or theme they're fond of that they haven't seen in the other sections and decide to go for it. Let's say somebody has been really itching for say, a Mega Man RP. They can't find it in the Casual Section, but they notice an Interest Check for it in Advanced so they decide to go for it. What's the worst that can happen, right?
2) How can GMs deal with these kinds of "out of place" writers in their RPs without creating a problem?
I've gotten into the habit of writing out EXACTLY the expectations and standards I am looking for in the RP, and have pretty stern requirements when it comes to a character sheet. If I can get someone to write out a detailed biography for their character, it shows they've put a lot of thought into it and because they've invested so much time and effort into their character and the setting, I can get an immediate indication of what their writing standards are going to be, and ensuring that all the players are on the exact same level when we start. You may have looked at my OOC posts and seen a wall of text of rules, there's a reason for that. For the most part, it works. The players you describe tend to realize they may be a bit over their head where there's a detailed wall-o-text describing exactly what's expected of players and going over a detailed story and setting.
For those that apply and aren't really meeting the expectations, I'm more than willing to go over a CS and work with somebody who's put an obvious amount of effort into it and genuinely wants to get their sheet up to snuff. It's part of the reason I do high-casual RPs that often cross into Advanced-level writing because I like to encourage people to do their best when they're making posts, but I also like them to feel comfortable if there's times where they feel they can only write two paragraphs or whatever because of writer's block or a longer post being inappropriate for that situation. Always give people room to grow and try new things, but don't put them under pressure. For the most part, I think I've had really good luck with players, and most of the time, I meet one or two new people for game that I generally seek out to roleplay with later on.
But when it comes to people who, as you put it, can't be helped, the best you can do is explain what's expected for the game and politely turn them down. As long as you're not being rude or condescending, most people tend to be pretty understanding. As long as you can nip the bud right away, usually any conflicts can be prevented before they get out of hand. However, if you end up accepting a player's character sheet and you notice problems arising, the best thing to do is bring it up with them in PMs with that player and express your concerns. If the person seems to be genuinely trying, it never hurts to give that player a hand and some suggestions for meeting the expectations. I found usually people can be reasoned with with diplomacy and any friction taken care of as long as you're on the ball. However, if the person's still not working out, the best thing to do is politely tell them they're not a good fit for the game and that they should move on. I don't really have any experience with that case, usually I have people who drop on their own if they're not content. At which point, it's usually easiest just to ignore their character or write them out offhandedly.
Anyways, that's my perspective on it. Most people can be helped, others just need a guiding hand to somewhere that's more their speed.