That depends entirely okn how many women are being taken, again there were medieval kingdom's that had populations over 1 million, they could sustain losing a few dozen women every year (this world is already better off than the actual medieval world because there doesn't appear to be as much disease...) Major cities such as Paris had over 100 thousand people. If you spread the number out of women taken out across an entire country, the impact would be negligible
Even the black death, which would have killed way more people in a far quicker time span than the Drakken have taken over the hundreds of years, did not outright destroy any European country. Let's say they take 100 a year (which I think is reasonable for an upper boundary; if you want to keep something as being prized it has to be rare, you don't want any old sod to have one so you probably only take a free each year and reward them to people who have done something particularly loyal or impressive in that year) it'd take 50 years to take 5000. The black death killed a total of 50 million people in 11 years. You said England was quite a bit smaller than Drakka and Gemminia (if you are impling England is small then thats rather silly, by the way; for a long time it was only dwarfed in europe by the Holy Roman Empire, the Scandinavian Kingdom's of Norway and Sweeden, and the Eastern European Kingdom's that would become Russia and Poland; and its population was quite large as well. For a Western kingdom, England was big for quite a while. It was only when France united (and taken England's French possessions) and Spain pushed out the moors that England dropped to being a middle power, and it still wasn't a small kingdom) and if they could survive losing hundreds of thousands of people to plague in a dozen years, they could have survived giving up some of their daughters every year. Maybe it would slowly wear them down... I don't think the Drakken care too much, that could be slowly happening but the Drakken don't care and the Gems doing have a choice.
So to go back to your first point... Maybe the Drakken are more holy Roman Empire than England, yes, but England was one of the big dogs of Europe for a time (for example, the Angevin Empire, which basically owned half of France and made the Scots, Welsh and Irish do whatever they said.) And even after the loss of this Empire, the Kingdom of England wasn't just England but encompassed Wales and a couple of French places like Calais, had the Irish under their control (much to the annoyance of the Irish) so apart from Scotland they basically controlled the British Isle still. I do see where you are coming from, of course; If he was some absolutely insignificant cousin of some backwater castle owner then yeah, I guess he could get away with it... Providing he doesn't bump into said backwater castle owner or any other relatives of Lord Backwater, both of whom would probably have something to say about it.
I want to point out that you live in a country whose borders were not defined by the medieval period. I can go outside now and just walk out of my town and into one of the nearby villages in what, 15 minutes. If I keep walking I'll leave my county fairly quickly. You can do the same in any European nation. The reason is because you are entirely right, it took a long time to get anywhere... And so most nations weren't really that big because you don't just have to defend them from foreign invaders you also have to defend them from unruly vassals who would rather like to be more powerful and independent themselves. If I decided to rebel with my earldom and you would have to match weeks and weeks and weeks to get there well... You probably weren't going to be able to do so because those soldiers will want to be paid every day and generally you have to get back in time for harvest and planting seasons, campaigning in winter is fairly suicidal in some places etc it's an absolute nightmare! So Gemminia might be large by European standards but it's certainly not going too be extremely big; it might have particularly fertile land given its description,in which case it might support a somewhat larger population
Also, who said the Drakken collect all the brides themselves? It seems pretry ineffective for the reasons you mentioned. Those Drakken warriors would have to be paid for all that travelling as well, no no no, the Drakken King isn't going to want to do that when he can have the Gem king use their men (it was said he has a royal guard, wasn't it?) to collect the brides from the further off parts of Gemminia, and the Drakken would really only collect those from near the pass themselves. It would also make more sense from the Drakken perspective because it would degrade Gem respect for their lords and King, helping to divide them and make it less likely they would be able to mount a substantial resistance; plus, a bunch of peasants might riot and kill the soldiers when they see them dragging a girl off if the soldiers were Drakken, that's less likely if they were actually Gem authority figures and even if they did lash out, that way it's no loss to the Drakken