Danish is supposed to hard to learn? Both of the Estonians who speak fluent Danish I know (and one of them speaks both Danish and Swedish) have at some point stated that they found the language rather easy to learn... Oppositely, Estonian is supposed to be (have seen it cited as such in plenty of places) a very hard language to learn, and the person who lived/studied in Denmark for a while and has a Danish husband commented that it was absolute hell for him [the husband; they live in Estonia] to learn Estonian ... and that her first name literally turned into [i]Kh'ht[/i] for the duration of her stay, since Danes apparently cannot pronounce any but the first and last letter of it. Whether it was a doctor's appointment or anything else, whenever she saw someone staring at a name with an utterly exasperated expression, she [i]knew[/i] it was her. (Except once, when there apparently was a person with a just as odd name.) And English-speakers typically find the Estonian grammar hard to get accustomed and pronunciation ... will stay extremely strongly accented for a long time, to say the least. Have heard plenty of frankly rather horrible-sounding accents of my own language over time. 
 - How difficult a language will be to learn tends to very heavily rely on what language-space one is from, I've noticed. So what might be fairly hard to learn for someone from the Germanic branch might not be so for the Finno-Ugric branch, and perhaps even more arduous for one of the Slavic people to pick up, for instance.
Oddly enough, Hungarians can speak Estonian almost accent-free, even when they have never seen the language before and you just give them a page with Estonian text on it, and ask them to read it aloud. I have personally tested this. Which is odd, since the languages do not look or sound anything alike (there were four words total which were the exact same, and some we found that sound slightly similar? If either looks at a text written in another, or hears them speak, nothing at all would be understood). But I guess the languages are related, after all... Then again, with a Finnish person speaking Estonian and vice versa, there usually is an accent (when both are speaking English, the accent is somewhat alike, though you can still fairly easily tell one from another). And Finnish and Estonian are [i]extremely[/i] close (well, Finnish will generally sound almost indistinguishable from some South-Estonian dialects when the person can speak neither), to the point you can compose entire sentences which are almost the same with only very minor differences in spelling and pronunciation. Perhaps just a conjugation or two... Of course, there are some words which look exactly the same, but mean different things, so if you went word-lending between the two too liberally you could end up saying something really stupid. 
 - Uhh, and I am a silly foreigner, so I first identify an accent as Scandinavian and then typically either Swedish or non-Swedish. From the two Danes I had to work together for a while - both of them very very fluent in English, though, so the accent was perhaps not as apparent? - I'd still put the accent far closer to Swedish than German. I never even made the association with German accent, and I have heard German accent in English *a lot*. ...I still can't make the association with German accents. Norweigan and Danish are very easy to tell apart when I hear them spoken, though, and I don't exactly understand much of either (well, I won't understand anything that isn't a lend or doesn't look like Swedish - and admittedly it is far easier with text than spoken word). I have picked up the understanding of some Swedish from being around it (if you recall, one of my aunts has a Swedish husband; they have residences in both countries ... all of my aunts have foreign husbands, actually), but I haven't really as much as tried to speak it aside of common pleasantries. As far as I can tell, I am trying to assign it the same unique-to-me accent I use with English and eh ... with the specific pronunciation-patterns I can only conclude it would sound truly weird if I went with it. (Ironically, with all the talk of my odd English accent, I appear to have opted to go with some some Australian-tinted UK English accent instead the past couple of days.)
(I wasn't wondering what you sounded like because I was wondering what Danish accent sounded like. Mostly I was just curious whether the accent you personally speak in with English comes across as unusual to me as it apparently is to others, or whether it might actually resemble something to me.)