Victory left a markedly stale taste in your mouth when you were part of the reason that you ended up having some massive fight against an alien race. Without Natasha there, he was sure things would've ended badly. For him, probably. Fury wasn't above having people killed, after all. And Clint knew, no matter what Nat had told him time and time again, he was not the biggest asset in the team. He wasn't even particularly 'super'.
'Super' was Nat and Cap, with the serum. The Hulk with Gamma radiation. Tony with ridiculous amounts of technology and that core in his chest. Thor, he was a god! That's what Clint had to match up to. A regular guy with a bow and arrow, just waiting to use what was basically two decades of training in archery and otherwise. Theft, showmanship, among other things. It wasn't like that was a problem, he didn't really need to use those anymore, but they were still skills, ones he knew hadn't gone off despite regular use.
Loki was still a sore spot with Clint - they didn't talk much about Loki, but there was plenty of talk as to how best to avoid... that kind of mind control. Never mentioning names. Clint wasn't sure he appreciated the handicap it seemed to provide. His apartment wasn't exactly a place he expected to be disturbed. At least, not by anyone but his neighbors. It was that first shout of his name that alerted Clint what was going on. Who it was outside his door. He didn't need to think twice about who that voice belonged to either.
He was up, and heading for his bow - which did not have arrows stored nearby, of course, because getting attacked at his apartment didn't really happen, and he never really had to worry about it - when the door was blasted in. No. Clint wasn't happy to see him at all. In fact, there was a deep sense of resentment, of hate even, that Clint didn't often feel at the sight. But he stopped in his tracks. Exposing your back to Loki was asking for death.
Too bad it seemed it might have been pointing in that direction anyway. Good job, Barton.