Ugh, Skrulls are part and parcel of the FF continuity, I hate to see them trawled out for another franchise.
Did the Avengers even have a recurring nemesis?
Yes, the Skrulls first appeared in FF, but outside of that the Avengers have been heavily involved with Skrull activity over the years. They were the major players in the Kree/Skrull war, The iluminati attack on their homeworld was led by Iron Man and Reed Richards (among others), and the more recent Secret Invasion was far more tied to the Avengers than the FF, with a Skrull replacing Spiderwoman to kick things off and Skrulls impersonating Hank Pym and Mockinbird for several years of the timeline. The invasion brings the core Avengers back together, reseats Iron Man from national security director to outlaw, puts the Dark Avengers into play, and kills off the Wasp...pretty major implications for Earth's Mightiest Heroes.
As for their own nemeses, the Avengers have quite a few: Ultron would be a great villian, but probably requires Hank Pym to be more established (he built Ultron to make himself look good in front of the Avengers, then the robot self evolved). Kang the time traveler is another solid villian, but time travel is another concept that might be best saved for a sequel. The Masters of Evil were a collection of baddies bent on destroying the Avengers as well. Would be cool to see a group of supervillians to fight, but then you end up having to spread screen time across a number of baddies. Loki is the villian who brought them together in the comics, so it would make sense to use him since he's already established. Maybe they don't want to showcase one villian so much though, tying the group so closely to Thor.
In the comics, the Skrulls are the ones who developed the Cosmic Cube, so if they are going this route then it makes sense to include them as well. Thanos is the one who wielded the infinity gauntlet, so will be interesting to see where they go with that. I'm a little dissapointed that they are tackling Aliens who can transform into anyone AND a device that can alter reality all in one movie - seems like an awfully big scope for one movie. Loki will probably fit in there somehow as well given the last scene in Thor.