I agree that there's been a lot of divergences with both movie universes. With the Avengers, the bones of the original team were largely there with the original members: Cap, Thor, Hulk & Iron Man. The SHIELD angle is something that came from the Ultimate universe, so I kind of can deal with that take on the story. As for Ultron, Vision, Pym, et al? I can see the concerns.
The X-Men films started with something of a hodge-podge team that was more of a composite of the team over the years rather than something that was true to the books. Cyclops and Jean Grey were the only original members on the team. Storm was with the 3rd wave that came in during Giant Sized X-Men, as was Wolverine. Iceman was a younger student at the school and not a member of the team, and there was no presence of Angel or Beast until the 3rd film. Say what you will, but the original X-Men team actually had some decent stories. And they also began to incorporate characters like Banshee, Polaris, Havok and Magneto well before Storm, Wolverine, Nightcrawler and Colossus ever made it into any books. Could that original team have carried a movie or two before introducing the Giant Sized X-Men team? It all would have depended on multiple factors: the director, script and cast. But if you did that I could imagine the fan outcry: where's Wolverine? And I think that Fox wanted to make sure that he was in the mix from the get-go purely from a marketing standpoint.
But what about the possibility of building a universe? I don't think that Fox really had the foresight to see it through. But imagine two 'First Class' sort of films that might have led up to a third movie that would finally introduce the Storm/Wolverine/Colossus/Nightcrawler, etc team? That could have been something great. And then you could potentially get another few films after that with the Phoenix saga, DoFP, and maybe eventually Apocalypse without it being such a ******* mess.