Physically, Hugh Jackman didn't really match up with the image of Wolverine for the first X-Men. In fact, he wasn't the first choice for the role. That had actually been Dougray Scott. But somewhere along the way the role of Wolverine got recast, and as a result Jackman has essentially been carrying the franchise.
Dougray Scott was originally cast as Wolverine but he couldn't do it because he was tied up with Mission Impossible II.
I'd argue that anybody basing casting judegements off of height in this day and age is a moron. They easily make tall guys look short and a midget look like the tallest guy in the room. I think Jackman's facial expressions and overall look made him a clear choice from the start, but maybe I was in the minority with that opinion.
I don't disagree, but you know how comic geeks are, they want the movies to be exactly like the comics until they don't...which usually only happens if the movie ends up exceeding their expectations.
I think Jackman has done an excellent job as Wolverine, don't get me wrong, but he didn't necessarily stand out as an obvious choice the first time. Unlike Downey as Stark, though, I think Jackman could be replaced as Wolverine at some point. It's harder for me to picture someone else in Downey's role, though.