I think that too sometimes, then I think the longer you survive the longer you're likely to survive... It's like fighter pilots in WW2. As Germany or Japan were throwing men into the air quickly their rates of survival plummeted (poor choice of words?) with their lack of experience. Even inferior planes could outperform the Zero because the man at the stick was seasoned while the Japanese pilot was young and made costly mistakes.
I think similar things would happen with the group here where say Glenn "grew some hair on his balls" after that winter out in the wild, and is a more seasoned killer/survivor... Some people have obvious handicaps to that, like someone missing a limb, or they've been relatively protected from the dirty work (beth), so they still would possibly make worse decisions or are forced into bad situations. People like Rick, or Glenn, or Tyrese could really survive a LONG time though just based on their experiences I'd think. I'd find it believable anyway.
They're gonna need more fodder here.