In a life or death situation, yeah, I'd give it up. Afterall, my hobby is really just bits and pieces of plastic and cardboard.
My wife doesn't like Star Wars and my collecting drives her nuts. That said, she gets the time with me she needs and she understands that I've been a fan all my life and was collecting for years before she and I met. She understands that it's just part of me. She also knows that there are numerous other hobbies I could have that would be detrimental to our marriage.
When I spend money on my hobby, she usually will spend money on her hobby. All in all, it evens out...but I do know my wife is tired of SW and thinks I'm a nerd for chatting on message boards.