Doing this. Small breaks throughout my workday that I roam the Internet and sporadically post. Other than that, really no "spare" time at all. Typical weekday:
4:45am rise and shine
5:30am arrive at work
3:30pm leave work
4:00pm arrive home
4-8pm I take over watching our 15-month old, which usually involves copious amounts of reading, songs, chasing and tickling, as well as a walk and fixing her dinner and feeding it to her. I consider this my entertainment time because, though it's tiring, it's fun.

8-10pm I can usually try to get some stuff done like bills, garbage, dishes, etc. as long as it's not noisy enough to wake the daughter. If I do manage to sit during this time, I'll doze off within a couple minutes.
For weekends, I'm the main caregiver, so days are mostly spent with my daughter. If a home-improvement project is needed, I'll squeeze it in.
So work is generally the only time I can actually relax...a bit.
And like
hansolo_506 said, I also feel guilty and selfish for any downtime I don't spend with the family. But it's worth it.