So far, Serenity's earned $17,600,000, which ain't much, but it only cost $40,000,000 to make. They always say opening weekend is the magic number as far as sequels are concerned, but I don't know if its as big a deal nowadays as it used to be. Overseas markets and DVDs will bring in more than $40m I think, and maybe enough to justify a sequel. I think Joss has got it in him to continue the story, and I know he's got enough creative control to not let it go to a hack writer or director.
Univeral has the rights to Serenity movies, but part of the terms of the contract were that they can't make a TV show for 10 years (non-compete with FOX.)