When you get to the end of this blog, you’ll see advertisements for books of mine. I know from experience, that the advertisement will sell, over the next week, a dozen copies of the books mentioned. The ones sold off my website will pay me 95% of the asking price immediately. The ones sold through Amazon will make me 70% which gets paid in 60 days.
A dozen books a week is 624 a year. Stackpole prices his books at $5. This results in an annual gross of $3,120. Stackpole's net will be between $2,964 and $2,184, depending on whether the books sell off his website or at Amazon.
Kelly and Lyda and Doug and Naomi can correct me, but my understanding is the current New York advance for a book by a new author is $5,000. I think there's an argument for selling to the New York houses, if you can.
The old advice for SF short fiction was start with the top markets, the ones that pay the best and are the most visible, and then work down.
It would seem to me that the hierarchy for novels starts with the New York houses, then goes to the SF specialty presses. Like the New York houses, the specialty presses produce books that look like science fiction. Some of them can get their books into chains. All of them (I think) are on Amazon. They have websites and catalogs. Many of them sell at conventions. And the best of them have good reputations. What they publish is worth looking at.
I would put self-publishing last. You get to keep more the revenue, if you self-publish, but you also do more work; and you are alone, without whatever help a publisher can provide.
When does one self-publish? My tendency right now would be to self-publish work that cannot be otherwise sold. Short story collections are very hard to sell, especially to the New York houses. Out-of-print novels have almost no market. You aren't going to make a lot of money, unless you have an extraordinary back list. But money is money, and it will mean that people who like your work can find it.
(Having said that, all of my out-of-print novels are available on Amazon, along with the three small press books I have done. All of Lyda's Archangel books are available, as are all of Naomi's books. Of course, if they are being sold by used book sellers, we don't get a cut.)
I would also argue that it's important to keep publishing, to remind yourself and the world that you are alive and writing. So if you hit a dry spell and can't sell, it may be a really good idea to self-publish a chapbook or collection or novel. It's another line in your bibliography. It's a new and recent publication date at Amazon.