In most cases, the first step in self-publishing is developing an idea for your book. You can self-publish almost anything you want, but if you want to make a profit, it helps to consider your book not just as a piece of art but also as a sellable product. What audience is interested in the subject and how do you get their attention?
Everybody has an opinion on what sells, and we won't get into that too much here -- it's part of the individual creative process that self publishers go through. The important point is that as a self-publisher, you have to consider sales just as a large publisher would. Step 1 is arriving at an approach to the book that will make it valuable to an audience. Among other things, that means seeing what similar books are out there, and seeing how they've sold (checking Amazon rankings is a good place to start).
Money isn't everything, of course. Few books are going to be blockbusters, and many self-publishers aren't that concerned with making money at all. But even setting profit aside, it is essential that you have a business plan based on what you reasonably believe you can sell. To put it another way, there's no point in printing 10,000 books if books like yours typically take three years to sell 1,000 copies.
In addition to the subject matter and approach, a self-publisher needs to think about the book as an actual, physical product. Will it be a trade paperback or a hardback book? How many pages will it have? What will it cost? All these questions are interrelated, as we'll see in the next section.