Whether you're a big AAA company creating the next blockbuster hit or the tiniest hobbyist creating their gem in spare time between two jobs you have constraints that limit your creative output. In the former case, it might be budget and need to earn money back, in the latter the physical amount of time you can spend and dwindling motivation. And if you want to ever release your game, at some point you have to say STOP.
When should you stop
One could naively say "You should stop when your product is finished, duh" but like with anything, things are not as simple.
- The first obstacle is you'll always have ideas for new features, mechanics or levels and if you give in to the urge you'll drown in feature creep. Whenever a new thing is added you may have to rebalance existing content at best or completely break it and have to redo it from scratch at worst. Or sometimes even remove something that's already there. And don't get me started on what happens when new features are added after you have a lot of levels done.
- The second obstacle is fixing bugs and covering edge cases. If you'll let anyone other than yourself play the game they are bound to find problems. If you let the game loose on more people, especially ones who know games or have various types of hardware, you'll have a steady flow of bugs. Do you think you can fix them? Every single one? Is your game resistant to sudden loss of GPU? Audio driver? No memory? No hard disk space? Crashed thread? Corrupted game files? Cosmic ray flipping a bit? It's physically impossible to cover every case.
- The third obstacle is polish. No matter how great your UI is, you can certainly improve it by tiniest of bits. That level you're proud of? Maybe move the cloud in the background 5 pixels to the left? You have a nice theming in levels? But are you sure you're not underusing or overusing certain tiles? Is the sound volume exactly right everywhere and in every case? Maybe you can improve the writing just a tiny little bit more.
That's a lot of things, If I didn't know when to say stop I'd probably make it twice as long.
But there is still a naive solution to this. Chose an arbitrary point to stop, whenever you feel like it, whenever you think the game is good enough or you are sick of it. And you know what? It can work. You are probably going to produce a game that is obvious it missed the polish stage, will have some annoying bugs, will miss handy usability features and might have unbalanced levels, or maybe everything will be fine. It's a bit like playing solitaire by randomly moving cards and hoping you win.
Let's stop working around the problem and concentrate on a solution - a solution that will make you turn up a better game and also will make the process smoother.
You need an issue tracker of any kind, something as simple as TODO.txt or as complex as JIRA or anything in between. Having a system like that in place solves three big issues:
- You always know the current state of the project and what is still left to be done.
- You can consciously and knowingly prioritize so that whatever is not very important is done last (or never if you have to ship before).
- Humans work better when their work is organized.  Not only you'll no longer have to face these pesky what-to-do-next? moments, there is actual research suggesting that just the act of planning improves our capabilities.
Moreover, if you make a habit of writing down every issue you encounter you'll never again forget to fix something. If you work with other people everyone can chime in to the common pool and you don't risk multiple people working on the same things needlessly. And you get this really nice kick each time you close a task, which makes it easier to stay motivated and disciplined.
It all sounds really great and all, but there is a small downside - keeping your system up to date is definitely more work. You need to set up the system, learn to use it and actually use it. If your system is slow to input new tasks and manage the old ones you'll stop using it sooner or later. There is a reason big and small software developing companies track development that way, it just works pretty darn good.
When and how to prioritize
The answer to the question when is very straightforward - always when the project is larger than what you can fit in your head at any time. If you work with other people you can't fit it all in your head. If you ever forgot to fix or implement something, you can't fit it all in your head. Even if you can fit it all, all you have to lose is a tiny fraction of your time on managing tasks. It's also fine to skip writing things down in the early stage of development when there are just so many things huge to be done that decomposing them would take forever.
How to approach it is a bit more complex. For readability I'll stick to bullet points:
- Divide tasks into two categories, "must be done" (MBD) and "nice to have" (NTH)
- MBDs, as the name suggests, absolutely have to be done before the game ships. It will include critical bugs that could affect more than a marginal number of people. Features that touch the core of the game or are very likely to positively impact reception. Things that become impossible to change post-release (eg. because you want to ensure 100% backward compatibility with game mechanics). Anything that you can't imagine your game not implementing goes here.
- NTHs are harder to manage, but basically what you have to do is order them by importance, possibly using a formula akin to:
<Affected people> x (<Joy added> + <Sorrow removed>) / <Time to implement>Or a simpler version
<Increase in sales> / <Time to implement>. Since those tasks are basically a layer of polish in the order of importance do them top to bottom.
- When all must-haves are done you are basically in a state where you can just release the game as-is, because the most important things are done. That's, of course, a terrible idea, as it will be obvious it lacks the finishing touches, but the game will work for most people. If by some insane chance you will be forced to ship the game there and then, you are covered. But in the more realistic scenario, you'll be able to realistically gauge the amount of work left, check it against the planned/expected release date, define a cutoff for nice-to-haves and then do some heavy testing, console certifying, marketing (actually you should start marketing way earlier) and so on.
The next time you play a game and wonder "Boy I why didn't they do X, it's not like it's a lot of work" the answer is probably "Because X was less important than all the other things that were actually shipped".
- The Guardian article "The psychology of the to-do list – why your brain loves ordered tasks"
- Screenshot by Retro_Apocalypse taken from their Steam screenshots directory.