If you have no idea what a bug hunt is and how it can help your engineering team, I recommend reading this first. If you know you want to raise your confidence in releasing a new feature by hosting a bug hunt — let’s get started.
Your teammates’ time is precious, so preparations should be done in advance to make sure you’re maximizing any bug hunt minute. I suggest creating an organized document with all relevant information and instructions, to easily share with participants.
Define the Scope
“Neither too big nor too small” — what is a suitable feature for a bug hunt?
A single bug hunt session should focus on a specific feature or a well-defined part of the product. If you want to test several features, product areas, or user flows, I suggest breaking it into consecutive bug hunts sessions.
List the main user flows you want to cover and their relevant variations or flavors if you have any. For example, is testing different operating systems important? Should we check the feature’s behavior in different languages or locales?
If there are many variations, I suggest listing them in a matrix representing all combinations, so you can assign it for participants to explore and make sure no path is left uncovered.
Who should participate
One of the benefits of a bug hunt is having the people who built a feature to be the ones testing it — engineers, designers, product managers, and other contributors. Even if they’re not all always a part of the release process, this is a great opportunity to give them a closer look at it.
To maximize the efficiency keep the number of participants between 4–8. With too many people you will most likely have less communication and end up with a ton of duplicate bugs you’ll have to filter later, which is very time-consuming.
Set up a testing environment
Decide on the relevant environment in which your feature should be tested, build, and deploy your feature to this environment.
Make sure all participants have access to that environment (especially if they’re not usually testing features) and ask them to download or install anything needed beforehand, so you won’t waste the actual hunting time.
Choose a bug reports collection method
The output of a bug hunt is, well, a list of bugs.
We want to make sure we collect the relevant information for every reported bug so it can be later easily reproduced, investigated, and fixed, but we don’t want to exhaust people by filling out long reports with unnecessary information. Define the minimal but crucial data you want to be collected for every bug found (OS, user ID, app version, etc.). Steps to reproduce and a screenshot are usually a must.
There are many ways to document bugs during the session. For example, you can create a shared spreadsheet, a form, a designated slack channel with a built-in form, or a message template.
Choose whatever tool you find most convenient, just make sure that it’s not too tedious to fill and that it will also be easy for you to transfer it later to your task management system.
Regarding prioritizing, I found it useful to ask participants to rank bugs they found based on a very simple scale — Critical / Not, rather than giving them a broader range to choose from.
Location & Duration
Book a room or an online meeting where you will all be during the session. If you are physically in the same place, snacks are always encouraged.
In my experience, an hour and a half is a good amount of time, including initial explanations and setup.
During the Bug Hunt
Congratulations, your bug hunt session is about to begin! Give participants the scope, assign different paths to explore from the variation matrix you created, and explain how to document found bugs.
Share any known issues you are aware of and encourage people to share issues they tackle out loud. This will save time for other people coming across the same ones, and create fewer duplicates for you to filter out later.
After the Bug Hunt
Go over the create list of bugs, filter duplicates, refine priorities, and transform them into tickets.
Thank the participants for their time and start planning your next bug hunt!
