So, you have found a bug. What Next ?

You have been testing and found a bug. No question about it, clearly a bug. Job done? No, there are few more things to do and to be aware of.

Firstly, of-course, reporting the bug.

The bug report needs to have enough information to reproduce it, but also to make it clear what impact does the bug have on the product.

Managers are busy, thus tend to read only the headline – make sure it is unique and contains the digest of the problem and impact, if possible. This also helps to keep the number of duplicate reports low.

Attach the debug logs, if possible, they are of great help to developers. Specially so if it is a hardly reproducible bug, or only seen once.

Try to make it clear, if the problem appears all the time, or only sometimes. How many times you tried and how often did it occur.

Before reporting, make sure there isn’t already similar one reported. Then try to understand if it is the same bug or not. Sometimes that is not possible – same result, similar sequence. Then report it as a new bug, but link to the existing one.

If it is already reported, add your comments to the report, bring out the new info or difference of experience. Maybe the report was not written as clearly as it could be. Or you found a way of reproduce something that had originally been a ‘seen once’ issue.

Make sure to go over ‘shelved’ or “rejected” errors also – maybe the bug you found had already been ‘dismissed’. And maybe for a reason that it was badly described.

Remember, if a bug is dismissed and you don’t agree – it is probably that you have not worded the impact clearly enough. (Or that you were not aware of the context, which is something you need to know in order to test.)

Don’t relay on developers memory – when you demo the problem, they might say, yes, it is a known error. They mean, ‘I know that it might happen‘. Still you have to report it.

Secondly, follow-up on the bug.

Keep your eyes on the report – read through the comments that are applied, to whom it is assigned, etc – in order to spot if it dismissed (rejected, shelved, or just forgotten). Make sure you understand why, and if you agree.

If you see the bug again, and have new information, update the report. Also, when you don’t see it any more, update accordingly, and investigate with developers.

If it is not reproducible at the developers site, investigate why that happens, and find the missing detail (either in the report or (more probably) in the developers setup.)

Update the report headline to best reflect the problem (and impact if possible), when new information is available.

Lot of issues tend to have similar result (crash-to-desktop) but are very different internally. To keep the picture clear it is sometimes good to group those bug-reports, if the identified cause is (or is believed to be) the same. Possible risk is that, the problem is actually something else and that is left unfixed (and untested).

Keeping the picture clean is very important when software changes have impact on a wide range of errors, which might have been fixed. Changed communication flow, for example.

Couple of small problems might add up to a big issue for customers, reflect that possibility in meetings, so that they are not forgotten. Also, critical bugs might be hiding behind them.

Thirdly, testing the fix of the bug.

While testing the fix, find out what else has been changed (or broken). Thinking through the changes with developers if possible.

What to do with bugs that were seen only rarely, how to say that it has been fixed? May-be you just have not seen it yet. Even so, update the error report with that – so people who only read the reports (project managers) get the latest info.

There are number of problems, mostly stability, that cannot be fixed as such, only by making communication more reliable, robust, etc. Those kind of errors need close inspection with every new version, and updated reports, so every new time it happens is not lost. Automation also helps in this case.

Sometimes, when the product is going through rapid development, a lot of issues might have become fixed or irrelevant. But no-one has changed the bug reports to reflect that, and they are cluttering the picture of open errors. They should be covered quickly, but also those areas noted for future testing.

Understanding how to bug was fixed goes a long way in getting better at testing (the product).

Fourthly, making sure it stays fixed.

Usually fixed bugs stay that way, but its good idea to test some of the more difficult ones again.

Every reappearance needs investigation – was it a lapse by individual or a more serious issue, a design problem perhaps. Or a problem software package building procedure.

Whenever you think that was an easy fix, think again.

In conclusion, there are lots of testings activities after finding a bug. Worst bug is the found-but-forgotten bug. This is where reputation is lost. Keep sharp and follow-up.

One good way to do the follow-up is to use the “issues” (by Micheal Bolton, here, and James Bach, here). Take notes on during a day and then analyze them later, so that following up on bug reports does not interfere with testing sessions.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s