PETS5 Workshop – Bug triage

This is one of the assignments used in PEST5, the post contains all the info for the all the roles. If you plan to use it, make sure you don’t spoil yourself.

Roles:

Product Owner – Business and Market Quality representative
Project Manager – Development side manager, knows the team and development problems
Test Lead – Testing side, knows the team and testing problems

Assignment:

Analyze your info, talk to instructor regarding details. Think through what you want to achieve and how.

Your assignment is to get these bugs classified ( Critical, High, medium, Low) and agreed if they will be (planned to) solved in this Release. Critical means it must be solved *now* with a hotfix. High means, it must be fixed for current release. Medium means, it must be fixed in next release, unless market incidents grow, then it is High. Low means, we fix it when we get to it.

General details:

Product:  High-end television

Current plan was to have software update launched in 3 weeks.
Code Freeze in 2 weeks.  1 week of testing on final software needed to be sure the software is good enough for release.

Situation:
Planned  features are coming along nicely and seem to be in good enough state to be ready for release.There are 4 important issues to investigate and prioritize – should they be included in the current release or not.

4 issues:

  • No. 10313 – Intermittent no sound on new track
  • No. 11613 – Endless playback stops sometimes after 4 hours
  • No. 11541 – Sometimes lipsync missing
  • No. 10711 – Does not play FLAC audio files

Role specific details, others should not see these:

Product Owner:

  • “Must be out at that time” and with important fixes – it is very important for business.
  • Actually, some agreements can be made – some things can be updated later. But you know if you say that, developers will start dragging their feet with everything.
  • And testers complain all the time anyway about not enough quality even though we don’t get that feedback from the market.
  • No. 10313 – Intermittent no sound on new track –
    • Reported from market
    • Really important problem, without feedback that what is going wrong this gives bad experience. When away from TV, or screensaver mode.
    • “No sound is bad, but if in complicated circumstances then not a problem. If workaround is simple”.
    • No logs available from market, at the moment about 0.2% of users have complained so far. Getting them might take a week.
  • No. 11613 – Endless playback stops sometimes after 4 hours
    • Reported by Testing team
    • Media player playback is important, but not critical.
    • Remembers after a bit – or when asked – Huge problem for dealers who keep products on display with endless video running.
  • No. 11541 – Sometimes lipsync missing
    • Reported by Testing team
    • Huge problem. No way can we live with this problem. Must be fixed now. LipSync (Audio & video playback in sync) is elementary and must be working 100%.
    • Critical issue (in the way it is described).
  • No. 10711 – Does not play FLAC audio files
    • Reported by Testing team
    • Our products should support FLAC if possible, our customers expect that so if anything can be done it would be very very very good.
    • Eventually can agree that – Not in ‘approved list’, so don’t care, as long as it fails gracefully.

 

Project manager, development:

  • We should be ready in time with all the other things for this release, so working on these bug-fixes issues is not a priority and might affect everything else.
  • You have 1 week of “buffer” time within the 2 weeks before code freeze. This is ‘unofficial’, something that you planned to use to finish everything.
  • No. 10313 – Intermittent no sound on new track
    • Long description: Sometimes when starting new song on Spotify no sound. Have to start the next song to get sound.
    • Reported from clients, and the information is not clear.
    • Very hard to fix without logs. Must be environmental. Request logs from market.
    • Current estimation: 4 weeks.
  • No. 11613 – Endless playback stops sometimes after 4 hours
    • Reported by testing team
    • Long description: Leaving product playing endless videos, after 4 or 5 hours, sometimes more, the screen is black. And playback must be restarted.
    • Probably something we can do, but investigation takes time due to long trigger time.
    • With right debug info 2-3 days investigation, and then 3-4 days of fixing time.
  • No. 11541 – Sometimes lipsync missing
    • Reported by Testing team
    • Difficult to fix due to archidecture, might not be solvable. Not a regression, the area has not been touched with this update. Could also be present on market software.
    • Investigation 3-4 days, and then 1 week at least for fixing.
  • No. 10711 – Does not play FLAC audio files
    • Reported by Testing team
    • Not important as we don’t officially support that format
    • The playback is in 3rd party component and needs to be investigates what went wrong. don’t know if it can be fixed, Component rollback might bring back other problems (the component notes said they optimized the decoding and fixed a lot of small issues).
    • High risk of regressions in other formats playback. Rollback 1 day of work.

 

Test Lead

  • Any additional changes after 2 weeks, makes it really difficult to test in time for release. Minor changes (if safe) can be accepted.
  • Overall the status is actually good, so low regression risk issues should be fine.
  • No. 10313 – Intermittent no sound on new track
    • Long description: Sometimes when starting new song on Spotify no sound. Have to start the next song to get sound.
    • Difficult to pinpoint and reproduce, could be environmental, etc. We can reproduce the problem in very special circumstances, but that might not be the same issue.
    • Testing a fix takes 1-2 days, but 1 week observance is needed to be sure to have no regressions.
  • No. 11613 – Endless playback stops sometimes after 4 hours
    • Reported by testing team
    • Long description: Leaving product playing endless videos, after 4 or 5 hours, sometimes more, the screen is black. And playback must be restarted.
    • Can be tested on “automated” setup, similar stops can be reproduced with different sequences, but those situations are very dependent on 3rd party content, specific NAS/USB stick, environment, etc. So these might not be the same problem.
    • Takes a while to test, but can be run in parallel, so 1-2 days. Observance at least 1 week.
  • No. 11541 – Sometimes lipsync missing
    • Reported by Testing team, was not seen in previous market release, seems to be regression.
    • No real sequence, seems to be dependent on 3rd party content which we can’t control. Takes a lot of effort get more information.
    • “Sometimes” means that 4 man testing team, focusing on other areas has seen this in total of three times in one week. But it happened on simple usage sequence, when it happened.
    • Direct testing 1 day, 2 weeks of observance to be sure.
  • No. 10711 – Does not play FLAC audio files
    • Reported by Testing team, not present in market software
    • Easy to test for, can be automated. Possible impacts from a fix are more difficult to test, although they are, to some extent, automatable.
    • Instead of playback there is silence, sometimes (heard twice with different content) white noise.
    • 1 day of testing, straightforward.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s