Flexible Test Cases
If you are stuck with a manager, who does not see the waste that using detailed test cases creates, here is one solution.
Propose to use flexible test cases in future. The manager will be happy as you still use the magic word ‘test case’ and can give the status based on ‘executed’ test cases. But in reality you will have lot more time for actual testing, more ways for being creative and more info to report than number of passed test cases.
(Long post describing the concept. Or read the pdf – Flexible Test Case)
FTC is a simple description of usage scenario on with relevant information and suggestions for in-depth testing, see structure and example below. It is up to tester to decide how detailed to be – regarding priority of the feature, available time, test cycle, etc. They can be used as ‘normal’ test cases only they give much more in return if executed as described below.
- Test cases are not too specific – can be created early, without too much effort.
- Test cases evolve with testing
- Low number of test cases to maintain
- Flexible detail level of testing
- Visible testing – gives clear picture of progress and status.
- Easy to share with other testers
- Powerful with both novice and skilled testers
- Test cases are hard to create in early stages of development
- Test cases are hard to create without knowledge of product
- Test cases need skilled testers to create them
- Test execution gives best results if tester knows the product
Example is given below.
Name / Short Description: Clear and readable, reflects content (and configuration if needed)
Description depends on the framework for FTC organization – in order to make them easy to find, to understand the content by short description, to filter and group them by configuration, area or feature, etc.
- Environment setup – OS, browser, other programs, etc
- Setup of product under test – Settings, user access levels, input data, etc
- Preconditions – (Getting system into) specific state for stating with scenario.
Add/remove these according to project and test case specifics.
- Login and do something
- Do something else after that
- Do something third with info from previous step(s)
Usage scenario with basic description of activities to fulfill. It should be more than just one item or specific action. The scenario should follow through a certain activity, process or logical part of the that. The goal is to cover how users would use a feature, business process, how data flows through the system, etc. Use different tours to get as much coverage as needed.
The description of each action should be simple and concrete to allow for ‘happy-path’ execution and leave room for expanding the scenario.
- Login – use wrong password, no password, use ‘back’ after logout instead of login.
- “Do something” – cancel and start again with different parameters
Description of various test ideas to use in scenario actions. Based on test design techniques and/or business goals/risks. They should be geared to be test case specific – not general copy-past of all possible test design ideas.
The goal is for tester to mix these into the scenario with subsequent runs in order to gain maximum test coverage. Tester is in no way limited by these and is welcomed to expand it.
Optionally, the advanced section can include “Things to note”
A list of specific items that should be observed during testing of certain actions or usage of advanced testing ideas. Business risks, technical aspects, etc. This is not the ‘expected results’ – that should be in testers head and evaluated on the fly.
Optionally, the advanced section can include “References”
A list of references to specifications, business rules, test heuristics, test programs, input data generators, etc.
The Flexible Test Case (FTC) is a simple description of usage scenario with relevant information and suggestions for in-depth testing. The goal is to have a set of test cases that cover main functionality of the product – how users would use a feature, business process, how data flows through the system, etc. Each test case should follow through a certain activity, process or logical part of the that. These test cases can be used in various ways (with variable detail level) for testing the product from quick smoke tests to thorough inspection.
Planning and estimation can be done using not only which test cases to run but also how detailed to run them. And the detail level can be adjusted along the way. Estimation of needed time for each test case is depending on the detail level, but is similar in different test runs of the same level.
‘Execution’ of test cases is not static, but driven by test ideas in advanced section, information gathered during execution and testers skill and knowledge.
Results of test ‘exaction’ includes error reports, comments, questions, test case status and updates to advanced section for future runs.
Tracking ‘executed’ test cases, when combined with detail level, results and the status (blocked / failed / problematic / promising / passing) for each test case gives clear picture of testing progress and status. And helps to provoke the questions for framing the testing story.
FTCs are most useful in later stages of software testing: acceptance or user-acceptance testing (and alike). Although they can be used earlier as well, the effort of keeping them up-to-date is greater and that time might be better used for actual testing.
Creating FTCs involves the business side of project – without that input the usage scenarios will be incomplete. While scenarios based on technical aspects are important the critical business cases need to be covered as well. Plus, their input helps to highlight the main concerns that business side has with different features. In addition, it is suggested to use different tours for creating FTCs to cover other aspects of the product as well.
FTC can also be shared easily with new testing partners or extra 3rd party testers, who can use them as guidelines for learning the product. Or for testing main usage according to ‘happy-path’ scenario. And then, if needed, gradually moving on to more detailed testing, using advanced section as idea generator.
‘Execution’ – does not mean blind execution, but using the scenario and ideas for testing specific part the product. See below.
For more info regarding touring see: http://www.developsense.com/blog/2009/04/of-testing-tours-and-dashboards/
Execution of FTC means going through the scenario with given configuration more than once. Results are evaluated on the fly according to testers knowledge. First run should be ‘happy path’ to see if the scenario is usable in good conditions. Every subsequent run should use the test ideas given in the ‘advanced’ section to expand the scenario for more detailed coverage. Additionally, testers are encouraged to use information that has been observed during previous runs and any other ideas tester has.
The number of runs to do in each ‘execution’ depends on:
- Available time
- Priority of the feature covered in FTC
- Test cycle and objective
- Testers knowledge and skill
- Information gained during ‘execution’
Results of ‘execution’ should contain:
- Error reports – to error tracking system (possibly with backtrack link to FTC)
- Comments and questions – to responsible persons (test lead, project manager, etc)
- Blocked / Failing / Problematic / Promising / Passing – status attached to FTC
- How detailed the testing run was – happy-path only / somewhat / very thorough.
- Updates to FTC
Problematic – meaning that the scenario is failing in most conditions, but working as happy-path.
Promising – meaning that the scenario is not failing in most conditions, but still has problems in certain situations.
In this example the product under test is an iDevice speaker-dock that support Apple airplay functionality.
FTC Name: Pausing and starting music playback over ‘airplay’
- Iphone 4, iOS 5.01
- Cisco E4200, router Wireless N, unsecured network “Test1″
Setup of product
- Preferred network is “Test1″
- No equalizer
- Start playing music via airplay from iPhone 4.
- Pause on product
- Unpause on product
- Select next track on product
- Pause on iDevice
- Select next track on product
- Pause on product
- Unpause on iDevice
- Vary timing between pausing-unpausing, try pausing just as track has started or ended.
- Add winding / rewinding into scenario
- Add mute and unmute volume when paused and unpaused
- Simultaneous operation on iDevice and Product
Keep in mind that the product must not lose airplay connection to iDevice, unless it has been disabled on iDevice, or track has been paused for 10 minutes.
Timeout values are found : https://local-svn-server/repository/spec/Timouts.xlsx
Apple certificate demands that operations on iDevice are with higher priority than commands via dock/airplay.
Planning and Tracking example:
A simple version of how tracking of executed FTCs looks. The is no algorithm for assigning ‘pass/fail/blocked/problematic/promising’ for test groups – that is up to test lead/manager to figure out in each case. All other details, how to group tests, how to manage configurations, etc., are so project dependent that there is no need to discuss them here.
Incomplete list of test groups (with number of FTC):
- iDevice connecting – 4
- iDevice interrupts – 3
- Playback control – 4
- 1: Pausing and starting music playback over ‘airplay’
- 2: Winding and rewinding music over ‘airplay’
- 3: Pausing and starting music playback when docked
- 4: Winding and rewinding music when docked
- Volume control – 6
For software version 0.19d
|Test area/FTC||Which TC to run||Priority||Detail|
|iDevice Connecting – 4||1,2||Low||Happy path|
|iDevice interrupts – 3||All||Medium||Somewhat|
|Playback control – 4||All||High||Very thorough|
|Volume control – 6||3,4,5||High||Very thorough|
|Test area/FTC||Priority||Result||Detail||SW version|
|iDevice Connecting – 4||Low||Passing||Happy path||0.17a|
|iDevice interrupts – 3||Medium||Problematic||Somewhat||0.18b|
|Playback control – 4||High||Blocked/Fail||Thorough||0.17a|
|2: Winding…airplay||Promising||Very thorough||0.19d|
|3: Pausing…docked||Passing||Very thorough||0.19d|
|Volume control – 6||High||Failing||Very thorough||0.19d|
Day – for prompting this idea to me by analyzing SC2 build orders in his daily show.
Oliver Vilson – for feedback and pushing me to write it out more formally.