As testers we have long wrestled with the tensions between the agile approach to projects and how we can use exploratory testing in a commercial project environment where we need to be able to provide test scripts as a deliverable to our customer and show executed test runs in order to provide an audit trail to demonstrate to our customers and their auditors that test coverage has been achieved. Measurements from projects that I’ve worked on over the last two or three years have shown that we spend approximately 40%-50% of our test effort on designing and writing test scripts. But, as our developers make use of more sophisticated development tools and reduce their development times it then appears that testing is taking a bigger and bigger proportion of the overall effort – or in practice we are being asked to reduce our testing effort by the same degree. In order to make test execution faster we need fewer defects in the applications leading to less need for retesting – and there is little evidence to suggest that this is the case. Even where the solution is a package implementation (and therefore little custom code to contain coding errors) we still see similar levels of defects in translating the requirements into the package configuration. Of course automation can reduce execution time in the longer term but when we are looking the cost of a project from start to implementation then the benefits of automation have not been realised – and indeed the investment costs of automation would inflate the testing development and execution in this time window. So the obvious area to look for improvement then becomes the test design and scripting and I’ve tried a number of techniques over the years which have had some small success:
- Test design templates to give more consistency and a more rigorous approach
- Introducing two stage design process with a review after the high level design in order to minimise wasted effort on incorrect test design
- Designing high level test cases which document the overall purpose of the test and expected results and documenting the detailed instructions for each test step when the test is run for the first time.