“Testing is the key bottleneck standing in the way of truly implementing DevOps and continuous delivery.” This is one of the opening statements of the Continuous Testing Report (CTR) 2019. Why don’t we do it like Bruce Lee – be water instead and flow effortlessly through the pipeline?
The Continuous Testing Report (CTR) is based on survey data and subject matter expert contributions. According to the CTR, the biggest challenge is that organizations don’t have an overview of their test automation landscape. Significant gaps in automation are revealed. For example, only 24% of functional and performance testcases are automated.
What is continuous testing?
Continuous testing is more than just a matter of automating tests. It is the integration of test automation in the deployment pipeline, focused on getting feedback on the software quality as soon as possible. It is the art of seeing the bigger picture of test automation and its place in the deployment pipeline. In order to achieve continuous testing, it is important to focus on executing the right tests at the right time and speed.
The right tests at the right time
Once it becomes clear how quickly tests can be performed when they are automated, there is a tendency to automate all tests. Therefore, automated GUI tests that last longer than an hour are unfortunately more the rule than the exception. It is not feasible to integrate them in a deployment pipeline.
It’s not possible to test everything, so choices need to be made. The aim is to achieve as much test coverage as possible with as few tests as possible. Risk is based on expected business impact and failure probability and is a great way to make those choices. A Product Risk Analysis according to TMap is a classic but effective method to determine the risk of applications: select the most important tests and create a smoke test set to check the testability after a deployment. The rest of the GUI tests can be split up based on risk and can be run after the smoke test. It is also possible to run these tests during a nightly build.
Testing different branches
In an Agile environment, developers often work with branching strategies. Feature branching is one of those strategies. In a feature branch all code is committed for one story, and when it’s done it’s merged back to the master branch. When there are different pipeline processes, it’s possible to implement different test strategies – one for the feature branch and one for the master branch. It would be feasible to do unit, API tests, and maybe a small smoke test during the feature branch and do more extensive testing in the master branch.
A developer doesn’t want to wait more than 10–15 minutes before getting feedback on a change. That means that tests in the deployment pipeline must be quick – very quick. There are several options to speed up tests.
Test automation pyramid
The test automation pyramid helps to find the ideal distribution of the number of automated tests based on their characteristics. Unit tests are a matter of milliseconds, API tests of seconds, and GUI tests of minutes. Calculations and scenarios that can be covered by unit tests are scenarios that don’t have to be covered in the layers higher in the pyramid. The different layers of test automation depend on the technology stack used. A unit test tests the smallest testable unit, which for example, in Java is a method, but in Oracle SOA Suite (a layer above Java) it is a component or service.
A degree of speed can be gained with parallel testing. Imagine a test set of 60 test cases taking three hours to run, the longest execution time of a test is four minutes. When running 60 parallel sessions, the total execution time will be the time of the longest test – four minutes! But parallel testing is easier said than done. Parallel testing requires a proper test strategy. The tests must be independent, both in execution and test data. Tests can’t use the same test data and when two tests require a database mutation, running those tests parallel can be a challenge.
What does Bruce Lee have to do with this?
In addition to the options above, there are many more possibilities to achieve continuous testing. How to apply it in practice is the big challenge here, and this is where Bruce Lee can help! Bruce Lee learned about the many possibilities in different Kung Fu styles, adapted the elements that worked for him, and rejected the elements that didn’t. Easier said than done if you ask me. Especially that last part; because of pride we sometimes hold on too long to something that doesn’t work.
The relevance of the principles and beliefs of Bruce Lee in agile software development can be seen in phenomena such as Bruce Lee-Driven Development and Kung Fu Coding.*
In my current assignment as a test automation coach I aspire towards this approach with the different teams I work with. In every team, I encourage a clear vision and strategy for test automation. A retrospective is a start of the analysis of the current situation to determine the most beneficial improvements best suited for a specific team. Recently, I created a lightweight offering at Capgemini to help the customer identify these improvements and create a continuous improvement environment.
Do you want to be water instead of a bottleneck in your DevOps journey? Then do like Bruce Lee and apply the possibilities and strategies that suit you best!
“Adapt what is useful, reject what is useless, and add what is specifically your own.” – Bruce Lee
* Do you want to find out more? Check out the links below!
Bruce Lee-Driven Development (slideshare): https://www.slideshare.net/JeroenvanderGulik/2018-0309-bruce-lee-driven-development-confoo
Kung Fu Coding: https://f3yourmind.net/2007/06/04/software-developmentkung-fu-coding-bruce-lee-style/