An intelligent approach to continuous testing

“If it hurts, do it more frequently, and bring the pain forward”.
― David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

                <p>All our clients today are taking steps to integrate testing as part of the continuous delivery pipeline.</p>

This was emphasized in our recent World Quality Report, a joint publication of Capgemini, Microfocus and Sogeti, where 88 percent of our financial service respondents stated that they are integrating or planning to integrate automated tests in the delivery pipeline. Forty-one percent of the surveyed respondents also stated that they are planning to use predictive analytics in testing.

Predictive analytics and Artificial intelligence is the science of drawing intelligent insights from raw sources. There are three sources of data namely code, test case data, and production from which insights can be generated in three ways with the help of analytics and machine learning algorithms. These are explained in brief below:

1. Code analytics. This assists in determining which test cases are impacted, once the code is changed. This can be achieved by mapping lines of code with test scenarios to generate a 360-degree code map using tools such as Corbeteura and SonarQube.
2. Test optimization: In this scenario, historical test case data can be analyzed by a machine learning algorithm, for example, natural language processing, and the random forest algorithm. These algorithms analyze test case and defect data and assign a quantitative risk index to each test case, as well as remove duplicate test cases.
3. Production analytics – Machine learning engines that perform sentimental analysis are used to analyze the user and usage behavior of the application in production. Insights from these are then used to optimize test cases.

Let us take an example of how the above three modules would aid in integrating test types in the delivery pipeline as indicated by the figure below:

1. When a developer checks in code the Code analytics module analyzes impacted test cases. Let us assume theoretically that these are 1,000 impacted test cases.
2. These 1,000 test cases serve as an input to the Optimization machine learning module. This identifies high priority test cases and removes duplicates optimizing the test pack to 800 test cases.
3. These tests are run as part of the pipeline and assuming that all tests pass the code is deployed in production.
4. Once the scripts are deployed in production the Production Analytics module tracks application usage and generates heat maps that provide information back to the development and operations team on most used scenarios in production leading to further 10 percent optimization to 700 test cases.

In summary, analytics and machine learning algorithms as illustrated in the scenario above would effectively optimize the test suite by a minimum of 30 percent. The key to above as in all analytics and artificial intelligence initiatives is the availability of reliable data.

To learn more about how predictive analytics and machine learning can be used for continuous testing free to get in touch with Akash Sinha or me.

Download the first edition of our Continuous Testing Report to learn more about the latest testing trends.

Co-authored by: Aakash Sinha