Intelligent testing

Product testing is become increasingly complex. Not so long ago, products were physical things that underwent physical tests. A heating system was run under different conditions to check it worked in all likely end-use environments. A car was crashed to see how safe it kept its passengers responded.

As products have become more complex, connected and data-driven, testing has also become much more complex. Most modern products – from Smart Buildings to autonomous vehicles – are connected to internal and external ecosystems of similar devices, and cloud-based service platforms. And the products themselves are increasingly complex, especially those that embrace elements of AI. Over the last 20 years, for instance, the number of lines of code in a single vehicle has multiplied by a factor of four. Added to this is the risk of cyber threats from attackers looking for a way into a company’s network via connected devices.

Necessary testing

Nonetheless, product testing is essential. Businesses face regulatory pressure to make their products safer and more sustainable. User expectations are becoming ever more sophisticated, too. They expect the addition of new features, often updated over-the-air at a rapid pace, and they expect these to work first time without jeopardizing their security and the overall user experience . Fail to meet these expectations, and they’ll take to social media. Total integrity is needed in product design, which means rigorous testing.

A direct effect of this is that tests have become difficult to execute, and difficult to plan – after all, it’s important to define which types of tests are needed to ensure that integrity. And this affects cost – testing the interoperability of a product with millions of lines of code and being sure everything definitely works could become prohibitively expensive.

It’s essential, therefore, to optimize testing activities and increase the value they deliver.

Define what to test

The desired future of testing is data-driven.

As noted above, testing is getting longer and more expensive.  What’s needed, therefore, is a thorough, reliable, smart solution that will cut time to market, without compromising product integrity.

Achieved this desired future begins with an AI-driven test definition. This leverages AI to define and optimize test plans to ensure full domain coverage and the bare minimum number of tests. So, for example, test plans for a new connected medical device would be issued based on patients’ age, gender, body mass, and the severity of their condition. The new device can then be trialed on those conditions – testing its UX, APIs connectivity, etc – to identify likely problems. Those problems can then be prioritized for more detailed testing, whilst areas that all integrate seamlessly will need less testing.

Test automation and simulation

Following a complete definition of the necessary tests, those tests must be executed as quickly and cost-effectively as possible. This requires test automation, an infrastructure that automates activity which, traditionally, would mostly be performed manually. Previously, for example, testing a connected heating system would require an employee to heat it up, switch it on and off, and check that everything worked properly. Now, though, this is all done automatically. A test execution platform will perform a thorough library of previously defined tests, often using robotics to manhandle it and computer vision technology to see how it responds. In other cases, it could simulate digital elements in the cloud against 100 different virtualized mobile devices – a significantly more efficient method than performing those tests in the real world.

This automation and simulation decreases costs and reduces time to market, both for the initial launch and for product upgrades.

Data value enhancement 

The final step is to extract deeper value from the testing data to improve product and design quality.

Testing produces a large set of data. Typically, when a test fail is added to a test set, it requires a deep dive into the data to try and understand why that test failed. If the test is passed, however, the data tends to be filed away and forgotten about – after all, everything’s fine. But this leaves a large set of underused data.

More tests can then be added, and the test path refined for those areas where there was previously no information or validation. It’s an iterative process, in which test run will be performed after test run to continually improve coverage and, simultaneously, decrease the number of test steps.

Analyzing the results of every test will allow you to identify opportunities to improve both the product and future testing. AI can look at this test data to gather deeper insights such as small signals in the data that may hint at rare points of failure. It can then define and refine the test set within these ranges to expand test areas where data coverage is insufficient.

Confidence and efficiency

The growing complexity of product development and connectivity means effective testing is becoming ever more time-consuming and expensive. There are three key benefits: to ensure confidence in validation, to improve operational efficiency of the testing program, and to use data-driven testing to better understand potential issues.  By defining what to test, optimizing testing, and deriving value from the testing data, intelligent testing can be used to reduce costs and shorten time to market, whilst ensuring maximum robustness in the testing strategy.

The increased confidence and efficiency this approach allows not only addresses existing validation challenges, but also opens new opportunities for product development – continuous improvement of product testing, accelerated introduction of new features or product evolution and, perhaps most importantly, improved user experience.