The World Quality Report in Focus: Acing the Test – Managing Test Automation, Data and Environments (Part 4)

Publish date:

The Keys to Business Alignment It will probably be no surprise that that the World Quality Report shows that more than a third of the overall testing budget is allocated to testing hardware and infrastructure. As quality and speed to market are the keys to successful app development, IT leaders know that creating an accurate, […]

The Keys to Business Alignment

It will probably be no surprise that that the World Quality Report shows that more than a third of the overall testing budget is allocated to testing hardware and infrastructure. As quality and speed to market are the keys to successful app development, IT leaders know that creating an accurate, reusable, automated and agile test environment is of paramount importance. What may be more surprising is that this expenditure has actually dropped 7% since last year.

So why is this? Well the good news is that this smacks of a slight increase in test maturity and also shows that cloud adoption, virtualisation and tighter integration are all starting to lower overall testing costs. On the flip side, the WQR shows that organisations are still encountering several barriers to creating and managing the ideal test environment as, for example, 35% of organisations find it difficult to complete the set up in a timely manner to keep the project on track. Delays in configuration and build equate to lost productivity and wasted costs. So what can organisations do to ensure their test environment is aligned with the needs of the business? The key lies in creating strategies and processes for virtualisation, automation and data management.

Costs and Carbon

The benefits of virtualisation include a faster set up, scalability, increased test coverage, greater flexibility (even with more complex apps) and time and cost savings. It’s therefore a good sign that 32% of IT leaders say that network and service virtualisation is a top priority, with 62% already using virtualisation to create network conditions for load and performance testing, 59% to validate app functionality and 54% to predict app security and risk. More than 2/3 of IT leaders embarking on a virtualisation strategy favour working with a carefully selected outsourced partner; with 23% favouring using a provider’s data centre, 22% using remote access and 23% utilising their service provider’s cloud.

An increasing number of companies are finding that a hybrid test solution works best with physical environments at their core, augmented by virtualisation:
Eastern European food giant Podravka was already automating its key business practices to great positive effect, but found that its server facility was mushrooming. They decided to turn to virtualisation to combat the problem, with excellent results; they were able to deploy virtual machines in hours instead of weeks, improve app availability and create a lower cost disaster recovery strategy. Their new virtualisation strategy will save them a total of $2m in 5 years and reduced their carbon emissions by 78%.
Sky High Cloud Adoption

The World Quality Report shows that a pretty impressive 32% of all testing now takes place in the cloud, enabling shorter lead times for hardware configuration, meaning projects are more likely to run on time and apps can be delivered faster, while the cost can be switched from CapEx to OpEx. To capitalise on these benefits more and more organisations are turning to the cloud, with testing in the cloud expected to increase to 49% of organisation by 2017.

So, why the cloud? Well as mobile continues to explode, the competitive differentiators are quality and time to market which means accurate performance testing is imperative to app success and reputation management. To see the importance of testing we only have to look at roll out disasters such as the US debacle during which stress tests, done far too late just 1 day before the launch date, revealed that the site virtually came to a grinding halt with only 1,100 simultaneous users when  50,000-60,000 were expected and 250,000 actually logged on! The cloud offers the scalable, cost-effective, fast, data rich environment that’s required to make projects like this successful instead of a PR disaster.
Sogeti’s OneShare is a comprehensive cloud platform for development and testing based on Microsoft Azure. Find out more about this service here.

TDM Grows Up

A major area for concern in 2013, test data management is growing in maturity and becoming a specialised role in QA and testing teams. Optimum efficiency can only be achieved through reusable test data sets and transactional test data to create reusable test scenarios and enable high levels of automation. Test data sets need to be backed up and restored frequently and there are strict privacy and security regulations about storing data. Therefore, anonymisation, data obfuscation and data masking techniques are an important requirement. The WQR shows test management is maturing as in 2014 only 41% of IT leaders had difficulty synchronising multiple versions of test data with multiple versions of test systems, as opposed to 65% last year. Virtualisation is also helping to reduce the complexity of data sets by allowing QA and test teams to validate customer credit in a virtual environment. Overall, test data management processes are becoming more structured as test teams define appropriate test data up front and create truly reusable test data sets. Sogeti’s Test Data Management service comprises a comprehensive 4 stage approach of current state analysis, tool selection and proof of concept, implementation and compliance operations. The full process and associated business benefits are set out in more detail here.


As we saw in our previous WQR blog [here], Agile is leading to a considerable growth in automation. Organisations have automated 25% of all test cases but IT leaders would like to see an even bigger rise to 35% by 2015 and 40% by 2017. The degree of successful automation varies from business to business with 11% saying they don’t automate at all and 7% saying they have automated over 70% of their test cases. This is dependent on their commitment to Agile, their Agile maturity and the types of tools and apps they use. The benefits of automation are clear and manifold including reduced cost, increased quality, freeing up resources and expedited development. Automation is not always the best solution though. For example, while load and stress testing and the simulation of services and networks all need to be automated to be affordable and effective, exploratory end user and validating usability testing and some legacy applications will need to remain manual.

A case in point is Vodafone Ireland who, with 56% of mobile market and 1.8 million customers, wanted to capitalise on their existing success and deliver new technology and products to their customers faster and more efficiently with focus on, of course, quality and time to market. Their development and test strategy was not coping with increased demand and new technology so they turned to a risk-based enterprise-wide testing strategy including KPI testing metrics, automated regression testing, delivery risk analysis and a team of 40 staff to implement it. The partially automated, risk based strategy yielded excellent results including a 50% reduction of post development bugs and a reduction of an estimated 170 days of testing! 

Overall virtualisation, test automation, test data and test environment management are growing in maturity. There is still some way to go before companies are confident that they have the right people, processes and tools in place to optimise these areas, but with the right partnerships and level of commitment IT leaders will be able to devise more efficient testing solutions that give rise to better quality products.

Related Posts


Site reliability engineering

Genesis Robinson
Date icon August 7, 2020

Due to the current state of how we monitor, alert, and log our digital ecosystem, it takes...


Building a culture of quality transformation

Deepika Mamnani
Date icon December 20, 2019

Transforming from traditional testing organizations to quality engineering organizations with...


Zombies, wizards, werewolves, and a test automation silver bullet

Grant Volker
Date icon November 21, 2019

Expectations of technology have dramatically changed over the years, creating a demand for...