I got inspired to cover this topic as a blog based on one of the replies to my earlier post on “New Breed of Testers in DevOpsWorld” which discussed how testing careers would change with evolving Agile/Dev-Ops discipline.
The query was “It would be good to add 'what happens to Testing' instead of 'what happens to the tester'. You mentioned that tester would handle automated code deployment and high level coding. This is a view to engage testers but would Dev-Ops shrink the amount of testing that is done in current world? If so, what would be the impact? I recently read that testing spend would increase from 35% to 40% by organizations. Wouldn't it contradict the 'Dev-Ops effect'?”
In this Point of View, I would like to discuss the future of testing engagement models in context of current shift to Agile/DevOps and would like to relate this to predictions of increased QA spends.
Let’s combine this data point prediction of increased QA spend to 35-40% with few more observations from the latest Capgemini World Quality Report (WQR) 2015.
(1) How is QA Effort Spread
(a) Split of QA Effort between Development and Production Support
- Share of QA spend in new development projects (as against enhancements/maintenance projects) has grown from 2012-13 to 2013-14 to 2014-15 from 41% to 46% to 52%.
- In 2015-16, the QA spend is split 50:50 between new development and production support.
- In 2014-15, for the first time, the QA spend for Development exceeded, the QA spend for production support. This is in line with more development projects due to organizations undertaking more Digital Transformation related development work.
Point of View:
- It is evident that when QA is done for new code (as against QA for maintaining/enhancing existing code), QA effort would be higher.
- Organizations are practicing Agile/DevOps to achieve cycle time reduction (faster time to production or time to market objective) and not necessarily a QA cost reduction objective but also to achieve competitive advantage. If code is released more often and tested in shorter cycles, one would achieve objective of cycle time, and not necessarily cost reduction.
- Continuous Integration and Continuous Deployment are practiced. One can’t integrate or deploy less tested faulty code, so continuous testing means more iterations of testing.
- The testers are carrying out executions up until production deployment and in live environment aligning to extreme right. Analysts such as Forrester and Gartner are recommending, not only Shift left to test early in life-cycle but also shift right to extend tester involvement till deployment.
(b) Increase of QA Effort in Design Phase
- 2014-15 WQR report highlighted Increased Involvement of QA in high level design phase to 14% (as against 6% in previous Year). What is being achieved by QA involvement in Design phase?
Point of View:
- Shift-Left practices are being leveraged to have testers contribute their domain expertise in design phase.
- In line with trend of continuous testing to support continuous integration and deployment, there is increased automation in silo phases of test design and development (to match software design phase).
- With automated tools supporting test models and test script generation and their integration with automated test execution, reporting and deployment tool have further pushed the shift left theory right to requirement phase.
- This increase of the effort is being debated testing functions becoming costly however based on the ROI’s this has proven to be more efficient both on cycle time and cost parameters because of reduction in Manual interventions, reworks and accelerating time to delivery.
(2) How are QA Organizations going to be structured?
(a) QA organizations would follow Hybrid QA Teams (Centralised and Decentralized). Use of internal independent industrialized test TCOE has stabilized and may see decline. To Cite Data from WQR
- Use of a combination of Centralised and Decentralised Teams (Hybrid) for QA grew from 35% in 2013-14 to 43% in 2014-15.
WQR 2015 reported 24% respondents citing plan to have internal TCOE as against 24% in 2014-15 and 23% in 2013-14.
- 18% cited no plan for TCOE in 2015-16 as against 20% in 2014-15 and 26% in 2013-14.
- 11% cited plan to use 3rd Party TCOE in 2015-16, as against 15% in 2014-15 and 19% in 2013-14.
Point Of View:
- The above observations point to need for testers to work with developers through life-cycle – shift-left as well as shift-right. Use of dedicated TCOEs would not continue to rise as in past and shall need to scale to provide services in platform as service model.
- In coming times, if not already, we need to be prepared to see a cross functional team which is leveraging the automated toolsets optimally to push the tested functions into production leveraging 6 C’s of DevOps (Continuous Planning, Continuous Integration, Continuous Testing, Continuous Deployment, Continuous Monitoring and Continuous Improvements) to deliver highly performing system yet reducing time to market considerably.
- A tester would be expected to be aware of the functional details and help developer by isolating the bugs to the extent of pointing to the line of code, indexing structure of the database, pinpointing the infrastructure that shall or is becoming the bottleneck etc. in meeting business objective
(3) What does this mean for Future of Managed Testing Services (MTS)?
Analysts such as Forrester believe Managed Test Services (MTS) deals in future may be fewer. Gartner believes Managed Test Services (MTS) market would see 15% growth. Organizations that are yet to mature in rolling out managed testing functions shall need to look at establishing MTS organization while the organization that have MTS organizations or equivalent need to look at Managed Test Operation (MTO) with aim to mature testing functions to Quality Assurance. There is little doubt that organizations specializing in quality / test service offerings are looking at ways to align their models to client needs who in turn has become more demanding due to awareness with expectation that Service Providers need to mature in becoming business partner in this space.
- Provider Perspective - What do our clients need? Also, what are providers offering?
- Client Perspective – What do I need? What do I want my providers to offer?
Point Of View:
Providers as well as our clients have three plateaus in their businesses. I describe these plateaus below–
Business As Usual (BAU) Plateau
Clients want to keep lights on for section of their BAU portfolios, manage their legacy applications to continue to service customers. Many of these engagements have QA & Testing “embedded” into the maintenance/ enhancement projects. The share of these engagements has continued to shrink over the years making a way for independent testing.
Capgemini addresses this plateau with “Test Factory” model to pool testing for many engagements, with ability to offer continuation of knowledge, availability of tools, centralized governance and capability planning, service based on work delivered rather than continuing dedicated practitioners to engagement during lean periods.
Clients want to industrialize their testing, get cost advantage, and route the savings from this segment to be able to spend into new initiatives. Testing is easier to separate out as validation step through the life-cycle, to be carried out by dedicated test practitioners, using best of the process, methods and tooling, and offering QA service to meet the agreed SLAs/KPIs.
Capgemini offers Managed Test Service (MTS) that have industrialized the manner in which testing service is offered and has offered cost advantage. Managed test services leverage offshore, core/flex, process and tool assets and transformational levers to carry out testing in an industrialized manner. Capgemini has also evolved Agile to be carried out remotely through “distributed agile” method and has pioneered “Agile MTS” offering in line with Agile/DevOps world. I have a separate blog on “MTS in the Agile/DevOps World - Capgemini’s Agile MTS Offering”. What is considered “Innovative” and “Specialised” today, needs to be offered as “Industrialized” in future, and soon to become “BAU” in our fast evolving IT landscape.
Clients need to address to digital business and be present in multiple-modes, and multiple speeds to cater to varied customer segments. Speed does not always come without cost. Clients seek specialized services (New Developments in Agile/DevOps mode) based on the needs. To address this segment, customers need business assurance (e.g. assurance of time to production, time to market, reduced failures,..). High quality, and timeliness are most important for this plateau. Cost reduction, while a relevant need, may not be addressed for this segment due to time pressures.
Capgemini has Specialized Test Services (STS) to cater to innovative projects QA need of (i) Agile/DevOps Testing with leading edge automation tools (ii) Business Assurance to partner with clients in the journey to reduce risk and share the risks and rewards through agreed rewards and penalties.
(4) Conclusion – Would DevOps Shrink the amount of Testing? Are Predictions on increasing QA spends justified?
DevOps is actually spreading testing across the Software Development Lifecycle. The builds are smaller and so are QA lifecycles. There are continued multiple builds though offering advantage of speed, which does not necessarily mean less QA or lesser QA spend. With new development projects on increase for Digital transformation projects, overall QA spends would be on the rise. One of the challenges to address would be to achieve a balance of the triple objectives (Time, Cost, and Quality) in the evolving QA/DevOps world.