Making your QA teams succeed – Five questions to ask

As one of the two lead authors, I’ve just added the finishing touches to the twelfth edition of the World Quality Report. For this edition, we polled 1,750+ respondents, of whom approximately 25% are chief information officers. In addition, we surveyed the CTOs, product and engineering heads, and key leaders in the QA and the development teams. One noticeable feature of this edition of the report is the ever-growing expectations from the QA teams.

As you can see from the graph, there’s an upward trend in almost every expectation from the QA teams compared to that in the previous years. The top expectations this year are related to the contribution of QA to drive business outcomes and ensuring end-user satisfaction. Both top expectations indicate that QA is no longer expected to be a backroom discipline but rather at the forefront of business growth.

While it is good to see this change, and although a large part of meeting the expectations will depend on how QA teams rise to this challenge, it is equally important to examine whether they are getting the right support from CIOs and business leaders. We suggest that CIOs ask these five key questions to increase the probability of success of their QA teams:

1. Is your QA team empowered to drive business outcomes and customer experience?

It is a good idea to examine the responsibilities and activities of the QA team. If you find that their activities mainly relate to testing, which is by design meant to determine whether the actual software product matches the defined requirements, then it would be a stretch to say that they can drive business outcomes. You can also examine the features added to the product based on changes suggested by the QA teams. There are several teams, such as sales and customer servicing teams, business analysts, developers, operations teams, etc. that influence quality. You must examine whether the QA teams are empowered to influence these quality influencers. If the developers aren’t doing unit testing, does the QA team have the authority to reject the build, and have they ever done that? Please also confirm whether your QA team is carrying out experiential testing along with the actual users to enhance your customer experience. These are just a few pointers, but the key is to do an assessment and ascertain whether the QA teams are empowered to meet your expectations.

2. Have your QA teams started putting less emphasis on business expertise in the name of adopting a Quality Engineering way of working?

We often come across instances where the focus of the QA teams has shifted to technical expertise. While there is benefit in adopting and following quality engineering practices, this should not be done at the expense of business expertise and focus. Business knowledge is one of the key skills that every QA team member should possess. Often, automation is seen as a solution to all problems, but test automation can only verify the product against the specifications. So, please consider automation as a means of achieving business outcomes rather than an end objective. Again, a few simple things can be done to validate this. If you look at the training data of the QA team members or do a sampling of the skill description of QA team members, it may tell you the reality on the ground. One must also keep in mind that expecting each team member to have significant technical expertise, business knowledge, and soft skills is unrealistic. Only a few team members will be that well rounded. Hence, getting the right team composition is very important.

3. Do your QA teams have expertise beyond testing or are these the same set of testers expected to perform differently?

Albert Einstein once said, “we cannot solve our problems with the same thinking we used when we created them.” This holds true for team skills and experience too. We cannot expect a different set of results unless the skill composition of the QA team changes in keeping with the new set of expectations. Advances in analytics, machine learning, and artificial intelligence are making QA efficient and effective. There are new tools in the market, which ensure that the quality is integral to the entire product development lifecycle. Test environment and test data management are becoming core to the success of QA teams. QA teams need to have members with expertise in these different areas to succeed. So, embedding data scientists or UX designers as part of your QA team is going to help them meet your expectations.

4. When it comes to cost optimization, do you look at the QA team as the first place to cut the budget from?

We have seen instances where certain parts of the organization dismantled the testing team, stating that since quality is everyone’s responsibility, they don’t need a dedicated QA team. And then quality issues started cropping up and they started putting the team back. Often, when quality becomes everyone’s responsibility, it ends up being no one’s responsibility. The second point on this is related to the unit cost of QA team members. If one expects QA team members to have technical skills, business skills, and soft skills, they aren’t going to come at a cost lower than the rest of the engineering teams. So, instead of looking at the unit cost per QA team member, you should focus on the value addition per unit cost. The third aspect is the cost related to the testing tools and environments. While open source solutions are great in many cases, there is merit in adopting commercial tools in several other scenarios. The evaluation should be done keeping the cost-benefit in mind rather than always going with open source solutions. In summary, evaluate if your expectations from QA has changed without the corresponding investment that comes with it.

5. Do you have the right measurements in place to drive the desired behavior?

This one sounds simple, yet, if you look at the metrics on which the QA teams are measured, the irony soon becomes evident. Seldom are business outcomes or end-customer satisfaction part of QA team performance metrics. For example, expecting the QA teams to ensure quality throughout the product lifecycle, but measuring them on testing cycle time or expecting them to be proactive about defect prevention but measuring their performance on the number of defects detected, isn’t going to produce desired results.

In summary, I would say that while it is great to set high expectations from the QA team, it is equally important to ensure that they get the right support and are measured on the right metrics to meet your expectations.

To find out more about what your industry peers are saying about QA, please visit this page and register to download the World Quality Report 2020-21.

Dhiraj Sinha

Expert in Agile, Application Modernization, Application Testing, Cloud Computing, DevOps, IOT

As organizations undertake digital transformation journey, quality assurance becomes a key aspect to achieve end customer delight. I work with my customers to drive transformation in the way their organization assures quality to achieve accelerated business outcomes.
I have significant experience in IT services industry having strong technology focus coupled with business acumen. In my role, I have managed large scale delivery and operations while keeping a strong customer relationship and focus on business growth. Have lead globally distributed work-force (in US, Europe, Mexico, Brazil, Malaysia and India) and adept at working in multi-cultural setting.
In my career spanning about 25 years, I have lead large delivery teams, set-up and grown various technology practices, managed important customer accounts and handled large scale operations (approx. 1B USD business portfolio). In my 9+ years tenure at a previous organization, I set-up multiple practices (Business Intelligence, Testing, Enterprise Applications and Apps Packaging), and played a key role in scaling up their global delivery capabilities.
I worked in client facing as well as service delivery roles for some of the marquee customers in Banking & Finance, Insurance, Hi-tech and Media industry verticals.