Crowd sourcing: Enabling Flexible, On-Demand Testing COEs

Publish date:

“While a managed service model provided by an expert QA vendor ranks the highest in importance (49%), the use of a crowd sourced solution model for quick test capacity is used by 47% of the respondents. This puts crowdsourcing ahead of a captive TCOE factory (45%), Hybrid TCOE factory model (45%) and a TCOE in […]

“While a managed service model provided by an expert QA vendor ranks the highest in importance (49%), the use of a crowd sourced solution model for quick test capacity is used by 47% of the respondents. This puts crowdsourcing ahead of a captive TCOE factory (45%), Hybrid TCOE factory model (45%) and a TCOE in an offshore location (42%).” World Quality Report 2016-17

As cited in the latest World Quality Report, crowd sourcing offers quick, on-demand testing capabilities. The demand for crowd sourcing is fueled by the growth in digital transformation engagements that require applications to be available across multiple channels, creating a bi-modal business. As a result, the service providers are designing the operating models using hybrid TCOE (combining service provider staff in both managed service model and crowd-sourced testers). This hybrid model has helped address the need for both a flexible and a managed-service operating model.

(1) What is Crowd-Testing and how does it Work?

“If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it?” (Paul Herzlich)

Crowd sourced testing is the process of using a tester pool to provide both formal and informal feedback on a product. Crowd-testing pilots software under real-world conditions with real users, allowing companies to gain insight and gather feedback to identify defects quickly and intelligently.

The figure below provides a simplified model of crowd-testing with the following key elements: crowd-testing platform, crowd, crowd-testing champion, and crowd-testing Services.



1. Crowd-testing platform: a cloud based platform, it manages testers, tracks their testing activities, curates their rankings based on the feedback given, manages compensation based on an output-based model, and integrates with standard test management tools.

2. Crowd: this represents the independent, geographically distributed testers who are part of the network and provide services on invitation. These SMEs are willing to provide services at a precise cost and services output. 

3. Crowd-testing champion: As part of the crowd-testing management team, a champion manages the deliverables meeting the client expectations. The champions could come from the customer end or be deployed from a crowd-testing service vendor.

4. Crowd-testing services: These involve testing devices, sites, content, and other elements as requested by the customer.

The crowd can be public (e.g., a subject specific community) or private (a client workforce community or a clients’ customer community). The public crowds provide global coverage on-demand and on their own terms. Public crowd subjects or SMEs can be utilized for deeper analytics and improvement potential identification. The “private crowd” makes up part of the client work-force community with product knowledge and provides user experience. They are motivated by appreciation and personal or organizational brand identification.

(2) Crowd-testing Operating Models

The two broad operating models prevailing for crowd-sourced testing include:

1. Testing as a service: the QA vendor provides internal employees as private crowd along with a crowd-testing platform and testing tools to enable the client to manage end-to-end testing using their own crowd-testing champions. This is the most common format of managed crowd testing services, and the client pays only for the outcome of the testing, e.g., the number of critical bugs identified in a given time frame and the severity of the bugs.

2. On-demand tester staffing: This is applicable when the scope of testing includes both private and public crowd, and where the client is looking to hire testers on weekly or monthly basis as retainers instead of using an output based model.

3. Crowd-testing as part of Testing Center of Excellence (TCOE): Crowd testing can be used to provide flexible staffing in TCOEs.   

When and where should crowdsourcing be introduced in a typical end-to-end testing engagement? Crowd-testing is often embedded in an end-to-end testing scope and employed during user acceptance testing. Crowd-testing is also used for niche testing needs such as localization, translation, content testing, and accessibility testing.

(3) The Key Challenges of Adopting Crowd-testing

Crowd-sourced testing utilizes testers spread across the globe. However, typical problems with these distributed teams include:

  • Communication challenges
  • Increased management overhead
  • Difficulty in obtaining stakeholder buy-in
  • Problems synchronizing all stakeholders on product updates      

Additional issues encountered in crowd sourcing include:

  • The need to ensure security and confidentiality of the product
  • Keeping the crowd motivated and encouraged regardless of monetary returns
  • Difficulty analyzing and interpreting the varied results of a crowd’s efforts
  • The complexity of using multi-platform, multi-geographic, and multi-lingual environments
  • The presence of a diverse user demographic with low tolerance and a potentially negative reputation
  • Concerns with quality, cost, and speed
  • Issues in providing secure, dedicated access to a client network.

(4) The Benefits of Crowd-testing

Crowd-sourced testing has many benefits, including the following:

  • Native testers using local devices for testing offer more reliable application behavior
  • Crowd testers bring the advantage of covering multiple devices, OS, configurations, and languages.
  • Crowd testers bring in knowledge of real-world scenarios for given end-user locations.
  • Crowd sourced testing offers a more predictable market acceptance, as the crowd acts as a beta-user.
  • Crowd can be leveraged for user assessments across different stages of application development and serve as structured beta testing
  • Crowd-testing can offer time-to-market advantages and support faster launch times

(5) Conclusion

The World Quality Report 2016-17 predicts that a centralized TCOE operating model will shift towards a hybrid TCOE in order to address both a flexible and a managed services operating model. Crowd-testing will offer flexible, on-demand testing capabilities as part of a hybrid TCOE model. Outsourcing vendors are either partnering with crowdsourcing platform providers or developing the ability to address crowd-sourcing methodology. The digital transformation will drive the need for enterprise crowd testing and disrupt the existing outsourcing models. Client will use different sourcing models to manage their testing needs while increasing costs, time to market, and contract duration.  

(6) References

World Quality Report 2016-17 –



Main Author: Renu Rajani, Vice president, Testing Global Service Line, Capgemini India;

Contributing Author: Manojkumar Nagaraj, Associate Director; Testing Global Service Line, Capgemini India;


For an in-depth look at the key trends in Testing and QA, download the World Quality Report 2016

Related Posts

Artificial Intelligence

Intelligent automation is driving digital transformation of Financial Services – Part 2

Date icon November 30, 2021

From faster response times with RPA, optimizing pricing and underwriting, and data-error...


Can AI save the security operations center?

Geert van der Linden
Date icon November 30, 2021

AI technologies can also perform threat modeling and impact analysis, activities that...

Accounts receivable

YayPay drives frictionless order-to-cash

Date icon November 29, 2021

Divya Turner talks to Sarah-Jayne Martin about how YayPay’s next-generation order-to-cash...