Crowdsourced Testing | Of the Software, By the Crowd, For the User

Recently my elder brother (and a guide) messaged me a link to LinkedIn Company page for Qualitrix, just the page link & nothing else – no text. It was to highlight the Crowdsourced Testing idea and the potential it holds, if we can crowd source anything out of Software Testing Studio network. I have still not figured it out, working on it will take some time. In the meantime I thought to at least cover the concept of crowdsourced testing.

TCL’s (Transition Consulting Limited) chairman Stewart Noakes once said,

“Not only is software testing being revolutionized by the idea of using a global community of professional testers, but the concept has proven to deliver compelling, real world business benefits at a fraction of the costs of traditional software testing.”

To get started: The concept has been around from a long time. The most famous example of Crowd Sourcing is “Wikipedia”. Wikipedia, the most comprehensive encyclopedia is the result of the information created by writers and editors from the crowd. The various Bug Bounty programs run by Tech giants is a wonderful practical application of crowdsourced testing which reward researchers and software hobbyists for finding software bugs. Beta testing (many apps have beta versions available in play store / app store including Whatsapp Beta program) is also a form of crowdsourced testing where a set of end-users can access the application & provide their feedback.

Crowdsourced Testing

“When Talent is recognized as Service.”

What I like most about Software Testing concepts is the simple terminology. Most of the terms are self-explanatory. Crowd as in mob, mass or a multitude. Sourced as in obtained, tracked or located. Testing as in ensuring quality. Taking a cue from the terminology itself,

“Crowdsourced Testing ensures product quality by sourcing the feedback from the crowd.”

  • Crowd: a community of expert quality assurance testers (team)
  • Sourced: tracking (or delegating) the test activities or process
  • Testing: identify issues to fix (feedback)

Note: Crowd is not a bunch of voluntary testers. Crowd refers to sourcing relevant and skilled testers from every disciplines and levels, appropriate for the given task. Task results turns into superior quality when there are combined ideas, skills and participation of people from varied background. Crowd is built on the basis of analysis of their performance over time. These testers can provide feedback on diverse phases of product development.

Crowdsourced Testing model

“Increase revenue by finding bugs before they cost you customers!”

Crowdsourced testing models can work in several ways, let’s have a look at the basic high-level flow:

  1. Crowdsourced Testing companies provide the platform for the testing cycles. It typically includes a cloud deployment for the application-under-test, a crowd of voluntary testers and a means to connect both.
  2. Client gets into an agreement (contract) with the CT Company to avail its services and the product is deployed on the platform.
  3. The product is then crowdsourced to a community of ‘suitable’ testers who register for testing the software voluntarily by completing their profile, i.e. experience, skill set, availability of devices, background, qualifications, etc.
  4. Crowd testers may use their own devices and environments to test the software, or they may access virtual machines (VMs) or device emulators via the crowdsourced testing platform.
  5. Tests can range from broad exploratory testing to highly specific functional verification testing, i.e. provided with instructions, detailed test plans, scripts etc.
  6. The testers provide feedback on the software, usually including a list of bugs and defects found while executing the tests. Discussion of relevant issues and concerns can also be done through forums.
  7. Crowd testers are generally paid per bug, depending upon its priority & severity. Some Crowd source companies also have rating systems which determine the quality of job delivered.
  8. The project coordinator then validates the testing outcome and prepares the final set of deliverables and its results.
  9. The final outcome is then shared with the customer where the statistics play a major role denoting the crowdsourced project being a success.
  10. Depending on the task, some platforms offer access to their crowd through a self-service model, a managed service by the provider, or both.

Benefits of Crowdsourced Testing

“Soliciting human feedback before you ship and fixing the major emotional pain points is the single biggest thing you can do to improve customers’ perception of your product.”

Crowdsourced Testing offers multiple benefits,

  • Unlocks the collective creative power latent in consumers to accelerate marketing and innovation. Additionally Crowdsourced Testing services are available on short notice and allow you to get results in as little as a few hours.
  • Real-world Tests: Crowdsourced testing ensure that the product is tested in various real-world situations instead of in-house controlled testing.
  • Mix-and-Match Multi-platform testing: Don’t tell me enterprises manage all range of hardware & software combinations in-house, i.e. systems, devices, operating systems, browsers, bandwidths, networks, etc. Stop the ‘But it’s working on my machine’ syndrome J Crowdsourced Testing offers a large variety of hardware and software combinations.
  • Flexibility: Crowd encompasses testers from different locations & time zones, thereby offering great flexibility, i.e. conduct tests when it is most convenient for crowd, no matter the time or day of the week.
  • Time-to-market: If a team of four testers take 2 weeks’ time to test an application, calculating time-to-market with 100’s of testers is not a scientific math.
  • Effective & Efficient: Crowdsourced Testing is done by a crowd of real-brains where everybody has his/her own perspective (or approach) to testing. It’s cost effective since you pay only for the valid bugs reported, that too without hiring full-time software testers. In today’s industry, faster time to market and cost effectiveness are key differentiators for any brand.
  • Unbiased: No Managerial pressure. No company guidelines. No process adherence. Crowd is fresh & free to test the product their own way thus providing more objective feedback & focusing on just one thing that matters – bugs!
  • Local Testing: The crowd is diverse in languages as well as locales which helps in testing local & global applications.
  • Skill-based: The payout is not fixed and vary depending upon the skill set & merits of a crowd tester.
  • Crowdsourced testing enables applications or solutions to be tested remotely without the constraints and costs associated with static offices and locations.
  • Crowdsourced Testing provides a bandwidth boost that can help improve product quality, while also freeing up in-house resources for other quality-driving activities.


Crowdsourcing was coined in 2005 as a portmanteau of crowd and outsourcing. This mode of sourcing, which is to divide work between participants to achieve a cumulative result, was already successful prior to the digital age (i.e., “offline”). Jeff Howe, who coined the term crowd sourcing, suggested that it encouraged the best-qualified and most creative participants to give input to a project. Wikipedia reveals some interesting events,

  • 1783 – King Louis XVI offered an award to the person who could ‘make the alkali’ by decomposing sea salt by the ‘simplest and most economic method.’
  • 1884 – Publication of the Oxford English Dictionary: 800 volunteers catalogued words to create the first fascicle of the OED
  • 1975 – ‘Manthan’ movie directed by Shyam Benegal about the story of Amul brand was funded by 500,000 farmers who contributed Rs. 2 each.
  • 1997 – British rock band Marillion raised $60,000 from their fans to help finance their U.S. tour.
  • 2000 – UNV Online Volunteering service launched: Connecting people who commit their time and skills over the Internet to help organizations address development challenges
  • 2001 – Launch of Wikipedia: “Free-access, free content Internet encyclopedia”
  • 2004 – Toyota’s first “Dream car art” contest: Children were asked globally to draw their ‘dream car of the future.’
  • 2006 – Jeff Howe coined the term crowd sourcing in Wired.

Crowdsourced Testing vs. Outsourced Testing

Crowdsourced Testing differs from traditional outsourced testing in that the testing is carried out by a crowd of different testers from different places, instead of hired professionals. Unlike hired tester a crowd tester might not get paid if he/she is unable to find any defect, i.e. it’s a bug-bounty. Crowdsourced Testing allows to conduct tests under diverse realistic platforms which makes it more reliable, cost-effective, fast, and bug-free.

Challenges to Crowdsourced Testing

  • Confidential: Crowd tests must be managed closely as the number of non-internal individuals looking at the application-under-test increases. Tests are often executed within virtual machines (VMs) that provide a clean, controlled environment for testing and secure access to web applications.
  • How do you manage the quality of services provided by a crowd? By building a limited, elite group of freelance software testers who are passionate about what they do and equally passionate about doing it right.
  • Test coverage: How do you ensure requirements coverage since it is not planned or tracked? Crowd source the Test case execution along with exploratory tests.
  • Reporting? Review the work of every tester participating in our software testing projects to ensure great results.
  • Communication: You cannot ‘expect’ immediate & prompt communication with a crowd of testers
  • Un-organized: Since testing is happening at different locations, devices and time zones – management needs a way to organize all the details for reporting.
  • Reliability: How do you know the Test results reliability? Crowdsourced testers may not be immediately available for communication and clarification. As a result, some crowdsourced testing solutions can offer unreliable or incomplete software feedback.

Few cases for Crowdsourced Testing

“Before rollout, what’s better than getting your product tested from a ‘varied set’ of expert testers?”

Enlisted below are few cases where Crowdsourced Testing is most useful:

User-centric: Crowdsourced Testing is considered when the software is more user-centric: i.e., software whose success is determined by its user feedback and which has a diverse user space. It is frequently implemented with gaming, mobile applications, when experts who may be difficult to find in one place are required for specific testing, or when the company lacks the resources or time to carry out the testing internally.

Fragmentation: Another usage of crowdsourced testing is when the permutations and combinations of hardware, software & network increases. Crowd provides multiple combinations of devices, operating systems, browsers, networks and thus certain specific issues can be determined.

Load Test: As multiple testers will hit the application at same time, intentionally or unintentionally, this form of testing can be proved as very useful in determining the behavior of application when multiple users hit at the same time.

Usability: Since crowdsourced testing involves multiple live users, it is best suited to get a different feedback on the application usability.

Key Pointers

  • A holistic approach would be to complement dedicated team with crowdsourced testing instead of replacing it, allowing enterprises to scale the resources up and down in a fluid manner.
  • Recruit right testers and put a check on the work quality delivered to meet client’s requirement and trust.
  • Reward for the efforts and quality delivered to maintain tester’s trust and interest.
  • Money should not be the only factor. Testers should be passionate enough regarding their work to prove their worthiness for the work.
  • Positioned as an extension to an in-house QA (quality assurance) team rather than a replacement, crowd sourcing attracts and creates an interconnected community of testers who seek fame and recognition within the marketplace based on the quality and quantity of the bugs they are able to find.

Conclusion | Welcome to the Crowd

“Testing with real crowd under real world conditions is best way to build successful software.”

Today, crowdsourced testing has become a popular trend in software testing (to augment in-house QA team) which exploits the benefits, effectiveness, and efficiency of crowd sourcing and the cloud platform. It provides Testing-as-a-Service functionality for many types of software applications using web-based, mobile, or front-end applications. The flexibility provided to the testers in terms of pay, time, work location, diverse range of testing options for testing diverse range of applications and many more have convinced many testers to select Crowdsourced Testing as their second job. Companies with limited testing bandwidth may benefit from using crowdsourced testing to increase test scope and coverage without hiring a large full-time team. It can be leveraged to achieve business objectives such as speed-to-market, improved quality of applications and cost savings. Crowdsourced Testing ensures the application to be robust and cost effective, especially for applications like mobile platform, distributed applications and globally available web applications which are deployed on cloud platforms.

Now that you know about Crowdsourced Testing, be a part of the crowd. You can register with any of these –,,,,,,, and

Note: I have no affiliation or connection to any of these products.



Leave a Reply

Your email address will not be published.