Challenges in implementing test automation

Publish date:

We conducted a research about implementing test automation within the organizations. Read this blog to learn about the key findings.

The software has changed dramatically from small programs to complex comprehensive IT systems over the last decade. Our lives are increasingly determined by its presence everywhere – right from our toothbrush to our cars. The quality of the software, therefore, stands to be of crucial importance. Organizations, which rely on manual testing methods, have to deal with an increasing pressure on faster time-to-market cycles. Software testers have to work iteratively to ensure the product continues to work even when there are multiple deliveries, resulting in tremendous pressure on the test process. This is where test automation plays an important role.

We conducted research about implementing test automation within the organizations.  Based on this research we see that the organizations start enthusiastically with the implementation of test automation, but as time progresses, the enthusiasm decreases and the organization ceases.

Research approach

We examined 17 different projects regarding the implementation of test automation within test and QA organizations of different companies in the Netherlands. As a research method, we used semi-structured interviews.  In this case study, the Test Automation Consultants (TACs) who implement or work with test automation were interviewed. This research focused on the question: „What are the positive experiences and challenges with Test Automation within the different organizations in the long run?” Thus, what has happened with the efforts from the past, and what are the changes the test automation will keep running after that one consultant leaves.

Based on interviews, we had positive experiences but faced challenges regarding test automation. The chart below lists the positive experiences and challenges.

Key results of the research: 

  • There was only one successful application of test automation, which is not a lot and means that test automation needs work, hard work.
  • Test automation is started using a project-based approach in various organizations, where the organization and the test professionals are enthusiastic about automating current tests on different levels from a unit to a Graphical User Interface (GUI). The first projects are started successfully, and the agreed deliverables are delivered. The test professionals who are to work with the delivered tool or framework are trained. After the implementations of the tools, some challenges arise, such as maintenance on test scripts or code. This is due to the fact that not all test professionals have the skills to be able to do the maintenance of the test scripts or code. The organizations, however, keep trying test automation and see it as continuous improvement. In an ideal scenario, all the three wheels (as shown in the figure below) need to turn, to have continuous improvement within the organization.

  • Most test professionals hired by an organization are outsourced. Thus, test automation remains dependent on individuals or external parties. Transferring the knowledge to the rest of the organization, therefore, becomes a challenge.
  • It was also noticeable that not all organizations have a specific test strategy or architecture when implementing test automation. The organizations focus more on automating test scripts instead of thinking of an overall test strategy or architecture for the organization. This results in using tools that do not align with the other tools within the architecture of the organization. And worse – we start testing the wrong things.

The next practical recommendations are based on this research to help the QA & Testing professionals further with test automation.

  • Start with the WHAT. What do you want to test and by whom?
  • Identify in this the regressions and routine tasks, which can be automated by using test automation tools.
  • Maintenance of your test automation is part of your test architecture.
  • Make sure that you are doing enough unit testing to avoid finding defects on higher levels.
  • Is there enough knowledge regarding test automation within the organization?
  • Use it often in your tests to overcome the maintenance bottlenecks.

The learning points from the past are still valid for many organizations[1]&[2].

We know it is not easy to do test automation, however, it is important to have a closer look at your organization and decide what makes sense.

[1] Testen met test tools – Slapend werken, Maurice Siteur, Academic Services, 2000

[2] Automate your Testing! – Sleep while you are working, Maurice Siteur, Academic Services, 2005

Co-authored by Mehmet Duran                                                                                                          Email:


Powiązane posty


Smart Factories and Industry 4.0 – Is it really about human interaction?

Nigel Thomas
Date icon 2019-12-23

In an intelligent enterprise of the future where IT (the ability to provide increasingly...


Making the workplace more human with AI and automation

Lukasz Ulaniuk
Date icon 2019-12-02

Humans will be augmented and enabled with AI capabilities to better perform their jobs


Zombies, wizards, werewolves, and a test automation silver bullet

Grant Volker
Date icon 2019-11-21

Expectations of technology have dramatically changed over the years, creating a demand for...