“Automate all the things,” I’ve heard on occasion – and from my experience there is so much more to this than meets the eye!
Expectations of technology have dramatically changed over the years, creating a demand for increasingly innovative software that delivers a superior user experience. These expectations have changed the methods of software manufacturing whereby businesses demand delivery at pace and reduced cost while maintaining quality. To meet these demands, developers are doing more testing and testers are involved much earlier in the software development lifecycle, sometimes even participating in development activities. There is also a greater focus on continuous delivery and testing.
Just over 15 years ago, I was introduced to Cruise Control – a framework for the continuous build process – and an iterative development to deliver a working build each day. It wasn’t a practice I had seen before and since then, so much has changed. The introduction of agile was hailed as the solution to reducing lengthy release cycles, and delivering what the consumer of the product wanted at a higher quality. With the focus now firmly set, automation was considered the key to productionizing the creation of bespoke and often wholly new software products. You might have seen in our World Quality Report that the level of automation is still very low, between 14% to 18%, with 61% of survey respondents reporting difficulties with automation. The report provides insights into the challenges of implementing automation and the ones I’ve experienced generally involve data, environments, and people.
Certainly, let’s try to achieve a great level of automation coverage in a maintainable, scalable way. To achieve these ambitions, the challenges mentioned earlier need to be considered.
You can have the best technology in the world, but without the right people and culture your efforts will likely be disappointing. You could also have the best people in the whole world and without the right tools and methods you are likely, again, to be disappointed.
Identifying the most suitable tests to apply and at which level of the technology stack is another source of difficulty I’ve seen in automation. Usually it involves selecting the right tool at the right price point (yes, open source tools have their hidden costs too) and one which your team has the capability to work with. By this I mean being knowledgeable about the use of the tool – of what it can and cannot do – as well as suitable programming skills.
The skill to build a scalable framework incorporating maintainable scripts and to an agreed level of quality is vitally important too! I’ve experienced test packs containing a high volume of tests where:
- Some tests were not doing anything at all (zombies) and unnecessarily adding to the run time of the pack.
- Some frameworks and tests made for difficult reading few could fully understand, requiring a wunderkind (wizard) to figure things out.
- Some tests would come out at night to fail intermittently, upon a full blue moon, when the stars were aligned (werewolves).
The silver bullet of automation was clearly not delivering to expectations and I’m sure we all have our tales from the crypt of automation nightmares to tell.
The way we work also contributes to the success or struggles of automation. Which brings me to behavior-driven development (BDD). A common misconception I come across is that Selenium is BDD, when the essence is a cultural aspect of communication and shared understanding; a preference for bringing people together, breaking down silos, and fostering a collaborative environment in which people can experiment and learn from each other without fear. We do after all work in Information Technology (IT), dealing with data (information), an area identified in the World Quality Report as contributing to the challenges in delivering effective automation.
Data has always been important and more so when applying machine learning, which requires a lot of data and good data at that. For example:
- Test data for checking your product under development
- Training and verification data for a model being developed in an AI system
- Data for cognitive QA test tools planning, creating, and executing tests.
Modern test data and provisioning tools are another form of automation we can use to support, automation and the testing of machine learning systems themselves. However, the tools, programming languages, and technical (hard) skills – to name a few – are only part of the whole. Getting the balance of people, practices, and technology appears to be the true key to success.
Have any of these experiences come to haunt you? Our extensive experience in test process and automation improvement projects across a wide range of industries and organizations might interest you. We’re also working with leading-edge technologies to help customers improve efficiencies and customer delight of their products and services. Our people are great too, and would love to talk with you to know more about what you need. You can reach us by one of these fabulous technologies: phone, email, or social media.