Testing of chatbots – the art of conversation, interpretation and validation

Publish date:

Chatbots are set to grow exponentially in the near future. Here’s what QA professionals need to keep in mind while building their testing strategy.

Hello Siri, may I have a strategy to test you?”

“Chatbots,” are essentially software that simulate human behavior and conduct conversations as a human would. Chatbots can be classified according to industry, and include banking chatbots, medical chatbots, personal finance chatbots, etc.

Depending on the industry, the mechanism to test the functionality will differ, but certain principles always remain the same. Below are important considerations for QA professionals to keep in mind as they build their testing strategy:

  1. Start with identifying use cases for the chatbot. List questions and potential responses for every scenario and prioritize them according to importance.
  2. Two aspects are important from a testing standpoint: the conversational capability of the chatbot, and the degree of intelligence the user expects from it. Most chatbots allow different types of data and these should be clearly identified and documented. For each use case, clearly define the testable requirement and the key performance indicator (KPI).
  3. From a technology vantage, chatbot KPIs include number of steps to perform a request as well as an average number of users. Business KPI examples include self-service rates (i.e., the extent to which a chatbot can resolve a request without human interaction), the average customer rating, and the sales conversion rate (i.e., the extent to which a chatbot can convert an online conversation into business).
  4. Once the testable requirements are defined, understand the underlying architecture and technology that the chatbot will use for each use case. Essentially, Chatbots are built on natural language processing (NLP). NLP is a way for computers to analyze and derive meaning from human language. For example, one use case could be an architecture that integrates an AI engine (like IBM Watson) with a custom NLP speech engine; another could be leveraging an existing API such as Google Cloud Natural Language. Understanding the architecture will be the crux for designing test cases.
  5. Test scenarios should encompass conversation and voice testing (i.e., the ability to recognize speech patterns and interpret non-verbal cues). Test scenarios should be designed with variations of the same input. Scenarios to handle multiple instructions in a request, conversations with background noises, different dictions, and localization needs are a must. In addition, tests to validate the ability of the chatbot to help in user navigation and the ability to handle errors are also vital. Omni channel compatibility tests to ensure the same look and feel and responses are required if the chatbot is expected to be used across multiple channels.
  6. From a non-functional standpoint performance testing i.e., the speed at which the chatbot responds and security testing including authentication, authorization, encryption of conversations, and adherence to compliance are key.

In summary, Chatbots are poised to exponentially grow in use and the technology to build them will rapidly evolve. Quality from a usability and functionality standpoint is a must and will be key for success.

 

 

Related Posts

AI

16 ways we use Watson now

Anne Aussems
March 13, 2018
AI is opening up countless possibilities in many areas—augmenting employees, automating processes, transforming customer experience and decision-making, or shaping new businesses.
Artificial Intelligence

Building a document-based chatbot

Reinoud Kaasschieter
March 5, 2018
When used for information transfer, chatbots can be used to direct the user to the information he or she wants. Using question–answer pairs, the user can traverse the knowledge captured in the chatbot.
AI

Intelligent automation

Alex Vayner
March 1, 2018
While the emergence of domain-specific chatbots is not a particularly recent development, the methodology by which bots consume and interpret human input has changed dramatically, and consequently the level of depth of bot comprehension has grown tremendously.
cookies.

By continuing to navigate on this website, you accept the use of cookies.

For more information and to change the setting of cookies on your computer, please read our Privacy Policy.

Close

Close cookie information