On track to the future: AI in compliance auditing

With a mission to deliver a safe, reliable, and efficient railway, Network Rail is one of the most important players in the UK transport sector, responsible for billions of passenger journeys and tons of freight each year. To do so, Network Rail maintains more than 20,000 miles of railway network in Great Britain, including tracks, switches, crossings, and other components of the rail infrastructure. Thousands of Network Rail engineers work around the clock to maintain this vast and complex railway network so that passengers are safe, and the railway operates smoothly.

Throughout its daily maintenance work, track works, and signaler teams exchange approximately five thousand phone calls. These communications must comply with Network Rail’s safety-critical communication (SCC) protocol, meaning that they must be accurate, brief, clear, and professional. These protocols help ensure that Network Rail achieves their vision of getting everyone home safe every day.

Network Rail wanted to conduct a feasibility experiment in order to develop an artificial intelligence (AI)-based solution that would audit calls and provide valuable insight into how compliant specific calls were. This would enable the organization to improve these vital conversations and more accurately determine their employees’ training needs to ensure the maintenance of SCC standards. In addition, such a critical discovery project would enable Network Rail to continuously improve and learn lessons that would create a safer working environment.

Using AI to analyze calls

To properly determine and test the viability of an AI-based auditing solution, Network Rail engaged Capgemini to combine its industry expertise with extensive technical knowledge and a lengthy history of successful innovation. Working together, the partners initiated a review of the intended program, its primary objectives, and the available technology in order to develop a clear roadmap. During this process, Network Rail leaders explained their current manual procedures and provided two hundred call recording files with which to train the AI models as well as the organization’s SCC manual, which helped to map the conversations.

With stakeholders from both partners on board with this plan, the project team then developed a conversation mapping tool that could test the feasibility of the approach. As part of this solution, Network Rail and Capgemini designed a custom speech-to-text (STT) recognition model, which included a custom Natural Language Processing (NLP) algorithm and calculation methods that analyze calls on four parameters: Clarity, Completeness, Compliance, and Focus.

At the end of the feasibility study, the project team presented a detailed report that contained findings, recommendations, guidelines, and AI best practices to analyze safety critical communications. Following this, Network Rail used the tool on a regular basis over a three-month period to analyze recorded calls.

Planning a journey into the future

Based on several business and technical analyze that were conducted along the way, Network Rail and Capgemini were able to draw certain conclusions about the solution’s viability. First, the NLP models trained on test data performed well in analyzing non-compliant calls and were able to detect missing information from calls in the test data set. While excessive background noise could affect voice clarity, the NLP models coped well with such distractions. In addition, the AI-based solution was able to detect the portion of a call duration that was dedicated to safety critical communication.

Following the feasibility study, Network Rail and Capgemini concluded that a successful, AI-driven solution has the potential to support the assessment of compliance with SCC standards and can drive substantially greater efficiency. They also agreed that AI models can help the organization accurately identify training recommendations for enhancing safety-critical communications, resulting in safer and more timely maintenance work. Finally, this experiment provided Network Rail with guidelines for ethically using AI technology to analyze safety-critical communications using established best practices.

With these recommendations, including guidance for how to scale the PoC for enterprise-wide deployment, Network Rail is now prepared to draw upon this AI-based technology to ensure communication compliance, resulting in increased passenger safety and operational efficiency.