Should we use generative AI for embedded and safety software development?

The idea of deploying generative AI (Gen AI) in software for safety critical systems may sound like a non-starter. With AI coding implicated in declines in code quality, it’s hard to imagine it playing a role in the safety-critical or embedded software used in applications like automatic braking, energy distribution management, or heart rate monitoring.

Engineering teams are right to be cautious about Gen AI. But they should also keep an open mind. Software development is about much more than coding. Design, specification, and validation can collectively consume more time than actual coding, and here, Gen AI can significantly reduce overall development time and cost. It could even improve quality.

Incorporating Gen AI in safety-critical environments

Before we come onto these areas, let’s quickly address the elephant in the room: Gen AI coding. AI code generation for safety-critical software is not impossible, but it would need extensive training of the AI algorithms, rigorous testing processes, and will bring a lot of complexities. Right now, Gen AI should never directly touch a safety-critical line of code. But we should certainly keep an eye on it, as Gen AI code writing as it advances in other sectors.

However, other areas – from specification to validation – are ripe for Gen AI innovation. Our recent Capgemini Research Institute report, Turbocharging software with Gen AI, found that software professionals felt Gen AI could assist with 28% of software design, 26% of development, and 25% of testing in the next two years. In the report, one Senior Director of Software Product Engineering at a major global pharmaceutical company was quoted as saying: “use cases like bug fixing and documentation are fast emerging, with others like UX design, requirement writing, etc. just around the corner.”

Software design

Let’s consider how the software development journey may look, just a few years from now. Let’s say you are designing a control system for car steering, plane landing gear, or a medical device (pick a product in your industry).

Right at the start, you probably have a project brief. Your company or customer has given you a high-level description of the software’s purpose. Gen AI can analyze this, alongside regulatory standards, to propose functional and non-functional requirements. It will still need work to get it perfect, but it has saved you a lot of time.

However, you want to go beyond technical requirements and ensure this works for the user. Thus, you ask Gen AI to develop a wide range of user stories, so you can design solutions that pre-empt problems. That includes the obvious ones you would have come up with, Gen AI just writes them more quickly. But it includes all the weird and wonderful ways that future customers will use and abuse your product, ways that never would have occurred to a sensible software engineer like you.

In most cases, this is about improving the user experience, but it could also prevent disasters. For example, many of Boeing’s recent troubles stem from its MCAS software, which led to two crashes. While the software was a technically well-designed safety feature, its implementation overlooked pilot training requirements and risks from sensor failures. This is the sort of real-world possibility that Gen AI can help identify, getting engineers who are laser-focused on a specific problem to see the bigger picture.

Armed with this insight, you start writing the code. While the AI doesn’t have any direct influence on the code, you may let it take a hands-off look at your code at each milestone, and make recommendations for improvements against the initial brief, which you can decide whether to act upon.

Test and validation

Once you have a software product you are happy with, Gen AI is back in the game for testing. This is perhaps one of its most valuable roles in safety-critical systems. In our CRI report, 54% of professionals cited improved testing speed as one of the top sources of Gen AI productivity improvements.

Gen AI can start the verification process by conducting a first code review, comparing code industry standards (eg. MISRA for automotive, DO-178 for aerospace), to check for errors, bugs, and security risks. You still need to review it, but a lot of the basic stuff you would have spent time looking for has been sorted in the first pass, saving you time, and giving you more headspace to ensure everything is perfect.

Once you are satisfied with the product, you want to test it. Your Gen AI assistant can quickly generate test cases – sets of inputs to determine whether a software application behaves as expected – faster and more accurately than when you did it manually. This is already a reality in critical industries, as Fabio Veronese, Head of ICT Industrial Delivery at Enel Grids noted in our report that his company uses generative AI for user acceptance tests.

And, when you are confident your software product is robust, Gen AI can help generate the ‘proofs’ to show it works and will function under all specified conditions. For example, in the rail industry, trains rely on automated systems to process signals, ensuring trains stop, go, or slow down at the right times. Gen AI can look at data readouts and create ‘proofs’ that show each step of the signal processing is done correctly and on time under various conditions – and generate the associated documents.

In fact, as you progress through these processes, Gen AI can expedite the creation and completion of required documentation, by populating predefined templates and compliance matrices with test logs. This ensures consistency and accuracy in reporting and saves engineering time.

Automating processes

Gen AI can also help you automate many laborious processes that can be so mundane that human brains struggle to stay focused, thus creating the risk of error.

Take the example of the process used in the space industry for addressing software defects. When a defect is discovered, developers must create a report documenting this defect, develop a test to reproduce the defect, correct the defect in a sandbox, put the updated software through a verification process, reimplement the corrected code back into the main project, and finally test it in within the product.

A five-minute code fix may take hours of meetings and tens of emails. This is exactly the sort of task Gen AI is well suited to support. Any organization writing safety-critical software will have hundreds of such tedious documentation and procedural compliance processes. We believe (in some cases) that as much as 80% of the time could be saved in such processes by deploying Gen AI for routine work.

Don’t just take our word for it. Speaking to us for our report, Akram Sheriff, Senior Software Engineering Leader at Cisco Systems notes that, “One of the biggest drivers of generative AI adoption is innovation. Not just on the product side but also on the process side. While senior professionals leverage generative AI combined with their domain expertise for product innovation, junior professionals see value in AI process and tool innovation, and in automation and productivity optimization.”

Managing the risks to get the rewards

Despite all these opportunities, we must acknowledge that this is a new and fast-moving field. There are risks, including the correctness of outputs (Gen AI can hallucinate plausible but wrong answers), inherited risk from underlying models, and bias in training data. But there are also risks of not acting out of fear, and missing out on huge rewards while your competitors speed ahead.

Gen AI needs safeguards, but also a flexible architecture that allows companies to quickly adopt, test, and use new Gen AI technologies, and evolve their uses as needs demand.

In our report, we propose a risk model (see image 1). It states that any use of Gen AI requires (a) a proper assessment of the risks and (b) that – where mistakes could have serious consequences – you have the expertise to assess whether the outputs are correct.

For now, safety-critical code creation will fall into ‘Not safe to use’, because the consequence of error is high, and the expertise needed to assess the code would probably be more of a burden than starting from scratch. However, testing would fall into ‘Use with caution’, because it would provide valuable insights about software behavior, that experts can assess.

Finally, a key part of managing risks is comprehensive user training to understand how Gen AI works and its strengths and weaknesses. In our research, 51% of senior executives said that leveraging Gen AI in software engineering will require significant investment to upskill the software workforce. Yet only 39% of organizations have a generative AI upskilling program for software engineering.

There is a real risk of becoming overly reliant on, or trusting of, Gen AI. We must ensure that humans retain their ability to think critically about the fundamental nature of software and safety. Software engineers must be well-informed and remain actively engaged in verification and decision-making processes, so they can spot problems and be ready to step in if Gen AI reaches its limits.

In conclusion

While Gen AI won’t be building safety-critical software on its own anytime soon, it has the potential to enhance development, documentation, and quality assurance right across the software development lifecycle. In doing so, it can not only save time and money, and speed time to market, but it can even improve safety.

Companies like Capgemini can help shape achievable, phased roadmaps for Gen AI adoption. We guide organizations to integrate AI carefully, following sensible adaption and risk management frameworks and deploying appropriate training, ensuring both its potential and limitations are carefully navigated.

Download our Capgemini Research Institute report Turbocharging software with Gen AI to learn more.