Assessing AI Value: Measuring the value of AI solutions

The number of possible applications of Artificial Intelligence (AI) can sometimes seem endless, with some estimates predicting that AI alone can unlock up to $5.8 trillion worth of value every year, roughly equivalent to the GDP of Japan. Look beyond the tech players of Silicon Valley, Beijing or London however, and you will find few businesses are leveraging AI successfully. In fact, Gartner estimates that 80% of AI applications in business require expert maintenance and cannot be scaled across the organisation.

So why is it that businesses cannot find their feet with AI?

It’s certainly not due to boardroom ambition; 88% of executives acknowledge their organisations need to be more data driven. And it is not due to the availability of data: it is widely known that data is cheaper and more abundant than ever before. The issue we see businesses struggle with is connecting c-suite ambition and abundant data with actual business value. Without being able to quantify AI value, it is hard to justify future development spend. AI applications predominantly need to be scaled to provide significant benefits, but without being able to measure value throughout the proof of concept and proof of value stage, they rarely get to this stage.

Measuring value through the development stage

Benefits, in particular financial ones, are not realised until an AI application has reached production, and to get to this development stage the application must pass through both the pilot and the optimisation phase. Value – and by extension RoI – can be measured in all three phases, but rarely will significant business benefit be realised before an application is industrialised and scaled. This results in applications being halted before they can demonstrate their business value.

For an AI application in the pilot phase, it’s value is measured through its functionality. Does the application produce the output that is expected, and can it function with real-world data and at real-world speed, even if it is being run in a development environment? If the answer to these questions is yes, then the investment on pilot development can be judged as a success – the funding has resulted in the creation of an application that works as expected. However, this is no more than a proof of concept, and is unlikely to produce any marked business benefit.

Measuring value through the industrialisation phase

Once functionality has been proven, development focus shifts to optimisation. Refining the application should serve to improve AI performance. Measurement is straightforward – has the algorithm’s performance improved following tweaking and additional training? Other considerations are also necessary, such as assessing whether the algorithm is ethical, such as whether it has learnt bias against protected characteristics from its training dataset. Again, these are measures of value, but would rarely interest a CFO.

CFO interest is seldom gained before AI applications can demonstrate its tangible value, and this rarely can be demonstrated until the application is industrialised. This requires project spend, resource and support from different business units, in contrast to AI pilots that can be developed by a single data scientist. But there are rewards for those companies who overcome these hurdles and can successfully leverage their applications for their use case. An AI application’s contribution to business value is measured in the same way as any other business initiative – tying the application’s value to expected benefits and monitoring the release of those benefits over a set period of time. Once the application has proved its value with its use case, businesses can then consider scaling: as with economies of scale, the wider the application is scaled, the greater the realised benefit.

A real-world example of realising the value of AI at scale is that of self-driving cars; to an individual, the likelihood of being in a car accident is already low, and the safety benefits of developing one driverless car are negligible. Scale the projected safety improvements across the population of a country such as the USA, and you should see a huge reduction in accidents and the benefits become clear. The disconnect between these macro benefits and micro benefits persist in business – while significant numbers of businesses have ambitions to leverage AI, lack of understanding of AI value at a micro-level sees few initiatives get the funding, support or resource to realise the full value an AI application can bring.

Joel Brocklehurst

Leader with A&AI’s advisory practice

Joel is a leader with A&AI’s advisory practice, with experience delivering data and AI strategy projects across a number of sectors. Joel’s experience enables him to marry innovation with business value, helping his clients use data solutions to drive value.