Facebook and Google have both prospered by exploiting the interconnected nature of our world – people, events and information. This is true at a global scale, national and local. The UK government could benefit from unlocking the value of their interconnected data and this blog will start by explaining this problem in more detail and then discuss proven solutions that are based upon graph technologies.
For government Artificial Intelligence (AI) strategies to be fully realised data must first be in a usable form. However, most UK government departments are facing a monumental task of breaking down archaic data structures in silos. Putting AI aside, simply reducing manual data tasks and ensuring that decisions are based on all the evidence will immeasurably help most departments facing rising workloads due to several factors like changes in the socio – political environment and causes like a response to COVID 19 pandemic . No-one joined the civil service to copy data from one screen to another, no matter how comfortable their swivel chair is.
Example; would a stranger be able to understand all your social, familial and professional connections (i.e. your network) by reading your contact lists?
They might be able to guess some of the connections based upon surnames and locations but generally the answer is No. You’d have to go through and describe how contacts relate to each other and to you. The “lists of contacts” is how most data is stored in government – i.e. the relational (SQL) database. To unlock value, you need to manually map the connections but this relies on having access to the right data and business experts. And often you repeat this mapping every time you exploit that data (e.g. to train an AI model).
Your supply network, client base, internal organisation, competitors and adversaries are complex, and this complexity cannot be adequately represented in neat tables. Making these tables longer, wider and normalising them is a delay tactic, not a holistic solution. And have you ever tried analysing multiple inter-connected tables each with billions of rows? Adding more compute nodes is another delay tactic, not a holistic solution.
Relational data structures aren’t the enemy and aren’t dying out; they provide an unsurpassed capability, for certain problems. By solving certain problems extremely well for 50yrs+ we believe they’re the solution to every problem. Graph technologies provide a complementary capability that is often a better fit for the large-scale analysis of highly connected data; which is prevalent in public sector organisations. You probably have highly connected data if your domain covers:
- Groups of people or organisations
- Managing a complex portfolio of products, services, cases or requests
- Tracking and tracing a global pandemic
- Managing GDPR compliance and FOI requests
- Supply chains and transport networks
- Computer networks which includes IoT
- Managing data access and permissions for knowledge stores
- Fraud analysis (covering all aspects of the previous bullets, depending on your jurisdiction and with example in Figure 2)
There are numerous examples of graphs being deployed across central government departments, many of which aren’t public but here is an example from the Cabinet Office.
When the real world that you’re representing is highly connected then your data needs to be highly connected. And to obtain value from this data you need to be able to search it quickly. In order to respond to emerging demands queries should run in seconds, not hours or days.
“We’re not having to torture our data to fit the database and then torture our queries to fit the data we’ve tortured to fit our database – we’re doing it in a way that naturally fits the problem and as a result, it is much, much quicker to execute.”
– Harry Powell, Director of Data and Analytics, Jaguar Land
Graph database structures provide this solution. This technology has been around for centuries and has been mainstream since the mid-2000’s when Neo4J was released. This is still the market leader for Labelled Property Graphs but modern, distributed alternatives are emerging (e.g. Cosmos DB in Azure, Neptune in AWS and TigerGraph, all of which integrate with Python and logos are in Figure 3).
A highly complex system with multiple interconnected data sources is the UK Border (details in this 272 page report) and my colleague Jake Luscombe will shortly be publishing a blog on this subject.
More details can be found in Introducing Capgemini’s UK Graph Guild so here we’ll wrap up with integrating this technology in your business for tangible benefit.
- Clearly articulate the business outcomes that you expect and aspire to achieve. These should link to cash amounts gained or saved, or changes to the metrics upon which your department is judged by parliament and the public.
- Agree the payback period. Is it in-year, within the Spending Round (SR), longer-term (e.g. Defra’s 25-year strategy)? If budgets allow, I recommend targeting multiple timeframes so some benefits will be realised in the short-term whilst others build long-term capability.
- Look at your current systems, processes and workforce to pinpoint changes required to deliver benefits in (1) within timeframes in (2)
- Based on the outcomes of (3); consider technology solutions. In my experience the exploitation of data is often an area of improvement for government departments and frequently this exploitation is aided or enabled by graph technologies…but it might not be so don’t assume every challenge is met be graph technologies!
- Now build something. A prototype is worth 1,000 video calls.
- Learn and evolve the solution into something your end-users easily want to consume. Prototypes expand your knowledge but you must develop them into a Beta or Live solution to get business value.
- Share your experiences. Help other departments to catch-up. Draw upon the experiences of others. Share with the general public; this is your currency for attracting top talent and expert input. If MOD and GCHQ can open-source stuff then so can you!
If anything in this blog seems unachievable then take the path of least resistance. (temporarily) buy in expertise to accelerate.
Dave work with clients designing and building useful and effective AI and Machine Learning solutions to meet their business needs. I help clients navigate the wide range of commercial and open source solutions. I help them see past the sales material by rapidly prototyping to guide future strategy and budgetary decisions.