Thriving on Data #3 – Real Real Time
With advances in Big Data such as the ability to store extremely large volumes of data, to deal with various formats – both structured and unstructured – and to apply complex analytics to up-to-date operational data, we have reached a true inflection point: the ‘end of batch’ makes us think far beyond just stepwise improvements of existing business processes, as the need for queuing, waiting times and intermediates disappears completely. Organizations that have the imagination to reinvent themselves can make a decisive leap forward.
The first time I worked as an IT engineer was for a large bank and everything was done on a big IBM mainframe. Every program was designed to be transactional (i.e., in direct link with a human, with an average performance goal of seconds) or batch (i.e., no interaction needed with a human, average performance goal was an hour).
Technological evolutions have led us to go faster and faster, but at the same time to deliver more complex processes or with much more data, but IT still kept a gap between transactional and batch. Today, technological developments enable us to really change the game.
In-memory computing and data storage combined with event processing technologies and real-time replication allow us to say that it is easier, faster, and cheaper to nowadays deliver in real-time what used to be batch. This is where the new technology underneath SAP’s HANA and Oracle’s Exadata/Exalytics (to mention only the most obvious specimens) starts to shine. If we can use one and the same system – all in-memory or whatever does the job – not only to store transactional data in real time but also to analyze it in a split second so that we can act in real time, something truly transformational is lurking around the corner.
Consider an ERP system (like those from Oracle or SAP). Now try to guess what has happened during the last hour on this system. This is a very difficult question. And ‘difficult’ in IT means costly, highly intrusive and slow. And once we have the complete list of all the events (or more probably a long list of events where maybe we have missed some), it will take time applying our business rules to all these events.
Conversely, ask the same ERP system to send you, in real time, a copy of each event it is processing. Then you do not have to dig into historical events, then you have one isolated event you’d like to apply a business rule to (or not) and then you’re able to go fast, very fast, really very very fast. And it is cheaper too.
Finally, you’re able to do something truly new: reacting early, sometimes proactively is about new opportunities, new services. If you’re with a bank, before each transaction, you can detect fraud or money laundering in order to cancel the transaction immediately. If you’re with a retailer, you’re able to push a targeted promotion based on current basket. If you’re with a telco company, you’re able to react immediately after a VIP customer meets network issues. If you’re with an energy provider, you’re able to optimize your smart grid in real time.

The challenge to both IT and business is not to consider the new capabilities as just another trigger for stepwise improvement. It requires fresh thinking, potentially leaving behind most of what we think we know about a process. What does it really mean if we can perform profitability or cash analysis at any time of the day, based on the actual transactions going on? What does a supply chain look like if it is driven by the mood of the hour – or minute – at the points of sale? What about credit scoring if a request can be analyzed in depth on the spot, in seconds?

SAP’s CTO Vishal Sikkanot not so long ago announced ‘the end of batch’ during a keynote. And indeed, if our systems are powerful enough to store and process any input the moment it is created, there is no more need for piling up, no more waiting queues, no more intermediates.

Clearly, the impact of this ‘end of batch’ goes much further than IT processes and systems. It indicates nothing less than an inflection point for business, and we need to imagine where it can lead us if there are no waiting times. For anything.

To finalize, I would like to discuss the ‘Real real time’ expression. I still remember a customer who wanted to do BI in real time. Ten years ago, one of my architects started by designing a very complex and expensive solution to deliver KPIs in real time, and came to me to validate the solution. I asked her what the customer meant by ‘real time.’ She said “probably a few seconds.” Then we called her customer. His current BI system was computing KPIs only once a month, and the whole process took about 15 days. So there were 45 days of delay. He was dreaming about having the KPIs in ‘real time,’ which meant for him in 1 or 2 days.
Formally speaking, real time doesn’t exist in IT: the minimum response time is a CPU cycle after all. I have heard customers or vendors speaking about ‘business time,’ ‘right business,’ ‘close to real time.’ Some (of the more old fashioned, no doubt) speak about ‘mini batches’ or even ‘fast batches.’
Whatever your definition, start imagining what Fast Data and real time would mean to your business. Not just speeding it up. But actually redefining it.
This contribution by Manuel Sevilla 
Part of Capgemini’s TechnoVision 2014 update series. See the overview here.