Thriving On Data #2: Real Real Time
It’s all true after all: size doesn’t matter. ‘Big Data’ really is about the ability to analyze and act in real time, using data from sensors, transactions or interactions, both from inside as well as outside the organization. This ‘Fast Data’ can be used to solve tougher business problems, create more competitive advantage and make much better, more informed decisions in a tightly connected world. With this comes the opportunity to create ultra-fast insights, within one single CPU cycle, used by business users or even automatically. If there is no longer a need to wait, the opportunities for radical business reinvention are limitless.
Werner Heisenberg was a German physicist and one of the key creators of quantum mechanics. In 1927 he published his uncertainty principle for which he is best known. This principle states: “It is impossible to determine accurately both the position and the velocity of a particle at the same instant.” Position is the identification of the relative location, in other words: where you are. Velocity is the speed and direction, in other words: where you are going. It is rumored that Heisenberg went for a drive one day and got stopped by a traffic cop for speeding. The cop asked, “Do you know how fast you were going?” and Heisenberg replied, “No, but I know where I am.”
The same seems to apply for many organizations today. They know where they are (or have been) but often they do not know where they are going. The main reason for this is that their data is at rest, even when it’s Big’. It is mostly inactive data that is stored physically in any digital form, for example a database or a data warehouse. Also it is used primarily for historic reporting or analysis on mostly internal data done by the IT department. Although the quality is high (data warehouses for example are often associated with a high level of data quality and creation of the single version of the truth), the time-to-market is often low (batch-oriented overnight architectures) and the value of the data is therefore relatively low.
To be truly successful, many organizations need to transform from hindsight analysts to foresight action takers. This implies that data is no longer in rest but in motion or even in use. It should flow in real time through the organization and change business outcomes on the fly. Data streaming technology from vendors like the SAP Event Stream Processor or Informatica VIBE data stream, allows enterprises to collect and deliver small data packages accumulating into one large ‘data lake’. Software like IBM Infosphere streams or the SAS Event Stream Processing engine, allows bringing complex analytics to operational data, creating much faster insights and interactive visualization for supporting business decisions.
‘Big Data’ has created a paradigm shift in the way we look at decision making today. Traditionally, structured data from internal systems like ERP has been the main source for this. Now unstructured data comes from sensors in machines, planes, trains, automobiles or even the fridge at home. It allows companies to optimize their client’s travel or create a predictive shopping list: they add to the amount of data available. But this is also the time where external data, from websites or social media, tells enterprises much more about its own performance. Not with facts or dimensions from the IT data warehouse but with opinions from Twitter and likes on Facebook by customers. This is the time where Facebook can predict when somebody is about to cheat or commit suicide, where Google can predict a flu outbreak, or retailers can predict that somebody’s teenage daughter is pregnant.
Big Data is not only about volume, as the name suggests. Volume is about data at rest. It is about storing massive volumes of data against lower cost, for example in a Hadoop environment, like Cloudera or Pivotal. However Big Data is also about where you are (position) and where you are going (velocity) with speed as the deciding factor. Research (for example our latest report ‘Big & Fast Data: The Rise Of Insight-driven Business‘) shows that C-level executives are convinced that value from data can be mostly found in real time. In other words, Fast Data (insight) is even more valuable than ‘just’ Big Data.
Fast data has been made possible by advances in technology – like in-memory or real time replications– allowing business users to quickly find usable insights by exploring big data sets (often in Hadoop nowadays) from inside as well as outside their own organization. New insights are only one CPU cycle away. SAP for example with their in memory SAP HANA platform or IBM DB2 BLU are databases that support such a real time environment. Also companies like Teradata support the need for speed with their Massive Parallel Processing Database appliances.
In order to be competitive in increasingly more complex business environments, organizations need to predict possible future outcomes such as customer behavior. This has to be done based on all available data, both historic as well as current. It is important to analyze this data – through advanced analytics – but even more to act even before an event takes place.
A credit card transaction in South America of a European citizen shows possible fraudulent behavior. Do we need to block the card right away? An online retail competitor changes the pricing for their top 3 popular products. Do we need to change our pricing policy on the spot? A railroad switch suddenly shows increasing energy consumption. Do we need to proactively perform asset maintenance?
When real time becomes real – with no more need for waiting – the event and the action become one. And not necessarily in that order. It’s not unlike that famous movie Minority Report, in which police forces use data to predict where a crime will take place and send officers to the scene proactively. Talking about new business models.
Say my name, Big Data: it’s Fast Data.