Gold is heavy, oil is dirty and diamonds are shiny. Each one of these follows a similar process, it’s extracted from the ground, its transformed into a more useful form and then loaded into something, for instance a car or an engagement ring, where it’s considered useful. This process is ultimately destructive as the source material is gone forever and not renewable. This matches how IT today deals with information, we extract the information in a specific form, we transform it to what we want and the we load it into a schema that represents our best current view on what might be required.
The problem is that if we want new information we have to repeat the process and often we lose temporal information, information that only existed for a time, because where we get to see the information is only after its been through the full refining process. So in essence we get to see the engagement ring, we don’t however get to choose the setting, the design or indeed whether we wanted a pair of earrings instead.
What is required is a new approach, an approach that enables us to repeatedly go from the raw ingredients and mix up what we want when we want it. Sometimes we need rigor; sometimes we just need something quick and dirty that works for the next few hours. Its here that the concept of distillation comes in, instead of just seeing everything after its been refined we get access to the raw materials, the raw data that our organization thrives upon and we get to choose the type of process it needs to go through and the specific view we want today.
Do we need the rigor and sophistication of aged single malt? In business terms this represents the financial reporting of an organization where consistency and quality is paramount to deliver something that is repeatedly brilliant. Do we just need a decent blend? Something we can mix up with other lower quality sources to help us make an good operational decision. Or is this something where its just about getting some sort of discovery, an ad-hoc exploration of what could be discovered, the daily moonshine of business decisions.
The point of distillation when looking at data is that you choose the level or rigor and refinement that makes sense in your circumstances, its not something imposed from outside its an intrinsic part of the business challenge you face.
Extract, Transform and Load was great when we considered data to be something that was mined, something where the raw material had little value, only the final product mattered. In the world of predictive analytics and where insight represents the difference between success and failure of a business however this simplistic view no longer works. Distillation, a process that values the raw materials, retains those raw materials and enables the business to drive new value all the time in new ways as new demands come in represents the only realistic way for information to become a valuable asset.
Distillation is about flexibility, it’s about being able to change the rigor, change the ingredients and change the type of insight being delivered. In today’s information driven world that ability to adapt is essential to survival.