Four hundred years ago the Management Team of Dutch East Indian Company (VOC) was struggling with some important business issues in Finance (Dividend), Fleet Management (big slow ships or small fast ships), Portfolio (salt, pepper or tulips) and Risk Management (pirates). A complicated task because the only data available to base their decisions on was the logbooks from the ships which came in every other month at best. Back then there was little information available at a low frequency. Today we have the opposite. We have a lot of information available at a high frequency. Many organizations have built datawarehouses to deal with capturing this data in a structured way.
But it is not only the volume of data that has increased, it’s also the complexity of data. There are various sources, both inside and outside (!) your organization, that generate data in all kinds of shapes and forms. This data can be structured (like data from your ERP system or point of sales) but also unstructured (like document, photos or even movies). This increase in volume, velocity and variety of data is often called Big Data. In terms of volume, Big Data can be generated by humans through social networks, but also through the explosion of internet-connected machines, and this has led to more and more data being generated every day. The variety in data comes from social media, video, text, but also Machine 2 Machine, which are continuously generating new types of data. Last but certainly not least, is velocity. Data is being generated faster than ever, so that traditional batch loads are not practical. And as data generated 1 minute ago is more important than data generated 1 day ago, real-time processing (or right-time) is becoming more and more important.
So is Big Data a threat which we have to fight with even bigger machines and smarter software? According to a recent article IT will be faced with many issues: http://www.computing.co.uk/ctg/opinion/2175813/opinion-raises-issues-departments . However the cost of storage is decreasing, we have all kinds of interesting development in processing and memory, solutions like Hadoop and In-Memory technology (datawarehouse appliances) make using this data practical. This allows us to deal with issues like performance and scalability.
Or is Big Data about finding value in data that you have not found before? We can use data to create new insights and take actions to improve business performance. Solutions like data mining and advanced analytics can help us in the right direction. On top of that many organizations struggle with other issues like how to create a single view of the organization, or how to fix the basics.
The solution lies in the combination of innovation and just plain old hard work. But also in combining both technology and business solutions. It’s about organizing your data ownership and execution (for example in a Big Data Service Center). But also about dealing with the increase in volume using solutions like Hadoop or datawarehouse appliances, or making the unstructured meet the structured world with smart integration engines, or dealing with velocity by creating self service architectures making data available anytime, anywhere on any device. But most of all it’s about managing your business information because: “Content at rest is cost, content in motion is value”.