The big data market is exploding in terms of marketing hype turning big data into a big promise. However to quote a prospective customer: “we have lots of data but little results”. When talking about Big Data the issue of increasing volume is widely addressed. With transactional data coming from ERP (traditionally the source for BI), CRM (with the increasing interest in customer behavior), interactional data from the WEB (facebook updates) and even observational data (like sensors, think internet of things), the volume has increased from megabytes, to terabytes and even petabytes. This causes increased complexity for our customer that need to handle this data variety in increasingly faster response times.
In addressing the Big Data challenge we see three dimensions dominating the discussion: how do we store, use and govern big data.
First of all storage. Traditionally we have built datawarehouses for this often in the form of a relational database with a dimensional model. This datawarehouse is maintained on premise. With big data the new way forward seems to distribute the data across multiple locations and servers (think cloud) using for example technologies like Hadoop and MapReduce.
Secondly, using big data. While storage is mainly an IT discussion this turns out to be the big debate among the business people. They wonder what could be the real advantage from having all this new, complex, still unknown and maybe even strange data ? Most people see the value of big data but are unable to make hard business cases. The new way forward seems to be to replace the business case with a business plan.
Thirdly, Big Data governance. This is where Business and IT actually should find each other but in reality are often on opposite sites (even physically) . Challenges in this domain are: security, privacy and data ownership (amongst many). The new way forward seems to be a full stop!
So we find IT, looking for ways to reduce the costs of acquiring and marshalling data and/or funding for a Big Data architecture. But we also see the business struggling with way to generate more revenue. Here lies the big opportunity for the CIO. My advice: start with a Proof-of-concept that shows the actual benefit of big data for the business . They actually need to see what they can do with data analysis and intelligence led decision making. I am sure this will get you a BIG BONUS.