IT has been forced into this strategy because, historically, the costs of storage and data movement, via ETL or in real-time, have limited what can be achieved within the IT budget. That budget has been limited in part because of the perceived value of information being relatively low when viewed at a corporate level as opposed to a local operational level where it rapidly becomes essential to effective operations.IT has a strategy to drive single solutions around data - EDWs and the Single Canonical form, while the business has a culture of heterogeneity and likes local solutions for infomation
While corporate views are often happy to provide simple financial consolidation this doesn't deliver the sort of detail, and pace of change, that helps the business work day to day. These challenges have led to EDWs which meet the needs of finance and some corporate KPIs and a fragmented estate of multiple data marts, Excel spreadsheets and ad-hoc solutions to help run the day-to-day business.
At Capgemini in the last few years we've started using Hadoop quite extensively in specific areas, normally areas where the cost of storage made traditional approaches unfeasible. We've also been using Hadoop to offload processing from existing data warehouses to reduce costs and also enable information to be provisioned for new uses.
That led to the recognition that the two historical constraints of storage and data movement costs were not really constraints in today's technologies. Pivotal's technology stack also gave another key build which is via Gemfire and Spring it becomes possible to do data movement in real-time as well.
So with these two constraints removed what is the possibility?
Well the first piece is that we need to meet the business challenge still of having both local and corporate views. This wasn't a big challenge as its something that we've been doing in our Master Data Management practice for several years. Together with Pivotal this led to the concepts behind the Business Data Lake:
- Store Everything
- Encourage Local
- Govern only the common
- Treat Global as a Local view
Often there is a challenge with working at an SI that some things can be more theory that practice when it comes to detailed technology, by co-innovating with Pivotal we've managed to deliver both the theory and practice of how it will work.