In my last blog item I discussed root causes for unacceptable amount of waste in traditional BI projects, the time we spend creating unused functionality and working on projects that will never deliver or under deliver on the promised benefits.
The root of this failure, I suggested, is in sticking with the large Enterprise Data Warehouse paradigm and traditional waterfall development methods.
The Naturalist’s View
There’s another way of approaching this. Biomimicry[i] is about using insights from natural systems and applying to human challenges. I’ve discussed my fascination with this simple idea and application to consulting in previous posts. Nature solves problems using the following techniques:
Evolution: Winning solutions are replicated, losing solutions are terminated. Mutation means innovation and unexpected solutions. Feedback is used extensively.
Resource Efficiency: Materials are recycled, energy is used efficiently and components are parsimonious.
Adaptable to Change: Self renewal. Resilience is created by decentralisation, redundancy, and diversity.
Development & Growth: Design is component based with nested hierarchies, it is bottom-up and components are self organising.
So How Do Agile & Data Lake Fit In Here?
What is an Enterprise Data Lake? It’s a fairly new thinking paradigm in BI and is a whole other set of papers that I urge you to explore. But to quote from Capgemini’s Steve Jones[ii]
“The Business Data Lake is a new approach enabled by a new generation of technologies. No longer do organizations need to be weighed down by the technology constraints of the past. But more than a technical solution the Business Data Lake is a new approach to the information challenge based on the principle that collaboration and access to information lie at the heart of effective business.”
See some excellent material at http://www.capgemini.com/big-data-analytics/business-data-lake.
Let’s look at this as a Data Lake Vs Traditional EDW comparison:
|Traditional EDW||Data Lake|
|Evolution||Data that will be used is fixed and prescribed at start.||Add data to the lake as required and as new sources emerge.|
|Parsimony||Prescribe full canonical model up front and make sure all data is clean, integrated, and ready to use.||Manage only the common master data elements. Monitor MD pain points and correct them.|
|Change Adaptation||Complicated change management in hands of centralised IT governance function.||Change managed at a local system level where possible. Control resides with data consumers.|
|Development & Growth||Design and use of information is centrally governed, often by IT.||Governance of priorities and effort typically determined locally and by the business owner.|
With the data lake approach the locus of control is with the local business user, who knows what they want. This allows them to adapt rapidly, taking and reusing what they need from the lake.
How about Agile Vs Waterfall? Agile is evolutionary in nature, continually iterating to an improved solution. It’s inherently adaptive – change is accepted as inevitable and a source of improvement.
|Traditional Waterfall||Agile Approach|
|Evolution||Requirements set at start and not tested until many months later.||Requirements to realisation cycle is weeks.|
|Parsimony||Design everything up front, and then build it.||Only design and build what is needed, then iterate to improve and add.|
|Change Adaptation||Discouraged, complicated and requires contract management.||Encouraged, inevitable, users are not expected to know exactly what they want in advance.|
|Development & Growth||Design, build, end users are different teams and involved at different times.||Self organising small ‘one team’ mentality. Business perspective integral to team.|
Taking agile development and a Business Data Lake philosophy to information provision is often a better way of solving the Information Management problem. There will be less waste because evolutionary mechanisms are employed, it’s parsimonious, allows change, and allows decentralised bottom up decision making.
Did you ever notice that there isn’t much waste in nature? Shall we do the same in Business Information?