How Data Lake and Agile Reduce Waste In BI

Publish date:

In my last blog item I discussed root causes for unacceptable amount of waste in traditional BI projects, the time we spend creating unused functionality and working on projects that will never deliver or under deliver on the promised benefits. The root of this failure, I suggested, is in sticking with the large Enterprise Data […]

In my last blog item I discussed root causes for unacceptable amount of waste in traditional BI projects, the time we spend creating unused functionality and working on projects that will never deliver or under deliver on the promised benefits.

The root of this failure, I suggested, is in sticking with the large Enterprise Data Warehouse paradigm and traditional waterfall development methods.


The Naturalist’s View

There’s another way of approaching this.  Biomimicry[i] is about using insights from natural systems and applying to human challenges.  I’ve discussed my fascination with this simple idea and application to consulting in previous posts.  Nature solves problems using the following techniques:

Evolution:  Winning solutions are replicated, losing solutions are terminated.  Mutation means innovation and unexpected solutions.  Feedback is used extensively.

Resource Efficiency:  Materials are recycled, energy is used efficiently and components are parsimonious.

Adaptable to Change: Self renewal.  Resilience is created by decentralisation, redundancy, and diversity.

Development & Growth:  Design is component based with nested hierarchies, it is bottom-up and components are self organising.


So How Do Agile & Data Lake Fit In Here?

What is an Enterprise Data Lake?  It’s a fairly new thinking paradigm in BI and is a whole other set of papers that I urge you to explore.  But to quote from Capgemini’s Steve Jones[ii]

“The Business Data Lake is a new approach enabled by a new generation of technologies. No longer do organizations need to be weighed down by the technology constraints of the past. But more than a technical solution the Business Data Lake is a new approach to the information challenge based on the principle that collaboration and access to information lie at the heart of effective business.”

See some excellent material at http://www.capgemini.com/big-data-analytics/business-data-lake

Let’s look at this as a Data Lake Vs Traditional EDW comparison:

  Traditional EDW Data Lake
Evolution Data that will be used is fixed and prescribed at start. Add data to the lake as required and as new sources emerge.
Parsimony Prescribe full canonical model up front and make sure all data is clean, integrated, and ready to use. Manage only the common master data elements.  Monitor MD pain points and correct them.
Change Adaptation Complicated change management in hands of centralised IT governance function. Change managed at a local system level where possible.  Control resides with data consumers.
Development & Growth Design and use of information is centrally governed, often by IT. Governance of priorities and effort typically determined locally and by the business owner.

 

With the data lake approach the locus of control is with the local business user, who knows what they want.  This allows them to adapt rapidly, taking and reusing what they need from the lake.

How about Agile Vs Waterfall?  Agile is evolutionary in nature, continually iterating to an improved solution.  It’s inherently adaptive – change is accepted as inevitable and a source of improvement.

  Traditional Waterfall Agile Approach
Evolution Requirements set at start and not tested until many months later. Requirements to realisation cycle is weeks. 
Parsimony Design everything up front, and then build it. Only design and build what is needed, then iterate to improve and add.
Change Adaptation Discouraged, complicated and requires contract management. Encouraged, inevitable, users are not expected to know exactly what they want in advance.
Development & Growth Design, build, end users are different teams and involved at different times. Self organising small ‘one team’ mentality.  Business perspective integral to team.

 

Taking agile development and a Business Data Lake philosophy to information provision is often a better way of solving the Information Management problem.  There will be less waste because evolutionary mechanisms are employed, it’s parsimonious, allows change, and allows decentralised bottom up decision making.

Did you ever notice that there isn’t much waste in nature?  Shall we do the same in Business Information?

Related Posts

big data

Moving Big Data delivery from the West Coast to the East Coast—part 1

Simon Turnbull
Date icon January 17, 2018

Over the past few years we have started to see a paradigm shift in the capabilities that...

Business Information Management

SAP and Today’s Business to Business Transaction

Kenneth Van Meter
Date icon April 25, 2017

When companies begin to think about how they can reach their prospects, they often turn to...

Business Information Management

How Should a Tester Adapt to Cloud – Call for Change of Mindset Amongst Testers

renu.rajani
Date icon January 23, 2017

It’s time that we as testers prepare ourselves to master and test this inevitable...