How to ‘hit the ground running’ with Big Data
Traditional approaches to Information Management usually do not apply when attempting to implement a Big Data initiative in your organisation. That does not mean it needs to be more expensive, rather, that the approach to scoping, delivering and supporting your information outcome, is likely to traverse and ‘adapt’ your organisation’s current funding, sourcing and delivery processes potentially for the better.
Big trouble Little sign off?
Well this isn’t necessarily an issue as, the bulk of software and hardware expenditure need not be capitalised. In fact, open source capabilities and/or cloud infrastructure (given your organisations risk profile) will undoubtedly be sufficient for the task at hand, be more cost effective and, will provide an agile and segregated ‘hotbed’ of intelligence which, can be extended and exploited free from the shackles of current information provisioning mechanisms in your business.
In essence operate a segregated Big Data factory as a startup inside your business.
As your insights become useful, a mechanism for promotion of this sandbox activity will need to be funded and implemented into your existing IT environment yet, crucially not until the intelligence you have fostered, is stabile and providing real and repeatable business insight and value.
Much Hadoop about nothing?
Big data does not necessarily imply Hadoop and, it is not the ‘be all and end all’ for Big Data implementations.
It should also be stated that learning the nuances of HDFS and, more recent wrapper technologies such as Hive and Pig, are a major mindset change for traditional data and business professionals weaned on SQL. However, help is at hand as emerging vendor toolsets and languages are starting to diminish the need to learn new ways of parallel processing and, therefore, taking this type of processing into the mainstream as a ‘Dropbox-like’ lift-and-shift’ addition to your current data pipeline.
Think of Hadoop as a parallel plug-and-play processing app that accelerates unstructured analysis in complement to traditional data and modelling approaches.
Become the Big Data ‘outsider’ in your organisation.
The case for an ‘outside in’ approach to Big Data has been stated many times and is unsurprisingly absolutely critical in the Big Data startup.
Big Data is more than simply SQL, joins and internal databases exploitation within your organisation,
Big Data Success = (Inside Out (What you know about yourself?) + Outside In (What the market thinks of you?)) * (Data Science + Data Art)
it is also about external ‘triggers’ that capture sentiment, awareness and perception of your organisation using algorithmic approaches from a wide variation of information sources often, outside the control of your organisational boundaries. The ambiguity of this data exploitation journey, suggests it would operate more effectively in a ‘startup’ culture, free from your current operating model and funding constraints at the outset at least.
The analogy of an outside-in perspective should not be underestimated. Never has the concept of ‘walking in your customers shoes’ been more important.
Furthermore, a world-class ‘Big Data Factory‘ should be delivered using a segregated and multi-faceted team composed of data specialists, business SMEs, social media experts, mobile device practitioners, customer experience (CX) experts and algorithmic statisticians (such as quants or actuaries). Its operation should be free of the daily concerns such as solution scalability, operational alignment and, continuous self-justification of the teams effort. Additionally, your marketing and sales functions should have regular interactions with communities of interest across digital and physical channels of importance to your brand to ‘effectively complete the feedback loop’.
The startup team’s primary goal is provide insight into a critical business issue and, as such, to use internal and external data sources (without restriction) to achieve their aim. This will represent some challenges especially when we consider data protection, regulation and privacy laws yet, with the right outcome in mind and careful use of the data at hand, these restrictions are usually surmountable.
Action and reaction are equal and opposite
Big Data solutions are a series of mistake-ridden iterations leading to a target result of deep business insight. The data sets are constantly changing, the algorithms are constantly adapting and, the team is also re-assessing its journey and vision against a level of tolerance that continuously makes sense to the outcome being sought.
In short, Big Data operates in a cauldron of ambiguity requiring as much ‘data art’ as ‘data science’ to achieve continuous, incremental and actionable insight.
In consequence, it is critical to have a control cycle that reduces the possibility of your insights ‘missing the mark’, that prevents over-engineering and, that permits continuous adaptability to changing market needs and circumstances.
Actionable Metrics, the lifeblood of Big Data success
In the book ‘The Lean Startup‘ by Eric Ries, he describes an iterative feedback loop based on actionable metrics that constitute the inflection point for developing the right product changes for the target market and customer. This concept of actionable versus vanity metrics from Ries has profound implications to the Big Data insight cycle.
The basis for your Big Data calculations and the improvement process you adopt, must be based on a few commonly agreed metrics (7 to 10 KPIs maximum) which have a demonstrable alignment (and therefore auditable impact) on key customer touch points, as well as the efficiency of operational processes that provide said service.
The resulting insight outcome needs to be actionable and, as such, each process of gaining information insight must be demonstrable to the bottom-line of your business. (Did it reduce churn by x%, did it improve sales in territory A to demographic B by Y% etc).
After all, if your Big Data efforts to reduce costs, increase revenue or reduce risk, are not flexible, repeatable and actionable leading to continuous business improvement, why bother at all?