Is IT risking irrelevancy? We have heard this all before; Cloud will make IT redundant; outsourcing and the rise of the virtual IT department, etc. Are we seeing something different with the rise of big data?
In our recent study – Big & Fast Data: The Rise of Insight-Driven Business – one alarming statistic leapt out that should be a flag to any IT leader:
Over a third (36%) of the most senior non-IT decision-makers surveyed (C-level/director-level) say that their business unit has circumvented IT in order to carry out the data analytics it requires.
IT being bypassed: this is likely to be an increasing rather than decreasing trend.
What are the likely drivers? If we cast our eyes at the wider statistics, it is likely to be multiple drivers; both in the business and as discrete IT challenges:
Almost half (45%) of respondents consider the current development cycle for new analytics to be too long and not matching their business requirements. Another gap is a shortage of skilled people to analyze the data properly (mentioned as a top three issue by 37% of respondents).

  • IT has a crisis of skills – Big Data technologies have a relatively low crossover with existing skill sets; the market is under supplied with the key software skills.
  • The business has a crisis of skills – and I place data science and analytics skills in business, not in IT.  The HBR had it right: Data Scientist is The Sexiest Job of the 21st Century; it’s certainly in high demand.
  • There is an expectations gap – with Big Data at a high level of interest, corresponding boardroom expectations are high and need careful management to properly align the “art of the possible.”

Over half (52%) consider the speed of their organization’s insight generation to be constrained by its IT development process.  The high cost of storing and manipulating large data sets 
was mentioned by 33% of respondents 
and the time taken to analyze large data sets was mentioned by 32%.

  • Big data foundations are still being evaluated and initiated; 3rd parties can often provide it “ready to go.” IT is keen to understand the integration of the new substrates and technologies with the legacy environments.
  • Constant IT budget pressures mean that the CMO often has as much direct discretionary spend as the CIO and is not mandated to use it internally – instead they will buy a “business outcome” rather than IT provision.
  • The adoption approach for big data adoption is proving different to classic ERP deployments– typically moving from a proof of concept, “fail fast, try many” approach lends to very agile development cycles; you need people that can simultaneously think through the technology, data analytics and business scenarios.

Combined, these factors create a real challenge for IT leaders; there is no panacea, but a need for aligning around a centrally driven big data strategy is clearly a must for many.