While the potential for big data to impact business decisions is well understood, a joint Capgemini EMC study highlighted impediments to implementation.  45% of respondents stated that “The current development cycle for new analytics insights is too long and does not match my business requirement”, an additional 52% respondents blamed the developmental process.
To enable the agility that businesses are today looking for, cultural shifts in thinking and approaches are required. It is important to approach an analytics and data science project as an innovation project. Too often, people are looking for certainty, a decisive project plan and guaranteed outcome. Your insights are only as good as your data…if we accept that, then elimination(rejecting a hypothesis) also brings us closer to selection.(finding the relevant insights) Hence governance around these projects has to shift from a pure IT governance process to one that is more akin to governing research projects. One of the relevant quotes was from Forrester’s Gualtieri sums it up very nicely. He suggests that businesses approach big data projects the way a venture capitalist invests in companies: “A venture capitalist will invest in 10 companies hoping one or two will hit, and that’s how companies have to view advanced analytics and predictive modeling.”
In addition to changing the way we approach these projects, there are two additional enablers that will impact agility and productivity. One is environmental setup and the second is design of the analytics processes. Data scientist and predictive modelers have long been lone warriors, each following an arduous path of exploration and elimination to get to relevant insights.  Big data environments can offer the data scientist easier access to data, (both structured and unstructured) and faster turnaround time. Environments and sandboxes have to set up in such a way so as to wean away your data scientist from the silo to the collaborative environment. It is essential to set a vision, get people on the same page, understand scalability and commonalities in mission and code, to set up an environment that allows them to leverage and collaborate. Each person in a team need not have to tread through each part of the maze but can successfully rely on the wisdom of another. Such collaborations also enable teams to define and leverage a wide variety of skills…allowing a data scientist to really spend more time on insights
The path to analytics insights is an iterative path; hence our vision, approach and processes also need to be re-aligned to facilitate collaboration and innovation, while also meeting the business requirements of agility and productivity.