Recently, a very interesting article was released in the New York Times, with the title “Sure, Big Data Is Great. But So Is Intuition.” From my point of view this perspective towards Business Intelligence and in this case, Big Data, is very important. I wrote my Master Thesis of the Radboud University in 2008 on the subject of tacit knowledge and BI, emphasizing that decision making in organizations is all about combining explicit knowledge (data, numbers) with intuition, gut feeling, experience and values (summarized as tacit knowledge).

Because let’s face it: why would an organization invest in Big Data? Because we want to take better decisions to improve our business (not just for a new ‘gadget’). If you want to improve your decision making, you need to realize that the process of decision making has an explicit and a tacit component that need to be addressed. So what does that mean for Big Data’s future?

Explicit knowledge, tacit knowledge and decision making

Professor Wegman (2006) stated the following formula for Knowledge: K = I * EVA. Translated to English this means: Knowledge = Information multiplied by Experience, Skills and Attitude. Intelligence can be seen as the application of knowledge in daily practice. Following this definition and the definition of Knowledge from professor Wegman, Intelligence contains an important Information component but also a so-called ‘tacit knowledge’ component.

So Business Intelligence should have an explicit and a tacit component. That may seem theoretically correct based upon the formulas stated above, but it also holds true in daily practice. BI is used to support decision making in organizations. Many scholars have stated that the importance of intuition in decision making is very high: “The effective use of intuition has even been seen as critical in differentiating successful top executives and board members from lower-level managers and dysfunctional boards (Agor, 1986;Barnard, 1938; Harper, 1989)” (Dane & Pratt, 2007). So if you want your organization to take better decisions to bring value for the future, your decisions will need to be based on both explicit information (like data) but also on intuition (tacit knowledge).

So what does all of this mean for Big Data?

The Big Data challenge

Big Data will give organizations the opportunity to collect a massive amount of (explicit) information about their customers, their competitors and the trends in the market. Data scientists create predictive models and are able to help organizations move from looking at the past to looking at the future. This is a very positive trend, but there can be a downside to it. In the article in the New York Times, miss Claudia Perlich is quoted. She is the chief scientist at Media6Degrees, an online ad-targeting start-up in New York. “You can fool yourself with data like you can’t with anything else. I fear a Big Data bubble.”

Predictive models cannot always predict the, sometimes unlogical, behaviour of people. If not all relevant factors are included in a model, the information that you see can even steer you in the wrong direction. In the article in the New York Times, the importance of using your experience and intuition when interpreting the data from Big Data projects is formulated like this: “A major part of managing Big Data projects, (…), is asking the right questions: How do you define the problem? What data do you need? Where does it come from? What are the assumptions behind the model that the data is fed into? How is the model different from reality?”

The future of Big Data: more room for your intuition

So now that we know that intuition forms an important part of decision making based upon Business Intelligence and Big Data, what’s next? First of all, data scientists in organizations need to have both analytical skills but should also be able to ask the right questions and bring in their intuition and experience to interpret certain results. BI and Big Data tooling should facilitate these types of processes. And that is very well possible!

For example, Oracle has bought Endeca Information Discovery last year. Analysts can use this product to analyze both structured and unstructured data. The way this can be done is quite new, because no pre-fixed data model is being used. This means that the data model can be created by the analyst while performing the analysis. This gives more space to his or her intuition and experience in interpreting the data, while combining data from for example ERP systems with Facebook information.

In the future of Big Data more of these types of tooling need to become available. They will facilitate the decision makers in the future to use both the ‘hard’ data and their own gut feeling and experience of how their market is behaving. That will prevent the pitfall of Big Data: only looking at the data without asking the right questions on how it was modeled and taking the wrong decisions because of it.

Product developments like Endeca Information Discovery will help organizations use both their explicit and tacit knowledge in order to improve their decision making.  And that should be exactly the purpose of Business Intelligence and Big Data!