This is the second post in a series on the next wave of innovation in BI. Part of this post was published in Information Management on May 3, 2011. In this post we will discuss a new paradigm in Business Intelligence. Before presenting the future, we will discuss the evolution of BI in large organizations from an historical perspective.

The evolution of BI in large organizations goes back to the 1970s. In an increasingly competitive and global environment, business managers were looking for tools to support their decision making processes. The early BI tools were used to extract data from source systems and to produce reports with performance indicators. The tools used in those early days were mostly custom-made applications developed by internal IT specialists. To satisfy the needs of a growing number of business managers, specific queries were launched overnight against the production systems. The objective was to get business information out of the production systems in the form of fixed-format standard reports (“print-outs”). On a regular basis, the printed information was manually aggregated and keyed into presentation templates and data sheets.

Some years later, new concepts, like the “Information Support Database” or the “Infocenter” were introduced to offload querying on the production systems and to improve the performance of the overall BI solution. As a response to the growing need for management support and reporting tools, software vendors like Pilot Software, Information Resources, and Comshare jumped on the occasion. The first generation of BI tools is often identified with the term EIS (Executive Information Systems). The early BI tools included ETL functionality, merged data from multiple sources, used relational databases, included what we later called Star-schemas, and built cubes for fast data retrieval. The EIS revolution can be considered as the first wave of innovation in Business Intelligence.

Despite some high expectations, in the early 1990s, the EIS pioneers fell on hard times. The costs to implement corporate EIS systems were way too high. The required technical infrastructure wasn’t there, so the EIS tools had to include their own. In addition, EIS didn’t target and serve enough end-users because of the “Executive” connotations. At the same time, new innovations like Data warehousing began broadening the realm of decision support and initiated a larger category of BI tools. The Data warehousing model was further popularized as a mean to describe a new set of concepts and methods to improve decision making by using fact-based decision support systems. New industry leaders like Bill Inmon (“The father of the Data warehouse”) and Ralph Kimball (“the inventor of the star schema”) actively promoted the Data warehouse by using relational database technologies.

Looking back we can say, that the Data warehousing model (ETL, DWH, DM, OLAP, etc) was a true revolution in Business Intelligence. During the second wave of innovation in BI, the production of management information was being industrialized by means of sequentially scheduled batch-processes (Information Logistics). The entire production process, from the extraction of source data to the generation of reports, was being automated by means of specialized BI tools. The Data warehousing model, as introduced in the early 1990s, has shaped the BI landscape ever since. Because of its proven concepts and technologies, the traditional Data warehousing model is still today the guiding principle for designing new BI architectures in large organizations. All major players and software vendors in today’s market place are still on the Data warehousing bandwagon, but for how long?

The Data warehousing model has enabled significant advances in the integration and use of business information, but its underlying architectural approach is now being questioned. In order to support future business requirements and to cater for the need of decision support in a 24/7 environment, BI needs to extend beyond the traditional Data warehousing model and include automated decision-making and real-time technologies. In addition, Business Intelligence needs to become an interconnected business function at the center of the enterprise architecture to make business processes more flexible and smarter. Will Search-based BI be the next revolution in Business Intelligence? In the next posts, we will explore some new groundbreaking innovations in BI.