Sparking the data science of optimization

Standing in line at the coffee shop, Blair’s phone buzzed in his pocket. Fishing it out of his coat, he noted that it was his regional vice president and boss, so he quickly answered, “Hi Tom, how’s it going?” Tom Callahan was a card-carrying member of the old generation of product sales guys who loved to sell and hated paperwork and planning. “Hey, Blair,” he responded. “I’m going to need your annual plan updated by the end of the day; and oh, by the way, corporate now wants it optimized for both profit and POS, OK?”

Blair has worked for this company for nine years. He remembers the total fear and dread of having to make even a minor change in his annual promotion and sales forecast plan. But today?  Well, today is different. He smiles. “No problem, Tom,” he says. “Let me take care of that now.”

Picking up his coffee, he walks to the back of the room and grabs an easy chair and fishes his iPad out of his briefcase. Two clicks and he’s into his promotion plan. At the top of the screen, he selects the OPTIMIZATION tab and scrolls down to his 2019 annual plan, which he had already submitted the previous week. Double-clicking on it, the plan appears in a new window with a group of functional radio buttons aligned with key objectives that he can chose individually or together. He had already optimized his plan for achievement of maximum point-of-sale (POS) performance, so he clicks the radio button for “Profit” and clicks “Optimize.” He picks up his cup and takes a couple of sips of his coffee. The screen displays a group of constraints, including one for volume and one for revenue. He has already optimized against the volume constraint, so he goes with the minimum revenue requirement and clicks “OK.” The screen returns with a successful update and he clicks the command to forward the plan to the approval routing process.

Done.

He clicks the home button and slides the iPad back into his briefcase.  He has a few minutes to spare, so he decides to read the paper left by the guy who had the chair before him.

He smiles again.

Blair knew that with the previous system this process would have likely taken him the rest of his week and potentially into his weekend. Today, however, he is confident that the technology is smarter than he is, and that his company’s clean and harmonized historical and downstream data is accurate and trustworthy.

This is not science fiction – it’s data science, and the technology exists today to make it happen. Trade promotion optimization (TPO), as a concept and process, has been a buzzword in the consumer products industry for almost 25 years. The first real public mention of the modern use cases for TPO was by the late Bob Houk in his 1993 paperback “MEDIANET Guide to Promotion Allowance Marketing” and later in his book, Co-op Advertising (1995, Association of National Advertisers).  Houk showed the early use cases for how technology could leverage historical trade promotion, POS, and syndicated consumer research data to create computer-generated “what-if” scenarios that predicted ROI (incremental sell-in volume and sales, and incremental sell-out POS) the promotion planner could turn into a finished promotion. Most of those scenarios he covered are still very much at the forefront of the metrics for trade promotion today.

In 2008, Michael Kantor and his partners created the Promotion Optimization Institute (www.poinstitute.com) dedicated to improving the performance and financial return on investment of trade spending. Kantor’s organization began by offering academic-accredited courses that focused on the growing technology and process trends of promotion optimization and has created one of the most successful industry trade organizations dedicated to trade promotion in the world.

What POI has done is to elevate the discussion of trade promotion optimization and thrust it into the center mass of digital transformation in consumer products. Bringing together the key revenue and promotion management executives all discussing TPO, the relevance of promotion optimization has grown to become a mandate for companies planning to upgrade their trade promotion management technology. The Promotion Optimization Institute’s (POI) 2019 State of the Industry report (Pam Brown, author) is breaking misperceptions: The majority (of over 100 respondents) felt it was over two years before artificial intelligence (AI) would impact CPG planning. The reality is that AI is already represented well with a few of the vendors in the POI TPx Vendor Panorama.[1]  “AI advances a company’s ‘prescriptive planning’ ability by using data to propose promotions, time frames, discounts, pricing, revenue growth management, etc. for the teams to evaluate,” says Kantor about POI’s promotion of TPO.

As with any new technology, however, there are misconceptions about the true functionality and value proposition of TPO. Unfortunately, many CPG companies are being presented to by vendors who claim to have rich TPO tools that may not actually use data science to arrive at predicted ROI. The functionality can vary significantly in data science and capability. There is clear segmentation between classes of TPO functionality that we will call basic, advanced, and highly advanced:

• Basic – The ability to provide simple reporting of historical trade promotion performance data to sort and display measured metrics (e.g., sell-in volume, revenue, and profit) based on promoted products, timing, tactics and costs. The planner would review the report to provide manual insights into how they would eventually configure the promotion.
• Advanced – Deeper (non-AI or cognitive computing technology) reporting and display capability to leverage both internal (trade promotion performance history, revenues, profit, and volume) and external (downstream POS and syndicated data) that deliver optimal results in the form of ranked metrics from which the planner can infer predicted ROI and select the best choice to create the promotion. The technology may be legitimate Bayesian or other similar algorithms that produce automated predictive results.
• Highly Advanced – Using artificial intelligence and machine-learning data science that leverage deep historical promotion results, and external data including both downstream and syndicated data to generate optimal predictive elements of promotions that the planner’s system can automatically accept, allow for editing or adjustment to current conditions. Outcomes are prescribed predicted metrics, including incremental ROI in both sell-in and final POS sales results.

In the latest CGT/Capgemini 2018 Trade Promotion Survey, the bulk of the respondents (42%) indicated that the best they had was a basic TPO functionality. However, notice that almost 40% of the respondents had some form of advanced TPO, and 11% claimed to have sophisticated AI, machine learning or other form of cognitive computing data science driving their promotion optimization engines.

With all this good news, however, it should be noted that, as with any high-tech engine, the fuel is critically important. One of the most difficult problems CPGs face today is bad data. We all know that trade promotion performance over the last several years shows that more than one-third of all promotions fail to achieve positive ROI. While the argument about sampling methodology and the appropriate need for both good and bad results may be mathematically accurate, most of my colleagues disagree. Why include promotions that fail? Other problems with unclean data, lack of harmonized product and account data, or syndicated data with extrapolated results to fill in market and location gaps with filler data all contribute to a base of more than acceptable margins of error.

Therefore, we are seeing more data lake projects take on cleansing and harmonization than before. It is also why many CPG companies are looking toward the acquisition of direct POS where they can get not only 100% coverage of every location and market, but also deeper intelligence, such as inventory tracking, pricing, assortment, and category planning.  “It’s the data that drives the TPO,” says the revenue manager for a major produce supplier. “I have to make sure that my [sales planners] are confident in the output from our TPO, because it costs a ton of my budget to get in here, and I keep tight reign over our database guys to make sure our data is clean and aligned.”

Michael Kantor’s POI organization pushes this issue harder than most.  Says Kantor, “AI advances a company’s ‘prescriptive planning’ ability by using data to propose promotions, time frames, discounts, pricing, revenue growth management, etc. for the teams to evaluate.” As we move toward higher levels of integrated data from marketing, including consumer profiles, online footprint tracking, coupon redemption performance, and ecommerce promotional performance, the promise of advanced data science is

As we see more advanced data science embedded into TPO systems and the increase in accuracy and trust of the data driving the intelligence, the promise of a more precise and trustworthy outcome from these tools becomes evident. Trade promotion optimization is rapidly becoming a mandate and, soon enough, a standard.

[1] The POI “Vendor Panorama” is an annual trade promotion vendor technology report comparing and ranking vendors according to several performance metrics manufacturers typically evaluate in a selection process.