Naked forecasts (Supply Chain Antipattern)

No one would let it be perceived that he could see nothing, for that would have shown that he was not fit for his office, or was very stupid. No clothes of the Emperor’s had ever had such a success as these. (The Emperor’s New Clothes, by Hans Christian Andersen, 1909)

Alias: Gosplan (sovietic planning)

Category: organization

Problem: a company faces recurrent stock-outs and excess inventory. Those problems are very costly. Clients are defecting to competitors due to stock-outs, but excessive inventory invariably ends up being costly to liquidate. While macro-forecasts, at the network level, or per product categories, are relatively accurate and unbiased, many mistakes are made at the SKU level, forecasting either too much or too little. The company has already undergone several iterations with software vendors, and yet, even though each vendor claims to have improved the forecasting accuracy compared to the previous system, excessive inventory and stock-outs remain more prevalent than ever.

Anecdotal evidence: the forecasts are always wrong, everybody knows that, but the planning folks seem to have an endless stream of excuses to cope with the situation.

Context: the company has several teams to orchestrate its supply chain, most notably: the planning team, the purchasing team, the production team, the replenishment team and the pricing team. The planning team produces the primary demand forecast for every single product to be launched and sold by the company. As the forecast has to cover a sizable portion of the product’s life cycle, the forecasting horizon is long - at least 3 months and frequently over 1 year. The primary demand forecast, the “plan”, is first transformed into purchased quantities, then into produced quantities, then into stock allocations, etc. Finally, depending on whether stock levels are fluctuating above or below levels set by the plan, prices are adjusted, sometimes up, but mostly down.

Supposed solution: the “plan” - i.e. the forecast produced by the planning team - has accuracy problems as products get sold either faster or slower compared to the original forecasts. Yet, the forecasting methods used by the company are somewhat crude, partly done with spreadsheets, and surely there must be more accurate ways of producing those forecasts. The management decides that something has to be done about those forecasts, and triggers an initiative to improve the forecasting accuracy. At this point, a third-party vendor typically gets involved - as advanced statistics aren’t exactly a core competency in the company - either to deliver a piece of software, or to deliver some training to the planning staff.

Resulting context: a lot of efforts are invested into improving the forecasts. According to some metrics, forecasts are improving. On the other hand, all the other teams, outside of planning, were used to the flaws of the old forecasting ways, and had already developed their own ways to cope with the limitations. As the planning team changes its recipe, all the other teams have to learn to cope with the new flaws of the new forecasting recipe. This causes a lot of friction for a while. Then, while revising all the supply chain processes driven by the forecasts yields a few low hanging fruits - completely unrelated with forecasting per se - the management doesn’t see any measurable outcome from the initiative. Excess stocks are still a problem, stock-outs are still as frequent as ever. Putting aside the fancy mathematical metrics, the loose perception within the company remains that forecasts are still as bad as before. Some key employees involved in the forecasting initiative have now moved to greener fields, often in other companies. No one really owns the results of the defunct forecasting initiatives, but vestiges remain both in the processes and the software tools being used by the company.

Seductive forces: a more accurate forecast looks like a silver bullet. Everybody, from the purchasing team to the store merchandising team, agrees that it would ease nearly all the company’s pain points: only push the top sellers to the market, keep just enough capacity to support demand but not more, stop giving away discounts, … It’s also a nice one-dimensional problem: reduce the forecasting error. It’s easy to convey the intent of the initiative to all stakeholders, and it feels like a rational - scientific even - way to improve the company. Also, it fundamentally does not touch the status quo in any meaningful way. Nobody gets their position threatened by the potential advent of more accurate forecasts, nobody gets to rethink their purpose in the company either. As far as digital transformation goes, it’s expected to be as straightforward as moving from one computer screen to a bigger one.

Positive patterns to address the problem: the only way to fix the “naked forecasts” problem is to put some clothes on them; more specifically, the supply chain decisions attached to the forecasts should be treated as intrinsically entangled with their underlying forecasts. The forecasting accuracy should be treated as a “debugging” artefact - which helps by pinpointing modelization problems - but not as a KPI to be optimized. The only metrics that matter are measured in dollars or euros and are associated to the mundane decisions such “how much to buy?”, “how much to push in the store?”, “how much to discount?”, etc.

Example: Contoso, a large fashion brand operating its own retail network is facing excess inventory at the end of every season, which results in hefty discounts being offered to customers to liquidate the excess during the sale. Worse, over the years, the average rate of discount has been steadily ramping up, and a growing portion of the customer base is now delaying their purchases until the sale period. While macro-forecasts are satisfying, many mistakes are made every season for many products, forecasting either too much or too little. Contoso has already undergone several in-house iterations to improve the forecasts. Those initiatives felt like the natural continuation of the ERP customization initiative that took place a few years back.

The roll-out of a new collection follows a well-established process. First, the planning team defines the range and depth of the collection, with target quantities for every product. The purchasing team follows, by applying further adjustments: MOQs (minimum order quantities) have to be met, and they have to ventilate the quantities across the sizes, as the original forecasts are at the product level. Then, the merchandising team and the store allocation teams establish the initial quantities to be pushed at the beginning of the season in every store. As the season progresses, the replenishment team steers the replenishment, trying to maintain the alignment with the forecast. Finally, at the end of season, and sometimes even before, the pricing team orchestrates the markdowns, in order to reestablish alignment with the plan wherever the excessive inventory has gone completely out-of-sync with the original forecast.

Contoso’s directors realize that the in-house initiative at improving the forecasting accuracy didn’t yield the intended benefits. The planning team is still struggling to get seasonality right. The CEO of Contoso is approached by the CEO of Genialys, a heavily funded Californian startup that has developed the next-gen of forecasting. Their technology is not only capable of processing all of Contoso’s sales data in real-time, but they are also integrating real-time weather data and real-time social media data. A few reference calls demonstrate that they have already validated the technology with some very big names. All of it is very impressive.

Thus, with the direct support of the CEO, the big initiative with Genialys emerges, with the goal of dramatically improving the forecasting accuracy. The first few weeks are going well, but after two months, it appears that Contoso’s IT teams are really struggling to extract all the relevant data. Many seemingly small problems turn out to be complicated. For example, Genialys’ team isn’t too sure what to do with “buy one, get one free” promotions that Contoso routinely performs. After 6 months of relatively intense struggle on both sides, Genialys is now delivering its forecasts. However, the planning team doesn’t really trust those numbers. Simple manual reviews of the numbers produced by Genialys show that the numbers are sometimes completely off. Genialys’ teams keep pointing out problems with the data, which seem to explain those forecast problems, but overall the situation is murky.

Not knowing who to trust, the supply chain management of Contoso decides to put KPIs in place to quantitatively assess the respective accuracies of Genialys and of the “old” forecasting system. The idea seems simple enough: let’s do a backtesting, it will clarify who is the most accurate. Unfortunately, 3 months later, dozens of meetings and hundreds of hours of effort later, the situation is still murky. It turns out that the historical forecasting process used by Contoso is impossible to backtest because the planning team has been manually adjusting many of the forecasts. Thus, they can’t really “replay” their historical forecasts, it’s just too much effort. On the other hand, Genialys has performed many backtests, but it’s unclear how many of those numbers are real. While the accuracy metrics of Genialys appear to be OK in aggregate, the planning team keeps discovering insanities in the numbers routinely produced by Genialys.

18 months down the road, Genialys is now used in production for a few stable product lines - like men’s underwear - which were never really much of a challenge to forecast in the first place. Tough categories like women’s shoes or men’s suits are still operated manually by the planning team with the old process. The original ambition to leverage weather and social data now belongs to a distant past. The Genialys solution is barely able to cope with the simplest categories anyway. The plan remains to increase the scope of categories covered by Genialys, but the teams are somewhat exhausted. Some people have left already. Business-wise results are mitigated. Men’s underwear availability has increased by 2% and markdowns have been reduced by 1%, however as the number of references has been reduced in this category, it’s unclear whether the extra (never measured) forecasting accuracy has anything to do with this favorable evolution. Officially, the forecasting initiative is still moving forward, but the top management don’t expect anything anymore from it.