Assessing the success of Quantitative Supply Chain
It might seem as somewhat of a paradox, but while the Quantitative Supply Chain puts significant emphasis on numerical methods and measurements, our experience tells us that metrics tend to tell us too little, and often too late, about whether an initiative is on the right track . Nearly all metrics can be gamed and this usually comes at the expense of the chosen approach’s sustainability. Thus, Quantitative Supply Chain seeks obvious improvements: if improvements are so subtle that it takes advanced measurements to detect them, then, the initiative was most likely not worth the effort and should be considered as a failure. On the contrary, if improvements are noticeable and consistent across many metrics, and the supply chain as whole feels more agile and more reactive than ever, then the initiative has most likely succeeded.
Metrics can be gamed
There is a reason why engineers are rarely assessed based on metrics: they are just too good at gaming the metrics, that is, taking advantage of the metrics for their own interests rather than serving the interests of the company. Supply chains are complex and nearly all simple metrics can be taken advantage of, in ways that might be thoroughly destructive for the company. It might feel that this issue is just a matter of closing the loopholes that lurk within the metrics. Yet, our experience indicates that there is always one more loophole to be found.
A tale of metric reverse engineering
Let’s take a fictitious e-commerce as an example. The management decides that service levels need to be improved and thus the service level becomes the flagship metric. The supply chain team starts working in accordance with this metric, and comes up with a solution, which consists of vastly increasing the stock levels, thereby incurring massive costs for the company.
As a result, the management changes the rules, the maximum amount of stock is defined, and the team has to operate within this limit. The team revises its figures, and realises that the easiest way to lower the stock levels is to flag large quantities of stocks as “dead” , which triggers aggressive promotions. Stock levels are indeed lowered, but gross margins are also significantly reduced in the process.
Once again, the problem doesn’t go unnoticed, and rules are changed one more time. A new limit is introduced on the amount of stock that can end up being marked as “dead”. Implementing this new rule takes a lot of effort because supply chain suddenly struggles with “old” stock that will need to be heavily discounted. In order to cope with this new rule, the team increases the share of air transport in relation to sea transport. Lead times are reduced, stocks are lowered, but operating costs are rising fast.
In order to deal with the operating costs that are getting out of control, management changes the rules once more, and puts an upper bound to the percentage of goods that can be transported by air. Once again, the new rule wreaks havoc, because it triggers a series of stock-outs that could have been prevented by using air transport. As a result of being forced to operate under increasingly tight constraints, the team starts giving up on leveraging the price breaks offered by suppliers. Purchasing smaller quantities is also a way of reducing lead times. Yet, once again, gross margins get reduced in the process.
Getting the purchase prices back on track proves to be a much more elusive target for management. No simple rule can cope with this challenge, and a myriad of price targets for each product subcategory are introduced instead. Many targets turn out to be unrealistic and lead to mistakes. All in all, the supply chain picture is less and less clear. Pressured from many sides, the supply chain team starts tweaking an obscure feature of the demand planning process: the product substitution list.
Indeed, management realized early on in the process that some stock-outs were not nearly as impacting as others, because some of the products that were missing had multiple near-perfect substitutes. Consequently, everybody agreed that stock-outs on those products could be largely discounted when computing the overall service level. However, the supply chain team, which is now operating under tremendous pressure, is starting to stretch the purpose of this list one to two notches beyond its original intent: products that are not that similar get listed as near-perfect substitutes. Service level metrics improve, but the business does not.
The pitfall of success
Metrics can be gamed and if teams are given toxic incentives, metrics will most likely be leveraged in a misleading manner. However, the situation is not nearly as bad as it might seem. In fact, our experience indicates that except for really dysfunctional company cultures, employees don’t generally tend to sabotage their line of work. Quite the contrary, we have observed that most employees take pride in doing the right thing even if it means that company policies need to be stretched a little.
Therefore, instead of taking freedom away from the team in charge of implementing the supply chain optimization strategy, it’s important to encourage the team to craft a set of metrics that sheds light on the supply chain initiative as a whole. The role of management is not to enforce rules built on the basis of those metrics, but rather to challenge the strategic thinking that underlies those metrics. Frequently, the immediate goal should not even be to improve the metric values, but to improve the very definition of the metrics themselves.
In reality, all metrics are not equally valuable for a business. It usually takes considerable effort to craft metrics that give a meaningful perspective on the business. This work requires not only a good understanding of the business strategy , but also a profound knowledge of underlying data, which comes with a myriad of artifacts and other numerical oddities. Thus, metrics should above all be considered as work in progress .
We have found that a strong indicator of success in any supply chain project is the quality of the metrics that are being established throughout the initiative. Yet, it’s a bit of a paradox, but there isn’t any reasonable metric to actually assess the relevance of those metrics. Here are a few elements that can help evaluate the quality of metrics:
- Is there a consensus within the different supply chain teams that the metrics capture the essence of the business? Or that the business perspectives implicitly promoted by the metrics are neither short-sighted nor blindsided?
- Do the metrics come with a real depth when it comes to reconciling the numbers with economic drivers? Simplicity is desirable, but not at the expense of getting the big picture wrong.
- Are the data artifacts properly taken care of? Usually, there are dozens of subtle “gotchas” that need to be taken care of when processing the data extracted from the company systems. Our experience tells us to be suspicious when raw data appears to be good enough, as this usually means that problems haven’t been even identified as such.
- Do decisions generated from the chosen metrics make sense? If a decision, that is otherwise aligned with the metrics, does not feel like it makes any sense, then, it most probably doesn’t; and the problem frequently lies in the metric itself.
In many ways, crafting good metrics is like orienting gravity towards the pit of success: unless something intervenes, the natural course of action is to roll down the slope towards the bottom, which happens to be precisely where success lies. Knowing the exact depth of where the bottom lies is not even strictly required, as long as every step of the journey towards the bottom is making things better for the company.
Sane decisions lead to better performance
In supply chain, even the best metrics come with a major drawback: numbers are usually late to the party. Lead times might be long and the decisions made today might not have any visible impact for weeks, if not months. In addition, Quantitative Supply Chain, which puts significant emphasis on iterative and incremental improvements, complicates this matter further. Yet, using non-incremental methods would be even worse, albeit for other reasons. Therefore, metrics can’t be the only signals used for assessing whether the initiative is on the right track.
Generating sane decisions is a simple, yet underestimated, signal of superior performance. Indeed, unless your company is already doing exceedingly well with its supply chain, it is most likely that the systems keep producing “insane” decisions that are caught and fixed manually by the supply chain teams. The purpose of all the “alerts”, or similar reactive mechanisms, is precisely to mitigate the ongoing problems through ongoing manual corrective efforts.
Bringing the Quantitative Supply Chain initiative to a point where all the decisions - generated in a fully robotized manner - are deemed sane or safe is a much bigger achievement than most practitioners realize. The emphasis on “robotized” decisions is important here: to play by the rules, no human intervention should be needed. Then, by “sane”, we refer to decisions that still look good to practitioners even after spending a few hours on investigating the case; which naturally can’t be done on a regular basis, due to the sheer amount of similar decisions to be made every day.
Our experience indicates that whenever the automated decisions are deemed reliable, performance materializes later on when those decisions actually get put to the test of being used “in production”. Indeed, the “sanity” test is a very strict test for the decision-making logic. Unless your company is already leveraging something very similar to Quantitative Supply Chain, then, most likely, the existing systems your company has in place are nowhere near passing this test. As a result, uncaught mistakes are being made all the time, and the company ends up paying a lot for this ongoing stream of problems.
Then, from an operational viewpoint, as soon as supply chain decisions become automated, the supply chain teams become free from the servitude of feeding their own system with a never-ending stream of manual entries. Those productivity gains can be reinvested where it actually matters: to refine the fine print of the supply chain strategy itself, or to monitor suppliers more closely in order to address supply chain problems that originate from their side. The increase in performance, achieved through pure quantitative optimization of the supply chain, is intensified by the gains obtained by the supply chain teams who can finally find the time to improve the processes and workflows.