Are your KPIs, Scoreboards and other metrics safe from the Simpson's paradox?

“Simpson’s paradox or Yule-Simpson effect is a paradox in probability and statistics, in which a trend appears in different groups of data but disappears or reverses when these groups are combined. It is sometimes given the descriptive title reversal paradox or amalgamation paradox”. (Extract from Wikipedia)

KPI, dashboard, other types of analytics that finance and FP&A use for communication may include or generate such paradox polluting the decision making process. Typically, those tools aggregate i.e. combine data in the perspective to give simpler and quicker ways to manage and communicate performance and take decisions. The general trend in management practices is to highly recommend limitation of indicators’ number, and as such to increase the aggregation level. It more or less focuses the attention of top and middle management depending on company culture and on rewards and recognition systems.

It is then highly possible for such aggregation to induce directly or indirectly “false” understanding or certitudes just like Simpson’s paradox would do. It should then be FP&A role to find and to fight such occurrence.

Still, we hardly hear or see mention of this concept during professional studies and life. 
Practically, teaching and promotion of such tools includes few recommendations that would potentially exclude dropping in that trap but without formally researching or verifying whether it exists or not.

Example

A sales director is looking at the ratio of commercial contact turned into sales in order to follow the performance of two teams and (among few other things) reward them on this basis. 

Team 2 had better rewards periods after periods.  

Frustrated Team 1 leader ask their FP&A to analyze in more details and feedback is 
“Sorry, Simpson paradox!! Here is a better view of the relative performance! In fact, the two teams had the same performance over last 6 periods“ 

Of course, here I forced the figures to a ridiculously simplistic example to illustrate my point and make it easy to understand. 

In real life, it is generally far more complex to identify such issue. My example is a reversal case (or nearly) which is more easy to spot. Still, in many cases it will not be so extreme, it will only hide (make disappear) the real evolution or performance. 

The paradox appears thanks to few main causes:

1. Dependent variables are hidden through the aggregation of data,
2. Difference of distribution in time (the example is it),
3. Difference of size between the data groups entering in the calculation.

Similar examples can be found with return on investments, any given type of expenses (say marketing) versus revenues, product lines revenue distribution or growth, etc.… (In fact, many of the widely known ratios or indicators may very well include or generate such effects if not properly customized and tested).   

This brings few questions:

1. Up to what point this paradox is known (as such or in another form)?
2. How is it reflected in the definition and implementation of the company tools and processes?
3. How is the verification done?

In my example, FP&A should have to make sure the KPI was not reduced (aggregated) to such a level and demonstrate that a more comprehensive set of figures needs to be considered, this should have been made during the set-up of the company systems and FP&A should have “tell the story” behind those figures all along the process in such a way that recognition should have been more in line with performance. A period to date indicator (or a weighted indicator) would have been more representative e.g.

In this example, it is highly intuitive, i.e. even without knowing about the paradox, a reasonably able person shall have made it happen. Still, reality can be far more complex and intuitive reasoning might not be sufficient to identify and adopt the right approach in each and every case, in particular, when the context is strongly oriented towards limiting the number of KPI, Indicators,… used by management. Who has not heard a manager saying “This is too much detail/figures, I just need to know (i.e. my compensation is only based on) these x indicators”. 

Obviously, this brings into picture a different subject i.e. the reward and recognition based on KPI, scoreboard and other metrics. Still, is it really different, if the metrics can reveal to be paradoxical?!

Knowing (be conscious of) this paradox and systematically making sure to avoid its trap is then a key role for FP&A (i) when elaborating the systems and process that will be used (ii) when analyzing and commenting the ongoing results. 
 
We may try to draw a list of a potential common case but it remains a high-level approach that cannot exempt from doing effectively the exercise. 

• Volume vs price evolution i.e. where your Revenue Indicators may hide more or less important shift in the price and/or volumes. Similar for external expenditures.
• Regional differences, RI covering wide geographic reach may hide major evolutions in the respective areas,
• Customer profiles differences, RI covering wide customers profiles may hide customer profile shift,
• Product or market maturity phase, RI covering wide range may hide product or market maturity shift,
• Currency effect, in particular when pricing currency is different from the entities currency and from the consolidation currency. Indicators expressed in consolidation currency will aggregate and then hide any currency effects and shift. This also applies to most costs.
• Risk assessment or indicators,
• Quality indicators,
• Productivity indicators, 
• …

Those applicable to revenues (but not only) may very well combine all together. 

A very simple  example would be a global company management concentrating on an overall revenue indicator mixing revenues from different geographic areas with product and markets in different maturity phase where customer profiles are evolving rapidly and where business is done in few pricing currencies but over a large number of legal entities (with different currencies then) and consolidating their results in its home currency. Such a global revenue indicator would mix so many effects and potentials evolutions that it may reveal paradoxical. Even more dangerous, as it may not have been a problem for few years i.e. whereas the potential of an issue was there but the reality of the markets was not challenging (activating) it. 

We could bet in such a case that the information systems will “lose” the pricing currency within the consolidation process, potentially shortcut the entity currency to consolidation currency effect too; will aggregate the product into a few product lines historically driving the company development, will probably lose track of the customers profiles among few other “simplifications” justified by the necessity to keep things more effective. Furthermore, the FP&A people may be located in a limited number of shared service centers loosening the contact with local reality. The end result would be that the whole organization IS and processes, (probably culture) are cut from reality thanks to a basic misconception (lack of understanding/verification) during the process of defining and implementing tools.  

FP&A peoples do not need to know all the mathematics about this paradox. They still need to understand the concept, recognize that it may “slip in” when defining their tools and processes, KPI, Scoreboard and other metrics and then make sure it is not the case.