Performance Reports (and Other Lies)

Roger   Tregear
Roger Tregear Principal Advisor, TregearBPM Read Author Bio || Read All Articles by Roger Tregear

George Bernard Shaw told us that the single biggest problem in communication is the illusion that it has taken place. Obviously, he was talking about performance reporting[1] and the many ways that performance outcomes are misrepresented.

Sometimes this is deliberate; often it is inadvertent. The result is to distort the data or make erroneous comparisons. An illusion, indeed.

If we are going to the trouble of collecting and reporting performance data (and, of course, I'm thinking of process performance data), then we should do so accurately, transparently, and in context. The presentation of the data should not distort or confuse.

There are many ways in which data can be manipulated in its presentation — for good or evil. Some common problems (opportunities) are discussed below.

That's a stretch!

Figure 1. Which is better, A or B?

Figure 1 depicts three different groups operating in two scenarios. Group 1 does so much better than the others in Scenario A. Let's investigate and find out why … or not.

Both scenarios use the same data with the only difference being that the Scenario A chart uses a different vertical axis that is emphasizing a relatively small difference between group performances.

Figure 2. We have a winner!?

Wow, Figure 2 shows Team A is outperforming Team B by a long way in every month. Let's investigate and find the magic. Nope, same data, different scale. Team A's magic is in their knowledge of visual presentation hacks!

Smooth operations

Figure 3. Smoothing away the problems.

While we don't want to chase shadows, smoothing out the variations might hide some important details.

The moving average on the left in Figure 3 looks OK; it might be dipping down a bit but it's above the target in every month.

Hang on, if we look at the monthly data, we see sales were below target in five months of the year. It also seems that the monthly data has been much worse in the second half of the year.

That's a very different message. Maybe that's OK; it might be what we expect in those months, but we don't see that in the chart of monthly averages.

Points of light

Traffic light performance indicators (aka red, amber, green (RAG) reporting) have the benefit of being simple to use and interpret.

They can also have the drawback of being simplistic. There's not much room for nuance in the no–maybe–yes sequence.

Inevitably the data represented is an aggregation of outcomes and the detail is lost. The thresholds set for each color are also unforgiving — for example, 80% might be amber, but 79.9% is red.

Depending on the range of scores that define the green zone, we might have a strong trend of deteriorating performance, but the light is still green — there are no shades of color in a traffic light system.

2 points is not a trend

Comparing two points of data — e.g., this month and last month, this month to last year, etc. — might be of limited interest. It also has the potential to be very dangerous.

There just isn't enough information to allow you to make anything other than a tentative and superficial assumption about actual performance.

Sometimes you just must wait for more data points before any viable conclusion can be drawn.

What effective management needs to know is what is likely to happen next, not just what has already happened.

Heat but no light

Heatmaps can provide striking visual representations, and they can work well. They pack a lot of information into a small space, and that can be both negative and positive.

A lot of assumptions are necessary to create an aggregated graphic such as this one. For example, we might get the impression that half (more?) of the team are not meeting quota, but are all quota shortfalls equal?

Of course, any reporting method that relies on color will be problematic for people with color vision deficiency.

The detail is the devil

Some of you might find a table like the one below (Figure 4) informative and engaging. The rest of us struggle to form a coherent view of what we are being shown.

Figure 4. Hiding in plain sight.

While it is useful and appropriate to provide the raw data along with any graphic to give context and transparency, the raw data alone will not highlight key messages.

Of course, if you want to hide information, the middle of a large, dull table is a great place to do so!

The devil is in the detail

Sometimes an extra level of granularity is required to gain insight into performance. Weekly performance data may look fine, while daily outcomes are problematic. Daily peaks and troughs are being averaged out of sight in this weekly aggregation.

Figure 5. A question of detail.

Half of you are below average!

On average, averages can be very misleading.

This is another example of how calculated numbers can become an anodyne representation of the original dataset with most, if not all, variation hidden.

We are often very interested in those variations because they give better insight into performance.

In practice …

Effective measurement and reporting of process performance is fundamental to process-based management. We must report all outcomes (the good, the bad, and the ugly) fairly and transparently. Deliberate or unintentional distortion of the performance data will leave us with just the illusion of management.

References

[1] No, of course he wasn't — but he could have been. OK, he likely knew nothing about performance reporting, but he did know a lot about (mis)communication.

# # #

Standard citation for this article:


citations icon
Roger Tregear, "Performance Reports (and Other Lies)" Business Rules Journal, Vol. 22, No. 1, (Jan. 2021)
URL: http://www.brcommunity.com/a2021/c056.html

About our Contributor:


Roger   Tregear
Roger Tregear Principal Advisor, TregearBPM

Roger Tregear is Principal Advisor at TregearBPM (www.tregearbpm.com) with 30 years of BPM education and consulting assignments in Australia (where he lives) and 14 other countries including Bahrain, Belgium, Jordan, Namibia, Nigeria, Netherlands, Saudi Arabia, South Africa, South Korea, Switzerland, New Zealand, United Arab Emirates, UK, and USA. He spends his working life talking, thinking, and writing about the analysis, improvement, innovation, and management of business processes.

Roger has authored or co-authored several books: Practical Process (2013), Establishing the Office of Business Process Management (2011), chapter Business Process Standardization in The International Handbook on BPM (2010, 2015), Questioning BPM? (2016), Reimagining Management (2017), Process Precepts (2017), and Process Provocations (2020). His video series, Process Insights, is on his YouTube channel https://www.youtube.com/TregearBPM.

Read All Articles by Roger Tregear
Subscribe to the eBRJ Newsletter
CONTRIBUTOR ARCHIVES
Performance — The Lifeblood of Process Management
Activating Process-Based Management
The Big Picture
Process Owners — the Daze of our Lives
BPM by Any Other Name
In The Spotlight
 Silvie  Spreeuwenberg
 Jim  Sinur

Online Interactive Training Series

In response to a great many requests, Business Rule Solutions now offers at-a-distance learning options. No travel, no backlogs, no hassles. Same great instructors, but with schedules, content and pricing designed to meet the special needs of busy professionals.