The Misery of Matrices

Key Takeaways

• Limitations of Heat Maps: Heat maps often fail to provide a true picture of risk by focusing on surface-level assessments, leaving out crucial data on risk frequency and impact.
• Quantitative Risk Assessment: A more effective method for ERM, quantitative risk analysis offers deeper insights into risk behavior, helping organizations prioritize and manage risks more accurately.
• Risk A vs. Risk B: A clear comparison between two risks shows that Risk A, despite having a lower likelihood, can cause significantly higher losses, illustrating the importance of factoring in both likelihood and impact in risk assessments.
• Long-Tailed Risks: Some risks, like Risk B, may have a long tail where the worst-case scenario can result in catastrophic loss, a nuance that heat maps fail to capture effectively.
• Need for Analytical Tools: Moving beyond heat maps to more sophisticated, data-driven tools enables a more informed and strategic approach to risk management.

Deep Dive

In Graeme Keith's latest article, he explores the limitations of heat maps in risk assessment and why quantitative risk analysis is essential for effective Enterprise Risk Management (ERM). By using two hypothetical risk scenarios, Keith highlights the significant gaps in traditional risk matrices and advocates for a more rational, analytical approach to risk prioritization and aggregation. Through his analysis, he emphasizes the need for a deeper understanding of risk impacts, beyond surface-level assessments.

Why Quantitative Risk Assessment Outperforms Heat Maps in Effective Risk Management

Look at the picture above. The assessment of likelihood is for all occurrences of the risk. The impact is an assessment of a credible worst-case for economic loss. These are real assessments using a real assessment workflow with a solid definition of credible worst-case.

Which risk, A or B, is losing you the most? Can you tell? Which risk, when it occurs, will, on average, lose you the most? Out of, say, 10 occurrences of each risk, which risk, statistically, will have the worst economic impact?

That B isn't the answer to any of these questions is why you need quantitative risk assessment in your ERM program. (To be clear, using quantitative risk assessment in your ERM program isn't the same as introducing decision analytics through risk modeling. You should do that too, but this is much easier.)

It turns out that on average, Risk A will lose you close to twenty times what Risk B will lose you on an annual basis. When Risk A occurs, it will lose you, on average, more than twice what Risk B loses you. Out of 10 occurrences of each risk, statistically, the two worst economic impacts will be similar, but Risk A's is slightly worse.

Part of the confusion arises from the wide interval in the rate of occurrence. Risk A has a probability of occurrence slightly under 75%, Risk B slightly over 25%, but they still have the same likelihood assessment.

The other problem is that Risk B has a long tail. Like the girl with a curl, when it's good, it's very good indeed, but when it's bad, it's horrid. This horrid is what is captured on the heat map. The rest of the information is thrown out.

Heat Maps Can’t Even Be Used for Triage

Now look at the picture here, which shows the top 10 risks from two business units.

Article Content
The risks of which of these two departments incur the greatest losses, A or B?

B, right? Department A has 3 greens, 7 yellows, and no reds, while Department B has no greens, 7 yellows, and 3 reds. Also, B’s yellows are nearly all much closer to the red than A’s. In fact, the expected annual losses from A are almost 250% of those from B.

Part of the story here is that the heat map is showing an assessment of a credible worst-case scenario. We might have long-tailed risks, which score badly on impact, even though the impacts are relatively benign most of the time and the expected loss is low. (The heat map can’t show both.)

But then we would expect a bad year for B to be worse than a bad year for A, right? In fact, the average of the worst 10% of annual losses for A is three times that of B.

Colors Have No Calculus
Counting doesn’t cut it. That the answer to neither of these questions is, in fact, B is why quantitative assessment is essential in an ERM program.

There is no upper limit to the rate of occurrence of risks assessed to the top row in the matrix. A has a couple of high-rate, medium-impact risks, but you can’t see that on a matrix. Nor is there an upper limit on the impact of risks assessed to the rightmost column of the matrix. A has a couple of low-rate, high-impact risks, but you can’t see that either. The heat map hides the most relevant information about your risk profile.

There is a huge body of literature making these and other arguments for the complete inadequacy of heat maps for their allotted task. I have taken these examples as they very clearly show how heat maps fail to deliver on the two most important of these tasks on which defenders often claim validity: prioritization and aggregation.

The GRC Report is your premier destination for the latest in governance, risk, and compliance news. As your reliable source for comprehensive coverage, we ensure you stay informed and ready to navigate the dynamic landscape of GRC. Beyond being a news source, the GRC Report represents a thriving community of professionals who, like you, are dedicated to GRC excellence. Explore our insightful articles and breaking news, and actively participate in the conversation to enhance your GRC journey.