Visualising content and context using issue maps – an example based on a discussion of Cox’s risk matrix theorem

Introduction

Some time ago I wrote a post on a paper by Tony Cox which describes some flaws in risk matrices (as they are commonly used) and proposes an axiomatic approach to address some of the problems.  In a recent comment on that post, Tony Waisanen suggested that someone take up the challenge to map the content of the  post and the ensuing discussion using issue mapping.   Hence my motivation to write the present post.

My main aims  in this post are to:

1. Create an issue map visualising the content of my post on  Cox’s paper.
2. Incorporate points raised in the comments into the map,  and show how they relate to Cox’s arguments.

A quick word about the notation and software before proceeding. I’ll use the IBIS (Issue-based information system) notation to map the argument.  Those unfamiliar with IBIS will find a   quick introduction here.  The mapping is done using Compendium, an open source issue mapping tool (that can do other things too).   I’ll provide a commentary as I build the map, because the detail behind the map cannot be seen in the screenshot

First map: the flaws in risk matrices and how to fix them

Cox ask’s the question: “What’s wrong with risk matrices?” – this is, in fact, the title of the paper in which he describes his theorem. The question is therefore an excellent starting point for our map.

As an answer to the question, Cox lists the following points as problems/flaws in risk matrices:

1. Poor resolution: risk matrices use qualitative categories (typically denoted by colour – red, green, yellow). Risks within a category cannot be distiguished.
2. Incorrect ranking of risks: In some cases, risks can end up in the wrong qualitative category – i.e. a quantitatively higher risk can be mistakenly categorised as a low risk and vice versa. In the worst case, this can lead to suboptimal resource allocation – i.e. a lower risk being given a higher priority.
3. Subjective inputs: Often, the criteria used to rank risks are based on subjective inputs. Such subjective inputs are prone to cognitive bias. This leads to inaccurate and unreliable risk rankings.

See my posts limitations of scoring methods in risk analysis and cognitive biases as project meta-risks for more on the above points.

The map with the root question, problems (ideas or responses, in IBIS terminology)  and their consequences is shown in Figure 1. Note that I’ve put numbers (1), (2) etc. against the points so that I can refer to them by number in other nodes.

Figure 1: What's wrong with risk matrices?

The next question suggests itself:   we’ve asked “What’s wrong with risk matrices?” so an obvious follow-up question is, “What can be done to fix risk matrices?”  There are a few approaches available to address the problems. These are dicussed in my post and the discussion following it. The approaches can be summarised as follows:

1. Statistical approach:  This involves obtaining the correct statistical distributions for probability of the risk occuring and the impact of the risk. This is generally hard to do because of the lack of data. However, once this is done, it obviates the need for risk matrices. Furthermore, it warns us about situations in which risk matrices may mislead.   In Cox’s words, “One (approach) is to consider applications in which there are sufficient data to draw some inferences about the statistical distribution of (Probability, Consequence) pairs. If data are sufficiently plentiful, then statistical and artificial intelligence tools … can potentially be applied to help design risk matrices that give efficient or optimal (according to various criteria) discrete approximations to the quantitative distribution of risks. In such data-rich settings, it might be possible to use risk matrices when they are useful (e.g., if probability and consequence are strongly positively correlated) and to avoid them when they are not (e.g., if probability and consequence are strongly negatively correlated).”  This is, in principle,  the best approach.
2. Qualitative approach: This approach was discussed by Glen Alleman in this comment. It essentially involves characterising impact using qualitative information –  i.e. narrative descriptions of impact. To quote from Glen’s comment, “...the numeric value of impacts are replaced by narrative descriptions of the actual operational impacts from the occurrence of the risk. These narratives are developed through analysis of the system…the quantitative risk as a product is abandoned in place of a classification of response to a predefined consequence.” This approach side steps a couple of the issues with risk matrices. Further, many risk aware organisations have used this method with great success (Glen mentions that NASA and the Department of Defense use such an approach to analyse risks on spaceflight/aviation projects)
3. Axiomatic approach: This is the approach that Tony Cox discusses in his paper. It has the advantage of being simple – it assumes that the risk function (defined as probability x impact, for example) is continuous whilst also ensuring consistency to the extent possible (i.e. ensuring a correct quantitative ranking of risks). The downside, as Glen emphasises in his comments, is that risk functions are actually discrete, as discussed in (1) above.  Cox’s arguments hinge on the continuity of the risk function, so they do not apply to the discrete case.

The map with these approaches added in is depicted in Figure 2. Note that I’ve added Cox’s theorem in as a map node, indicating that a detailed discussion of the theorem is presented in a separate map.

Figure 2: ...and what can be done to fix them.

Note also, that I have added an idea node representing how the issue regarding subjective inputs can be addressed. I will not pursue this point further in the present post as it did not come up in the discussion. That said, I have discussed this point in some detail in an article on cognitive bias in project risk management.

Second map: Cox’s risk matrix theorem

Since the entire discussion is based on Cox’s arguments, it is worth looking into his paper in some detail – in particular, at the axioms  and the theorem itself. It is convenient to hive this material off into a separate map, but one connected to the original map  (see the map node representing the theorem in Figure 2 above).

The root question  of the new map would be,  “What is the basis of Cox’s theorem?” Answer: the theorem is based on the axioms and other (tacit) assumptions.

Now, my earlier post on Cox’s theorem contains a very detailed treatment of the axioms, so I’ll offer only a one-line explanation for each here. The axioms are:

1. Weak consistency – which states that all risks in the highest category (red) must represent quantitatively higher risks than those in the lowest category (green).
2. Consistent colouring – As far as possible, risks with the same quantitative value must have the same colour.
3. Between-ness – small changes in probability or impact (i.e. the risk function) should not cause a risk to move from the highest (red) to lowest (green) or vice versa.

The axioms are intuitively appealing – they express a basic  consistency that one would expect risk matrices to satisfy.  The secondary map, with the three axioms shown is depicted in Figure 3.

Figure 3: Cox's Axioms

Cox’s theorem, which  essentially follows from these axioms, can be stated as follows: In a risk matrix that satisfies the three axioms, all cells in the bottom row and left-most column must be green and all cells in the second from bottom row and second from left column must be non-red.

The theorem has two corollaries:

1. 2×2 matrices cannot satisfy the theorem.
2. 3×3 and 4×4 matrices which satisfy the theorem have a unique colouring scheme.

These are rather surprising conclusions, arrived at from some very intuitive axioms. The secondary map, with the theorem and corollaries added in is shown in Fig. 4.

Figure 4: The theorem, corollaries and axioms

That completes the map of the theorem. However, in this comment Glen Alleman pointed out that the assumption of a continuous function to describe risk  (such as risk = probability x impact, where both quantities on the right hand side are continuous functions) is questionable. He also makes the point the probability is specified by a distribution, and numerical values that come out of distributions  cannot be combined via arithmetic operations. The reason that folks make the simplifying assumptions  (of continutity and ignoring the probabilistic nature of the variables)   is that it is intuitive and easy to work with. As I mentioned in one of my responses to the comments, one can choose to define risk this way although it isn’t logically sound.  Cox’s theorem essentially specifies consistency conditions that need to be satisfied when such ad-hoc approaches are used. The map with this discussion included is shown in Figure 5 (click anywhere on figure to view a full-sized image)

Figure 5: The theorem in context (Axioms and assumptions made explicit)

That completes the mapping exercise: Figures 2 and 5 represent a fairly complete map of the post and the discussion around it.

Caveats and conclusions

At the risk of belaboring the obvious, the maps represent my interpretation of Cox’s work and my interpretation of others’ comments on my post on Cox’s work. Further, the discussion on which the maps are based is far from comprehensive because it did not cover other limitations of risk matrices. Please see my post on limitations of scoring methods in risk analysis for a detailed discussion of these.

Before closing, it is worth looking at the Figures 2 and 5 from a broader perspective: the figures make clear the context of the discussion in a way that is simply not possible through words. As an example, Figure 2 lays bare the context of Cox’s theorem –  it emphasises, for example, that Cox’s approach  isn’t the only  method to fix what’s  wrong with risk matrices. Further, Figure 5 distinguishes between explicitly declared and tacit assumptions.  Examples of the former are the three axioms and that of the latter is the assumption of continuity.

In this post  I’ve  summarised the content and context of Cox’s risk matrix theorem via issue mapping. The maps  provide an “at a glance” summary of the theorem alongwith supporting assumptions and axioms.  Further,  the maps also incorporate key elements of readers’  reaction regarding the post. I hope this example clarifies the content and context of my earlier post on Cox’s risk matrix theorem, whilst also serving as a demonstration of the utility of the IBIS notation in mapping complex arguments.

Acknowledgements:

Thanks go out to Tony Waisanen for suggesting that the post and comments be issue mapped, and to Glen Alleman, Robert Higgins and Prakash Vaidhyanathan for their contributions to the discussion.