Data visualisation is sometimes (often?) approached as a bag of tricks to be learnt individually, with no little or no reference to any underlying principles. Reading Hadley Wickham’s paper on the grammar of graphics was an epiphany; it showed me how different types of graphics can be constructed in a consistent way using common elements. Among other things, the grammar makes visualisation a logical affair rather than a set of tricks. This post is a brief – and hopefully logical – introduction to visualisation using ggplot2, Wickham’s implementation of a grammar of graphics.

In keeping with the practical bent of this series we’ll focus on worked examples, illustrating elements of the grammar as we go along. We’ll first briefly describe the elements of the grammar and then show how these are used to build different types of visualisations.

Most visualisations are constructed from common elements that are pieced together in prescribed ways. The elements can be grouped into the following categories:

**Data**– this is obvious, without data there is no story to tell and definitely no plot!**Mappings**– these are correspondences between data and display elements such as spatial location, shape or colour. Mappings are referred to as*aesthetics*in Wickham’s grammar.**Scales**– these are transformations (conversions) of data values to numbers that can be displayed on-screen. There should be one scale per mapping. ggplot typically does the scaling transparently, without users having to worry about it. One situation in which you might need to mess with default scales is when you want to zoom in on a particular range of values. We’ll see an example or two of this later in this article.**Geometric objects**– these specify the geometry of the visualisation. For example, in ggplot2 a scatter plot is specified via a*point*geometry whereas a fitting curve is represented by a*smooth*geometry. ggplot2 has a range of geometries available of which we will illustrate just a few.**Coordinate system –**this specifies the system used to position data points on the graphic. Examples of coordinate systems are Cartesian and polar. We’ll deal with Cartesian systems in this tutorial. See this post for a nice illustration of how one can use polar plots creatively.**Facets**– a facet specifies how data can be split over multiple plots to improve clarity. We’ll look at this briefly towards the end of this article.

The basic idea of a *layered* grammar of graphics is that each of these elements can be combined – literally added layer by layer – to achieve a desired visual result. Exactly how this is done will become clear as we work through some examples. So without further ado, let’s get to it.

In what follows we’ll use the NSW Government Schools dataset, made available via the state government’s open data initiative. The data is in csv format. If you cannot access the original dataset from the aforementioned link, you can download an Excel file with the data **here** (remember to save it as a csv before running the code!).

The first task – assuming that you have a working R/RStudio environment – is to load the data into R. To keep things simple we’ll delete a number of columns (as shown in the code) and keep only rows that are complete, i.e. those that have no missing values. Here’s the code:

#set working directory if needed (modify path as needed)

#setwd(“C:/Users/Kailash/Documents/ggplot”)

#load required library

library(ggplot2)

#load dataset (ensure datafile is in directory!)

nsw_schools <- read.csv(“NSW-Public-Schools-Master-Dataset-07032017.csv”)

#build expression for columns to delete

colnames_for_deletion <- paste0(“AgeID|”,”street|”,”indigenous_pct|”,

“lbote_pct|”,”opportunity_class|”,”school_specialty_type|”,

“school_subtype|”,”support_classes|”,”preschool_ind|”,

“distance_education|”,”intensive_english_centre|”,”phone|”,

“school_email|”,”fax|”,”late_opening_school|”,

“date_1st_teacher|”,”lga|”,”electorate|”,”fed_electorate|”,

“operational_directorate|”,”principal_network|”,

“facs_district|”,”local_health_district|”,”date_extracted”)

#get indexes of cols for deletion

cols_for_deletion <- grep(colnames_for_deletion,colnames(nsw_schools))

#delete them

nsw_schools <- nsw_schools[,-cols_for_deletion]

#structure and number of rows

str(nsw_schools)

nrow(nsw_schools)

#remove rows with NAs

nsw_schools <- nsw_schools[complete.cases(nsw_schools),]

#rowcount

nrow(nsw_schools)

#convert student number to numeric datatype.

#Need to convert factor to character first…

#…alternately, load data with StringsAsFactors set to FALSE

nsw_schools$student_number <- as.numeric(as.character(nsw_schools$student_number))

#a couple of character strings have been coerced to NA. Remove these

nsw_schools <- nsw_schools[complete.cases(nsw_schools),]

A note regarding the last line of code above, a couple of schools have “np” entered for the* student_number* variable. These are coerced to NA in the numeric conversion. The last line removes these two schools from the dataset.

Apart from *student numbers* and location data, we have retained *level of schooling* (primary, secondary etc.) and* ICSEA ranking*. The location information includes attributes such as *suburb*, *postcode*, *region*, *remoteness* as well as *latitude* and *longitude*. We’ll use only *remoteness* in this post.

The first thing that caught my eye in the data was was the ICSEA ranking. Before going any further, I should mention that the Australian Curriculum Assessment and Reporting Authority (the organisation responsible for developing the ICSEA system) emphasises that the score is *not* a school ranking, but a measure of *socio-educational advantage* of the student population in a school. Among other things, this is related to family background and geographic location. The average ICSEA score is set at an average of 1000, which can be used as a reference level.

I thought a natural first step would be to see how ICSEA varies as a function of the other variables in the dataset such as *student number*s and *location* (*remoteness*, for example). To begin with, let’s plot ICSEA rank as a function of student number. As it is our first plot, let’s take it step by step to understand how the layered grammar works. Here we go:

#specify data layer

p <- ggplot(data=nsw_schools)

#display plot

p

This displays a blank plot because we have not specified a *mapping* and *geometry *to go with the data. To get a plot we need to specify both. Let’s start with a scatterplot, which is specified via a point geometry. Within the geometry function, variables are mapped to visual properties of the using aesthetic mappings. Here’s the code:

#specify a point geometry (geom_point)

p <- p + geom_point(mapping = aes(x=student_number,y=ICSEA_Value))

#…lo and behold our first plot

p

The resulting plot is shown in Figure 1.

At first sight there are two points that stand out: 1) there are fewer number of large schools, which we’ll look into in more detail later and 2) larger schools seem to have a higher ICSEA score on average. To dig a little deeper into the latter, let’s add a linear trend line. We do that by adding another layer (geometry) to the scatterplot like so:

#add a trendline using geom_smooth

p <- p + geom_smooth(mapping= aes(x=student_number,y=ICSEA_Value),method=”lm”)

#scatter plot with trendline

p

The result is shown in Figure 2.

The *lm* method does a linear regression on the data. The shaded area around the line is the 95% confidence level of the regression line (i.e that it is 95% certain that the true regression line lies in the shaded region). Note that *geom_smooth * provides a range of smoothing functions including generalised linear and local regression (loess) models.

You may have noted that we’ve specified the aesthetic mappings in both *geom_point* and *geom_smooth*. To avoid this duplication, we can simply specify the mapping, once in the top level ggplot call (the first layer) like so:

#rewrite the above, specifying the mapping in the ggplot call instead of geom

p <- ggplot(data=nsw_schools,mapping= aes(x=student_number,y=ICSEA_Value)) +

geom_point()+

geom_smooth(method=”lm”)

geom_point()+

geom_smooth(method=”lm”)

#display plot, same as Fig 2

p

From Figure 2, one can see a clear positive correlation between student numbers and ICSEA scores, let’s zoom in around the average value (1000) to see this more clearly…

#set display to 900 < y < 1100

p <- p + coord_cartesian(ylim =c(900,1100))

#display plot

p

The *coord_cartesian* function is used to zoom the plot to without changing any other settings. The result is shown in Figure 3.

To make things clearer, let’s add a reference line at the average:

#add horizontal reference line at the avg ICSEA score

p <- p + geom_hline(yintercept=1000)

#display plot

p

The result, shown in Figure 4, indicates quite clearly that larger schools tend to have higher ICSEA scores. That said, there is a twist in the tale which we’ll come to a bit later.

As a side note, you would use *geom_vline* to zoom in on a specific range of x values and *geom_abline* to add a reference line with a specified slope and intercept. See this article on ggplot reference lines for more.

OK, now that we have seen how *ICSEA scores* vary with *student number*s let’s switch tack and incorporate another variable in the mix. An obvious one is *remoteness*. Let’s do a scatterplot as in Figure 1, but now colouring each point according to its remoteness value. This is done using the colour aesthetic as shown below:

#Map aecg_remoteness to colour aesthetic

p <- ggplot(data=nsw_schools, aes(x=student_number,y=ICSEA_Value, colour=ASGS_remoteness)) +

geom_point()

geom_point()

#display plot

p

The resulting plot is shown in Figure 5.

Aha, a couple of things become apparent. First up, large schools tend to be in metro areas, which makes good sense. Secondly, it appears that metro area schools have a distinct socio-educational advantage over regional and remote area schools. Let’s add trendlines by remoteness category as well to confirm that this is indeed so:

#add reference line at avg + trendlines for each remoteness category

p <- p + geom_hline(yintercept=1000) + geom_smooth(method=”lm”)

#display plot

p

The plot, which is shown in Figure 6, indicates clearly that ICSEA scores decrease on the average as we move away from metro areas.

Moreover, larger schools metropolitan areas tend to have higher than average scores (above 1000), regional areas tend to have lower than average scores overall, with remote areas being markedly more disadvantaged than both metro and regional areas. This is no surprise, but the visualisations show just how stark the differences are.

It is also interesting that, in contrast to metro and (to some extent) regional areas, there *negative correlation* between student numbers and scores for remote schools. One can also use local regression to get a better picture of how ICSEA varies with student numbers and remoteness. To do this, we simply use the *loess* method instead of *lm:*

#redo plot using loess smoothing instead of lm

p <- ggplot(data=nsw_schools, aes(x=student_number,y=ICSEA_Value, colour=ASGS_remoteness)) +

geom_point() + geom_hline(yintercept=1000) + geom_smooth(method=”loess”)

geom_point() + geom_hline(yintercept=1000) + geom_smooth(method=”loess”)

#display plot

p

The result, shown in Figure 7, has a number of interesting features that would have been worth pursuing further were we analysing this dataset in a real life project. For example, why do small schools tend to have lower than benchmark scores?

From even a casual look at figures 6 and 7, it is clear that the confidence intervals for remote areas are huge. This suggests that the number of datapoints for these regions are a) small and b) very scattered. Let’s quantify the number by getting counts using the *table* function (I know, we could plot this too…and we will do so a little later). We’ll also transpose the results using *data.frame* to make them more readable:

#get school counts per remoteness category

data.frame(table(nsw_schools$ASGS_remoteness))

Var1 Freq

1 0

2 Inner Regional Australia 561

3 Major Cities of Australia 1077

4 Outer Regional Australia 337

5 Remote Australia 33

6 Very Remote Australia 14

The number of datapoints for remote regions is *much* less than those for metro and regional areas. Let’s repeat the loess plot with only the two remote regions. Here’s the code:

#create vector containing desired categories

remote_regions <- c(‘Remote Australia’,’Very Remote Australia’)

#redo loess plot with only remote regions included

p <- ggplot(data=nsw_schools[nsw_schools$ASGS_remoteness %in% remote_regions,], aes(x=student_number,y=ICSEA_Value, colour=ASGS_remoteness)) +

geom_point() + geom_hline(yintercept=1000) + geom_smooth(method=”loess”)

geom_point() + geom_hline(yintercept=1000) + geom_smooth(method=”loess”)

#display plot

p

The plot, shown in Figure 8, shows that there is indeed a huge variation in the (small number) of datapoints, and the confidence intervals reflect that. An interesting feature is that some small remote schools have above average scores. If we were doing a project on this data, this would be a feature worth pursuing further as it would likely be of interest to education policymakers.

Note that there is a difference in the x axis scale between Figures 7 and 8 – the former goes from 0 to 2000 whereas the latter goes up to 400 only. So for a fair comparison, between remote and other areas, you may want to re-plot Figure 7, zooming in on student numbers between 0 and 400 (or even less). This will also enable you to see the complicated dependence of scores on student numbers more clearly across all regions.

We’ll leave the scores vs student numbers story there and move on to another geometry – the well-loved bar chart. The first one is a visualisation of the remoteness category count that we did earlier. The relevant geometry function is *geom_bar*, and the code is as easy as:

#frequency plot

p <- ggplot(data=nsw_schools, aes(x=ASGS_remoteness)) + geom_bar()

#display plot

p

The plot is shown in Figure 9.

The category labels on the x axis are too long and look messy. This can be fixed by tilting them to a 45 degree angle so that they don’t run into each other as they most likely did when you ran the code on your computer. This is done by modifying the *axis.text* element of the plot theme. Additionally, it would be nice to get counts on top of each category bar. The way to do that is using another geometry function, geom_text. Here’s the code incorporating the two modifications:

#frequency plot

p <- p + geom_text(stat=’count’,aes(label= ..count..),vjust=-1)+

theme(axis.text.x=element_text(angle=45, hjust=1))

theme(axis.text.x=element_text(angle=45, hjust=1))

#display plot

p

The result is shown in Figure 10.

Some things to note: : *stat=count* tells ggplot to compute counts by category and the aesthetic *label = ..count.. *tells ggplot to access the internal variable that stores those counts. The the vertical justification setting, *vjust=-1*, tells ggplot to display the counts on top of the bars. Play around with different values of *vjust* to see how it works. The code to adjust label angles is self explanatory.

It would be nice to reorder the bars by frequency. This is easily done via *fct_infreq* function in the *forcats* package like so:

#use factor tools

library(forcats)

#descending

p <- ggplot(data=nsw_schools) +

geom_bar(mapping = aes(x=fct_infreq(ASGS_remoteness)))+

theme(axis.text.x=element_text(angle=45, hjust=1))

geom_bar(mapping = aes(x=fct_infreq(ASGS_remoteness)))+

theme(axis.text.x=element_text(angle=45, hjust=1))

#display plot

p

The result is shown in Figure 11.

To reverse the order, invoke *fct_rev,* which reverses the sort order:

#reverse sort order to ascending

p <- ggplot(data=nsw_schools) +

geom_bar(mapping = aes(x=fct_rev(fct_infreq(ASGS_remoteness))))+

theme(axis.text.x=element_text(angle=45, hjust=1))

geom_bar(mapping = aes(x=fct_rev(fct_infreq(ASGS_remoteness))))+

theme(axis.text.x=element_text(angle=45, hjust=1))

#display plot

p

The resulting plot is shown in Figure 12.

If this is all too grey for us, we can always add some colour. This is done using the *fill* aesthetic as follows:

#add colour using the fill aesthetic

p <- ggplot(data=nsw_schools) +

geom_bar(mapping = aes(x=ASGS_remoteness, fill=ASGS_remoteness))+

theme(axis.text.x=element_text(angle=45, hjust=1))

geom_bar(mapping = aes(x=ASGS_remoteness, fill=ASGS_remoteness))+

theme(axis.text.x=element_text(angle=45, hjust=1))

#display plot

p

The resulting plot is shown in Figure 13.

Note that, in the above, that we have mapped fill and x to the *same* variable, *remoteness *which makes the legend superfluous. I will leave it to you to figure out how to suppress the legend – Google is your friend.

We could also map fill to another variable, which effectively adds another dimension to the plot. Here’s how:

#map fill to another variable

p <- ggplot(data=nsw_schools) +

geom_bar(mapping = aes(x=ASGS_remoteness, fill=level_of_schooling))+

theme(axis.text.x=element_text(angle=45, hjust=1))

geom_bar(mapping = aes(x=ASGS_remoteness, fill=level_of_schooling))+

theme(axis.text.x=element_text(angle=45, hjust=1))

#display plot

p

The plot is shown in Figure 14. The new variable, level of schooling, is displayed via proportionate coloured segments stacked up in each bar. The default stacking is one on top of the other.

Alternately, one can stack them up side by side by setting the *position *argument to *dodge* as follows:

#stack side by side

p <- ggplot(data=nsw_schools) +

geom_bar(mapping = aes(x=ASGS_remoteness,fill=level_of_schooling),position =”dodge”)+

theme(axis.text.x=element_text(angle=45, hjust=1))

geom_bar(mapping = aes(x=ASGS_remoteness,fill=level_of_schooling),position =”dodge”)+

theme(axis.text.x=element_text(angle=45, hjust=1))

#display plot

p

The plot is shown in Figure 15.

Finally, setting the *position* argument to *fill* normalises the bar heights and gives us the proportions of* level of schooling* for each *remoteness* category. That sentence will make more sense when you see Figure 16 below. Here’s the code, followed by the figure:

#proportion plot

p <- ggplot(data=nsw_schools) +

geom_bar(mapping = aes(x=ASGS_remoteness,fill=level_of_schooling),position = “fill”)+

theme(axis.text.x=element_text(angle=45, hjust=1))

geom_bar(mapping = aes(x=ASGS_remoteness,fill=level_of_schooling),position = “fill”)+

theme(axis.text.x=element_text(angle=45, hjust=1))

#display plot

p

Obviously, we lose frequency information since the bar heights are normalised.

An interesting feature here is that the proportion of central and community schools increases with remoteness. Unlike primary and secondary schools, central / community schools provide education from Kindergarten through Year 12. As remote areas have smaller numbers of students, it makes sense to consolidate educational resources in institutions that provide schooling at all levels .

Finally, to close the loop so to speak, let’s revisit our very first plot in Figure 1 and try to simplify it in another way. We’ll use faceting to split it out into separate plots, one per remoteness category. First, we’ll organise the subplots horizontally using *facet_grid*:

#faceting – subplots laid out horizontally (faceted variable on right of formula)

p <- ggplot(data=nsw_schools) + geom_point(mapping = aes(x=student_number,y=ICSEA_Value))+

facet_grid(~ASGS_remoteness)

facet_grid(~ASGS_remoteness)

#display plot

p

The plot is shown in Figure 17 in which the different remoteness categories are presented in separate plots (facets) against a common y axis. It shows, the sharp differences between student numbers between remote and other regions.

To get a vertically laid out plot, switch the faceted variable to other side of the formula (left as an exercise for you).

If one has too many categories to fit into a single row, one can wrap the facets using *facet_wrap* like so:

#faceting – wrapping facets in 2 columns

p <- ggplot(data=nsw_schools) +

geom_point(mapping = aes(x=student_number,y=ICSEA_Value))+

facet_wrap(~ASGS_remoteness, ncol= 2)

geom_point(mapping = aes(x=student_number,y=ICSEA_Value))+

facet_wrap(~ASGS_remoteness, ncol= 2)

#display plot

p

The resulting plot is shown in Figure 18.

One can specify the number of rows instead of columns. I won’t illustrate that as the change in syntax is quite obvious.

…and I think that’s a good place to stop.

Data visualisation has a reputation of being a dark art, masterable only by the visually gifted. This may have been partially true some years ago, but in this day and age it definitely isn’t. Versatile packages such as ggplot, that use a consistent syntax have made the art much more accessible to visually ungifted folks like myself. In this post I have attempted to provide a brief and (hopefully) logical introduction to ggplot. In closing I note that although some of the illustrative examples violate the principles of good data visualisation, I hope this article will serve its primary purpose which is pedagogic rather than artistic.

**Further reading**:

Where to go for more? Two of the best known references are Hadley Wickham’s books:

I highly recommend his R for Data Science , available online here. Apart from providing a good overview of ggplot, it is an excellent introduction to R for data scientists. If you haven’t read it, do yourself a favour and buy it now.

People tell me his ggplot book is an excellent book for those wanting to learn the ins and outs of ggplot . I have not read it myself, but if his other book is anything to go by, it should be pretty damn good.

Filed under: Data Analytics, Data Science, Data Visualization, R ]]>

The metaphor of *time as a river* resonates well with our subjective experiences of time. Everyday phrases that evoke this metaphor include *the flow of time *and *time going by*, or the somewhat more poetic *currents of time*. As Heraclitus said, *no [person] can step into the same river twice* – and so it is that a particular instant in time …like right now…is ephemeral, receding into the past as we become aware of it.

On the other hand, organisations have to capture and quantify time because things have to get done within fixed periods, the financial year being a common example. Hence, key organisational activities such as projects, strategies and budgets are invariably time-bound affairs. This can be problematic because there is a mismatch between the ways in which organisations view time and individuals experience it.

The idea that time is an objective entity is most clearly embodied in the notion of a *timeline*: a graphical representation of a time period, punctuated by events. The best known of these is perhaps the ubiquitous Gantt Chart, loved (and perhaps equally, reviled) by managers the world over.

Timelines are interesting because, as Elaine Yakura states in this paper, “*they seem to render time, the ultimate abstraction, visible and concrete*.” As a result, they can serve as *boundary objects* that make it possible to negotiate and communicate what is to be accomplished in the specified time period. They make this possible because they tell a story with a clear beginning, middle and end, a narrative of what is to come and when.

For the reasons mentioned in the previous paragraph, timelines are often used to manage time-bound organisational initiatives. Through their use in scheduling and allocation, timelines serve to *objectify* time in such a way that it becomes a *resource* that can be *measured* and *rationed*, much like other resources such as money, labour etc.

At our workplaces we are governed by many overlapping timelines – workdays, budgeting cycles and project schedules being examples. From an individual perspective, each of these timelines are different representations of how one’s time is to be utilised, when an activity should be started and when it must be finished. Moreover, since we are generally committed to multiple timelines, we often find ourselves switching between them. They serve to remind us what we should be doing and when.

But there’s more: one of the key aims of developing a timeline is to enable all stakeholders to have a shared understanding of time as it pertains to the initiative. In this view, a timeline is a consensus representation of how a particular aspect of the future will unfold. Timelines thus serve as *coordinating mechanisms*.

In terms of the metaphor, a timeline is akin to a map of the river of time. Along the map we can measure out and apportion it; we can even agree about way-stops at various points in time. However, we should always be aware that it remains a *representation* of time, for although we might treat a timeline as real, the fact is *no one actually experiences time as it is depicted in a timeline*. Mistaking one for the other is akin to confusing the map with the territory.

This may sound a little strange so I’ll try to clarify. I’ll start with the observation that *we experience time through events and processes* – for example the successive chimes of a clock, the movement of the second hand of a watch (or the oscillations of a crystal), the passing of seasons or even the greying of one’s hair. Moreover, since these events and processes can be objectively agreed on by different observers, they can also be marked out on a timeline. Yet *the *actual experience of *living* these events is unique to each individual.

As we have seen, organisations treat time as an objective commodity that can be represented, allocated and used much like any tangible resource. On the other hand our experience of time is intensely personal. For example, I’m sitting in a cafe as I write these lines. My perception of the flow of time depends rather crucially on my level of engagement in writing: slow when I’m struggling for words but zipping by when I’m deeply involved. This is familiar to us all: when we are deeply engaged in an activity, we lose all sense of time but when our involvement is superficial we are acutely aware of the clock.

This is true at work as well. When I’m engaged in any kind of activity at work, be it a group activity such as a meeting, or even an individual one such as developing a business case, my perception of time has little to do with the actual passage of seconds, minutes and hours on a clock. Sure, there are things that I will do habitually at a particular time – going to lunch, for example – but my perception of how fast the day goes is governed not by the clock but by the degree of engagement with my work.

I can only speak for myself, but I suspect that this is the case with most people. Though our work lives are supposedly governed by “objective” timelines, the way we *actually* live out our workdays depends on a host of things that have more to do with our inner lives than visible outer ones. Specifically, they depend on things such as feelings, emotions, moods and motivations.

OK, so you may be wondering where I’m going with this. Surely, my subjective perception of my workday should not matter as long as I do what I’m required to do and meet my deadlines, right?

As a matter of fact, I think the answer to the above question is a qualified, “No”. The quality of the work we do depends on our level of commitment and engagement. Moreover, since a person’s perception of the passage of time depends rather sensitively on the degree of their involvement in a task, their subjective sense of time is a good indicator of their engagement in work.

In his book, Finding Flow, Mihalyi Csikszentmihalyi describes such engagement as an optimal experience in which a person is completely focused on the task at hand. Most people would have experienced flow when engaged in activities that they really enjoy. As Anthony Reading states in his book, Hope and Despair: How Perceptions of the Future Shape Human Behaviour, *“…most of what troubles us resides in our concerns about the past and our apprehensions about the future*.” People in flow are entirely focused on the present and are thus (temporarily) free from troubling thoughts. As Csikszentmihalyi puts it, for such people, “*the sense of time is distorted; hours seem to pass by in minutes*.”

All this may seem far removed from organisational concerns, but it is easy to see that it isn’t: a Google search on the phrase “increase employee engagement” will throw up many articles along the lines of “N ways to increase employee engagement.” The sense in which the term is used in these articles is essentially the same as the one Csikszentmihalyi talks about: deep involvement in work.

So, the advice of management gurus and business school professors notwithstanding, the issue is less about employee engagement or motivation than about *creating conditions that are conducive to flow*. All that is needed for the latter is a deep understanding how the particular organisation functions, the task at hand and (most importantly) the people who will be doing it. The best managers I’ve worked with have grokked this, and were able to create the right conditions in a seemingly effortless and unobtrusive way. It is a skill that cannot be taught, but can be learnt by observing how such managers do what they do.

Organisations tend to treat their employees’ time as though it were a commodity or resource that can be apportioned and allocated for various tasks. This view of time is epitomised by the timeline as depicted in a Gantt Chart or a resource-loaded project schedule.

In contrast, at an individual level, the perception of time depends rather critically on the level of engagement that a person feels with the task he or she is performing. Ideally organisations would (or ought to!) want their employees to be in that optimal zone of engagement that Csikszentmihalyi calls flow, at least when they are involved in creative work. However, like spontaneity, flow is a state that cannot be achieved by corporate decree; the best an organisation can do is to create the conditions that encourage it.

The organisational focus on timelines ought to be balanced by actions that are aimed at *creating the conditions that are conducive to employee engagement and flow*. It may then be possible for those who work in organisation-land to experience, if only fleetingly, that Blakean state in which eternity is held in an hour.

Filed under: General Management, Management, Organizational Paradoxes, Organizations ]]>

My main aim in this post is to provide a beginner level introduction to logistic regression using R and also introduce LASSO (Least Absolute Shrinkage and Selection Operator), a powerful feature selection technique that is very useful for regression problems. Lasso is essentially a regularization method. If you’re unfamiliar with the term, think of it as a way to reduce overfitting using less complicated functions (and if that means nothing to you, check out my prelude to machine learning). One way to do this is to toss out less important variables, after checking that they aren’t important. As we’ll discuss later, this can be done manually by examining p-values of coefficients and discarding those variables whose coefficients are not significant. However, this can become tedious for classification problems with many independent variables. In such situations, lasso offers a neat way to model the dependent variable while *automagically* *selecting significant variables by* *shrinking the coefficients of unimportant predictors to zero. A*ll this without having to mess around with p-values or obscure information criteria. How good is that?

In linear regression one attempts to model a dependent variable (i.e. the one being predicted) using the best straight line fit to a set of predictor variables. The best fit is usually taken to be one that minimises the root mean square error, which is the sum of square of the differences between the actual and predicted values of the dependent variable. One can think of logistic regression as the equivalent of linear regression for a classification problem. In what follows we’ll look at binary classification – i.e. a situation where the dependent variable takes on one of two possible values (Yes/No, True/False, 0/1 etc.).

First up, you might be wondering why one can’t use linear regression for such problems. The main reason is that classification problems are about determining *class membership* rather than predicting *variable values*, and linear regression is more naturally suited to the latter than the former. One could, in principle, use linear regression for situations where there is a natural ordering of categories like *High*, *Medium* and *Low* for example. However, one then has to map sub-ranges of the predicted values to categories. Moreover, since predicted values are potentially unbounded (in data as yet unseen) there remains a degree of arbitrariness associated with such a mapping.

Logistic regression sidesteps the aforementioned issues by modelling class probabilities instead. Any input to the model yields a number lying between 0 and 1, representing the probability of class membership. One is still left with the problem of determining the threshold probability, i.e. the probability at which the category flips from one to the other. By default this is set to p=0.5, but in reality it should be settled based on how the model will be used. For example, for a marketing model that identifies potentially responsive customers, the threshold for a positive event might be set low (much less than 0.5) because the client does not really care about mailouts going to a non-responsive customer (the negative event). Indeed they may be more than OK with it as there’s always a chance – however small – that a non-responsive customer will actually respond. As an opposing example, the cost of a false positive would be high in a machine learning application that grants access to sensitive information. In this case, one might want to set the threshold probability to a value closer to 1, say 0.9 or even higher. The point is, the setting an appropriate threshold probability is a business issue, not a technical one.

So how does logistic regression work?

For the discussion let’s assume that the outcome (predicted variable) and predictors are denoted by Y and X respectively and the two classes of interest are denoted by + and – respectively. We wish to model the conditional probability that the outcome Y is +, given that the input variables (predictors) are X. The conditional probability is denoted by p(Y=+|X) which we’ll abbreviate as p(X) since we know we are referring to the positive outcome Y=+.

As mentioned earlier, we are after the *probability of class membership* so we must ensure that the hypothesis function (a fancy word for the model) always lies between 0 and 1. The function assumed in logistic regression is:

You can verify that does indeed lie between 0 and 1 as varies from to . Typically, however, the values of that make sense are bounded as shown in the example (stolen from Wikipedia) shown in Figure 1. The figure also illustrates the typical S-shaped curve characteristic of logistic regression.

As an aside, you might be wondering where the name *logistic* comes from. An equivalent way of expressing the above equation is:

The quantity on the left is the logarithm of the odds. So, the model is a linear regression of the *log-odds*, sometimes called* logit*, and hence the name *logistic*.

The problem is to find the values of and that results in a that most accurately classifies all the observed data points – that is, those that belong to the positive class have a probability as close as possible to 1 and those that belong to the negative class have a probability as close as possible to 0. One way to frame this problem is to say that we wish to maximise the product of these probabilities, often referred to as the likelihood:

Where represents the products over i and j, which run over the +ve and –ve classed points respectively. This approach, called maximum likelihood estimation, is quite common in many machine learning settings, especially those involving probabilities.

It should be noted that in practice one works with the *log* likelihood because it is easier to work with mathematically. Moreover, one *minimises* the *negative * log likelihood which, of course, is the same as maximising the log likelihood. The quantity one minimises is thus:

However, these are technical details that I mention only for completeness. As you will see next, they have little bearing on the practical use of logistic regression.

In this example, we’ll use the logistic regression option implemented within the *glm* function that comes with the base R installation. This function fits a class of models collectively known as *generalized linear models*. We’ll apply the function to the *Pima Indian Diabetes* dataset that comes with the mlbench package. The code is quite straightforward – particularly if you’ve read earlier articles in my “gentle introduction” series – so I’ll just list the code below noting that the logistic regression option is invoked by setting *family=”binomial”* in the *glm* function call.

Here we go:

#set working directory if needed (modify path as needed)

#setwd(“C:/Users/Kailash/Documents/logistic”)

#load required library

library(mlbench)

#load Pima Indian Diabetes dataset

data(“PimaIndiansDiabetes”)

#set seed to ensure reproducible results

set.seed(42)

#split into training and test sets

PimaIndiansDiabetes[,”train”] <- ifelse(runif(nrow(PimaIndiansDiabetes))<0.8,1,0)

#separate training and test sets

trainset <- PimaIndiansDiabetes[PimaIndiansDiabetes$train==1,]

testset <- PimaIndiansDiabetes[PimaIndiansDiabetes$train==0,]

#get column index of train flag

trainColNum <- grep(“train”,names(trainset))

#remove train flag column from train and test sets

trainset <- trainset[,-trainColNum]

testset <- testset[,-trainColNum]

#get column index of predicted variable in dataset

typeColNum <- grep(“diabetes”,names(PimaIndiansDiabetes))

#build model

glm_model <- glm(diabetes~.,data = trainset, family = binomial)

summary(glm_model)

Call:

glm(formula = diabetes ~ ., family = binomial, data = trainset)

<<output edited>>

Coefficients:

Estimate Std. Error z value Pr(>|z|)

(Intercept)-8.1485021 0.7835869 -10.399 < 2e-16 ***

pregnant 0.1200493 0.0355617 3.376 0.000736 ***

glucose 0.0348440 0.0040744 8.552 < 2e-16 ***

pressure -0.0118977 0.0057685 -2.063 0.039158 *

triceps 0.0053380 0.0076523 0.698 0.485449

insulin -0.0010892 0.0009789 -1.113 0.265872

mass 0.0775352 0.0161255 4.808 1.52e-06 ***

pedigree 1.2143139 0.3368454 3.605 0.000312 ***

age 0.0117270 0.0103418 1.134 0.256816

—

Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

#predict probabilities on testset

#type=”response” gives probabilities, type=”class” gives class

glm_prob <- predict.glm(glm_model,testset[,-typeColNum],type=”response”)

#which classes do these probabilities refer to? What are 1 and 0?

contrasts(PimaIndiansDiabetes$diabetes)

pos

neg 0

pos 1

#make predictions

##…first create vector to hold predictions (we know 0 refers to neg now)

glm_predict <- rep(“neg”,nrow(testset))

glm_predict[glm_prob>.5] <- “pos”

#confusion matrix

table(pred=glm_predict,true=testset$diabetes)

glm_predict neg pos

neg 90 22

pos 8 33

#accuracy

mean(glm_predict==testset$diabetes)

[1] 0.8039216

Although this seems pretty good, we aren’t quite done because there is an issue that is lurking under the hood. To see this, let’s examine the information output from the model summary, in particular the coefficient estimates (i.e. estimates for ) and their significance. Here’s a summary of the information contained in the table:

- Column 2 in the table lists coefficient estimates.
- Column 3 list s the standard error of the estimates (the larger the standard error, the less confident we are about the estimate)
- Column 4 the z statistic (which is the coefficient estimate (column 2) divided by the standard error of the estimate (column 3)) and
- The last column (Pr(>|z|) lists the p-value, which is the probability of getting the listed estimate assuming the predictor has no effect. In essence, the
*smaller*the p-value, the*more significant*the estimate is likely to be.

From the table we can conclude that only 4 predictors are significant – *pregnant*, *glucose*, *mass *and *pedigree *(and possibly a fifth – *pressure*). The other variables have little predictive power and worse, may contribute to overfitting. They should, therefore, be eliminated and we’ll do that in a minute. However, there’s an important point to note before we do so…

In this case we have only 9 variables, so are able to identify the significant ones by a manual inspection of p-values. As you can well imagine, such a process will quickly become tedious as the number of predictors increases. Wouldn’t it be be nice if there were an algorithm that could somehow automatically shrink the coefficients of these variables or (better!) set them to zero altogether? It turns out that this is precisely what lasso and its close cousin, ridge regression, do.

Recall that the values of the logistic regression coefficients and are found by minimising the negative log likelihood described in equation (3). Ridge and lasso regularization work by adding a *penalty term* to the log likelihood function. In the case of ridge regression, the penalty term is and in the case of lasso, it is (Remember, is a vector, with as many components as there are predictors). The quantity to be minimised in the two cases is thus:

– for ridge regression,

and

– for lasso regression.

Where is a free parameter which is usually selected in such a way that the resulting model minimises the out of sample error. Typically, the optimal value of is found using grid search with cross-validation, a process akin to the one described in my discussion on cost-complexity parameter estimation in decision trees. Most canned algorithms provide methods to do this; the one we’ll use in the next section is no exception.

In the case of ridge regression, the effect of the penalty term is to shrink the coefficients that contribute most to the error. Put another way, it reduces the magnitude of the coefficients that contribute to *increasing* . In contrast, in the case of lasso regression, the effect of the penalty term is to set the these coefficients *exactly to zero! *This is cool because what it mean that lasso regression works like a feature selector that picks out the most important coefficients, i.e. those that are most predictive (and have the lowest p-values).

Let’s illustrate this through an example. We’ll use the glmnet package which implements a combined version of ridge and lasso (called elastic net). Instead of minimising (4) or (5) above, glmnet minimises:

where controls the “mix” of ridge and lasso regularisation, with being “pure” ridge and being “pure” lasso.

Let’s reanalyse the Pima Indian Diabetes dataset using glmnet with (pure lasso). Before diving into code, it is worth noting that glmnet:

- does not have a formula interface, so one has to input the predictors as a matrix and the class labels as a vector.
- does not accept categorical predictors, so one has to convert these to numeric values before passing them to glmnet.

The glmnet function model.matrix creates the matrix and also converts categorical predictors to appropriate dummy variables.

Another important point to note is that we’ll use the function cv.glmnet, which automatically performs a grid search to find the optimal value of .

OK, enough said, here we go:

#load required library

library(glmnet)

#convert training data to matrix format

x <- model.matrix(diabetes~.,trainset)

#convert class to numerical variable

y <- ifelse(trainset$diabetes==”pos”,1,0)

#perform grid search to find optimal value of lambda

#family= binomial => logistic regression, alpha=1 => lasso

# check docs to explore other type.measure options

cv.out <- cv.glmnet(x,y,alpha=1,family=”binomial”,type.measure = “mse” )

#plot result

plot(cv.out)

The plot is shown in Figure 2 below:

The plot shows that the *log *of the optimal value of lambda (i.e. the one that minimises the root mean square error) is approximately -5. The exact value can be viewed by examining the variable *lambda_min* in the code below. In general though, the objective of regularisation is to balance accuracy *and *simplicity. In the present context, this means a model with the* smallest number of coefficients that also gives a good accuracy*. To this end, the cv.glmnet function finds the value of lambda that gives the simplest model but also lies within one standard error of the optimal value of lambda. This value of lambda (*lambda.1se*) is what we’ll use in the rest of the computation. Interested readers should have a look at this article for more on* lambda.1se* vs* lambda.min*.

#min value of lambda

lambda_min <- cv.out$lambda.min

#best value of lambda

lambda_1se <- cv.out$lambda.1se

#regression coefficients

coef(cv.out,s=lambda_1se)

10 x 1 sparse Matrix of class “dgCMatrix”

1

(Intercept) -4.61706681

(Intercept) .

pregnant 0.03077434

glucose 0.02314107

pressure .

triceps .

insulin .

mass 0.02779252

pedigree 0.20999511

age .

The output shows that *only* those variables that we had determined to be significant on the basis of p-values have non-zero coefficients. The coefficients of all other variables have been set to zero by the algorithm! Lasso has reduced the complexity of the fitting function massively…and you are no doubt wondering what effect this has on accuracy. Let’s see by running the model against our test data:

#get test data

x_test <- model.matrix(diabetes~.,testset)

#predict class, type=”class”

lasso_prob <- predict(cv.out,newx = x_test,s=lambda_1se,type=”response”)

#translate probabilities to predictions

lasso_predict <- rep(“neg”,nrow(testset))

lasso_predict[lasso_prob>.5] <- “pos”

#confusion matrix

table(pred=lasso_predict,true=testset$diabetes)

pred neg pos

neg 94 28

pos 4 27

#accuracy

mean(lasso_predict==testset$diabetes)

[1] 0.7908497

Which is a bit less than what we got with the more complex model. So, we get a similar out-of-sample accuracy as we did before, and we do so using a way simpler function (4 non-zero coefficients) than the original one (9 nonzero coefficients). What this means is that the simpler function does at least as good a job fitting the signal in the data as the more complicated one. The bias-variance tradeoff tells us that the simpler function should be preferred because it is less likely to overfit the training data.

Paraphrasing William of Ockham: *all other things being equal, a simple hypothesis should be preferred over a complex one*.

In this post I have tried to provide a detailed introduction to logistic regression, one of the simplest (and oldest) classification techniques in the machine learning practitioners arsenal. Despite it’s simplicity (or I should say, because of it!) logistic regression works well for many business applications which often have a simple decision boundary. Moreover, because of its simplicity it is less prone to overfitting than flexible methods such as decision trees. Further, as we have shown, variables that contribute to overfitting can be eliminated using lasso (or ridge) regularisation, without compromising out-of-sample accuracy. Given these advantages and its inherent simplicity, it isn’t surprising that logistic regression remains a workhorse for data scientists.

Filed under: Data Analytics, Data Science, Predictive Analytics, Probability, R, Statistics ]]>

Throw three marbles onto a flat surface. When the marbles come to rest, you are most likely to end up with a random configuration as in Figure 1.

Indeed, you’d be extremely surprised if the three ended up being collinear as in Figure 2. Note that Figure 2 is just one example of many collinear possibilities, but the point I’m making is that if the marbles are thrown randomly, they are more likely to end up in a random state than a lined-up one.

This raises a couple of questions:

**Question**: On what basis can one claim that the collinear configuration is *tidier* or *more ordered* than the non-collinear one?

**Naive answer**: It *looks* more ordered. Yes, tidiness is in the eye of the beholder so it is necessarily subjective. However, I’ll wager that if one took a poll, an overwhelming number of people would say that the configuration in Figure 2 is more ordered than the one in Figure 1.

**More sophisticated answer **: The “state” of collinear marbles can be described using 2 parameters, the slope and intercept of the straight line that three marbles lie on (in any coordinate system) whereas the description of the nonlinear state requires 3 parameters. The first state is tidier because it requires fewer parameters. Another way to think about is that the line can be described by two marbles; the third one is redundant as far as the description of the state is concerned.

**Question**: Why is a tidier configuration less likely than a messy one?

**Answer**: May be you see this intuitively and need no proof, but here’s one just in case. Imagine rolling the three marbles one after the other. The first two, regardless of where they end up, will necessarily lie along a line (two points lie on the straight line joining them). Now, I think it is easy to see that if we throw the third marble randomly, it is highly unlikely end up on that line. Indeed, for the third marble to end up exactly on the same straight line requires a coincidence of near cosmic proportions.

I know, I know, this is not a proof, but I trust it makes the point.

Now, although it is near impossible to get to a collinear end state via random throws, it is possible to *approximate *it by changing the way we throw the marbles. Here’s how:

- Throw the marbles consecutively rather than in one go.
- When throwing the third marble, adjust its initial speed and direction in a way that takes into account the positions of the two marbles that are already on the surface. Remember these two already define a straight line.

The third throw is no longer random because it is designed to maximise the chance that the last marble will get as close as possible to the straight line defined by the first two. Done right, you’ll end up with something closer to the configuration in Figure 3 rather than the one in Figure 2.

Now you’re probably wondering what this has to do with *success*. I’ll make the connection via an example that will be familiar to many readers of this blog: an *organisation’s strategy*. However, as I will reiterate later, the arguments I present are very general and can be applied to just about any initiative or situation.

Typically, a strategy sets out *goals* for an organisation and a *plan* to achieve them in a specified timeframe. The goals define a number of *desirable* *outcomes, *or* states* which, by design, are constrained to belong to a (very) small subset of all possible states the organisation can end up in. In direct analogy with the simple model discussed above it is clear that, left to its own devices, the organisation is more likely to end up in one of the much overwhelmingly larger number of “failed states” than one of the successful ones. Notwithstanding the popular quote about there being many roads to success, in reality there are a great many more roads to failure.

Of course, that’s precisely why organisations are never “left to their own devices.” Indeed, a strategic plan specifies actions that are intended to make a successful state *more likely* than an unsuccessful one. However, no plan can guarantee success; it can, at best, make it more likely. As in the marble game, success is ultimately a matter of chance, even when we take actions to make it more likely.

If we accept this, the key question becomes: *how can one design a strategy that improves the odds of success*? The marble analogy suggests a way to do this is to:

- Define success in terms of an end state that is a
*natural extension of your current state*. - Devise a plan to (approximately) achieve that end state. Such a plan will necessarily
*build on the current state rather than change it wholesale*. Successful change is an*evolutionary*process rather than a*revolutionary*one*.*

My contention is that these points are often ignored by management strategists. More often than not, they will define an end state based on a textbook idealisation, consulting model or (horror!) *best practice*. The marble analogy shows why copying others is unlikely to succeed.

Figure 4 shows a variant of the marble game in which we have two sets of marbles (or organisations!), one blue, as before, and the other red.

Now, it is considerably harder to align an additional marble with both sets of marbles than the blue one alone. Here’s why…

To align with both sets, the new marble has to end up close to the *point* that lies at the intersection of the blue and red lines in Figure 5. In contrast, to align with the blue set alone, all that’s needed is for it to get close to *any* *point on the blue line*.

QED!

Finally, on a broader note, it should be clear that the arguments made above go beyond organisational strategies. They apply to pretty much any planned action, whether at work or in one’s personal life.

So, to sum up: when developing an organisational (or personal) strategy, the first step is to understand where you are and then identify the *minimal actions* you need to take in order to get to an “improved” state that is *consistent with your current one*. Yes, this is akin to the incremental and evolutionary approach that Agilistas and Leaners have been banging on about for years. However, their prescriptions focus on specific areas: software development and process improvement. My point is that the basic principles are way broader because they are a direct consequence of a fundamental fact regarding the relative likelihood of order and disorder in a toddler’s room, an organisation, or even the universe at large.

Filed under: Decision Making, Emergent Design, Management, Organizations, Probability ]]>

- Identify available options.
- Develop criteria for rating options.
- Rate options according to criteria developed.
- Select the top-ranked option.

Although this appears to be a logical way to proceed it is often difficult to put into practice, primarily because of *uncertainty* about matters relating to the decision.

Uncertainty can manifest itself in a variety of ways: one could be uncertain about facts, the available options, decision criteria or even one’s own preferences for options.

In this post, I discuss the role of uncertainty in decision making and, more importantly, how one can make well-informed decisions in such situations.

It is ironic that the term *uncertainty* is itself vague when used in the context of decision making. There are at least five distinct senses in which it is used:

- Uncertainty about decision options.
- Uncertainty about one’s preferences for options.
- Uncertainty about what criteria are relevant to evaluating the options.
- Uncertainty about what data is needed (data relevance).
- Uncertainty about the data itself (data accuracy).

Each of these is qualitatively different: uncertainty about data accuracy (item 5 above) is very different from uncertainty regarding decision options (item 1). The former can potentially be dealt with using statistics whereas the latter entails learning more about the decision problem and its context, ideally from different perspectives. Put another way, the item 5 is essentially a technical matter whereas item 1 is a deeper issue that may have social, political and – as we shall see – even behavioural dimensions. It is therefore reasonable to expect that the two situations call for vastly different approaches.

A common problem in project management is the estimation of task durations. In this case, what’s requested is a “best guess” time (in hours or days) it will take to complete a task. Many project schedules represent task durations by *point estimates*, i.e. by single numbers. The Gantt Chart shown in Figure 1 is a common example. In it, each task duration is represented by its expected duration. This is misleading because the single number conveys a sense of certainty that is unwarranted. It is far more accurate, not to mention safer, to quote *a range of possible durations*.

In general, quantifiable uncertainties, such as those conveyed in estimates, should always be quoted as ranges – something along the following lines: task A may take anywhere between 2 and 8 days, with a most likely completion time of 4 days (Figure 2).

In this example, aside from stating that the task will finish sometime between 2 and 4 days, the estimator implicitly asserts that the likelihood of finishing before 2 days or after 8 days is zero. Moreover, she also implies that some completion times are more likely than others. Although it may be difficult to quantify the likelihood exactly, one can begin by making simple (linear!) approximations as shown in Figure 3.

The key takeaway from the above is that *quantifiable uncertainties are shapes rather than single numbers*. See this post and this one for details for how far this kind of reasoning can take you. That said, one should always be aware of the assumptions underlying the approximations. Failure to do so can be hazardous to the credibility of estimators!

Although I haven’t explicitly said so, estimation as described above has a subjective element. Among other things, the quality of an estimate depends on the judgement and experience of the estimator. As such, it is prone to being affected by errors of judgement and cognitive biases. However, provided one keeps those caveats in mind, the probability-based approach described above is suited to situations in which uncertainties are quantifiable, at least in principle. That said, let’s move on to more complex situations in which uncertainties defy quantification.

The economist Frank Knight was possibly the first person to draw the distinction between quantifiable and unquantifiable uncertainties. To make things really confusing, he called the former *risk* and the latter *uncertainty*. In his doctoral thesis, published in 1921, wrote:

…it will appear that a measurable uncertainty, or “risk” proper, as we shall call the term, is so far different from an unmeasurable one that it is not in effect an uncertainty at all. We shall accordingly restrict the term “uncertainty” to cases of the non-quantitative type (p.20)

Terminology has moved on since Knight’s time, the term *uncertainty* means lots of different things, depending on context. In this piece, we’ll use the term *uncertainty* to refer to *quantifiable* uncertainty (as in the task estimate of the previous section) and use *ambiguity* to refer to *non*–*quantifiable* uncertainty. In essence, then, we’ll use the term *uncertainty* for situations where we know what we’re measuring (i.e. the facts) but are uncertain about its numerical or categorical values whereas we’ll use the word *ambiguity* to refer to situations in which we are uncertain about what the facts are or which facts are relevant.

As a test of understanding, you may want to classify each of the five points made in the second section of this post as either uncertain or ambiguous (Answers below)

** Answer**: 1 through 4 are ambiguous and 5 is uncertain.

The distinction between uncertainty and ambiguity points to a problem with quantitative decision-making techniques such as cost-benefit analysis, multicriteria decision making methods or analytic hierarchy process. All these methods assume that decision makers are aware of all the available *options*, their *preferences* for them, the relevant *evaluation criteria* and the *data* needed. This is almost never the case for consequential decisions. To see why, let’s take a closer look at the different ways in which ambiguity can play out in the rational decision making process mentioned at the start of this article.

- The first step in the process is to identify available options. In the real world, however,
*options often cannot be enumerated or articulated fully*. Furthermore, as options are articulated and explored, new options and sub-options tend to emerge. This is particularly true if the options depend on how future events unfold. - The second step is to develop criteria for rating options. As anyone who has been involved in deciding on a contentious issue will confirm, it is extremely difficult to agree on a set of decision criteria for issues that affect different stakeholders in different ways. Building a new road might improve commute times for one set of stakeholders but result in increased traffic in a residential area for others. The two criteria will be seen very differently by the two groups. In this case, it is very difficult for the two groups to agree on the relative importance of the criteria or even their legitimacy. Indeed,
*what constitutes a legitimate criterion is a matter of opinion*. - The third step is to rate options. The problem here is that
*real-world options often cannot be quantified or rated in a meaningful way*. Many of life’s dilemmas fall into this category. For example, a decision to accept or decline a job offer is rarely made on the basis of material gain alone. Moreover,*even where ratings are possible, they can be highly subjective*. For example, when considering a job offer, one candidate may give more importance to financial matters whereas another might consider lifestyle-related matters (flexi-hours, commuting distance etc.) to be paramount. Another complication here is that there may not be enough information to settle the matter conclusively. As an example, investment decisions are often made on the basis of quantitative information that is based on questionable assumptions.

A key consequence of the above is that such ambiguous decision problems are *socially complex* – i.e. different stakeholders could have wildly different perspectives on the problem itself. One could say *the ambiguity experienced by an individual is compounded by the group.*

Before going on I should point out that acute versions of such ambiguous decision problems go by many different names in the management literature. For example:

- Horst Rittel called them wicked problems..
- Russell Ackoff referred to them as messes.
- Herbert Simon labelled them non-programmable problems.

All these terms are more or less synonymous: the root cause of the difficulty in every case is ambiguity (or unquantifiable uncertainty), which prevents a clear formulation of the problem.

Social complexity is hard enough to tackle as it is, but there’s another issue that makes things even harder: ambiguity invariably triggers negative emotions such as fear and anxiety in individuals who make up the group. Studies in neuroscience have shown that in contrast to uncertainty, which evokes logical responses in people, *ambiguity tends to stir up negative emotions while simultaneously suppressing the ability to think logically*. One can see this playing out in a group that is debating a contentious decision: stakeholders tend to get worked up over issues that touch on their values and identities, and this seems to limit their ability to look at the situation objectively.

Summarising the discussion thus far: rational decision making approaches are based on the assumption that stakeholders have a shared understanding of the decision problem as well as the facts and assumptions around it. These conditions are clearly violated in the case of ambiguous decision problems. Therefore, when confronted with a decision problem that has even a hint of ambiguity, the first order of the day is to help the group reach a shared understanding of the problem. This is essentially an exercise in sensemaking, the art of collaborative problem formulation. However, this is far from straightforward because ambiguity tends to evoke negative emotions and attendant defensive behaviours.

The upshot of all this is that *any approach to tackle ambiguity must begin by taking the concerns of individual stakeholders seriously*. Unless this is done, it will be impossible for the group to coalesce around a consensus decision. Indeed, ambiguity-laden decisions in organisations invariably fail when they overlook concerns of specific stakeholder groups. The high failure rate of organisational change initiatives (60-70% according to this Deloitte report) is largely attributable to this point

There are a number of techniques that one can use to gather and synthesise diverse stakeholder viewpoints and thus reach a shared understanding of a complex or ambiguous problem. These techniques are often referred to as problem structuring methods (PSMs). I won’t go into these in detail here; for an example check out Paul Culmsee’s articles on dialogue mapping and Barry Johnson’s introduction to polarity management. There are many more techniques in the PSM stable. All of them are intended to help a group reconcile different viewpoints and thus reach a common basis from which one can proceed to the next step (i.e., make a decision on what should be done). In other words, these techniques help reduce ambiguity.

But there’s more to it than a bunch of techniques. The main challenge is to create a *holding environment* that enables such techniques to work. I am sure readers have been involved in a meeting or situation where the outcome seems predetermined by management or has been undermined by self- interest. When stakeholders sense this, no amount of problem structuring is going to help. In such situations one needs to first create the conditions for open dialogue to occur. This is precisely what a holding environment provides.

Creating such a holding environment is difficult in today’s corporate world, but not impossible. Note that this is not an idealist’s call for an organisational utopia. Rather, it involves the application of a practical set of tools that address the diverse, emotion-laden reactions that people often have when confronted with ambiguity. It would take me too far afield to discuss PSMs and holding environments any further here. To find out more, check out my papers on holding environments and dialogue mapping in enterprise IT projects, and (for a lot more) the Heretic’s Guide series of books that I co-wrote with Paul Culmsee.

The point is simply this: in an ambiguous situation, a good decision – whatever it might be – is most likely to be reached by a consultative process that synthesises diverse viewpoints rather than by an individual or a clique. However, genuine participation (the hallmark of a holding environment) in such a process will occur only after participants’ fears have been addressed.

Standard approaches to decision making exhort managers and executives to begin with facts, and if none are available, to gather them diligently prior to making a decision. However, most real-life decisions are fraught with uncertainty so it may be best to begin with what one doesn’t know, and figure out how to make the possible decision under those “constraints of ignorance.” In this post I’ve attempted to outline what such an approach would entail. The key point is to figure out the kind uncertainty one is dealing with and choosing an approach that works for it. I’d argue that most decision making debacles stem from a failure to appreciate this point.

Of course, there’s a lot more to this approach than I can cover in the span of a post, but that’s a story for another time.

**Note**: This post is written as an introduction to the Data and Decision Making subject that is part of the core curriculum of the Master of Data Science and Innovation program, run by the Connected Intelligence Centre at UTS. I’m coordinating the subject this semester, and am honoured to be co-teaching it with my erstwhile colleague Sean Heffernan and my longtime collaborator Paul Culmsee.

Filed under: Decision Making, General Management, Management, Organizations, sensemaking ]]>