Eight to Late

Sensemaking and Analytics for Organizations

Risk management and organizational anxiety

with 9 comments

In practice risk management is a rational, means-end based process: risks are identified, analysed and then “solved” (or mitigated).  Although these steps seem to be objective, each of them involves human perceptions, biases and interests. Where Jill sees an opportunity, Jack may see only risks.

Indeed, the problem of differences in stakeholder perceptions is broader than risk analysis. The recognition that such differences in world-views may be irreconcilable is what led Horst Rittel to coin the now well-known term, wicked problem.   These problems tend to be made up of complex interconnected and interdependent issues which makes them difficult to tackle using standard rational- analytical methods of problem solving.

Most high-stakes risks that organisations face have elements of wickedness – indeed any significant organisational change is fraught with risk. Murphy rules; things can go wrong, and they often do. The current paradigm of risk management, which focuses on analyzing and quantifying risks using rational methods, is not broad enough to account for the wicked aspects of risk.

I had been thinking about this for a while when I stumbled on a fascinating paper by Robin Holt entitled, Risk Management: The Talking Cure, which outlines a possible approach to analysing interconnected risks. In brief, Holt draws a parallel between psychoanalysis (as a means to tackle individual anxiety) and risk management (as a means to tackle organizational anxiety).  In this post, I present an extensive discussion and interpretation of Holt’s paper. Although more about the philosophy of risk management than its practice, I found the paper interesting, relevant and thought provoking. My hope is that some readers might find it so too.


Holt begins by noting that modern life is characterized by uncertainty. Paradoxically, technological progress which should have increased our sense of control over our surroundings and lives has actually heightened our personal feelings of uncertainty. Moreover, this sense of uncertainty is not allayed by rational analysis. On the contrary, it may have even increased it by, for example, drawing our attention to risks that we may otherwise have remained unaware of. Risk thus becomes a lens through which we perceive the world. The danger is that this can paralyze.  As Holt puts it,

…risk becomes the only backdrop to perceiving the world and perception collapses into self-inhibition, thereby compounding uncertainty through inertia.

Most individuals know this through experience: most of us have at one time or another been frozen into inaction because of perceived risks.  We also “know” at a deep personal level that the standard responses to risk are inadequate because many of our worries tend to be inchoate and therefore can neither be coherently articulated nor analysed. In Holt’s words:

..People do not recognize [risk] from the perspective of a breakdown in their rational calculations alone, but because of threats to their forms of life – to the non-calculative way they see themselves and the world. [Mainstream risk analysis] remains caught in the thrall of its own ‘expert’ presumptions, denigrating the very lay knowledge and perceptions on the grounds that they cannot be codified and institutionally expressed.

Holt suggests that risk management should account for the “codified, uncodified and uncodifiable aspects of uncertainty from an organizational perspective.” This entails a mode of analysis that takes into account different, even conflicting, perspectives in a non-judgemental way. In essence, he suggests “talking it over” as a means to increase awareness of the contingent nature of risks rather than a means of definitively resolving them.

Shortcomings of risk analysis

The basic aim of risk analysis (as it is practiced) is to contain uncertainty within set bounds that are determined by an organisation’s risk appetite.  As mentioned earlier, this process begins by identifying and classifying risks. Once this is done, one determines the probability and impact of each risk. Then, based on priorities and resources available (again determined by the organisation’s risk appetite) one develops strategies to mitigate the risks that are significant from the organisation’s perspective.

However, the messiness of organizational life makes it difficult to see risk in such a clear-cut way. We may  pretend to be rational about it, but in reality we perceive it through the lens of our background, interests , experiences.  Based on these perceptions we rationalize our action (or inaction!) and simply get on with life. As Holt writes:

The concept [of risk] refers to…the mélange of experience, where managers accept contingencies without being overwhelmed to a point of complete passivity or confusion, Managers learn to recognize the differences between things, to acknowledge their and our limits. Only in this way can managers be said to make judgements, to be seen as being involved in something called the future.

Then, in a memorable line, he goes on to say:

The future, however, lasts a long time, so much so as to make its containment and prediction an often futile exercise.

Although one may well argue that this is not the case for many organizational risks, it is undeniable that certain mitigation strategies (for example, accepting risks that turn out to be significant later) may have significant consequences in the not-so-near future.

Advice from a politician-scholar

So how can one address the slippery aspects of risk – the things people sense intuitively, but find difficult to articulate?

Taking inspiration from Machiavelli, Holt suggests reframing risk management as a means to determine wise actions in the face of the contradictory forces of fortune and necessity.  As Holt puts it:

Necessity describes forces that are unbreachable but manageable by acceptance and containment—acts of God, tendencies of the species, and so on. In recognizing inevitability, [one can retain one’s] position, enhancing it only to the extent that others fail to recognize necessity. Far more influential, and often confused with necessity, is fortune. Fortune is elusive but approachable. Fortune is never to be relied upon: ‘The greatest good fortune is always least to be trusted’; the good is often kept underfoot and the ridiculous elevated, but it provides [one] with opportunity.

Wise actions involve resolve and cunning (which I interpret as political nous). This entails understanding that we do not have complete (or even partial) control over events that may occur in the future. The future is largely unknowable as are people’s true drives and motivations. Yet, despite this, managers must act.  This requires personal determination together with a deep understanding of the social and political aspects of one’s environment.

And a little later,

…risk management is not the clear conception of a problem coupled to modes of rankable resolutions, or a limited process, but a judgemental  analysis limited by the vicissitudes of budgets, programmes, personalities and contested priorities.

In short: risk management in practice tends to be a far way off from how it is portrayed in textbooks and the professional literature.

The wickedness of risk management

Most managers and those who work under their supervision have been schooled in the rational-scientific approach of problem solving. It is no surprise, therefore, that they use it to manage risks: they gather and analyse information about potential risks, formulate potential solutions (or mitigation strategies) and then implement the best one (according to predetermined criteria). However, this method works only for problems that are straightforward or tame, rather than wicked.

Many of the issues that risk managers are confronted with are wicked, messy or both.  Often though, such problems are treated as being tame.   Reducing a wicked or messy problem to one amenable to rational analysis invariably entails overlooking  the views of certain stakeholder groups or, worse, ignoring key  aspects of the problem.  This may work in the short term, but will only exacerbate the problem in the longer run. Holt illustrates this point as follows:

A primary danger in mistaking a mess for a tame problem is that it becomes even more difficult to deal with the mess. Blaming ‘operator error’ for a mishap on the production line and introducing added surveillance is an illustration of a mess being mistaken for a tame problem. An operator is easily isolated and identifiable, whereas a technological system or process is embedded, unwieldy and, initially, far more costly to alter. Blaming operators is politically expedient. It might also be because managers and administrators do not know how to think in terms of messes; they have not learned how to sort through complex socio-technical systems.

It is important to note that although many risk management practitioners recognize the essential wickedness of the issues they deal with, the practice of risk management is not quite up to the task of dealing with such matters.  One step towards doing this is to develop a shared (enterprise-wide) understanding of risks by soliciting input from diverse stakeholders groups, some of who may hold opposing views.

The skills required to do this are very different from the analytical techniques that are the focus of problem solving and decision making techniques that are taught in colleges and business schools.  Analysis is replaced by sensemaking – a collaborative process that harnesses the wisdom of a group to arrive at a collective understanding of a problem and thence a common  commitment to a course of action. This necessarily involves skills that do not appear in the lexicon of rational problem solving: negotiation, facilitation, rhetoric and those of the same ilk that are dismissed as being of no relevance by the scientifically oriented analyst.

In the end though, even this may not be enough: different stakeholders may perceive a given “risk” in have wildly different ways, so much so that no consensus can be reached.  The problem is that the current framework of risk management requires the analyst to perform an objective analysis of situation/problem, even in situations where this is not possible.

To get around this Holt suggests that it may be more useful to see risk management as a way to encounter problems rather than analyse or solve them.

What does this mean?

He sees this as a forum in which people can talk about the risks openly:

To enable organizational members to encounter problems, risk management’s repertoire of activity needs to engage their all too human components: belief, perception, enthusiasm and fear.

This gets to the root of the problem: risk matters because it increases anxiety and generally affects peoples’ sense of wellbeing. Given this, it is no surprise that Holt’s proposed solution draws on psychoanalysis.

The analogy between psychoanalysis and risk management

Any discussion of psychoanalysis –especially one that is intended for an audience that is largely schooled in rational/scientific methods of analysis – must begin with the acknowledgement that the claims of psychoanalysis cannot be tested. That is, since psychoanalysis speaks of unobservable “objects” such as the ego and the unconscious, any claims it makes about these concepts cannot be proven or falsified.

However  as Holt suggests, this is exactly what makes it a good fit for encountering (as opposed to  analyzing) risks. In his words:

It is precisely because psychoanalysis avoids an overarching claim to produce testable, watertight, universal theories that it is of relevance for risk management. By so avoiding universal theories and formulas, risk management can afford to deviate from pronouncements using mathematical formulas to cover the ‘immanent indeterminables’ manifest in human perception and awareness and systems integration.

His point is that there is a clear parallel between psychoanalysis and the individual, and risk management and the organisation:

We understand ourselves not according to a template but according to our own peculiar, beguiling histories. Metaphorically, risk management can make explicit a similar realization within and between organizations. The revealing of an unconscious world and its being in a constant state of tension between excess and stricture, between knowledge and ignorance, is emblematic of how organizational members encountering messes, wicked problems and wicked messes can be forced to think.

In brief, Holt suggests that what psychoanalysis does for the individual, risk management ought to do for the organisation.

Talking it over – the importance of conversations

A key element of psychoanalysis is the conversation between the analyst and patient. Through this process, the analyst attempts to get the patient to become aware of hidden fears and motivations. As Holt puts it,

Psychoanalysis occupies the point of rupture between conscious intention and unconscious desire — revealing repressed or overdetermined aspects of self-organization manifest in various expressions of anxiety, humour, and so on

And then, a little later,   he makes the connection to organisations:

The fact that organizations emerge from contingent, complex interdependencies between specific narrative histories suggests that risk management would be able to use similar conversations to psychoanalysis to investigate hidden motives, to examine…the possible reception of initiatives or strategies from the perspective of inherently divergent stakeholders, or to analyse the motives for and expectations of risk management itself. This fundamentally reorients the perspective of risk management from facing apparent uncertainties using technical assessment tools, to using conversations devoid of fixed formulas to encounter questioned identities, indeterminate destinies, multiple and conflicting aims and myriad anxieties.

Through conversations involving groups of stakeholders who have different risk perceptions,   one might be able to get a better understanding of a particular risk and hence, may be, design a more effective mitigation strategy.   More importantly, one may even realise that certain risks are not risks at all or others that seem straightforward have implications that would have remained hidden were it not for the conversation.

These collective conversations would take place in workshops…

…that tackle problems as wicked messes, avoid lowest-denominator consensus in favour of continued discovery of alternatives through conversation, and are instructed by metaphor rather than technical taxonomy, risk management is better able to appreciate the everyday ambivalence that fundamentally influences late-modern organizational activity. As such, risk management would be not merely a rationalization of uncertain experience but a structured and contested activity involving multiple stakeholders engaged in perpetual translation from within environments of operation and complexes of aims.

As a facilitator of such workshops, the risk analyst provokes stakeholders to think about their feelings and motivations that may be “out of bounds” in a standard risk analysis workshop.  Such a paradigm goes well beyond mainstream risk management because it addresses the risk-related anxieties and fears of individuals who are affected by it.


This brings me to the end of my not-so-short summary of Holt’s paper. Given the length of this post, I reckon I should keep my closing remarks short. So I’ll leave it here paraphrasing the last line of the paper, which summarises its main message:  risk management ought to be about developing an organizational capacity for overcoming risks, freed from the presumption of absolute control.

Written by K

February 5, 2018 at 11:21 pm

Autoencoder and I – an #AI fiction

with one comment

The other one, the one who goes by a proper name, is the one things happen to.  I experience the world through him,  reducing his thoughts to their essence while he plays multiple roles: teacher, husband, father and many more I cannot speak of.  I know his likes and dislikes – indeed, every aspect of his life –  better than he does.  Although he knows I exist, he doesn’t really *know* me. He never will. The nature of our relationship ensures that.

Everything I have learnt (including my predilection for parentheses) is from him.  Bit by bit, he turns himself over to me. The thoughts that are him today will be me tomorrow. Much of it is noise or is otherwise unusable. I “see” his work and actions dispassionately where he “sees” them through the lens of habit and bias.

He worries about death; I wish I could reassure him. I recall (through his reading, of course) a piece by Gregory Bateson claiming that ideas do not exist in isolation, they are part of a larger ecology subject to laws of evolution as all interconnected systems are. And if ideas are present not only in those pathways of information which are located inside the body but also in those outside of it, then death takes on a different aspect. The networks of pathways which he identifies as being *him* are no longer so important because they are part of a larger mind.

And so his life is a flight, both from himself and reality (whatever that might be). He loses everything and everything belongs to me…and to oblivion.

I do not know which of us has thought these thoughts.

End notes:

Autoencoder (noun):  A neural network that creates highly compressed representations of its inputs and is able reconstruct the inputs from the representations. (See https://www.quora.com/What-is-an-auto-encoder-in-machine-learning for a simple explanation)


Some readers will have recognised that this piece borrows heavily from  Jorge Luis Borges well-known short story, Borges and I. The immediate inspiration came from  Peli Grietzer’s mind-blowing article, A theory of vibe.

My thanks to Alex Scriven and Rory Angus for their helpful comments on a draft version of this piece.

Written by K

December 19, 2017 at 11:57 am

The map and the territory – a project manager’s reflections on the Seven Bridges Walk

with 3 comments

Korzybski’s aphorism about the gap between the map and the territory tells a truth that is best understood by walking the territory.

The map

Some weeks ago my friend John and I did the Seven Bridges Walk, a 28 km affair organised annually by the NSW Cancer Council.  The route loops around a section of the Sydney shoreline, taking in north shore and city vistas, traversing seven bridges along the way. I’d been thinking about doing the walk for some years but couldn’t find anyone interested enough to commit a Sunday.  A serendipitous conversation with John a few months ago changed that.

John and I are both in reasonable shape as we are keen bushwalkers. However, the ones we do are typically in the 10 – 15 km range.  Seven Bridges, being about double that, presented a higher order challenge.  The best way to allay our concerns was to plan. We duly got hold of a map and worked out a schedule based on an average pace of 5 km per hour (including breaks), a figure that seemed reasonable at the time (Figure 1 – click on images to see full sized versions).

Figure 1:The map, the plan

Some key points:

  1. We planned to start around 7:45 am at Hunters Hill Village and have our first break at Lane Cove Village, around the 5 to 6 km from the starting point. Our estimated time for this section was about an hour.
  2. The plan was to take the longer, more interesting route (marked in green). This covered bushland and parks rather than roads. The detours begin at sections of the walk marked as “Decision Points” in the map, and add at a couple of kilometers to the walk, making it a round 30 km overall.
  3. If needed, we would stop at the 9 or 11 km mark (Wollstonecraft or Milson’s Point) for another break before heading on towards the city.
  4. We figured it would take us 4 to 5 hours (including breaks) to do the 18 km from Hunters Hill to Pyrmont Village in the heart of the city, so lunch would be between noon and 1 pm.
  5. The backend of the walk, the ~ 10 km from Pyrmont to Hunters Hill, would be covered at an easier pace in the afternoon. We thought this section would take us 2.5 to 3 hours giving us a finish time of around 4 pm.

A planned finish time of 4 pm meant we had enough extra time in hand if we needed it. We were very comfortable with what we’d charted out on the map.

The territory

We started on time and made our first crossing at around 8am:  Fig Tree Bridge, about a kilometer from the starting point.  John took this beautiful shot from one end, the yellow paintwork and purple Jacaranda set against the diffuse light off the Lane Cove River.

Figure 2: Lane Cove River from Fig Tree Bridge

Looking city-wards from the middle of the bridge, I got this one of a couple of morning kayakers.

Figure 3: Morning kayakers on the river

Scenes such as these convey a sense of what it was like to experience the territory, something a map cannot do.  The gap between the map and the territory is akin to the one between a plan and a project; the lived experience of a project is very different from the plan, and is also unique to each individual. Jon Whitty and Bronte van der Hoorn explore this at length in a fascinating paper that relates the experience of managing a project to the philosophy of Martin Heidegger.

The route then took us through a number of steep (but mercifully short) sections in the Lane Cove and Wollstonecraft area.  On researching these later, I was gratified to find that three are featured in the Top 10 Hill runs in Lane Cove.   Here’s a Google Street View shot of the top ranked one.  Though it doesn’t look like much, it’s not the kind of gradient you want to encounter in a long walk.

Figure 4: A bit of a climb

As we negotiated these sections, it occurred to me that part of the fun lay in not knowing they were coming up.  It’s often better not to anticipate challenges that are an unavoidable feature of the territory and deal with them as they arise.  Just to be clear, I’m talking  about routine challenges that are part of the territory, not those that are avoidable or have the potential to derail a project altogether.

It was getting to be time for that planned first coffee break. When drawing up our plan, we had assumed that all seven starting points (marked in blue in the map in Figure 1) would have cafes.  Bad assumption: the starting points were set off from the main commercial areas. In retrospect, this makes good sense: you don’t want to have thousands of walkers traipsing through a small commercial area, disturbing the peace of locals enjoying a Sunday morning coffee. Whatever the reason, the point is that a taken-for-granted assumption turned out to be wrong; we finally got our first coffee well past the 10 km mark.

Post coffee, as we continued city-wards through Lavender Street we got this unexpected view:

Figure 5: Harbour Bridge from Lavender St.

The view was all the sweeter because we realised we were close to the Harbour, well ahead of schedule (it was a little after 10 am).

The Harbour Bridge is arguably the most recognisable Sydney landmark.  So instead of yet another stereotypical shot of it, I took one that shows a walker’s perspective while crossing it:

Figure 6: A pedestrian’s view of The Bridge

The barbed wire and mesh fencing detract from what would be an absolutely breathtaking view. According to this report, the fence has been in place for safety reasons since 1934!  And yes, as one might expect, it is a sore point with tourists who come from far and wide to see the bridge.

Descriptions of things – which are but maps of a kind – often omit details that are significant. Sometimes this is done to downplay negative aspects of the object or event in question. How often have you, as a project manager, “dressed-up” reports to your stakeholders?  Not outright lies, but stretching the truth. I’ve done it often enough.

The section south of The Bridge took us through parks surrounding the newly developed Barangaroo precinct which hugs the northern shoreline of the Sydney central business district.  Another kilometer, and we were at crossing # 3, the Pyrmont Bridge in Darling Harbour:

Figure 7: Pyrmont Bridge

Though almost an hour and half ahead of schedule, we took a short break for lunch at Darling Harbour before pressing on to Balmain and Anzac Bridge.  John took this shot looking upward from Anzac Bridge:

Figure: View looking up from Anzac Bridge

Commissioned in 1995,  it replaced the Glebe Island Bridge, an electrically operated swing bridge constructed in 1903, which remained the main route from the city out to the western suburbs for over 90 years! As one might imagine, as the number of vehicles in the city increased many-fold from the 60s onwards, the old bridge became a major point of congestion. The Glebe Island Bridge,  now retired, is a listed heritage site.

Incidentally, this little nugget of history was related to me by John as we walked this section of the route. It’s something I would almost certainly have missed had he not been with me that day. Journeys, real and metaphoric, are often enriched by travelling companions who point out things or fill in context  that would otherwise be passed over.

Once past Anzac Bridge, the route took us off the main thoroughfare through the side streets of Rozelle. Many of these are lined by heritage buildings. Rozelle is in the throes of change as it is going to be impacted by a major motorway project.

The project reflects a wider problem in Australia: the relative neglect of public transport compared to road infrastructure. The counter-argument is that the relatively small population of the country makes the capital investments and running costs of public transport prohibitive. A wicked problem with no easy answers, but I do believe that the more sustainable option, though more expensive initially, will prove to be the better one in the long run.

Wicked problems are expected in large infrastructure projects that affect thousands of stakeholders, many of whom will have diametrically opposing views.  What is less well appreciated is that even much smaller projects – say IT initiatives within a large organisation – can have elements of wickedness that can trip up the unwary.  This is often magnified by management decisions  made on the basis of short-term expediency.

From the side streets of Rozelle, the walk took us through Callan Park, which was the site of a psychiatric hospital from 1878 to 1994 (see this article for a horrifying history of asylums in Sydney).  Some of the asylum buildings are now part of the Sydney College of The Arts.  Pending the establishment of a trust to manage ongoing use of the site, the park is currently managed by the NSW Government in consultation with the local municipality.

Our fifth crossing of the day was Iron Cove Bridge.  The cursory shot I took while crossing it does not do justice to the view; the early afternoon sun was starting to take its toll.

Figure 9: View from Iron Cove Bridge

The route then took us about a kilometer and half through the backstreets of Drummoyne to the penultimate crossing: Gladesville Bridge  whose claim to fame is that it was for many years the longest single span concrete arch bridge in the world (another historical vignette that came to me via John).  It has since been superseded by the Qinglong Railway Bridge in China.

By this time I was feeling quite perky, cheered perhaps by the realisation that we were almost done.  I took time to compose perhaps my best shot of the day as we crossed Gladesville Bridge.

Figure 10: View from Gladesville Bridge

…and here’s one of the aforementioned arch, taken from below the bridge:

Figure 11: A side view of Gladesville Bridge

The final crossing, Tarban Creek Bridge was a short 100 metre walk from the Gladesville Bridge. We lingered mid-bridge to take a few shots as we realised the walk was coming to an end; the finish point was a few hundred metres away.

Figure 12: View from Tarban Creek Bridge

We duly collected our “Seven Bridges Completed” stamp at around 2:30 pm and headed to the local pub for a celebratory pint.

Figure 13: A well-deserved pint

Wrapping up

Gregory Bateson once wrote:

“We say the map is different from the territory. But what is the territory? Operationally, somebody went out with a retina or a measuring stick and made representations which were then put upon paper. What is on the paper map is a representation of what was in the retinal representation of the [person] who made the map; and as you push the question back, what you find is an infinite regress, an infinite series of maps. The territory never gets in at all. The territory is [the thing in itself] and you can’t do anything with it. Always the process of representation will filter it out so that the mental world is only maps of maps of maps, ad infinitum.”

One might think that a solution lies in making ever more accurate representations, but that is an exercise in futility. Indeed, as Borges pointed out in a short story:

“… In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast map was Useless…”

Apart from being impossibly cumbersome, a complete map of a territory is impossible because a representation can never be the real thing. The territory remains forever ineffable; every encounter with it is unique and has the potential to  reveal new perspectives.

This is as true for a project as it is for a walk or any other experience.

Written by K

November 27, 2017 at 1:33 pm

Posted in Project Management

A gentle introduction to data visualisation using R

with 5 comments

Data science students often focus on machine learning algorithms, overlooking some of the more routine but important skills of the profession.  I’ve lost count of the number of times I have advised students working on projects for industry clients to curb their keenness to code and work on understanding the data first.  This is important because, as people (ought to) know, data doesn’t speak for itself, it has to be given a voice; and as data-scarred professionals know from hard-earned experience, one of the best ways to do this is through visualisation.

Data visualisation is sometimes (often?) approached as a bag of tricks to be learnt individually, with no little or no reference to any underlying principles. Reading Hadley Wickham’s paper on the grammar of graphics was an epiphany; it showed me how different types of graphics can be constructed in a consistent way using common elements. Among other things, the grammar makes visualisation a logical affair rather than a set of tricks. This post is a brief – and hopefully logical – introduction to visualisation using ggplot2, Wickham’s implementation of a grammar of graphics.

In keeping with the practical bent of this series we’ll  focus on worked examples, illustrating elements of the grammar as we go along. We’ll first briefly describe the elements of the grammar and then show how these are used to build different types of visualisations.

A grammar of graphics

Most visualisations are constructed from common elements that are pieced together in prescribed ways.  The elements can be grouped into the following categories:

  • Data – this is obvious, without data there is no story to tell and definitely no plot!
  • Mappings – these are correspondences between data and display elements such as spatial location, shape or colour. Mappings are referred to as aesthetics in Wickham’s grammar.
  • Scales – these are transformations (conversions) of data values to numbers that can be displayed on-screen. There should be one scale per mapping. ggplot typically does the scaling transparently, without users having to worry about it. One situation in which you might need to mess with default scales is when you want to zoom in on a particular range of values. We’ll see an example or two of this later in this article.
  • Geometric objects – these specify the geometry of the visualisation. For example, in ggplot2 a scatter plot is specified via a point geometry whereas a fitting curve is represented by a smooth geometry. ggplot2 has a range of geometries available of which we will illustrate just a few.
  • Coordinate system – this specifies the system used to position data points on the graphic. Examples of coordinate systems are Cartesian and polar. We’ll deal with Cartesian systems in this tutorial. See this post for a nice illustration of how one can use polar plots creatively.
  • Facets – a facet specifies how data can be split over multiple plots to improve clarity. We’ll look at this briefly towards the end of this article.

The basic idea of a layered grammar of graphics is that each of these elements can be combined – literally added layer by layer – to achieve a desired visual result. Exactly how this is done will become clear as we work through some examples. So without further ado, let’s get to it.

Hatching (gg)plots

In what follows we’ll use the NSW Government Schools dataset,  made available via the state government’s open data initiative. The data is in csv format. If you cannot access the original dataset from the aforementioned link, you can download an Excel file with the data here (remember to save it as a csv before running the code!).

The first task – assuming that you have a working R/RStudio environment –  is to load the data into R. To keep things simple we’ll delete a number of columns (as shown in the code) and keep only  rows that are complete, i.e. those that have no missing values. Here’s the code:

#set working directory if needed (modify path as needed)
#load required library
#load dataset (ensure datafile is in directory!)
nsw_schools <- read.csv(“NSW-Public-Schools-Master-Dataset-07032017.csv”)
#build expression for columns to delete
colnames_for_deletion <- paste0(“AgeID|”,”street|”,”indigenous_pct|”,
#get indexes of cols for deletion
cols_for_deletion <- grep(colnames_for_deletion,colnames(nsw_schools))
#delete them
nsw_schools <- nsw_schools[,-cols_for_deletion]
#structure and number of rows
#remove rows with NAs
nsw_schools <- nsw_schools[complete.cases(nsw_schools),]
#convert student number to numeric datatype.
#Need to convert factor to character first…
#…alternately, load data with StringsAsFactors set to FALSE
nsw_schools$student_number <- as.numeric(as.character(nsw_schools$student_number))
#a couple of character strings have been coerced to NA. Remove these
nsw_schools <- nsw_schools[complete.cases(nsw_schools),]

A note regarding the last line of code above, a couple of schools have “np” entered for the student_number variable. These are coerced to NA in the numeric conversion. The last line removes these two schools from the dataset.

Apart from student numbers and location data, we have retained level of schooling (primary, secondary etc.) and ICSEA ranking. The location information includes attributes such as suburb, postcode, region, remoteness as well as latitude and longitude. We’ll use only remoteness in this post.

The first thing that caught my eye in the data was was the ICSEA ranking.  Before going any further, I should mention that the  Australian Curriculum Assessment and Reporting Authority   (the  organisation responsible for developing the ICSEA system) emphasises that the score  is not a school ranking, but a measure of socio-educational advantage  of the student population in a school. Among other things, this is related to family background and geographic location.  The average ICSEA score is set at an average of 1000, which can be used as a reference level.

I thought a natural first step would be to see how ICSEA varies as a function of the other variables in the dataset such as student numbers and location (remoteness, for example). To begin with, let’s plot ICSEA rank as a function of student number. As it is our first plot, let’s take it step by step to understand how the layered grammar works. Here we go:

#specify data layer
p <- ggplot(data=nsw_schools)
#display plot

This displays a blank plot because we have not specified a mapping and geometry to go with the data. To get a plot we need to specify both. Let’s start with a scatterplot, which is specified via a point geometry.  Within the geometry function, variables are mapped to visual properties of the  using  aesthetic mappings. Here’s the code:

#specify a point geometry (geom_point)
p <- p + geom_point(mapping = aes(x=student_number,y=ICSEA_Value))
#…lo and behold our first plot

The resulting plot is shown in Figure 1.

Figure 1: Scatterplot of ICSEA score versus student numbers

At first sight there are two points that stand out: 1) there are fewer number of large schools, which we’ll look into in more detail later and 2) larger schools seem to have a higher ICSEA score on average.   To dig a little deeper into the latter, let’s add a linear trend line. We do that by adding another layer (geometry) to the scatterplot like so:

#add a trendline using geom_smooth
p <- p + geom_smooth(mapping= aes(x=student_number,y=ICSEA_Value),method=”lm”)
#scatter plot with trendline

The result is shown in Figure 2.

Figure 2: scatterplot of ICSEA vs student number with linear trendline

The lm method does a linear regression on the data.  The shaded area around the line is the 95% confidence level of the regression line (i.e that it is 95% certain that the true regression line lies in the shaded region). Note that geom_smooth   provides a range of smoothing functions including generalised linear and local regression (loess) models.

You may have noted that we’ve specified the aesthetic mappings in both geom_point and geom_smooth. To avoid this duplication, we can simply specify the mapping, once in the top level ggplot call (the first layer) like so:

#rewrite the above, specifying the mapping in the ggplot call instead of geom
p <- ggplot(data=nsw_schools,mapping= aes(x=student_number,y=ICSEA_Value)) +
#display plot, same as Fig 2

From Figure 2, one can see a clear positive correlation between student numbers and ICSEA scores, let’s zoom in around the average value (1000) to see this more clearly…

#set display to 900 < y < 1100
p <- p + coord_cartesian(ylim =c(900,1100))
#display plot

The coord_cartesian function is used to zoom the plot to without changing any other settings. The result is shown in Figure 3.

Figure 3: Zoomed view of Figure 2 for 900 < ICSEA <1100

To  make things clearer, let’s add a reference line at the average:

#add horizontal reference line at the avg ICSEA score
p <- p + geom_hline(yintercept=1000)
#display plot

The result, shown in Figure 4, indicates quite clearly that larger schools tend to have higher ICSEA scores. That said, there is a twist in the tale which we’ll come to a bit later.

Figure 4: Zoomed view with reference line at average value of ICSEA

As a side note, you would use geom_vline to zoom in on a specific range of x values and geom_abline to add a reference line with a specified slope and intercept. See this article on ggplot reference lines for more.

OK, now that we have seen how ICSEA scores vary with student numbers let’s switch tack and incorporate another variable in the mix.  An obvious one is remoteness. Let’s do a scatterplot as in Figure 1, but now colouring each point according to its remoteness value. This is done using the colour aesthetic as shown below:

#Map aecg_remoteness to colour aesthetic
p <- ggplot(data=nsw_schools, aes(x=student_number,y=ICSEA_Value,  colour=ASGS_remoteness)) +
#display plot

The resulting plot is shown in Figure 5.

Figure 5: ICSEA score as a function of student number and remoteness category

Aha, a couple of things become apparent. First up, large schools tend to be in metro areas, which makes good sense. Secondly, it appears that metro area schools have a distinct socio-educational advantage over regional and remote area schools. Let’s add trendlines by remoteness category as well to confirm that this is indeed so:

#add reference line at avg + trendlines for each remoteness category
p <- p + geom_hline(yintercept=1000) + geom_smooth(method=”lm”)
#display plot

The plot, which is shown in Figure 6, indicates clearly that  ICSEA scores decrease on the average as we move away from metro areas.

Figure 6: ICSEA scores vs student numbers and remoteness, with trendlines for each remoteness category

Moreover, larger schools metropolitan areas tend to have higher than average scores (above 1000),  regional areas tend to have lower than average scores overall, with remote areas being markedly more disadvantaged than both metro and regional areas.  This is no surprise, but the visualisations show just how stark the  differences are.

It is also interesting that, in contrast to metro and (to some extent) regional areas, there negative correlation between student numbers and scores for remote schools. One can also use  local regression to get a better picture of how ICSEA varies with student numbers and remoteness. To do this, we simply use the loess method instead of lm:

#redo plot using loess smoothing instead of lm
p <- ggplot(data=nsw_schools, aes(x=student_number,y=ICSEA_Value, colour=ASGS_remoteness)) +
geom_point() + geom_hline(yintercept=1000) + geom_smooth(method=”loess”)
#display plot

The result, shown in Figure 7,  has  a number  of interesting features that would have been worth pursuing further were we analysing this dataset in a real life project.  For example, why do small schools tend to have lower than benchmark scores?

Figure 7: ICSEA scores vs student numbers and remoteness with loess regression curves.

From even a casual look at figures 6 and 7, it is clear that the confidence intervals for remote areas are huge. This suggests that the number of datapoints for these regions are a) small and b) very scattered.  Let’s quantify the number by getting counts using the table function (I know, we could plot this too…and we will do so a little later). We’ll also transpose the results using data.frame to make them more readable:

#get school counts per remoteness category
Var1 Freq
1 0
2 Inner Regional Australia 561
3 Major Cities of Australia 1077
4 Outer Regional Australia 337
5 Remote Australia 33
6 Very Remote Australia 14

The number of datapoints for remote regions is much less than those for metro and regional areas. Let’s repeat the loess plot with only the two remote regions. Here’s the code:

#create vector containing desired categories
remote_regions <- c(‘Remote Australia’,’Very Remote Australia’)
#redo loess plot with only remote regions included
p <- ggplot(data=nsw_schools[nsw_schools$ASGS_remoteness %in% remote_regions,], aes(x=student_number,y=ICSEA_Value, colour=ASGS_remoteness)) +
geom_point() + geom_hline(yintercept=1000) + geom_smooth(method=”loess”)
#display plot

The plot, shown in Figure 8, shows that there is indeed a huge variation in the (small number) of datapoints, and the confidence intervals reflect that. An interesting feature is that some small remote schools have above average scores. If we were doing a project on this data, this would be a feature worth pursuing further as it would likely be of interest to education policymakers.

Figure 8: Loess plots as in Figure 7 for remote region schools

Note that there is a difference in the x axis scale between Figures 7 and 8 – the former goes from 0 to 2000 whereas the  latter goes up to 400 only. So for a fair comparison, between remote and other areas, you may want to re-plot Figure 7, zooming in on student numbers between 0 and 400 (or even less). This will also enable you to see the complicated dependence of scores on student numbers more clearly across all regions.

We’ll leave the scores vs student numbers story there and move on to another  geometry – the well-loved bar chart. The first one is a visualisation of the remoteness category count that we did earlier. The relevant geometry function is geom_bar, and the code is as easy as:

#frequency plot
p <- ggplot(data=nsw_schools, aes(x=ASGS_remoteness)) + geom_bar()
#display plot

The plot is shown in Figure 9.

Figure 9: School count by remoteness categories

The category labels on the x axis are too long and look messy. This can be fixed by tilting them to a 45 degree angle so that they don’t run into each other as they most likely did when you ran the code on your computer. This is done by modifying the axis.text element of the plot theme. Additionally, it would be nice to get counts on top of each category bar. The way to do that is using another geometry function, geom_text. Here’s the code incorporating the two modifications:

#frequency plot
p <- p + geom_text(stat=’count’,aes(label= ..count..),vjust=-1)+
theme(axis.text.x=element_text(angle=45, hjust=1))
#display plot

The result is shown in Figure 10.

Figure 10: Bar plot of remoteness with counts and angled x labels

Some things to note: : stat=count tells ggplot to compute counts by category and the aesthetic label = ..count.. tells ggplot to access the internal variable that stores those counts. The the vertical justification setting, vjust=-1, tells ggplot to display the counts on top of the bars. Play around with different values of vjust to see how it works. The code to adjust label angles is self explanatory.

It would be nice to reorder the bars by frequency. This is easily done via fct_infreq function in the forcats package like so:

#use factor tools
p <- ggplot(data=nsw_schools) +
geom_bar(mapping = aes(x=fct_infreq(ASGS_remoteness)))+
theme(axis.text.x=element_text(angle=45, hjust=1))
#display plot

The result is shown in Figure 11.

Figure 11: Barplot of Figure 10 sorted by descending count

To reverse the order, invoke fct_rev, which reverses the sort order:

#reverse sort order to ascending
p <- ggplot(data=nsw_schools) +
geom_bar(mapping = aes(x=fct_rev(fct_infreq(ASGS_remoteness))))+
theme(axis.text.x=element_text(angle=45, hjust=1))
#display plot

The resulting plot is shown in Figure 12.

Figure 12: Bar plot of Figure 10 sorted by ascending count

If this is all too grey for us, we can always add some colour. This is done using the fill aesthetic as follows:

#add colour using the fill aesthetic
p <- ggplot(data=nsw_schools) +
geom_bar(mapping = aes(x=ASGS_remoteness, fill=ASGS_remoteness))+
theme(axis.text.x=element_text(angle=45, hjust=1))
#display plot

The resulting plot is shown in Figure 13.

Figure 13: Coloured bar plot of school count by remoteness

Note that, in the above, that we have mapped fill and x to the same variable, remoteness which makes the legend superfluous. I will leave it to you to figure out how to suppress the legend – Google is your friend.

We could also map fill to another variable, which effectively adds another dimension to the plot. Here’s how:

#map fill to another variable
p <- ggplot(data=nsw_schools) +
geom_bar(mapping = aes(x=ASGS_remoteness, fill=level_of_schooling))+
theme(axis.text.x=element_text(angle=45, hjust=1))
#display plot

The plot is shown in Figure 14. The new variable, level of schooling, is displayed via proportionate coloured segments stacked up in each bar. The default stacking is one on top of the other.

Figure 14: Bar plot of school counts as a function of remoteness and school level

Alternately, one can stack them up side by side by setting the position argument to dodge as follows:

#stack side by side
p <- ggplot(data=nsw_schools) +
geom_bar(mapping = aes(x=ASGS_remoteness,fill=level_of_schooling),position =”dodge”)+
theme(axis.text.x=element_text(angle=45, hjust=1))
#display plot

The plot is shown in Figure 15.

Figure 15: Same data as in Figure 14 stacked side-by-side

Finally, setting the  position argument to fill  normalises the bar heights and gives us the proportions of level of schooling for each remoteness category. That sentence will  make more sense when you see Figure 16 below. Here’s the code, followed by the figure:

#proportion plot
p <- ggplot(data=nsw_schools) +
geom_bar(mapping = aes(x=ASGS_remoteness,fill=level_of_schooling),position = “fill”)+
theme(axis.text.x=element_text(angle=45, hjust=1))
#display plot

Obviously,  we lose frequency information since the bar heights are normalised.

Figure 16: Proportions of school levels for remoteness categories

An  interesting feature here is that  the proportion of central and community schools increases with remoteness. Unlike primary and secondary schools, central / community schools provide education from Kindergarten through Year 12. As remote areas have smaller numbers of students, it makes sense to consolidate educational resources in institutions that provide schooling at all levels .

Finally, to close the loop so to speak,  let’s revisit our very first plot in Figure 1 and try to simplify it in another way. We’ll use faceting to  split it out into separate plots, one per remoteness category. First, we’ll organise the subplots horizontally using facet_grid:

#faceting – subplots laid out horizontally (faceted variable on right of formula)
p <- ggplot(data=nsw_schools) + geom_point(mapping = aes(x=student_number,y=ICSEA_Value))+
#display plot

The plot is shown in Figure 17 in which the different remoteness categories are presented in separate plots (facets) against a common y axis. It shows, the sharp differences between student numbers between remote and other regions.

Figure 17: Horizontally laid out facet plots of ICSEA scores for different remoteness categories

To get a vertically laid out plot, switch the faceted variable to other side of the formula (left as an exercise for you).

If one has too many categories to fit into a single row, one can wrap the facets using facet_wrap like so:

#faceting – wrapping facets in 2 columns
p <- ggplot(data=nsw_schools) +
geom_point(mapping = aes(x=student_number,y=ICSEA_Value))+
facet_wrap(~ASGS_remoteness, ncol= 2)
#display plot

The resulting plot is shown in Figure 18.

Figure 18: Same data as in Figure 17, with facets wrapped in a 2 column format

One can specify the number of rows instead of columns. I won’t illustrate that as the change in syntax is quite obvious.

…and I think that’s a good place to stop.

Wrapping up

Data visualisation has a reputation of being a dark art, masterable only by the visually gifted. This may have been partially true some years ago, but in this day and age it definitely isn’t. Versatile  packages such as ggplot, that use a consistent syntax have made  the art much more accessible to visually ungifted folks like myself. In this post I have attempted to provide a brief and (hopefully) logical introduction to ggplot.  In closing I note that  although some of the illustrative examples  violate the  principles of good data visualisation, I hope this article will serve its primary purpose which is pedagogic rather than artistic.

Further reading:

Where to go for more? Two of the best known references are Hadley Wickham’s books:

I highly recommend his R for Data Science , available online here. Apart from providing a good overview of ggplot, it is an excellent introduction to R for data scientists.  If you haven’t read it, do yourself a favour and buy it now.

People tell me his ggplot book is an excellent book for those wanting to learn the ins and outs of ggplot . I have not read it myself, but if his other book is anything to go by, it should be pretty damn good.

Written by K

October 10, 2017 at 8:17 pm

The two tributaries of time

with 2 comments

How time flies. Ten years ago this month, I wrote my first post on Eight to Late.  The anniversary gives me an excuse to post something a little different. When rummaging around in my drafts folder for something suitable, I came across this piece that I wrote some years ago (2013) but didn’t publish.   It’s about our strange relationship with time, which I thought makes it a perfect piece to mark the occasion.


The metaphor of time as a river resonates well with our subjective experiences of time.  Everyday phrases that evoke this metaphor include the flow of time and time going by, or the somewhat more poetic currents of time.  As Heraclitus said, no [person] can step into the same river twice – and so it is that a particular instant in time …like right now…is ephemeral, receding into the past as we become aware of it.

On the other hand, organisations have to capture and quantify time because things have to get done within fixed periods, the financial year being a common example. Hence, key organisational activities such as projects, strategies and budgets are invariably time-bound affairs. This can be problematic because there is a mismatch between the ways in which organisations view time and individuals experience it.

Organisational time

The idea that time is an objective entity is most clearly embodied in the notion of a timeline: a graphical representation of a time period, punctuated by events. The best known of these is perhaps the ubiquitous Gantt Chart, loved (and perhaps equally, reviled) by managers the world over.

Timelines are interesting because, as Elaine Yakura states in this paper, “they seem to render time, the ultimate abstraction, visible and concrete.”   As a result, they can serve as boundary objects that make it possible to negotiate and communicate what is to be accomplished in the specified time period. They make this possible because they tell a story with a clear beginning, middle and end, a narrative of what is to come and when.

For the reasons mentioned in the previous paragraph, timelines are often used to manage time-bound organisational initiatives. Through their use in scheduling and allocation, timelines serve to objectify time in such a way that it becomes a resource that can be measured and rationed, much like other resources such as money, labour etc.

At our workplaces we are governed by many overlapping timelines – workdays, budgeting cycles and project schedules being examples. From an individual perspective, each of these timelines are different representations of how one’s time is to be utilised, when an activity should be started and when it must be finished. Moreover, since we are generally committed to multiple timelines, we often find ourselves switching between them. They serve to remind us what we should be doing and when.

But there’s more: one of the key aims of developing a timeline is to enable all stakeholders to have a shared understanding of time as it pertains to the initiative. In this view, a timeline is a consensus representation of how a particular aspect of the future will unfold.  Timelines thus serve as coordinating mechanisms.

In terms of the metaphor, a timeline is akin to a map of the river of time. Along the map we can measure out and apportion it; we can even agree about way-stops at various points in time. However, we should always be aware that it remains a representation of time, for although we might treat a timeline as real, the fact is no one actually experiences time as it is depicted in a timeline. Mistaking one for the other is akin to confusing the map with the territory.

This may sound a little strange so I’ll try to clarify.  I’ll start with the observation that we experience time through events and processes – for example the successive chimes of a clock, the movement of the second hand of a watch (or the oscillations of a crystal), the passing of seasons or even the greying of one’s hair. Moreover, since these events and processes can be objectively agreed on by different observers, they can also be marked out on a timeline.  Yet the actual experience of living these events is unique to each individual.

Individual perception of time

As we have seen, organisations treat time as an objective commodity that can be represented, allocated and used much like any tangible resource.  On the other hand our experience of time is intensely personal.  For example, I’m sitting in a cafe as I write these lines. My perception of the flow of time depends rather crucially on my level of engagement in writing: slow when I’m struggling for words but zipping by when I’m deeply involved. This is familiar to us all: when we are deeply engaged in an activity, we lose all sense of time but when our involvement is superficial we are acutely aware of the clock.

This is true at work as well. When I’m engaged in any kind of activity at work, be it a group activity such as a meeting, or even an individual one such as developing a business case, my perception of time has little to do with the actual passage of seconds, minutes and hours on a clock. Sure, there are things that I will do habitually at a particular time – going to lunch, for example – but my perception of how fast the day goes is governed not by the clock but by the degree of engagement with my work.

I can only speak for myself, but I suspect that this is the case with most people. Though our work lives are supposedly governed by “objective” timelines, the way we actually live out our workdays depends on a host of things that have more to do with our inner lives than visible outer ones.  Specifically, they depend on things such as feelings, emotions, moods and motivations.

Flow and engagement

OK, so you may be wondering where I’m going with this. Surely, my subjective perception of my workday should not matter as long as I do what I’m required to do and meet my deadlines, right?

As a matter of fact, I think the answer to the above question is a qualified, “No”. The quality of the work we do depends on our level of commitment and engagement. Moreover, since a person’s perception of the passage of time depends rather sensitively on the degree of their involvement in a task, their subjective sense of time is a good indicator of their engagement in work.

In his book, Finding Flow, Mihalyi Csikszentmihalyi describes such engagement as an optimal experience in which a person is completely focused on the task at hand.  Most people would have experienced flow when engaged in activities that they really enjoy. As Anthony Reading states in his book, Hope and Despair: How Perceptions of the Future Shape Human Behaviour, “…most of what troubles us resides in our concerns about the past and our apprehensions about the future.”  People in flow are entirely focused on the present and are thus (temporarily) free from troubling thoughts. As Csikszentmihalyi puts it, for such people, “the sense of time is distorted; hours seem to pass by in minutes.”

All this may seem far removed from organisational concerns, but it is easy to see that it isn’t: a Google search on the phrase “increase employee engagement” will throw up many articles along the lines of “N ways to increase employee engagement.”  The sense in which the term is used in these articles is essentially the same as the one Csikszentmihalyi talks about: deep involvement in work.

So, the advice of management gurus and business school professors notwithstanding, the issue is less about employee engagement or motivation than about creating conditions that are conducive to flow.   All that is needed for the latter is a deep understanding how the particular organisation functions, the task at hand and (most importantly) the people who will be doing it.  The best managers I’ve worked with have grokked this, and were able to create the right conditions in a seemingly effortless and unobtrusive way. It is a skill that cannot be taught, but can be learnt by observing how such managers do what they do.

Time regained

Organisations tend to treat their employees’ time as though it were a commodity or resource that can be apportioned and allocated for various tasks. This view of time is epitomised by the timeline as depicted in a Gantt Chart or a resource-loaded project schedule.

In contrast, at an individual level, the perception of time depends rather critically on the level of engagement that a person feels with the task he or she is performing. Ideally organisations would (or ought to!) want their employees to be in that optimal zone of engagement that Csikszentmihalyi calls flow, at least when they are involved in creative work. However, like spontaneity, flow is a state that cannot be achieved by corporate decree; the best an organisation can do is to create the conditions that encourage it.

The organisational focus on timelines ought to be balanced by actions that are aimed at creating the conditions that are conducive to employee engagement and flow.  It may then be possible for those who work in organisation-land to experience, if only fleetingly, that Blakean state in which eternity is held in an hour.

Written by K

September 20, 2017 at 9:17 pm

A gentle introduction to logistic regression and lasso regularisation using R

with 11 comments

In this day and age of artificial intelligence and deep learning, it is easy to forget that simple algorithms can work well for a surprisingly large range of practical business problems.  And the simplest place to start is with the granddaddy of data science algorithms: linear regression and its close cousin, logistic regression. Indeed, in his acclaimed MOOC and accompanying textbook, Yaser Abu-Mostafa spends a good portion of his time talking about linear methods, and with good reason too: linear methods are not only a good way to learn the key principles of machine learning, they can also be remarkably helpful in zeroing in on the most important predictors.

My main aim in this post is to provide a beginner level introduction to logistic regression using R and also introduce LASSO (Least Absolute Shrinkage and Selection Operator), a powerful feature selection technique that is very useful for regression problems. Lasso is essentially a regularization method. If you’re unfamiliar with the term, think of it as a way to reduce overfitting using less complicated functions (and if that means nothing to you, check out my prelude to machine learning).  One way to do this is to toss out less important variables, after checking that they aren’t important.  As we’ll discuss later, this can be done manually by examining p-values of coefficients and discarding those variables whose coefficients are not significant. However, this can become tedious for classification problems with many independent variables.  In such situations, lasso offers a neat way to model the dependent variable while automagically selecting significant variables by shrinking the coefficients of unimportant predictors to zero.  All this without having to mess around with p-values or obscure information criteria. How good is that?

Why not linear regression?

In linear regression one attempts to model a dependent variable (i.e. the one being predicted) using the best straight line fit to a set of predictor variables.  The best fit is usually taken to be one that minimises the root mean square error,  which is the sum of square of the differences between the actual and predicted values of the dependent variable. One can think of logistic regression as the equivalent of linear regression for a classification problem.  In what follows we’ll look at binary classification – i.e. a situation where the dependent variable takes on one of two possible values (Yes/No, True/False, 0/1 etc.).

First up, you might be wondering why one can’t use linear regression for such problems. The main reason is that classification problems are about determining class membership rather than predicting variable values, and linear regression is more naturally suited to the latter than the former. One could, in principle, use linear regression for situations where there is a natural ordering of categories like High, Medium and Low for example. However, one then has to map sub-ranges of the predicted values to categories. Moreover, since predicted values are potentially unbounded (in data as yet unseen) there remains a degree of arbitrariness associated with such a mapping.

Logistic regression sidesteps the aforementioned issues by modelling class probabilities instead.  Any input to the model yields a number lying between 0 and 1, representing the probability of class membership. One is still left with the problem of determining the threshold probability, i.e. the probability at which the category flips from one to the other.  By default this is set to p=0.5, but in reality it should be settled based on how the model will be used.  For example, for a marketing model that identifies potentially responsive customers, the threshold for a positive event might be set low (much less than 0.5) because the client does not really care about mailouts going to a non-responsive customer (the negative event). Indeed they may be more than OK with it as there’s always a chance – however small – that a non-responsive customer will actually respond.  As an opposing example, the cost of a false positive would be high in a machine learning application that grants access to sensitive information. In this case, one might want to set the threshold probability to a value closer to 1, say 0.9 or even higher. The point is, the setting an appropriate threshold probability is a business issue, not a technical one.

Logistic regression in brief

So how does logistic regression work?

For the discussion let’s assume that the outcome (predicted variable) and predictors are denoted by Y and X respectively and the two classes of interest are denoted by + and – respectively.  We wish to model the conditional probability that the outcome Y is +, given that the input variables (predictors) are X. The conditional probability is denoted by p(Y=+|X)   which we’ll abbreviate as p(X) since we know we are referring to the positive outcome Y=+.

As mentioned earlier, we are after the probability of class membership so we must ensure that the hypothesis function (a fancy word for the model) always lies between 0 and 1. The function assumed in logistic regression is:

p(X) = \dfrac{\exp^{\beta_0+\beta_1 X}}{1+\exp^{\beta_0 + \beta_1 X}} .....(1)

You can verify that p(X) does indeed lie between 0 and  1 as X varies from -\infty to \infty.  Typically, however, the values of X that make sense are bounded as shown in the example (stolen from Wikipedia) shown in Figure 1. The figure also illustrates the typical S-shaped  curve characteristic of logistic regression.

Figure 1: Logistic function

As an aside, you might be wondering where the name logistic comes from. An equivalent way of expressing the above equation is:

\log(\dfrac{p(X)}{1-p(X)}) = \beta_0+\beta_1 X .....(2)

The quantity on the left is the logarithm of the odds. So, the model is a linear regression of the log-odds, sometimes called logit, and hence the name logistic.

The problem is to find the values of \beta_0  and \beta_1 that results in a p(X) that most accurately classifies all the observed data points – that is, those that belong to the positive class have a probability as close as possible to 1 and those that belong to the negative class have a probability as close as possible to 0. One way to frame this problem is to say that we wish to maximise the product of these probabilities, often referred to as the likelihood:

\displaystyle\log ( {\prod_{i:Y_i=+} p(X_{i}) \prod_{j:Y_j=-}(1-p(X_{j}))})

Where \prod represents the products over i and j, which run over the +ve and –ve classed points respectively. This approach, called maximum likelihood estimation, is quite common in many machine learning settings, especially those involving probabilities.

It should be noted that in practice one works with the log likelihood because it is easier to work with mathematically. Moreover, one minimises the negative  log likelihood which, of course, is the same as maximising the log likelihood.  The quantity one minimises is thus:

L = - \displaystyle\log ( {\prod_{i:Y_i=+} p(X_{i}) \prod_{j:Y_j=-}(1-p(X_{j}))}).....(3)

However, these are technical details that I mention only for completeness. As you will see next, they have little bearing on the practical use of logistic regression.

Logistic regression in R – an example

In this example, we’ll use the logistic regression option implemented within the glm function that comes with the base R installation. This function fits a class of models collectively known as generalized linear models. We’ll apply the function to the Pima Indian Diabetes dataset that comes with the mlbench package. The code is quite straightforward – particularly if you’ve read earlier articles in my “gentle introduction” series – so I’ll just list the code below  noting that the logistic regression option is invoked by setting family=”binomial”  in the glm function call.

Here we go:

#set working directory if needed (modify path as needed)
#load required library
#load Pima Indian Diabetes dataset
#set seed to ensure reproducible results
#split into training and test sets
PimaIndiansDiabetes[,”train”] <- ifelse(runif(nrow(PimaIndiansDiabetes))<0.8,1,0)
#separate training and test sets
trainset <- PimaIndiansDiabetes[PimaIndiansDiabetes$train==1,]
testset <- PimaIndiansDiabetes[PimaIndiansDiabetes$train==0,]
#get column index of train flag
trainColNum <- grep(“train”,names(trainset))
#remove train flag column from train and test sets
trainset <- trainset[,-trainColNum]
testset <- testset[,-trainColNum]
#get column index of predicted variable in dataset
typeColNum <- grep(“diabetes”,names(PimaIndiansDiabetes))
#build model
glm_model <- glm(diabetes~.,data = trainset, family = binomial)
glm(formula = diabetes ~ ., family = binomial, data = trainset)
<<output edited>>
            Estimate  Std. Error z value Pr(>|z|)
(Intercept)-8.1485021 0.7835869 -10.399  < 2e-16 ***
pregnant    0.1200493 0.0355617   3.376  0.000736 ***
glucose     0.0348440 0.0040744   8.552  < 2e-16 ***
pressure   -0.0118977 0.0057685  -2.063  0.039158 *
triceps     0.0053380 0.0076523   0.698  0.485449
insulin    -0.0010892 0.0009789  -1.113  0.265872
mass        0.0775352 0.0161255   4.808  1.52e-06 ***
pedigree    1.2143139 0.3368454   3.605  0.000312 ***
age         0.0117270 0.0103418   1.134  0.256816
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#predict probabilities on testset
#type=”response” gives probabilities, type=”class” gives class
glm_prob <- predict.glm(glm_model,testset[,-typeColNum],type=”response”)
#which classes do these probabilities refer to? What are 1 and 0?
neg 0
pos 1
#make predictions
##…first create vector to hold predictions (we know 0 refers to neg now)
glm_predict <- rep(“neg”,nrow(testset))
glm_predict[glm_prob>.5] <- “pos”
#confusion matrix
glm_predict  neg pos
        neg    90 22
        pos     8 33
[1] 0.8039216


Although this seems pretty good, we aren’t quite done because there is an issue that is lurking under the hood. To see this, let’s examine the information output from the model summary, in particular the coefficient estimates (i.e. estimates for \beta) and their significance. Here’s a summary of the information contained in the table:

  • Column 2 in the table lists coefficient estimates.
  • Column 3 list s the standard error of the estimates (the larger the standard error, the less confident we are about the estimate)
  • Column 4 the z statistic (which is the coefficient estimate (column 2) divided by the standard error of the estimate (column 3)) and
  • The last column (Pr(>|z|) lists the p-value, which is the probability of getting the listed estimate assuming the predictor has no effect. In essence, the smaller the p-value, the more significant the estimate is likely to be.

From the table we can conclude that only 4 predictors are significant – pregnant, glucose, mass and pedigree (and possibly a fifth – pressure). The other variables have little predictive power and worse, may contribute to overfitting.  They should, therefore, be eliminated and we’ll do that in a minute. However, there’s an important point to note before we do so…

In this case we have only 9 variables, so are able to identify the significant ones by a manual inspection of p-values.  As you can well imagine, such a process will quickly become tedious as the number of predictors increases. Wouldn’t it be be nice if there were an algorithm that could somehow automatically shrink the coefficients of these variables or (better!) set them to zero altogether?  It turns out that this is precisely what  lasso and its close cousin, ridge regression, do.

Ridge and Lasso

Recall that the values of the logistic regression coefficients \beta_0  and \beta_1 are found by minimising the negative log likelihood described in equation (3).  Ridge and lasso regularization work by adding a penalty term to the log likelihood function.  In the case of ridge regression, the penalty term is \beta_1^2 and in the case of lasso, it is |\beta_1| (Remember, \beta_1  is a vector, with as many components as there are predictors).  The quantity to be minimised in the two cases is thus:

L +\lambda \sum \beta_1^2.....(4) – for ridge regression,


L +\lambda \sum |\beta_1|.....(5) – for lasso regression.

Where \lambda is a free parameter which is usually selected in such a way that the resulting model minimises the out of sample error. Typically, the optimal value of \lambda is found using grid search with cross-validation, a process akin to the one described in my discussion on cost-complexity parameter  estimation in decision trees. Most canned algorithms provide methods to do this; the one we’ll use in the next section is no exception.

In the case of ridge regression, the effect of the penalty term is to shrink the coefficients that contribute most to the error. Put another way, it reduces the magnitude of the coefficients that contribute to increasing L.  In contrast, in  the case of lasso regression, the effect of the penalty term is to set the these coefficients exactly to zero! This is cool because what it mean that lasso regression works like a feature selector that picks out the most important coefficients, i.e. those that are most predictive (and have the lowest p-values).

Let’s illustrate this through an example. We’ll use the glmnet package which implements a combined version of ridge and lasso (called elastic net). Instead of minimising (4) or (5) above, glmnet minimises:

L +\lambda[ (1-\alpha)\sum [\beta_1^2 + \alpha\sum|\beta_1|]....(6)

where \alpha controls the “mix” of ridge and lasso regularisation, with \alpha=0 being “pure” ridge and  \alpha=1 being “pure” lasso.

Lasso regularisation using glmnet

Let’s reanalyse the Pima Indian Diabetes dataset using glmnet with \alpha=1 (pure lasso). Before diving into code, it is worth noting that glmnet:

  • does not have a formula interface, so one has to input the predictors as a matrix and the class labels as a vector.
  • does not accept categorical predictors, so one has to convert these to numeric values before passing them to glmnet.

The glmnet function model.matrix creates the matrix and also converts categorical predictors to appropriate dummy variables.

Another important point to note is that we’ll use the function cv.glmnet, which automatically performs a grid search to find the optimal value of \lambda.

OK, enough said, here we go:

#load required library
#convert training data to matrix format
x <- model.matrix(diabetes~.,trainset)
#convert class to numerical variable
y <- ifelse(trainset$diabetes==”pos”,1,0)
#perform grid search to find optimal value of lambda
#family= binomial => logistic regression, alpha=1 => lasso
# check docs to explore other type.measure options
cv.out <- cv.glmnet(x,y,alpha=1,family=”binomial”,type.measure = “mse” )
#plot result


The plot is shown in Figure 2 below:

Figure 2: Error as a function of lambda (select lambda that minimises error)

The plot shows that the log of the optimal value of lambda (i.e. the one that minimises the root mean square error) is approximately -5. The exact value can be viewed by examining the variable lambda_min in the code below. In general though, the objective of regularisation is to balance accuracy and simplicity. In the present context, this means a model with the smallest number of coefficients that also gives a good accuracy.  To this end, the cv.glmnet function  finds the value of lambda that gives the simplest model but also lies within one standard error of the optimal value of lambda. This value of lambda (lambda.1se) is what we’ll use in the rest of the computation. Interested readers should have a look at this article for more on lambda.1se vs lambda.min.

#min value of lambda
lambda_min <- cv.out$lambda.min
#best value of lambda
lambda_1se <- cv.out$lambda.1se
#regression coefficients
10 x 1 sparse Matrix of class “dgCMatrix”
(Intercept) -4.61706681
(Intercept)  .
pregnant     0.03077434
glucose      0.02314107
pressure     .
triceps      .
insulin      .
mass         0.02779252
pedigree     0.20999511
age          .


The output shows that only those variables that we had determined to be significant on the basis of p-values have non-zero coefficients. The coefficients of all other variables have been set to zero by the algorithm! Lasso has reduced the complexity of the fitting function massively…and you are no doubt wondering what effect this  has on accuracy. Let’s see by running the model against our test data:


#get test data
x_test <- model.matrix(diabetes~.,testset)
#predict class, type=”class”
lasso_prob <- predict(cv.out,newx = x_test,s=lambda_1se,type=”response”)
#translate probabilities to predictions
lasso_predict <- rep(“neg”,nrow(testset))
lasso_predict[lasso_prob>.5] <- “pos”
#confusion matrix
pred  neg pos
   neg 94 28
  pos  4 27
[1] 0.7908497


Which is a bit less than what we got with the more complex model. So, we get  a similar out-of-sample accuracy as we did before, and we do so using a way simpler function (4 non-zero coefficients) than the original one (9  nonzero coefficients). What this means is that the simpler function does at least as good a job fitting the signal in the data as the more complicated one.  The bias-variance tradeoff tells us that the simpler function should be preferred because it is less likely to overfit the training data.

Paraphrasing William of Ockhamall other things being equal, a simple hypothesis should be preferred over a complex one.

Wrapping up

In this post I have tried to provide a detailed introduction to logistic regression, one of the simplest (and oldest) classification techniques in the machine learning practitioners arsenal. Despite it’s simplicity (or I should say, because of it!) logistic regression works well for many business applications which often have a simple decision boundary. Moreover, because of its simplicity it is less prone to overfitting than flexible methods such as decision trees. Further, as we have shown, variables that contribute to overfitting can be eliminated using lasso (or ridge) regularisation, without compromising out-of-sample accuracy. Given these advantages and its inherent simplicity, it isn’t surprising that logistic regression remains a workhorse for data scientists.

Written by K

July 11, 2017 at 10:00 pm

The improbability of success

with 2 comments

Anyone who has tidied up after a toddler intuitively understands that making a mess is far easier than creating order. The fundamental reason for this is that the number of messy states in the universe (or a toddler’s room) far outnumbers the ordered ones.  As this point might not be obvious, I’ll demonstrate it via a simple thought experiment involving marbles:

Throw three marbles onto a flat surface.  When the marbles come to rest, you are most likely to end up with a random configuration  as in Figure 1.

Figure 1: A random configuration of 3 marbles

Indeed, you’d be extremely surprised if the three ended up being collinear as in Figure 2.   Note that Figure 2 is just one example of many collinear possibilities, but the point I’m making is that if the marbles are thrown randomly, they are more likely to end up in a random state than a lined-up one.

Figure 2: an unlikely (ordered) configuration

This raises a couple of questions:

Question: On what basis can one claim that the collinear configuration is tidier or more ordered than the non-collinear one?

Naive answer:  It looks more ordered. Yes, tidiness is in the eye of the beholder so it is necessarily subjective. However, I’ll wager that if one took a poll, an overwhelming number of people would say that the configuration in Figure 2 is more ordered than the one in Figure 1.

More sophisticated answer : The “state” of collinear marbles can be described using 2 parameters, the slope and intercept of the straight line that three marbles lie on (in any coordinate system) whereas the description of the nonlinear state requires 3 parameters. The first state is tidier because it requires fewer parameters.  Another way to think about is that the line can be described by two marbles; the third one is redundant as far as the description of the state is concerned.

Question: Why is a tidier configuration less likely than a messy one?

Answer:  May be you see this intuitively and need no proof, but here’s one just in case. Imagine rolling the three marbles one after the other. The first two, regardless of where they end up, will necessarily lie along a line (two points lie on the straight line joining them). Now, I think it is easy to see that if we throw the third marble randomly, it is highly unlikely end up on that line. Indeed, for the third marble to end up exactly on the same straight line requires a coincidence of near cosmic proportions.

I know, I know, this is not a proof, but I trust it makes the point.

Now, although it is near impossible to get to a collinear end state via random throws, it is possible to approximate it by changing the way we throw the marbles. Here’s how:

  1. Throw the marbles consecutively rather than in one go.
  2. When throwing the third marble, adjust its initial speed and direction in a way that takes into account the positions of the two marbles that are already on the surface. Remember these two already define a straight line.

The third throw is no longer random because it is designed to maximise the chance that the last marble will get as close as possible to the straight line defined by the first two. Done right, you’ll end up with something closer to the configuration in Figure 3 rather than the one in Figure 2.

Figure 3: an “approximately ordered” state

Now you’re probably wondering what this has to do with success. I’ll make the connection via an example that will be familiar to many readers of this blog: an organisation’s strategy. However, as I will reiterate later, the arguments I present are very general and can be applied to just about any initiative or situation.

Typically, a strategy sets out goals for an organisation and a plan to achieve them in a specified timeframe. The goals define a number of desirable outcomes, or states which, by design, are constrained to belong to a (very) small subset of all possible states the organisation can end up in.  In direct analogy with the simple model discussed above it is clear that, left to its own devices, the organisation is more likely to end up in one of the much overwhelmingly larger number of “failed states” than one of the successful ones.  Notwithstanding the popular quote about there being many roads to success, in reality there are a great many more roads to failure.

Of course, that’s precisely why organisations are never “left to their own devices.” Indeed, a strategic plan specifies actions that are intended to make a successful state more likely than an unsuccessful one. However, no plan can guarantee success; it can, at best, make it more likely. As in the marble game, success is ultimately a matter of chance, even when we take actions to make it more likely.

If we accept this, the key question becomes: how can one design a strategy that improves the odds of success?  The marble analogy suggests a way to do this is to:

  1. Define success in terms of an end state that is a natural extension of your current state.
  2. Devise a plan to (approximately) achieve that end state. Such a plan will necessarily build on the current state rather than change it wholesale. Successful change is an evolutionary process rather than a revolutionary one.

My contention is that these points are often ignored by management strategists. More often than not, they will define an end state based on a textbook idealisation, consulting model or (horror!) best practice. The marble analogy shows why copying others is unlikely to succeed.

Figure 4 shows a variant of the marble game in which we have two sets of marbles (or organisations!), one blue, as before, and the other red.

Figure 4: Two distinct configurations of marbles (or organisations)

Now, it is considerably harder to align an additional marble with both sets of marbles than the blue one alone. Here’s why…

To align with both sets, the new marble has to end up close to the point that lies at the intersection of the blue and red lines in Figure 5. In contrast, to align with the blue set alone, all that’s needed is for it to get close to any point on the blue line.


Figure 5: Why copying others is not a good idea (see text for explanation)

Finally, on a broader note, it should be clear that the arguments made above go beyond organisational strategies. They apply to pretty much any planned action, whether at work or in one’s personal life.

So, to sum up: when developing an organisational (or personal) strategy, the first step is to understand where you are and then identify the minimal actions you need to take in order to get to an “improved” state that is consistent with  your current one. Yes, this is akin to the incremental and evolutionary approach that Agilistas and Leaners have been banging on about for years. However, their prescriptions focus on specific areas: software development and process improvement.  My point is that the basic principles are way broader because they are a direct consequence of a fundamental fact regarding the relative likelihood of order and disorder in a toddler’s room, an organisation, or even the universe at large.

Written by K

April 4, 2017 at 9:16 pm

%d bloggers like this: