Eight to Late

Sensemaking and Analytics for Organizations

Data science and sensemaking – tales from two hackathons

with 5 comments

It isn’t that they can’t see the solution. It is that they can’t see the problem” – GK Chesterton

Introduction

Examples of vendor-generated hype about data science are not hard to find,   I found one on the very first site I visited:  a large technology and services vendor who, in their own words, claim their analytics solutions help organisations “engage with data to answer the toughest business questions, uncover patterns and pursue breakthrough ideas.”  I’ve deliberately avoided linking to the guilty party because there are many others that spout similar rhetoric.

Unfortunately it seems to work:  according to Gartner, “by 2020, predictive and prescriptive analytics will attract 40% of enterprises’ net new investment in business intelligence and analytics.” This trend is accompanied by a concomitant increase in demand for data science education, fuelled by  remarks along the lines that data science is “The Sexiest Job of the 21st Century.”

By and large, data science education tends to focus on algorithms and technology, but its practice involves much more. The vendor who claims that technology can help organisations grapple with “toughest business questions” and “pursue breakthrough ideas” is singularly silent about where these questions or ideas come from. Data is meaningless without a meaningful hypothesis.  Problem is, in the real world questions or hypotheses aren’t obvious; one has to work to formulate them. As the management icon Russell Ackoff once said, “Outside of school, problems are seldom given; they have to be taken, extracted from complex situations…”

The art of taking problems is what sensemaking is all about.

Unfortunately, it is a skill that is typically ignored by data science educators.

Why?

Probably because it is hard to teach…but the good news is that it can be learnt. Like most tacit skills, sensemaking is best learnt by doing, that is, by formulating problems in real-world situations.  Before I get to that, however, let’s take a brief detour.

Real world problems are characterised by ambiguity

An important aspect of real-world problems – as opposed to classroom ones – is that they are invariably fraught with ambiguity. For example, a customer’s requirements may be vague or the available data incomplete and messy. What this means is that there is no guarantee one will be able to formulate a well-posed problem, let alone get a useful answer.   Worse, unlike a risk-based situation in which uncertainty can be quantified, one cannot even figure out the odds of success.

The human brain processes quantifiable uncertainty (aka risk) and ambiguity very differently. The former, which can be calculated, is dealt with by the prefrontal cortex which is responsible for decision making and goal-oriented thinking. Ambiguity, on the other hand, is processed by the amygdala, which deals with emotions.  The upshot of this is that ambiguity evokes an emotional response, the most common one being anxiety.

Although some people are innately better at coping with anxiety than others, it is possible to get better at it by repeatedly putting oneself in high-pressure (yet safe) situations that are ambiguous.  For data science students, hackathons provide a perfect opportunity to do this.

Ambiguity in data science – tales from two hackathons

Over the last two months, I’ve had the privilege of being a part of the Master of Data Science Innovation (MDSI) program run by the Connected Intelligence Centre at UTS.   The course director, Theresa Anderson, sees hackathons as a great way for students to learn how to handle ambiguity.  So, apart from regular coursework assignments, students are encouraged to participate in external hackathons sponsored by industry and government organisations.   This gives them opportunities to gain practical experience in formulating problems in ambiguous and high-pressure environments.

Datacake at GovHack

A few MDSI student teams participated in a GovHack event earlier this year. Here’s what William Azevedo,  a member of team that called themselves Datacake, wrote about his team’s problem formulation journey at the event :

The challenge is simple: the competitors should form teams, identify a problem and use data from government agencies from Australia and New Zealand to present a solution to the problem. Naturally, this solution should bring some benefit to the society.

I’m not sure I’d use the word simple…but the importance of problem formulation comes through quite clearly.  Here’s how he and his team (called Datacake) went about it:

 As a starting point, our team published an online survey to understand how safe people feel when walking on the streets, especially at night. As we didn’t have much time, we spread the message via social networks. In a couple of hours, we received 44 answers. It gave us enough information to back our idea.

Notice the process used in defining the problem – the team realised they did not know enough to define a meaningful problem so they went and got relevant data. Following this:

Our team analysed the answers of the survey, engaged in passionate discussions, took tips from the mentors, had lots of coffee and designed some cool diagrams on the blackboard.

…and then his description of the Aha moment when a good idea emerged:

Then the magic happened. We had this idea of merging information about crime, demographics, weather, land zoning and street illumination to provide a map of the safe and unsafe areas within a suburb.

An important point is that sensemaking is best done collaboratively. Since the problem is ambiguous or even undefined (as in this case) no individual has a privileged access to the “truth.” It is therefore important to bring diverse perspectives to bear on the problem. Indeed, sensemaking may be thought of as collaborative problem formulation and solving. In view of this it is interesting to hear what other members of Team Datacake had to say about their problem formulation process.  Here’s a comment from Anthony So:

During the whole weekend we really forced ourselves to go deep and asked “Why is it happening? Why is it happening? Why is it happening?” every time we found an interesting pattern. We really wanted to understand the true root causes of those accidents. We didn’t want to stay at a descriptive level. We knew the answers were behavioural. We knew there were multiple problems and therefore require different answers and solutions. We did different techniques to do so: machine learning, stats, data visualisation. It didn’t matter which we used the only important point was how can we get to answers of those questions.

The specific area they looked at was pedestrian safety. They found that obvious variables, such as driver fatigue and hazards were not significant, so they started looking for other potential factors. Here’s how Anthony put it:

For instance we built a classification model on the severity of the accidents involving children but we didn’t use it to make predictions. We used it to identify the important features (and unimportant) for those cases. We found out that some of the variables related to the environment (Primary_hazardous_feature, Surface_condition, Weather…) and to the drivers (Fatigue_involved_in_crash…) were not important. This gave us a good indication that those accidents are mostly related directly to the behaviour of the children. So we kept diving further and further and found 3 postcodes with higher numbers of accidents than others. We focused on those 3 areas and we kept going deeper and deeper…

In the end Datacake came up with a few suggestions for improving pedestrian safety. They were awarded a prize for their efforts, so the problem they formulated and solved was clearly valuable to the sponsors.

Peppermoney Hackathon

A couple of weekends ago, Pepper Money, Australia’s largest non-bank lender sponsored a day long internal hackathon for MDSI students, with a hefty winner-take-all prize as an incentive. The challenge was quite open-ended, and had to do with helping the organisation develop a consistent brand voice. Participants were given a small corpus of text files from the organisation’s public and social media sites and were given very general guidelines on how to proceed. Details were left entirely to the teams.

As one might expect, most teams spent the first few hours struggling to define a relevant and tractable problem – relevance being paramount for the client and tractability for the teams.  Being a mentor at the event, I was able observe how different teams handled this. Among other things, I was particularly impressed by how some teams with very little text mining experience were able to – in a few hours – come up with a good problem, an approach to solve it…and, most importantly, make decent progress by day’s end.

I won’t go into details except to say that the approaches were diverse, ranging from the somewhat philosophical to the very technical. A couple of examples:

I was amazed at the diversity of solutions the groups came up with, and so were the other mentors and the sponsor. Blair Hudson, Innovation Portfolio Manager at Pepper Money, summed the day up very well when he said:

#PepxUTS was our first hackathon event, challenging students to build data science solutions in a day to allow everyone at Pepper to communicate using a consistent brand voice. Our Co-Group CEOs both joined in for judging and awarded the winners. It was a rewarding day for all involved

(For some vignettes from the day, check out the #PepxUTS hashtag on Twitter.)

The day’s experiences left me ever more convinced that hackathons are an excellent vehicle for learning and demonstrating the practical utility of sensemaking skills.

Wrapping up

The two case studies highlight the benefits of sensemaking skills, both for students and organisations.  On the one hand,  students who participated got valuable experience in formulating problems collaboratively in high-pressure, high-ambiguity situations. This is a skill that cannot be learnt in classrooms, MOOCs or even in online data challenges (like Kaggle) where problems tend to be clearly defined. On the other hand, sponsoring organisations have benefited from new insights into longstanding problems.

Finally, it should be clear that although I’ve focused on educational settings,  what I’ve said for students applies to organisational settings too: there’s nothing to stop organisations from using hackathons as a means to help their employees learn sensemaking skills.

To conclude, the main point I want to make is that the most important situations we encounter at work (and even in our personal lives) are usually fraught with ambiguity. Our first reaction is to jump into problem solving mode because it feels like the right thing to do. In reality, one is generally better off stepping back and taking the time to think the situation through, preferably with a group of diversely skilled individuals. All too often this sensemaking step is neglected, and teams end up solving an irrelevant problem.

To paraphrase Chesterton, in order to see the right solution, one must first see the right problem.

Acknowledgements

Many thanks to Blair Hudson, William Azevedo and Anthony So for their contributions to this piece.

Written by K

October 18, 2016 at 6:23 pm

5 Responses

Subscribe to comments with RSS.

  1. […] Kailash Awati case studies two examples of sensemaking using data science from two hackathons. […]

    Like

  2. Kailash, thank you for another interesting post. I would like to know if there are any published texts you would recommend (apart from your own texts, which I have already read) that provide a very good analysis and explanation of what you call sensemaking (even if the texts do not use that term).

    Going back a century, Charles Sanders Peirce’s (1839–1914) notion of abduction or abductive reasoning (which he opposed to both deduction and induction) seems very close to the problem-structuring aspect of sensemaking. Some of John Dewey’s (1859–1952) later books seem relevant, such as How We Think (2nd ed. 1933) and Logic: The Theory of Inquiry (1938). Sami Paavola recently compared Peirce’s notion of abduction and Dewey’s account of inquiry in a paper titled “Deweyan approaches to abduction?” (2015).

    Your criticism of those who wish to substitute analytics for sensemaking could perhaps be restated in Peircean terms as: those who want to substitute deduction and induction (especially induction) for abduction.

    As I noted in a comment on your previous post on sensemaking, problem formulation in business or in other domains may not be extremely different from what is known as case formulation in the clinical professions. I’ve noticed a few papers that analyze clinical case formulation in terms of abductive reasoning. Doug Walton, a well-known argumentation theorist, wrote a book on abduction simply titled Abductive Reasoning (2004).

    Liked by 1 person

    Nathan

    October 25, 2016 at 12:24 am

    • Hi Nathan,

      Thanks for reading and commenting…and for putting me on the spot🙂 Before I say more, I should admit that I haven’t done anything close to what one could call a comprehensive review of the relevant literature. Further, my “definition” of sensemaking (if one can call it that) is still provisional. Everything below should be read with this in mind.

      You are absolutely right that abductive reasoning – the process of framing hypotheses from particular instances – is part of what I call sensemaking. I’ve alluded to this in my response to Tim van Gelder’s comment on a previous post on sensemaking. Indeed, I noticed that much of the work in the early hours of hackathons involves precisely this kind of reasoning.

      However, in general, there is more to it than abduction. Even before one gets to reason from particulars to a general (hypothesis), there is the issue of what to focus on. This is a matter that is highly dependent on the individual. Hence the need for collaborative work. The more perspectives one brings to bear on the situation, the greater the chances that particulars that turn out to be significant later will be picked up. The surprise for me is that is the case even in situations where one might think (at least at first sight) that the facts will “speak for themselves.” Indeed, barring socially trivial situations, it is almost always the case that data has to be given a voice. How this is done matters a lot. Indeed, one of the biggest problems with problem formulation is that this is not done in an inclusive manner. This is where sensemaking techniques can be very useful. Of course, one has to have the right conditions too, but that’s another story, and one you know well I reckon🙂

      I’m not sure what I can point you to in terms of references; you are far better read than I am in this area. A book that comes to mind – if you haven’t read it yet – is Ackoff’s Art of Problem Solving, which does mention the importance of getting diverse perspectives. There are a few other papers and articles in my collection that may be of interest, but I’d have to track them down. On a related note, I’d love to have a chat about this and other topics. Do drop me a note if you’re interested and we’ll take it from there.

      Thanks as always for your interest in my posts and your thought-provoking comments.

      Regards,

      Kailash.

      PS: Thanks for the link to Paavola’s paper and the Walton ref.

      Like

      K

      October 26, 2016 at 9:22 am

  3. Kailash, thanks for your generously lengthy reply. I agree with everything you said. I think the third paragraph of your reply identifies an important aspect of group sensemaking that was missing in my comment, and it’s possibly something was missing in those earlier thinkers that I mentioned: namely, how to deal with multiple perspectives “in an inclusive manner.” In fact, Donald “Mr. Reflective Practice” Schön (1930–1997) wrote an article in 1992 titled “The theory of inquiry: Dewey’s legacy to education” in which he noted that in the 1950s he wrote his doctoral thesis on Dewey’s Logic, and he mused:

    “Dewey never fully confronts the ontological differences in our ways of seeing situations and construing them as problematic or not (differences whose importance I shall discuss later in this article). […] He is well aware, it is true, that our constructed problems determine what facts we select for attention, and that our ways of constructing problems from problematic situations are subject to variation from culture to culture, person to person, time to time, and context to context. He appears, however, to hold a robust belief that ‘observed facts’ being just what they are, judgments about problems can be tested against them.”

    Schön notes in a footnote that this criticism of Dewey is open to dispute, but in any case what Schön said in this passage sounds like a paraphrase of your statement: “The surprise for me is that is the case even in situations where one might think (at least at first sight) that the facts will ‘speak for themselves’. Indeed, barring socially trivial situations, it is almost always the case that data has to be given a voice.”

    Thanks for mentioning Ackoff’s Art of Problem Solving; I’ve read a couple of articles by Ackoff but not his books, so I will put it on my reading list. Karl Weick’s book Sensemaking in Organizations (1995) could be considered as good a definition of sensemaking as any (although it is certainly not a “how to” manual).

    Like

    Nathan

    October 27, 2016 at 11:01 pm

  4. Since writing my previous comment, I happened to find a review article in Academy of Management Annals that has a nice collection of definitions of sensemaking: Sally Maitlis & Marlys Christianson, “Sensemaking in organizations: taking stock and moving forward” (2014). See especially the tables in the article.

    Like

    Nathan

    October 30, 2016 at 12:54 am


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: