Eight to Late

Sensemaking and Analytics for Organizations

Archive for the ‘Argumentation’ Category

Improving decision-making in projects

with 5 comments

An irony of organisational life is that the most important decisions on projects (or any other initiatives) have to be made at the start, when ambiguity is at its highest and information availability lowest. I recently gave a talk at the Pune office of BMC Software on improving decision-making in such situations.

The talk was recorded and simulcast to a couple of locations in India. The folks at BMC very kindly sent me a copy of the recording with permission to publish it on Eight to Late. Here it is:


Based on the questions asked and the feedback received, I reckon that a number of people found the talk  useful. I’d welcome your comments/feedback.

Acknowledgements: My thanks go out to Gaurav Pal, Manish Gadgil and Mrinalini Wankhede for giving me the opportunity to speak at BMC, and to Shubhangi Apte for putting me in touch with them. Finally, I’d like to thank the wonderful audience at BMC for their insightful questions and comments.

The Risk – a dialogue mapping vignette

with one comment

Foreword

Last week, my friend Paul Culmsee conducted an internal workshop in my organisation on the theme of collaborative problem solving. Dialogue mapping is one of the tools  of the tools he introduced during the workshop.  This piece, primarily intended as a follow-up for attendees,  is an introduction to dialogue mapping via a vignette that illustrates its practice (see this post for another one). I’m publishing it here as I thought it might be useful for those who wish to understand what the technique is about.

Dialogue mapping uses a notation called Issue Based Information System (IBIS), which I have discussed at length in this post. For completeness, I’ll begin with a short introduction to the notation and then move on to the vignette.

A crash course in IBIS

The IBIS notation consists of the following three elements:

  1. Issues(or questions): these are issues that are being debated. Typically, issues are framed as questions on the lines of “What should we do about X?” where X is the issue that is of interest to a group. For example, in the case of a group of executives, X might be rapidly changing market condition whereas in the case of a group of IT people, X could be an ageing system that is hard to replace.
  2. Ideas(or positions): these are responses to questions. For example, one of the ideas of offered by the IT group above might be to replace the said system with a newer one. Typically the whole set of ideas that respond to an issue in a discussion represents the spectrum of participant perspectives on the issue.
  3. Arguments: these can be Pros (arguments for) or Cons (arguments against) an issue. The complete set of arguments that respond to an idea represents the multiplicity of viewpoints on it.

Compendium is a freeware tool that can be used to create IBIS maps– it can be downloaded here.

In Compendium, IBIS elements are represented as nodes as shown in Figure 1: issues are represented by blue-green question markspositions by yellow light bulbspros by green + signs and cons by red – signs.  Compendium supports a few other node types, but these are not part of the core IBIS notation. Nodes can be linked only in ways specified by the IBIS grammar as I discuss next.

Figure 1: Elements of IBIS

Figure 1: IBIS node types

The IBIS grammar can be summarized in three simple rules:

  1. Issues can be raised anew or can arise from other issues, positions or arguments. In other words, any IBIS element can be questioned.  In Compendium notation:  a question node can connect to any other IBIS node.
  2. Ideas can only respond to questions– i.e. in Compendium “light bulb” nodes can only link to question nodes. The arrow pointing from the idea to the question depicts the “responds to” relationship.
  3. Arguments  can only be associated with ideas– i.e. in Compendium “+” and “–“  nodes can only link to “light bulb” nodes (with arrows pointing to the latter)

The legal links are summarized in Figure 2 below.

Figure 2: Legal links in IBIS

Figure 2: Legal links in IBIS

 

…and that’s pretty much all there is to it.

The interesting (and powerful) aspect of IBIS is that the essence of any debate or discussion can be captured using these three elements. Let me try to convince you of this claim via a vignette from a discussion on risk.

 The Risk – a Dialogue Mapping vignette

“Morning all,” said Rick, “I know you’re all busy people so I’d like to thank you for taking the time to attend this risk identification session for Project X.  The objective is to list the risks that we might encounter on the project and see if we can identify possible mitigation strategies.”

He then asked if there were any questions. The head waggles around the room indicated there were none.

“Good. So here’s what we’ll do,”  he continued. “I’d like you all to work in pairs and spend 10 minutes thinking of all possible risks and then another 5 minutes prioritising.  Work with the person one your left. You can use the flipcharts in the breakout area at the back if you wish to.”

Twenty minutes later, most people were done and back in their seats.

“OK, it looks as though most people are done…Ah, Joe, Mike have you guys finished?” The two were still working on their flip-chart at the back.

“Yeah, be there in a sec,” replied Mike, as he tore off the flip-chart page.

“Alright,” continued Rick, after everyone had settled in. “What I’m going to do now is ask you all to list your top three risks. I’d also like you tell me why they are significant and your mitigation strategies for them.” He paused for a second and asked, “Everyone OK with that?”

Everyone nodded, except Helen who asked, “isn’t it important that we document the discussion?”

“I’m glad you brought that up. I’ll make notes as we go along, and I’ll do it in a way that everyone can see what I’m writing. I’d like you all to correct me if you feel I haven’t understood what you’re saying. It is important that  my notes capture your issues, ideas and arguments accurately.”

Rick turned on the data projector, fired up Compendium and started a new map.  “Our aim today is to identify the most significant risks on the project – this is our root question”  he said, as he created a question node. “OK, so who would like to start?”

 

 

Fig 3: The root question

Figure 3: The root question

 

“Sure,” we’ll start, said Joe easily. “Our top risk is that the schedule is too tight. We’ll hit the deadline only if everything goes well, and everyone knows that they never do.”

“OK,” said Rick, “as he entered Joe and Mike’s risk as an idea connecting to the root question. “You’ve also mentioned a point that supports your contention that this is a significant risk – there is absolutely no buffer.” Rick typed this in as a pro connecting to the risk. He then looked up at Joe and asked,  “have I understood you correctly?”

“Yes,” confirmed Joe.

 

Fig 4: Map in progress

Figure 4: Map in progress

 

“That’s pretty cool,” said Helen from the other end of the table, “I like the notation, it makes reasoning explicit. Oh, and I have another point in support of Joe and Mike’s risk – the deadline was imposed by management before the project was planned.”

Rick began to enter the point…

“Oooh, I’m not sure we should put that down,” interjected Rob from compliance. “I mean, there’s not much we can do about that can we?”

…Rick finished the point as Rob was speaking.

 

Fig 4: Map in progress

Figure 5: Two pros for the idea

 

“I hear you Rob, but I think  it is important we capture everything that is said,” said Helen.

“I disagree,” said Rob. “It will only annoy management.”

“Slow down guys,” said Rick, “I’m going to capture Rob’s objection as “this is a management imposed-constraint rather than risk. Are you OK with that, Rob?”

Rob nodded his assent.

 

Fig 6: A con enters the picture

Fig 6: A con enters the picture

I think it is important we articulate what we really think, even if we can’t do anything about it,” continued Rick. There’s no point going through this exercise if we don’t say what we really think. I want to stress this point, so I’m going to add honesty  and openness  as ground rules for the discussion. Since ground rules apply to the entire discussion, they connect directly to the primary issue being discussed.”

Figure 7: A "criterion" that applies to the analysis of all risks

Figure 7: A “criterion” that applies to the analysis of all risks

 

“OK, so any other points that anyone would like to add to the ones made so far?” Queried Rick as he finished typing.

He looked up. Most of the people seated round the table shook their heads indicating that there weren’t.

“We haven’t spoken about mitigation strategies. Any ideas?” Asked Rick, as he created a question node marked “Mitigation?” connecting to the risk.

 

Figure 8: Mitigating the risk

Figure 8: Mitigating the risk

“Yeah well, we came up with one,” said Mike. “we think the only way to reduce the time pressure is to cut scope.”

“OK,” said Rick, entering the point as an idea connecting to the “Mitigation?” question. “Did you think about how you are going to do this? He entered the question “How?” connecting to Mike’s point.

Figure 9: Mitigating the risk

Figure 9: Mitigating the risk

 

“That’s the problem,” said Joe, “I don’t know how we can convince management to cut scope.”

“Hmmm…I have an idea,” said Helen slowly…

“We’re all ears,” said Rick.

“…Well…you see a large chunk of time has been allocated for building real-time interfaces to assorted systems – HR, ERP etc. I don’t think these need to be real-time – they could be done monthly…and if that’s the case, we could schedule a simple job or even do them manually for the first few months. We can push those interfaces to phase 2 of the project, well into next year.”

There was a silence in the room as everyone pondered this point.

“You know, I think that might actually work, and would give us an extra month…may be even six weeks for the more important upstream stuff,” said Mike. “Great idea, Helen!”

“Can I summarise this point as – identify interfaces that can be delayed to phase 2?” asked Rick, as he began to type it in as a mitigation strategy. “…and if you and Mike are OK with it, I’m going to combine it with the ‘Cut Scope’ idea to save space.”

“Yep, that’s fine,” said Helen. Mike nodded OK.

Rick deleted the “How?” node connecting to the “Cut scope” idea, and edited the latter to capture Helen’s point.

Figure 10: Mitigating the risk

Figure 10: Mitigating the risk

“That’s great in theory, but who is going to talk to the affected departments? They will not be happy.” asserted Rob.  One could always count on compliance to throw in a reality check.

“Good point,”  said Rick as he typed that in as a con, “and I’ll take the responsibility of speaking to the department heads about this,” he continued entering the idea into the map and marking it as an action point for himself. “Is there anything else that Joe, Mike…or anyone else would like to add here,” he added, as he finished.

Figure 11: Completed discussion of first risk (click to see full size

Figure 11: Completed discussion of first risk (click to view larger image)

“Nope,” said Mike, “I’m good with that.”

“Yeah me too,” said Helen.

“I don’t have anything else to say about this point,” said Rob, “ but it would be great if you could give us a tutorial on this technique. I think it could be useful to summarise the rationale behind our compliance regulations. Folks have been complaining that they don’t understand the reasoning behind some of our rules and regulations. ”

“I’d be interested in that too,” said Helen, “I could use it to clarify user requirements.”

“I’d be happy to do a session on the IBIS notation and dialogue mapping next week. I’ll check your availability and send an invite out… but for now, let’s focus on the task at hand.”

The discussion continued…but the fly on the wall was no longer there to record it.

Afterword

I hope this little vignette illustrates how IBIS and dialogue mapping can aid collaborative decision-making / problem solving by making diverse viewpoints explicit. That said, this is a story, and the problem with stories is that things  go the way the author wants them to.  In real life, conversations can go off on unexpected tangents, making them really hard to map. So, although it is important to gain expertise in using the software, it is far more important to practice mapping live conversations. The latter is an art that requires considerable practice. I recommend reading Paul Culmsee’s series on the practice of dialogue mapping or <advertisement> Chapter 14 of The Heretic’s Guide to Best Practices</advertisement> for more on this point.

That said, there are many other ways in which IBIS can be used, that do not require as much skill. Some of these include: mapping the central points in written arguments (what’s sometimes called issue mapping) and even decisions on personal matters.

To sum up: IBIS is a powerful means to clarify options and lay them out in an easy-to-follow visual format. Often this is all that is required to catalyse a group decision.

Improving decision-making in projects (and life)

with 4 comments

Note: This post is based on a presentation that I gave at BA World Sydney on 26th June. It draws from a number of posts that I have written over the last few years.

Introduction – the myth of rational decision making

A central myth about decision making in organisations is that it is a rational process.   The qualifier rational refers to decision-making methods that are based on the following broad steps:

  1. Identify available options.
  2. Develop criteria for rating options.
  3. Rate options according to criteria developed.
  4. Select the top-ranked option.

The truth is that decision making in organisations generally does not follow such a process. As I have pointed out in this post (which is based on this article by Tim van Gelder) decisions are often based on a mix of informal reasoning, personal beliefs and even leaps of faith. . Quoting from that post, formal (or rational) processes often cannot be applied for one or  more of the following reasons:

  1. Real-world options often cannot be quantified or rated in a meaningful way. Many of life’s dilemmas fall into this category. For example, a decision to accept or decline a job offer is rarely made on the basis of material gain alone.
  2. Even where ratings are possible, they can be highly subjective. For example, when considering a job offer, one candidate may give more importance to financial matters whereas another might consider lifestyle-related matters (flexi-hours, commuting distance etc.) to be paramount. Another complication here is that there may not be enough information to settle the matter conclusively. As an example, investment decisions are often made on the basis of quantitative information that is based on questionable assumptions.
  3. Finally, the problem may be wicked – i.e. complex, multi-faceted and difficult to analyse using formal decision making methods. Classic examples of wicked problems are climate change (so much so, that some say it is not even a problem) and city / town planning. Such problems cannot be forced into formal decision analysis frameworks in any meaningful way.

The main theme running through all of these is uncertainty. Most of the decisions we are called upon to make in our professional lives are fraught with uncertainty – it is what makes it hard to rate options, adds to the subjectivity of ratings (where they are possible) and magnifies the wickedness of the issue.

Decision making in projects and the need for systematic deliberation

The most important decisions in a project are generally made at the start, at what is sometimes called the “front-end of projects”. Unfortunately this is where information availability is at its lowest and, consequently, uncertainty at its highest.  In such situations, decision makers feel that they are left with little choice but to base their decisions on instinct or intuition.

Now, even when one bases a decision on intuition, there is some deliberation involved – one thinks things through and weighs up options in some qualitative way. Unfortunately, in most situations, this is done in an unsystematic manner. Moreover, decision makers fail to record the informal reasoning behind their decisions. As a result the rationale behind decisions made remain opaque to those who want to understand why particular choices were made.

A brief introduction to IBIS

Clearly, what one needs is a means to make the informal reasoning behind a decision explicit. Now there are a number of argument visualisation techniques available for this purpose, but I will focus on one that I have worked with for a while: Issue-Based Information System (IBIS). I will introduce the notation briefly below. Those who want a detailed introduction will find one in my article entitled, The what and whence of issue-based information systems.

IBIS consists of three main elements:

  • Issues (or questions): these are issues that need to be addressed.
  • Positions (or ideas): these are responses to questions. Typically the set of ideas that respond to an issue represents the spectrum of perspectives on the issue.
  • Arguments: these can be Pros (arguments supporting) or Cons (arguments against) an issue. The complete set of arguments that respond to an idea represents the multiplicity of viewpoints on it.

The best IBIS mapping tool is Compendium – it can be downloaded here.  In Compendium, the IBIS elements described above are represented as nodes as shown in Figure 1: issues are represented by green question nodes; positions by yellow light bulbs; pros by green + signs and cons by red – signs.  Compendium supports a few other node types, but these are not part of the core IBIS notation. Nodes can be linked only in ways specified by the IBIS grammar as I discuss next.

IBIS Elements

The IBIS grammar can be summarized in a few simple rules:

  • Issues can be raised anew or can arise from other issues, positions or arguments. In other words, any IBIS element can be questioned.  In Compendium notation:  a question node can connect to any other IBIS node.
  • Ideas can only respond to questions – i.e.  in Compendium “light bulb” nodes  can only link to question nodes. The arrow pointing from the idea to the question depicts the “responds to” relationship.
  • Arguments  can only be associated with ideas –  i.e in Compendium + and –  nodes can only link to “light bulb” nodes (with arrows pointing to the latter)

The legal links are summarized in Figure 2 below.

Figure 2: Legal Links in IBIS

The rules are best illustrated by example-   follow the links below to see some illustrations of IBIS in action:

  1. See or this post or this one for examples of IBIS in mapping dialogue.
  2. See this post or this one for examples of argument visualisation (issue mapping) using IBIS.

The case studies

In my talk, I illustrated the use of IBIS by going through a couple of examples in detail, both of which I have described in detail in other articles. Rather than reproduce them here, I will provide links to the original sources below.

The first example was drawn from a dialogue mapping exercise I did for a data warehousing project. A detailed discussion of the context and process of mapping (along with figures of the map as it developed) are available in a paper entitled, Mapping project dialogues using IBIS – a case study and some reflections (PDF).

The second example, in which I described a light-hearted example of the use of IBIS in a non-work setting,  is discussed in my post, What should I do now – a bedtime story about dialogue mapping.

Benefits of IBIS

The case studies serve to highlight how IBIS encourages collective deliberation of issues. Since the issues we struggle with in projects often have elements of wickedness, eliciting opinions from a group through dialogue improves our chances arriving at a “solution” that is acceptable to the group as a whole.

Additional benefits of using  IBIS in a group setting include:

  • It adds clarity to a discussion
  • Serves as a simple and intuitive discussion summary (compare to meeting minutes!)
  • Is a common point of reference to move a discussion forward.
  • It captures the logic of a decision (decision rationale)

Further still, IBIS disarms disruptive discussion tactics such as “death by repetition” – when a person brings up the same issue over and over again in a million and one different ways. In such situations the mapper simply points to the already captured issue and asks the person if they want to add anything to it. The disruptive behaviour becomes evident to all participants (including the offender).

The beauty of IBIS lies in its simplicity. It is easy to learn – four nodes with a very simple grammar. Moreover, participants don’t need to learn the notation. I have found that most people can understand what’s going on within a few minutes with just a few simple pointers from the mapper.

Another nice feature of IBIS is that it is methodology-neutral. Whatever your methodological persuasion – be it Agile or something  that’s  BOKsed – you can use it to address decision problems in your project meetings.

Getting started

The best way to learn IBIS is to map out the logic of articles in newspapers, magazines or even professional journals. Once you are familiar with the syntax and grammar, you can graduate to one-on-one conversations, and from there to small meetings. When using it in a meeting for the first time, tell the participants that you are simply taking notes. If things start to work well – i.e. if you are mapping the conversation successfully – the group will start interacting with the map, using it as a basis for their reasoning and as a means to move the dialogue forward. Once you get to this point, you are where you want to be – you are mapping the logic of the conversation.

Of course, there is much more to it than I’ve mentioned above. Check out the references at the end of this piece for more information on mapping dialogues using IBIS.

Wrap up

As this post is essentially covers a talk I gave at a conference, I would like to wrap up with a couple of observations and comments from the audience.

I began my talk with the line, “A central myth about decision making in organisations is that it is a rational process.”  I thought many in the audience would disagree. To my surprise, however, there was almost unanimous agreement! The points I made about uncertainty and problem wickedness also seemed to resonate. There were some great examples from the audience on wicked problems in IT – a particularly memorable one about an infrastructure project (which one would normally not think of as particularly wicked) displaying elements of wickedness soon after it was started.

It seems that although mainstream management ignores the sense-making aspect of the profession, many practitioners tacitly understand that making sense out of ambiguous situations is an important part of their work.  Moreover, they know that this is best done by harnessing the collective intelligence of a group rather than by enforcing a process or a solution

References:

  1. Jeff Conklin, Dialogue Mapping: Building Shared Understanding of Wicked Problems, John Wiley, New York (2005). See my review of Conklin’s book here
  2. Paul Culmsee & Kailash Awati, The Heretic’s Guide to Best Practices: The Reality of Managing Complex Problems in Organisations, iUniverse: Bloomington, Indiana (2011).

Not on the same page, not even reading the same book

with 4 comments

In the course of a project it is not uncommon to have stakeholders with conflicting viewpoints on a particular issue. Some examples of this include:

  • The sponsor who wants a set of reports done in a day and the report writer who reckons it will take a week.
  • The project manager who believes that tasks can be tracked to a very fine level and the developer who “knows” they can’t.
  • The developer who is convinced that method A is the best way to go and her colleague who is equally certain that method B is the way to go.

These are but a small selection of the conflicts I have encountered in my work. Most project professionals would undoubtedly have had similar experiences. It can be difficult to reconcile such conflicting viewpoints because they are based on completely different worldviews. Unless these are made explicit, it is difficult to come to for those involved to understand each other let alone agree.

Consider, for example, the first case above: the sponsor’s worldview is likely based on his reality, perhaps a deadline imposed on him by his boss , whereas the report writer’s view is based on what she thinks  is a reasonable time to create the reports requested.

Metaphorically, the two parties are not on the same page.  Worse, they are not even reading the same book. The sponsor’s reality – his “book” – is based on an imposed deadline whereas the report writer’s is based on an estimate.

So, how does one get the two sides to understand each other’s point of view?

The metaphor gives us a clue – we have to first get them to understand that they are “reading from different books.”  Only then do they have a hope in hell of understanding each other’s storylines.

This isn’t easy because people tend to believe their views are reasonable (even when they aren’t!). The only way to resolve these differences are through dialogue or collective deliberation. As I have written in my post on rational dialogue in project environments:

Someone recently mentioned to me that the problem in project meetings (and indeed any conversation) is that participants  see their own positions  as being rational, even when they are not.  Consequently, they stick to their views, even when faced with evidence to the contrary. However, such folks aren’t being rational because they do not subject their positions and views to “trial by argumentation.”  Rationality lies in dialogue, not in individual statements or positions. A productive discussion is one in which conflicting claims are debated until they converge on an optimal decision.  The best (or most rational) position is one that emerges from such collective deliberation.

The point is a simple one: we have to get the two sides talking to each other, with each one accepting that their views may need to be revised in the light of the arguments presented by the other.  Dialogue Mapping, which I have discussed in many posts on this blog is a great way to facilitate such dialogue.

In our forthcoming book entitled, The Heretic’s Guide to Best Practices, Paul Culmsee and I describe Dialogue Mapping and a host of other techniques that can help organisations tackle problems associated with people who are “not on the same page” or “reading different books.”

The book is currently in the second round of proofs. We’ll soon be putting up a website with excerpts, review comments, pricing, release dates and much more – stay tuned!

Written by K

November 11, 2011 at 6:03 am

Mapping project dialogues using IBIS – a paper preview

with one comment

Work commitments have conspired to keep this post short. Well, short compared to my usual long-winded essays at any rate. Among other things, I’m currently helping  get a biggish project started while also trying to finish my current writing commitments in whatever little free time I have.  Fortunately, I have a ready-made topic to write about this week:  my recently published paper on the use of dialogue mapping in project management.  Instead of summarizing the paper, as I usually do in my paper reviews, I’ll simply present some background to the paper and describe, in brief, my rationale for writing it.

As regular readers of this blog will know, I am a fan of dialogue mapping,  a conversation mapping technique pioneered by Jeff Conklin. Those unfamiliar with the technique will find a super-quick introduction here.  Dialogue mapping uses a visual notation called issue based information system (IBIS) which I have described in detail in this post.  IBIS was invented by Horst Rittel as a means to capture and clarify facets of   wicked problems – problems that are hard to define, let alone solve.  However, as I discuss in the paper, the technique also has utility in the much more mundane day-to-day business of managing projects.

In essence, IBIS provides a means to capture questions,  responses to questions and arguments for and against those responses. This, coupled with the fact that it is easy to use, makes it eminently suited to capturing conversations in which issues are debated and resolved. Dialogue mapping is therefore a great way to surface options, debate them and reach a “best for group” decision in real-time. The technique thus has many applications in organizational settings. I have used it regularly in project meetings, particularly those in which critical decisions regarding design or approach are being discussed.

Early last year I used the technique to kick-start a data warehousing initiative within the organisation I work for. In the paper I use this experience as a case-study to illustrate some key aspects and features of dialogue mapping that make it useful in project discussions.  For completeness I also discuss why other visual notations for decision and design rationale don’t work as well as IBIS for capturing conversations in real-time. However, the main rationale for the paper is to provide a short,  self-contained introduction to the technique via a realistic case-study.

Most project managers would have had to confront questions such as “what approach should we take to solve this problem?” in situations where there is not enough information to make a sound decision. In such situations, the only recourse one has is to dialogue – to talk it over with the team, and thereby reach a shared understanding of the options available. More often than not, a  consensus decision emerges from such dialogue.  Such a decision would be based on the collective knowledge of the team, not just that of an individual.  Dialogue mapping provides a means to get to such a collective decision.

Why deliberation trumps standard decision-making methods

with 10 comments

Wikipedia defines decision analysis as the discipline comprising the philosophy, theory, methodology, and professional practice necessary to address important decisions in a formal manner.  Standard decision-making techniques generally involve the following steps:

  1. Identify available options.
  2. Develop criteria for rating options.
  3. Rate options according to criteria developed.
  4. Select the top-ranked option.

This sounds great in theory, but as Tim van Gelder points out in an article entitled the The Wise Delinquency of Decision Makers, formal methods of decision analysis are not used as often as textbooks and decision-theorists would have us believe.  This, he argues, isn’t due to ignorance:  even those trained in such methods often do not use them for decisions that really matter. Instead they resort to deliberation –    weighing up options in light of the arguments and evidence for and against them. He discusses why this is so, and also points out some problems with deliberative methods and what can be done do fix them. This post is a summary of the main points he makes in the article.

To begin with, formal methods aren’t suited to many decision-making problems encountered in the real world. For instance:

  1. Real-world options often cannot be quantified or rated in a meaningful way. Many of life’s dilemmas fall into this category. For example, a decision to accept or decline a job offer is rarely made on the basis of material gain alone.
  2. Even where ratings are possible, they can be highly subjective. For example, when considering a job offer, one candidate may give more importance to financial matters whereas another might consider lifestyle-related matters (flexi-hours, commuting distance etc.) to be paramount. Another complication here is that there may not be enough information to settle the matter conclusively. As an example, investment decisions are often made on the basis of quantitative information that is based on questionable assumptions.
  3. Finally, the problem may be wicked – i.e. complex, multi-faceted and difficult to analyse using formal decision making methods. Classic examples of wicked problems are climate change (so much so, that some say it is not even a problem) and city / town planning. Such problems cannot be forced into formal decision analysis frameworks in any meaningful way.

Rather than rating options and assigning scores, deliberation involves making arguments for and against each option and weighing them up in some consistent (but qualitative) way. In contrast to textbook methods of decision analysis, this is essentially an informal process; there is no prescribed method that one must follow. One could work through an arguments oneself or in conversation with others.  Because of the points listed above, deliberation is often better suited to deal with many of the decisions we are confronted with in our work and personal lives (see this post for a real-life example of deliberative decision making)

However, as Van Gelder points out,

The trouble is that deliberative decision making is still a very problematic business. Decisions go wrong all the time. Textbook decision methods were developed, in part, because it was widely recognized that our default or habitual decision making methods are very unreliable.

He  lists four problems with deliberative methods:

  1. Biases – Many poor decisions can be traced back to cognitive biaseserrors of judgement based on misperceptions of situations, data or evidence. A common example of such a bias is overconfidence in one’s own judgement. See this post for a discussion of how failures of high-profile projects may have been due to cognitive biases.
  2. Emotions – It is difficult, if not impossible, to be completely rational when making a decision – even a simple one.  However, emotions can cloud judgement and lead to decisions being made on the basis of pride, anger or envy rather than a clear-headed consideration of known options and their pros and cons.
  3. Tyranny of the group – Important decisions are often made by committees. Such decisions are subjected to collective biases such as groupthink – the tendency of group members to think alike and ignore external inputs so as to avoid internal conflicts. See this post for a discussion of groupthink in project environments. 
  4. Lack of training – People end up making poor decisions because they lack knowledge of informal logic and argumentation, skills that can be taught and then honed through practice.

Improvements in our ability to deliberate matters can be brought about by addressing the above. Clearly, it is difficult to be completely objective when confronted with tough decisions just as it is impossible to rid ourselves of our (individual and collective) biases.  That said, any technique that lays out all the options and arguments for and against them in a easy-to-understand way may help in making our biases and emotions (and those of others) obvious. Visual notations such as  IBIS (Issue-Based Information Systems) and  Argument Mapping do just that.  See this post for more on why it is better to represent reasoning visually than in prose.

The use of techniques such as the ones listed in the previous paragraph can lead to immediate improvements in corporate decision making. Firstly,  because gaps in logic and weaknesses in supporting evidence are made obvious, those responsible for formulating, say, a business case can focus on improving the their arguments prior to presenting them to senior managers. Secondly, decision makers can see the logic, supporting materials and the connections between them at a glance. In short: those formulating an argument and those making decisions based on it can focus on the essential points of the matter without having to wade through reams of documentation or tedious presentations.

To summarise: formal decision-making techniques are unsuited to complex problems  or those that have  options that cannot be quantified in a meaningful way. For such issues, deliberation –  supplemented by visual notations such as IBIS or Argument Mapping – offers a better alternative.

Written by K

May 13, 2011 at 5:32 am

Value judgements in system design

leave a comment »

Introduction

How do we choose between competing design proposals for information systems? In principle this should be done using an evaluation process based on objective criteria. In practice, though, people generally make choices based on their interests and preferences. These can differ widely, so decisions are often made on the basis of criteria that do not satisfy the interests of all stakeholders. Consequently, once a system becomes operational, complaints from stakeholder groups whose interests were overlooked are almost inevitable (Just think back to any system implementation that you were involved in…).

The point is: choosing between designs is not a purely technical issue; it also involves value judgements – what’s right and what’s wrong – or even, what’s good and what’s not. Problem is, this is a deeply personal matter – different folks have different values and, consequently, differing ideas of which design ideal is best (Note: the term ideal refers to the value judgements associated with a design). Ten years ago, Heinz Klein and Rudy Hirschheim addressed this issue in a landmark paper entitled, Choosing Between Competing Design Ideals in Information Systems Development. This post is a summary of the main ideas presented in their paper.

Design ideals and deliberation

A design ideal is not an abstract, philosophical concept. The notion of good and bad or right and wrong can be applied to the technical, economic and social aspects of a system. For example, a choice between building and buying a system has different economic and social consequences for stakeholder groups within and outside the organisation. What’s more, the competing ideals may be in conflict – developers employed in the organisation would obviously prefer to build rather than buy because their employment depends on it; management, however, may have a very different take on the issue.

The essential point that Hirschheim and Klein make is that differences in values can be reconciled only through honest discussion. They propose a deliberative approach wherein all stakeholders discuss issues in order to come to an agreement. To this end, they draw on the theories of argumentation and communicative rationality to come up with a rational framework for comparing design ideals.  Since these terms are new, I’ll spend a couple of paragraphs in describing them briefly.

Argumentation is essentially reasoned debate – i.e. the process of reaching conclusions via arguments that use informal logic – which, according to the definition in the foregoing link, is the attempt to develop a logic to assess, analyse and improve ordinary language (or “everyday”) reasoning. Hirschheim and Klein use the argumentation framework proposed by Stephen Toulmin, to illustrate their approach.

The basic premise of communicative rationality is that rationality (or reason) is tied to social interactions and dialogue. In other words, the exercise of reason can  occur only through dialogue.  Such communication, or mutual deliberation, ought to result in a general agreement about the issues under discussion.  Only once such agreement is achieved can there be a consensus on actions that need to be taken.  See my article on rational dialogue in project environments for more on communicative rationality.

Obstacles to rational dialogue and how to overcome them

The key point about communicative rationality is that it assumes the following conditions hold:

  1. Inclusion:  includes all stakeholders.
  2. Autonomy: all participants should be able to present and criticise views without fear of reprisals.
  3. Empathy: participants must be willing to listen to and understand claims made by others.
  4. Power neutrality: power differences (levels of authority) between participants should not affect the discussion.
  5. Transparency: participants must not indulge in strategic actions (i.e. lying!).

Clearly these are idealistic conditions, difficult to achieve in any real organisation.  Klein and Hirschheim acknowledge this point, and note the following barriers to rationality in organisational decision making:

  1. Social barriers: These include inequalities (between individuals) in power, status, education and resources.
  2. Motivational barriers: This refers to  the psychological cost of prolonged debate.  After a period of sustained debate, people will often cave in just to stop arguing even though they may have the better argument.
  3. Economic barriers: Time is money: most organisations cannot afford a prolonged debate on contentious issues.
  4. Personality differences: How often is it that the most charismatic or articulate person gets their way, and the quiet guy in the corner (with a good idea or two) is completely overlooked?
  5. Linguistic barriers: This refers to the difficulty of formulating arguments in a way that makes sense to the listener. This involves, among other things, the ability to present ideas in a way that is succinct, without glossing over the important issue – a skill not possessed by many.

These barriers will come as no surprise to most readers. It will be just as unsurprising that it is difficult to overcome them. Klein and Hirschheim offer the usual solutions including:

  1. Encourage open debate – They suggest the use of technologies that support collaboration. They can be forgiven for their optimism given that the paper was written a decade ago, but the fact of the matter is that all the technologies that have sprouted since have done little to encourage open debate.
  2. Implement group decision techniques –  these include arrangements such as quality circles, nominal groups and constructive controversy. However, these too will not work unless people feel safe enough to articulate their opinions freely.

Even though the barriers to open dialogue are daunting, it behooves system designers to strive towards reducing or removing them.  There are effective ways to do this, but that’s a topic I won’t go into here as it has been dealt with at length elsewhere.

Principles for arguments about value judgements

So, assuming the environment is right, how should we debate value judgements?  Klein and Hirschheim recommend using informal logic supplemented with ethical principles. Let’s look at these two elements briefly.

Informal logic is a means to reason about human concerns. Typically, in these issues there is no clear cut notion of truth and falsity. Toulmin’s argumentation framework (mentioned earlier in this post) tells us how arguments about such issues should be constructed. It consists of the following elements:

  1. Claim:   A statement that one party asks another to accept as true. An example would be my claim that I did not show up to work yesterday because I was not well.
  2. Data (Evidence): The basis on which one party expects to other to accept a claim as true. To back the claim made in the previous line, I might draw attention to my runny nose and hoarse voice.
  3. Warrant:  The bridge between the data and the claim.  Again, using the same example, a warrant would be that I look drawn today, so it is likely that I really was sick yesterday.
  4. Backing: Further evidence, if the warrant should prove insufficient. If my boss is unconvinced by my appearance he may insist on a doctor certificate.
  5. Qualifier: These are words that express a degree of certainty about the claim. For instance, to emphasise just how sick I was, I might tell my boss  that I stayed in bed all day because I had  high fever.

This is all quite theoretical: when we debate issues we do not stop to think whether a statement is a warrant or a backing or something else; we just get on with the argument. Nevertheless, knowledge of informal logic can help us construct better arguments for our positions. Further, at the practical level there are computer supported deliberative techniques such as argument mapping and dialogue mapping which can assist in structuring and capturing such arguments.

The other element is ethics: Klein and Hirschheim  contend that moral and ethical principles ought to be considered when value judgements are being evaluated. These principles include:

  1. Ought implies can – which essentially means that one morally ought to do something only if one can do it (see this paper for an interesting counterview of this principle). Taking the negation of this statement, one gets “Cannot implies ought not” which means that a design can be criticised if it involves doing something that is (demonstrably) impossible – or makes impossible demands on certain parties.
  2. Conflicting principles – This is best explained via an example. Consider a system that saves an organisation money but involves putting a large number of people out of work. In this case we would have an economic principle coming up against a social one. According to the principle, a design ideal based on an economic imperative can be criticised on social grounds.
  3. The principle of reconsideration – This implies reconsidering decisions if relevant new information becomes available. For example, if it is found that a particular design overlooked a certain group of users, then the design should be reconsidered in the light of their needs.

They also mention that new ethical and moral theories may trigger the principle of reconsideration. In my opinion, however, this is a relatively rare occurrence:  how often are new ethical or moral theories proposed?

Summarising

The main point made by the authors is that system design involves value judgements. Since these are largely subjective,  open debate using the principles of informal logic is the best way to deal with conflicting values. Moreover, since such issues are not entirely technical, one has to use ethical principles to guide debate.  These principles – not asking people to do the impossible; taking everyone’s interests into account and reconsidering decisions in the light of new information – are reasonable if not self-evident. However, as obvious as they are, they are often ignored in design deliberations. Hirschheim and Klein do us a great service by reminding us of their importance.

Written by K

March 31, 2011 at 10:16 pm

%d bloggers like this: