Eight to Late

Sensemaking and Analytics for Organizations

Archive for the ‘sensemaking’ Category

Collaborative reasoning in the age of Covid

leave a comment »

Ever since the start of the pandemic, there have been no end of opinions, presentations and reports on how we might navigate our way out of the crisis. Much of this takes a narrow, discipline-centric view, which is inadequate because the problem is multifaceted and thus defies traditional disciplinary boundaries. It is therefore of urgent importance to chart a course that considers all aspects of recovery, not just those relevant to specific interests.  A recent report produced by the Australian Group of Eight does just that.   The key points of the report are concisely described in an executive summary and snapshot, so I will cover just the main points in this article. My focus instead is on the platform used to create the report, as it offers an effective collaborative approach to tackling complex issues in a broad range of contexts.

To me the most amazing thing about the 192-page report is that it was produced by a taskforce comprised of over a hundred academics and researchers across diverse disciplines, collaborating over a three-week period. As stated in the exec summary:

To chart a Roadmap to Recovery we convened a group of over a hundred of the country’s leading epidemiologists, infectious disease consultants, public health specialists, healthcare professionals, mental health and well-being practitioners, indigenous scholars, communications and behaviour change experts, ethicists, philosophers, political scientists, economists and business scholars from the Group of Eight (Go8) universities. The group developed this Roadmap in less than three weeks, through remote meetings and a special collaborative reasoning platform, in the context of a rapidly changing pandemic.

Those who have done any collaborative work involving large groups will have stories to tell about how challenging it is to get a coherent result.  This taskforce achieved this in part by working on an online collaborative reasoning platform called SWARM, described in this paper.  This post is mainly about what SWARM is and how it works, but I will also describe how the Roadmap taskforce used the platform to come up with a comprehensive recovery plan and the key recommendations made therein. I’ll end with some thoughts on the use of SWARM in broader organizational and business contexts.

The SWARM platform

The platform was designed and implemented by a team led by Drs. Tim van Gelder and Richard de Rozario as part of a large Intelligence Advanced Research Projects Activity (IARPA) initiative. In essence,  SWARM is a cloud collaboration environment designed to enhance evidence-based reasoning in teams. It does this by supporting an approach called contending analyses, wherein team members produce and refine multiple distinct analyses of a problem, and then select the best one as their collective response.

On SWARM, team members create artefacts that represent their reasoning. Additionally, they can rate, comment on and contribute to artefacts created by others through the course of the challenge. This enables a “best response” to emerge through an iterative process of discussion, refinement and evaluation.

To understand how it works, it is necessary to briefly describe the various ways in which users can interact and contribute to solving the problem with each other in SWARM. The user interface of the SWARM platform consists of three panes (Figure 1).

Figure 1: SWARM user interface

 

The left pane contains the problem description and links to related documents. In the centre pane, users can post and update responses. A response may be a Resource (e.g. a link to an external article, a visualisation or an analysis) that contributes to understanding or solving the problem, or it may be a Report, which is a draft candidate for the team’s final output. Users can then comment on and rate others’ responses and comments. The most highly rated Report at the conclusion of the problem is submitted as the result of the group’s collaborative reasoning.

The right pane is a streaming chat window through which users can interact in real-time. To summarise, SWARM users can:

  1. converse with team members via the chat feed.
  2. post or update a Resource or a Report
  3. comment on a Resource or a Report, or
  4. rate a Resource, Report or Comment.

By design, SWARM does not prescribe (or proscribe) any particular analytical process. As van Gelder, de Rozario and Sinnott (2018, pp. 22-34) note, contending analyses:

…promotes engagement by providing the opportunity for any participant to contribute their own thinking (autonomy), to think in a manner matching their natural expertise (mastery or competence), and to earn the respect of others by drafting a well-regarded response (relatedness)’ – thus meeting each of the three psychological needs identified by self-determination theory.

The idea is that teams should be free to work in ways that suit them collectively, with individuals given the choice to contribute as and when they please. That said, SWARM, via its Lens Kit (https://swarm-help.zendesk.com/hc/en-us/categories/360000312751-The-Lens-Kit), offers participants a compendium of structured analytical techniques and other “logical lenses” that may be useful in analysing complex and uncertain scenarios in which the available information is scarce or  ambiguous.

The Roadmap to Recovery project

The Roadmap project involved over a hundred academics from the Group of Eight – a coalition of the oldest, largest and most research-intensive Australian universities. Over three weeks in April 2020, the team worked on developing scenarios for national recovery from the COVID crisis. Their recommendations are available in a comprehensive report.  The report is unique in that it synthesises the knowledge of a range of experts and takes a systemic, evidence-based view of the problem.  In the words of the co-chairs of the project:

How this document differs from the hundreds of articles and opinion pieces on this issue is that this report specifies the evidence on which it is based, it is produced by researchers who are experts and leaders in their area, and it engages the broadest range of disciplines – from mathematicians, to virologists, to philosophers.

Over a three-week period, this taskforce has debated and discussed, disagreed, and agreed, edited and revised its work over weekdays and holidays, Good Friday and Easter. All remotely. All with social distancing…

…It is research collaboration in action – a collective expression of a belief that expert research can help Government plot the best path forward…

Given the wide geographical distribution of the team and the requirement for social distancing, it was clear that the team needed an online collaboration platform that enabled collective deliberation. Traditional online methods would not have worked for a group this large. As noted in the report:

Standard remote collaboration methods, such as circulating drafts by email, have many drawbacks such as the difficulty of keeping track of document versions, integrating edits and comments on many different versions, and ensuring that everyone can see the latest version. It seemed clear this approach would struggle with an expert group as large as the Roadmap Task Force.

The steering committee therefore decided to give SWARM a go.

As noted in the previous section, SWARM works on the principle that a group should canvas multiple approaches and then collectively settle on the best one, a principle summarised by the term contending analyses. The benefit of such an approach is evident in the report in that it outlines two distinct strategies for recovery:

  1. Elimination: as the term suggests, this strategy aims at eliminating the virus within the country. This is the lowest risk approach and is technically feasible for a relatively isolated country like Australia. However, the cost in terms of time, effort and money is substantial. Moreover, a strict implementation of this approach would bar international travel for an unrealistically long duration.
  2. Controlled Adaptation: this involves controlling the infection within the country to a level that does not overwhelm the healthcare system. This is less expensive in terms of time, effort and money, but the outcome is also less certain. However, as the taskforce points out, this could lead to restrictions being eased as early as May 15th, a choice that the government had made before the report was released. This decision is understandable given the cost of extended restrictions; however, it isn’t clear at all how they will handle the inevitable resurgence of the disease down the line. The report considers how things could develop as a result of this decision.

The report aims to provide a balanced case for the two options, and also emphasises that in terms of implementation, the options have considerable overlap. For instance, there are three requirements for the success of either:

  1. Early detection and supported isolation
  2. Travel and border restrictions.
  3. Public trust, transparency and civic engagement.

It should be clear that all three require massive government involvement and support. To this end, the taskforce has formulated an ethical framework that should guide government decision-making and policy. The framework comprises of the following six principles:

  1. Democratic accountability and the protection of civil liberties.
  2. Equal access to healthcare and social welfare.
  3. Shared economic sacrifice.
  4. Attentiveness to the distinctive patterns of disadvantage.
  5. Enhancing social well-being and mental health.
  6. Partnership and shared responsibility

An ethical framework should serve as a check on policy-making that might disadvantage specific groups. If followed, the six principles listed above will ensure that policies are fair to all sections of the community, both in terms of burdens and benefits This is perhaps the trickiest part of policy-making.

Finally, the taskforce has formulated six imperatives (essential rules) that should guide the actual implementation of a recovery. They are:

  1. The health of our healthcare system and its workers.
  2. Preparing for relaxation of social distancing.
  3. Mental health and wellbeing for all.
  4. The care of indigenous Australians.
  5. Equity of access and outcomes in health support.
  6. Clarity of communication.

Each of the above requirements, ethical principles and rules for action are unpacked in detail in the full report and summarised in the executive brief.

How the project unfolded

The Roadmap process was a bold experiment. The Group of Eight had never attempted to pull together such a large report, with so many participants and diverse perspectives, in such a short time, and where no face-to-face meeting was possible. The SWARM platform, still a research prototype, had never previously been used to address a real problem, let alone a problem of this scale and importance.

The project had a steering committee consisting of the project chairs, Professor Shitij Kapur and Go8 CEO Vicki Thomson, and two reasoning experts from the Hunt Lab, Drs. Tim van Gelder and Richard de Rozario.  The committee proposed a project design which would involve two weeks working on the SWARM platform, followed by a week of off-platform final report drafting by a small group from the Go8. The two weeks on SWARM would involve the panel of experts working on 9 major topics, corresponding to the anticipated major sections of the final report, such as “How and when to relax social distancing.” It was expected that the experts would distribute themselves across the topics, with “emergent teams” coalescing to work on producing a draft report for each section. Week 1 on SWARM would be mostly “exploratory” thinking, with panelists mostly posting Resources, comments and chat. Week 2 would be mostly “synthetic” thinking, with emergent teams posting early draft Reports for each topic, and collaboratively refining the most promising drafts. In Week 3, these draft section reports would be integrated into a single overall final report.

The steering committee planned to closely monitor progress over the first two weeks and, if/as necessary, modify the process. The project did unfold largely as planned, but the steering committee had to intervene mid-late in the second week when it was apparent that some topics lacked emergent teams with “critical mass,” and in some cases even where critical mass had developed, the teams needed some guidance and prodding to deliver an adequate section report. At this point, the committee, and in particular one of the Chairs, Shitij Kapur, convened a series of zoom meetings meetings the emergent teams, and developed with them a plan for finalising their section reports. From that point on, most work on the draft section reports was done, over just a few days, using more traditional collaboration techniques, such as as circulating a Word document and communicating by email.

Thus, as things turned out, the process was a novel hybrid of a pure SWARM platform-based approach, and more standard methods. The steering committee were committed from the outset to expediency in getting the intended result (a high-quality final report) rather than being “purist” about the approach being used. The use of more traditional collaboration tools and methods later in the process, was driven by a number factors, including some limitations in the SWARM platform (most importantly, the lack of a “track changes” function in the platform’s document editor), and the natural tendency for people to revert to habits and reflexive behaviours when under great pressure.  It was clear, however, that the SWARM platform played a crucial role in the first half, allowing participants range across all topics, share lots of ideas and discussion, form emergent teams, and at least start drafting reports.

Whither collaborative reasoning?

The Roadmap project highlights the value of collaborative reasoning platforms like SWARM. It is therefore appropriate to close with a few thoughts on how such platforms can help organisations build internal capability to deal with complex issues that they confront – for example, developing a strategy in an uncertain environment (such as the one we are in currently).

The first point to note is that such problems require stakeholders with diverse viewpoints and skills to work collaboratively to craft a solution. Long-time readers of this blog will know that I advocate tools like Issue-Based Information System (IBIS) to help such groups reach a consensus on problem definition, and thus settle issues around “Are we solving the right problem?” or “How should we approach this issue?” However, once the problem is defined by consensus, the group needs to solve it. This is where platforms like SWARM are particularly useful.

Although SWARM was designed for the intelligence community, the Roadmap  project shows that it can be used in other settings. As another example, Tim van Gelder notes  that citizen intelligence (where ordinary citizens collaborate on solving intelligence problems) is becoming a thing, but lacks a marketplace. As a possible solution, he envisages the creation of a Kaggle-like platform for complex problems (rather than data problems). He notes that there are challenges around setting up such platforms, but there is interest from large private (non-intelligence) organisations. New deployments of the platform are already underway.

The problems organisations confront in the post-Covid world will be more complex than ever before. There are those who believe such problems will yield to computational approaches that rely primarily on vast quantities of data.  However, complex situations cannot be characterised by data alone, so computational approaches will need to be augmented by human sensemaking and reasoning. The success of the Roadmap to Recovery project demonstrates that platforms like SWARM can help organisations tackle such problems by harnessing the power of collaborative reasoning.

Note: For more information on SWARM, please visit the Hunt Lab for Intelligence Research.

Acknowledgement: My thanks to Dr. Tim van Gelder for reviewing a draft version of this article and for contributing the section on how the project unfolded.

Written by K

May 26, 2020 at 8:39 am

The worth of an education – a metalogue

leave a comment »

To be prepared against surprise is to be trained. To be prepared for surprise is to be educated.” – James Carse in Finite and Infinite Games.

Prospective student: How much do your students earn after they graduate?

Teacher: I’m sorry, I don’t know.

Prospective student: What do you mean? Don’t you keep track of what students do after they complete the program?

Teacher: Yes, we do, but we don’t ask them how much they earn. It is not the kind of question one asks….

Prospective student:  I know that, but shouldn’t you be gathering hard data on outcomes?

Teacher:  Of course, and we gather a range of data including the roles they get on graduation and how they progress in their careers.

Prospective student:  OK, but that doesn’t help me. How do I know that your degree is worth the hefty fees you charge?  If I’m investing that much, I must be sure of a decent return.

Teacher:  One cannot be certain about anything in life, there are never any guarantees. However, it’s perfectly reasonable to expect value from your education, so let me ask you – what would you consider a decent return?

Prospective student:   Hmm, a good job, I guess.

Teacher: Well, you heard from some of our alumni this evening.  They are currently employed as professionals in the field, and most of them got their jobs while studying or soon after completing their degree.

Prospective student: Yes, but you are unlikely to invite those who would say negative things about the program or those who have failed to get jobs.

Teacher: Fair enough, there are a few of those.

Prospective student: Well, that’s just my point. How do I know it will work for me?

Teacher:  You don’t!  Getting a job is an indirect effect of a good education.

Prospective student: I’m not sure I follow.

Teacher: An analogy might help clarify what I mean by indirect, Bill Gates did not become a multi-billionaire by setting out to become one. He became one by following his interests, his passion.

Prospective student: [a tad irritated] I still don’t get it.

Teacher: The objective of education is not to train you for a vocation…although, if you do things right, you are almost certain to get a job, and a good one at that. The aim of education is personal transformation, to broaden your perspective and thus enable you to look anew at the things you do at work and, possibly, even in life. Another aim is to prepare you to become that buzzword: a lifelong learner. No university can teach you those things, but they can help you learn them.

Prospective student: “personal transformation” sounds wonderful but terribly vague. Could you give me an example?

Teacher: Well, you’ve just heard from a few of our alumni and I could tell you many more stories. The thing is, I don’t think it is helpful to hear second-hand stories – their journeys are theirs, not yours. The story I’m interested in right now is yours: what you’re thinking, what you do and where you want to go.

Prospective student: OK. I’m a financial analyst [Editor’s note: feel free to substitute your current profession here] and I want to be a data scientist. Will this course enable me to become one?

Teacher: It could, but whether or not it actually does depends largely on you.

Prospective student: That’s not an answer.

Teacher: It is, and it’s an honest one. No course will make you a data scientist. And if any university tells you they can, they’re lying. What a good university course will do is help you learn the technical and non-technical skills that will enable you to become a data scientist.  Whether you learn or not depends on you. The responsibility for your personal transformation lies largely with you. All we can do is show you the way.

Prospective student: so, do you cover … [student recites a litany of data science languages and techniques].

Teacher: Yes, we cover them.

Prospective student:  Won’t doing those make me a data scientist?

Teacher: No. If tech skills are all you are after, I’d strongly suggest you don’t join our program…or any other university program for that matter. Instead, head off to one of the good online data science education providers and save yourself a whole lot of money.

Prospective student: Huh?

Teacher:  A good face-to-face program at a university covers a whole lot more than tech. For example, there are certain tacit skills and dispositions that are critical to becoming a good data scientist. These skills have to do with problem finding rather than technical adeptness or problem solving.  A good university course will give you opportunities to gain experience in doing that.

Prospective student; Problem finding? What’s that?

Teacher: In university assignments you’re given readymade problems that you can go off and solve. In real life, however, you are rarely given a problem. More often, you are presented with a situation from which you must extract or formulate a problem before you solve it. That’s not always a straightforward process because every situation is unique in its details.

Prospective student: If every situation is unique then there is no formula to deal with it.

Teacher: Exactly! These skills have to be taught indirectly – by putting students in safe-to-fail situations in which they can learn how to deal with the ambiguity inherent in them

Prospective student: But won’t that be throwing students into a situation they’re unprepared for.

Teacher: Although they may not admit it, most consultants – even experienced ones – rarely feel totally in control when dealing with new clients. It’s good to experience that kind of ambiguity early in one’s career, even if it is a second career. Every consulting engagement is a learning experience. This ties in with what I mentioned earlier – becoming a lifelong learner.

Prospective student: So how do you prepare students to deal with these types of scenarios?

Teacher: Through carefully crafted technical and non-technical subjects, with assignments that make them think rather than just do. To do well in the assignments you will have to think things through, try different approaches and even make judgement calls.

Prospective student: Judgement calls?

Teacher:  Yes, that’s right. You will find that the biggest issues when doing data science in the real world are not technical, rather they are about dealing with ambiguous situations in which you don’t have a well-defined problem or adequate data.  Then there are ethical issues that are becoming ever more important today. There are big corporations that completely ignore the ethical implications of what they do. Just because you can do something, it doesn’t mean you should.  All these issues involve judgement calls in which data is of little or no help.

Prospective student: Hmm, I didn’t realise there were so many facets to being a data scientist. Thanks, you’ve given something to think about.

Teacher: No worries…and good luck, I hope you find what you’re looking for.

 

 

Afterword:

HR gurus and consultants continually pontificate about the future of work. The ground reality for many mid-career professionals is that the future of their work is highly uncertain, much of the uncertainty being fuelled by the perception that data-related technologies are going to “disrupt” established industries. Among other things, this has led to an unprecedented demand for courses that teach data-related skills.

What is often left unsaid, however, is the transition to data science – or any profession for that matter – involves more than just picking up technical skills. The biggest missing piece (in my opinion) is the ability to make sense of ambiguous situations. This is a tacit skill that is difficult, if not impossible, to teach but can be learnt given the right environment and attitude. The university ought to provide the environment, the student the attitude.

Note: A metalogue is a dialogue that unfolds in such a way that the structure of the conversation turns out to be illustrative of the issue being discussed. The anthropologist Gregory Bateson coined the term.  Here is a metalogue written by him.

Written by K

February 18, 2020 at 5:40 am

Posted in Metalogues, Organizations, sensemaking

Tagged with

Complex decision making as an infinite game

with 10 comments

A decision is the act of choosing between two or more options.

There are two kinds of decisions, computable and non-computable [1]. In the former, options are well-defined and finite in number, and there are unambiguous facts (data) available based on which options can be rated. In the latter, options are neither clear nor enumerable and facts, if available at all, are ambiguous.

Computable decisions are simple, non-computable decisions are complex. We’ll refer to the two decision types by these names in the remainder of this article.

An example of a simple decision is buying a product (TV, car or whatever) based on well-defined criteria (price, features etc.). An example of a complex decision is formulating a business strategy.

It should be clear that simple decisions involve smaller temporal and monetary stakes – i.e. the cost of getting things wrong is limited and the effects of a bad decision wear off in (a relatively short) time. Neither is true for complex decisions: the cost of a poor choice can be significant, and its negative effects tend to persist over time.

A key feature of complex decisions is that they (usually) affect multiple parties. That is, they are socially complex. This has implications regarding how such decisions should be approached. More on this later.

Conventional decision theory is based on the notion of maximizing benefit or utility. For simple decisions it is assumed that utility of each option can be computed; for complex decisions it is assumed they can be estimated, or at least ranked. The latter assumption is questionable because each party affected by a complex decision will have its own notion of utility, at least at the outset. Moreover, since neither options nor facts are unambiguous at the start, it makes little sense to attempt to estimate utility upfront.

The above being the case, it is clear that complex decisions cannot be made on the basis of maximizing utility alone.  Something else is needed.

–x–

James Carse’s classic book, Finite and Infinite Games, begins with the following lines:

There are at least two kinds of games. One could be called finite, the other infinite. A finite game is played for the purpose of winning, an infinite for the purpose of continuing the play.

A finite game ends when a player or team wins. However, “just as it is essential for a finite game to have a definitive ending, it must also have a precise beginning. Therefore, we can speak of finite games as having temporal boundaries.”

The parallel between simple decisions and finite games should be evident. Although less obvious, it is useful to think of a complex decision as an infinite game.

When making a complex decision – such as a business strategy – decision-makers will often focus on maximising potential benefits (aka utility). However, as often as not, the outcome of the decision will fall far short of the expected benefits and may, in some cases, even lead to ruin. This being so, it is perhaps more fruitful to focus on staying in the game (keep playing) rather than winning (maximising utility).

The aim of a complex decision should be to stay in the game rather than win.

How does one ensure that one stays in the game? Heinz von Foerster’s ethical imperative offers an answer”

Always act to increase your choices.

That is, one should decide in such a way that increases one’s options in the future thereby increasing chances of staying in the game. One can frame this discussion in terms of adaptability:  the greater the number of options the greater the ability to adapt to unexpected changes in the environment.

How can one “act to increase one’s choices”?

One way to do this is to leverage social complexity: get different parties to articulate their preferred options. Some of these options are likely to contradict each other. Nevertheless, there are ways to handle such a diversity of potentially contradictory views in an inclusive manner (for an example, see this paper; for more, check out this book). Such an approach also ensures that the problem and solution spaces are explored more exhaustively than if only a limited number of viewpoints are considered.

infinite_options_graphic

The point is this: there are always more options available than apparent. Indeed, the number of unexplored options at any stage is potentially infinite. The job of the infinite player (decision-maker) is to act so as surface them gradually, and thus stay in the game.

–x–

Traditionally, decision-making is seen as a logical undertaking based on facts or data. In contrast, when viewed as an infinite game, complex decision-making becomes a matter of ethics rather than logic.

Why ethics?

The answer lies in von Foerster’s dictum to increase one’s choices.  By doing so, one increases the chances that fewer stakeholders’ interests are overlooked in the decision-making process.

As Wittgenstein famously said, “It is clear ethics cannot be articulated.” All those tedious classes and books on business ethics miss the point entirely. Ethical matters are necessarily oblique:  the decision-maker who decides in a way that increases (future) choices, will be acting ethically without drawing attention to it, or even being consciously aware of it.

–x–

Any honest discussion of complex decision-making in organisations must address the issue of power.

Carse asserts that players (i.e. decision-makers in the context of this article) become powerful by acquiring titles (e.g. CEO, Manager etc.). However, such titles can only be acquired by winning a finite game– i.e. by being successful in competitions for roles. Power therefore relates to finite rather than infinite games.

As he notes in his book:

Power is a concept that belongs only in finite play. To speak meaningfully of a person’s power is to speak of what that person has already achieved, the titles they have already won.

Be that as it may, one cannot overlook the reality that those in powerful positions can (and often do) subvert the decision-making process by obstructing open and honest discussion of contentious issues. Sometimes they do so by their mere presence in the room.

How does a complex decision-maker deal with the issue of power?

Carse offers the following answer:

How do infinite players contend with power? Since the outcome of infinite play is endlessly open, there is no way of looking back to make an assessment of the power or weakness of earlier play. Infinite players look forward, not to a victory but toward ongoing play. A finite player plays to be powerful; the infinite player plays with strength. Power is concerned (and a consequence of) what has happened, strength with what has yet to happen. Power will be always restricted to a relatively small number of people. Anyone can be strong.

What strength means is context-dependent, but the following may help clarify its relationship to power:

Late last year I attended an end-of-year event at the university I teach at. There I bumped into a student I had mentored some time ago. We got talking about his workplace (a large government agency).

At one point he asked, “We really need to radically change the way we think about and work with data, but I’m not a manager and have no authority to initiate changes that need to be made.”

“Why don’t you demonstrate what you are capable of? Since you are familiar your data, it should be easy enough to frame and tackle a small yet meaningful data science problem.” I replied.

“What if my manager doesn’t like my taking the initiative?”

“It is easier to beg forgiveness than seek permission.”

“He might feel threatened and make life difficult for me.”

“If management doesn’t like you’re doing, it’s their loss. What’s the worst that could happen? You could lose your job. With what you are learning at university you should have no trouble moving on to another role. Indeed, by doing so, you will diversify your experience and increase your future options.”

–x–

To summarise:  when deciding on complex matters, act in a way that maximises possibility rather than utility. Such an approach is inherently ethical and enhances one’s chances of staying in the game.

Complex decision making is an infinite game.

[1] There are many other terms for this classification:  tame and wicked (Horst Rittel), programmed and non-programmed (Herbert Simon), complicated and complex (David Snowden). Paul Culmsee and I have, perhaps confusingly, used the terms uncertain and ambiguous to refer to these in our books.  There are minor contextual differences between how these different authors interpret these terms, but for the most part they are synonymous with computable/non-computable.

 

Written by K

January 21, 2020 at 4:09 am

%d bloggers like this: