Collaborative reasoning in the age of Covid

Ever since the start of the pandemic, there have been no end of opinions, presentations and reports on how we might navigate our way out of the crisis. Much of this takes a narrow, discipline-centric view, which is inadequate because the problem is multifaceted and thus defies traditional disciplinary boundaries. It is therefore of urgent importance to chart a course that considers all aspects of recovery, not just those relevant to specific interests.  A recent report produced by the Australian Group of Eight does just that.   The key points of the report are concisely described in an executive summary and snapshot, so I will cover just the main points in this article. My focus instead is on the platform used to create the report, as it offers an effective collaborative approach to tackling complex issues in a broad range of contexts.

To me the most amazing thing about the 192-page report is that it was produced by a taskforce comprised of over a hundred academics and researchers across diverse disciplines, collaborating over a three-week period. As stated in the exec summary:

To chart a Roadmap to Recovery we convened a group of over a hundred of the country’s leading epidemiologists, infectious disease consultants, public health specialists, healthcare professionals, mental health and well-being practitioners, indigenous scholars, communications and behaviour change experts, ethicists, philosophers, political scientists, economists and business scholars from the Group of Eight (Go8) universities. The group developed this Roadmap in less than three weeks, through remote meetings and a special collaborative reasoning platform, in the context of a rapidly changing pandemic.

Those who have done any collaborative work involving large groups will have stories to tell about how challenging it is to get a coherent result.  This taskforce achieved this in part by working on an online collaborative reasoning platform called SWARM, described in this paper.  This post is mainly about what SWARM is and how it works, but I will also describe how the Roadmap taskforce used the platform to come up with a comprehensive recovery plan and the key recommendations made therein. I’ll end with some thoughts on the use of SWARM in broader organizational and business contexts.

The SWARM platform

The platform was designed and implemented by a team led by Drs. Tim van Gelder and Richard de Rozario as part of a large Intelligence Advanced Research Projects Activity (IARPA) initiative. In essence,  SWARM is a cloud collaboration environment designed to enhance evidence-based reasoning in teams. It does this by supporting an approach called contending analyses, wherein team members produce and refine multiple distinct analyses of a problem, and then select the best one as their collective response.

On SWARM, team members create artefacts that represent their reasoning. Additionally, they can rate, comment on and contribute to artefacts created by others through the course of the challenge. This enables a “best response” to emerge through an iterative process of discussion, refinement and evaluation.

To understand how it works, it is necessary to briefly describe the various ways in which users can interact and contribute to solving the problem with each other in SWARM. The user interface of the SWARM platform consists of three panes (Figure 1).

Figure 1: SWARM user interface

 

The left pane contains the problem description and links to related documents. In the centre pane, users can post and update responses. A response may be a Resource (e.g. a link to an external article, a visualisation or an analysis) that contributes to understanding or solving the problem, or it may be a Report, which is a draft candidate for the team’s final output. Users can then comment on and rate others’ responses and comments. The most highly rated Report at the conclusion of the problem is submitted as the result of the group’s collaborative reasoning.

The right pane is a streaming chat window through which users can interact in real-time. To summarise, SWARM users can:

1. converse with team members via the chat feed.
2. post or update a Resource or a Report
3. comment on a Resource or a Report, or
4. rate a Resource, Report or Comment.

By design, SWARM does not prescribe (or proscribe) any particular analytical process. As van Gelder, de Rozario and Sinnott (2018, pp. 22-34) note, contending analyses:

…promotes engagement by providing the opportunity for any participant to contribute their own thinking (autonomy), to think in a manner matching their natural expertise (mastery or competence), and to earn the respect of others by drafting a well-regarded response (relatedness)’ – thus meeting each of the three psychological needs identified by self-determination theory.

The idea is that teams should be free to work in ways that suit them collectively, with individuals given the choice to contribute as and when they please. That said, SWARM, via its Lens Kit (https://lenskit.atlassian.net/wiki/spaces/LK/overview), offers participants a compendium of structured analytical techniques and other “logical lenses” that may be useful in analysing complex and uncertain scenarios in which the available information is scarce or  ambiguous.

The Roadmap to Recovery project

The Roadmap project involved over a hundred academics from the Group of Eight – a coalition of the oldest, largest and most research-intensive Australian universities. Over three weeks in April 2020, the team worked on developing scenarios for national recovery from the COVID crisis. Their recommendations are available in a comprehensive report.  The report is unique in that it synthesises the knowledge of a range of experts and takes a systemic, evidence-based view of the problem.  In the words of the co-chairs of the project:

How this document differs from the hundreds of articles and opinion pieces on this issue is that this report specifies the evidence on which it is based, it is produced by researchers who are experts and leaders in their area, and it engages the broadest range of disciplines – from mathematicians, to virologists, to philosophers.

Over a three-week period, this taskforce has debated and discussed, disagreed, and agreed, edited and revised its work over weekdays and holidays, Good Friday and Easter. All remotely. All with social distancing…

…It is research collaboration in action – a collective expression of a belief that expert research can help Government plot the best path forward…

Given the wide geographical distribution of the team and the requirement for social distancing, it was clear that the team needed an online collaboration platform that enabled collective deliberation. Traditional online methods would not have worked for a group this large. As noted in the report:

Standard remote collaboration methods, such as circulating drafts by email, have many drawbacks such as the difficulty of keeping track of document versions, integrating edits and comments on many different versions, and ensuring that everyone can see the latest version. It seemed clear this approach would struggle with an expert group as large as the Roadmap Task Force.

The steering committee therefore decided to give SWARM a go.

As noted in the previous section, SWARM works on the principle that a group should canvas multiple approaches and then collectively settle on the best one, a principle summarised by the term contending analyses. The benefit of such an approach is evident in the report in that it outlines two distinct strategies for recovery:

1. Elimination: as the term suggests, this strategy aims at eliminating the virus within the country. This is the lowest risk approach and is technically feasible for a relatively isolated country like Australia. However, the cost in terms of time, effort and money is substantial. Moreover, a strict implementation of this approach would bar international travel for an unrealistically long duration.
2. Controlled Adaptation: this involves controlling the infection within the country to a level that does not overwhelm the healthcare system. This is less expensive in terms of time, effort and money, but the outcome is also less certain. However, as the taskforce points out, this could lead to restrictions being eased as early as May 15^th, a choice that the government had made before the report was released. This decision is understandable given the cost of extended restrictions; however, it isn’t clear at all how they will handle the inevitable resurgence of the disease down the line. The report considers how things could develop as a result of this decision.

The report aims to provide a balanced case for the two options, and also emphasises that in terms of implementation, the options have considerable overlap. For instance, there are three requirements for the success of either:

1. Early detection and supported isolation
2. Travel and border restrictions.
3. Public trust, transparency and civic engagement.

It should be clear that all three require massive government involvement and support. To this end, the taskforce has formulated an ethical framework that should guide government decision-making and policy. The framework comprises of the following six principles:

1. Democratic accountability and the protection of civil liberties.
2. Equal access to healthcare and social welfare.
3. Shared economic sacrifice.
4. Attentiveness to the distinctive patterns of disadvantage.
5. Enhancing social well-being and mental health.
6. Partnership and shared responsibility

An ethical framework should serve as a check on policy-making that might disadvantage specific groups. If followed, the six principles listed above will ensure that policies are fair to all sections of the community, both in terms of burdens and benefits This is perhaps the trickiest part of policy-making.

Finally, the taskforce has formulated six imperatives (essential rules) that should guide the actual implementation of a recovery. They are:

1. The health of our healthcare system and its workers.
2. Preparing for relaxation of social distancing.
3. Mental health and wellbeing for all.
4. The care of indigenous Australians.
5. Equity of access and outcomes in health support.
6. Clarity of communication.

Each of the above requirements, ethical principles and rules for action are unpacked in detail in the full report and summarised in the executive brief.

How the project unfolded

The Roadmap process was a bold experiment. The Group of Eight had never attempted to pull together such a large report, with so many participants and diverse perspectives, in such a short time, and where no face-to-face meeting was possible. The SWARM platform, still a research prototype, had never previously been used to address a real problem, let alone a problem of this scale and importance.

The project had a steering committee consisting of the project chairs, Professor Shitij Kapur and Go8 CEO Vicki Thomson, and two reasoning experts from the Hunt Lab, Drs. Tim van Gelder and Richard de Rozario.  The committee proposed a project design which would involve two weeks working on the SWARM platform, followed by a week of off-platform final report drafting by a small group from the Go8. The two weeks on SWARM would involve the panel of experts working on 9 major topics, corresponding to the anticipated major sections of the final report, such as “How and when to relax social distancing.” It was expected that the experts would distribute themselves across the topics, with “emergent teams” coalescing to work on producing a draft report for each section. Week 1 on SWARM would be mostly “exploratory” thinking, with panelists mostly posting Resources, comments and chat. Week 2 would be mostly “synthetic” thinking, with emergent teams posting early draft Reports for each topic, and collaboratively refining the most promising drafts. In Week 3, these draft section reports would be integrated into a single overall final report.

The steering committee planned to closely monitor progress over the first two weeks and, if/as necessary, modify the process. The project did unfold largely as planned, but the steering committee had to intervene mid-late in the second week when it was apparent that some topics lacked emergent teams with “critical mass,” and in some cases even where critical mass had developed, the teams needed some guidance and prodding to deliver an adequate section report. At this point, the committee, and in particular one of the Chairs, Shitij Kapur, convened a series of zoom meetings meetings the emergent teams, and developed with them a plan for finalising their section reports. From that point on, most work on the draft section reports was done, over just a few days, using more traditional collaboration techniques, such as as circulating a Word document and communicating by email.

Thus, as things turned out, the process was a novel hybrid of a pure SWARM platform-based approach, and more standard methods. The steering committee were committed from the outset to expediency in getting the intended result (a high-quality final report) rather than being “purist” about the approach being used. The use of more traditional collaboration tools and methods later in the process, was driven by a number factors, including some limitations in the SWARM platform (most importantly, the lack of a “track changes” function in the platform’s document editor), and the natural tendency for people to revert to habits and reflexive behaviours when under great pressure.  It was clear, however, that the SWARM platform played a crucial role in the first half, allowing participants range across all topics, share lots of ideas and discussion, form emergent teams, and at least start drafting reports.

Whither collaborative reasoning?

The Roadmap project highlights the value of collaborative reasoning platforms like SWARM. It is therefore appropriate to close with a few thoughts on how such platforms can help organisations build internal capability to deal with complex issues that they confront – for example, developing a strategy in an uncertain environment (such as the one we are in currently).

The first point to note is that such problems require stakeholders with diverse viewpoints and skills to work collaboratively to craft a solution. Long-time readers of this blog will know that I advocate tools like Issue-Based Information System (IBIS) to help such groups reach a consensus on problem definition, and thus settle issues around “Are we solving the right problem?” or “How should we approach this issue?” However, once the problem is defined by consensus, the group needs to solve it. This is where platforms like SWARM are particularly useful.

Although SWARM was designed for the intelligence community, the Roadmap  project shows that it can be used in other settings. As another example, Tim van Gelder notes  that citizen intelligence (where ordinary citizens collaborate on solving intelligence problems) is becoming a thing, but lacks a marketplace. As a possible solution, he envisages the creation of a Kaggle-like platform for complex problems (rather than data problems). He notes that there are challenges around setting up such platforms, but there is interest from large private (non-intelligence) organisations. New deployments of the platform are already underway.

The problems organisations confront in the post-Covid world will be more complex than ever before. There are those who believe such problems will yield to computational approaches that rely primarily on vast quantities of data.  However, complex situations cannot be characterised by data alone, so computational approaches will need to be augmented by human sensemaking and reasoning. The success of the Roadmap to Recovery project demonstrates that platforms like SWARM can help organisations tackle such problems by harnessing the power of collaborative reasoning.

Note: For more information on SWARM, please visit the Hunt Lab for Intelligence Research.

Acknowledgement: My thanks to Dr. Tim van Gelder for reviewing a draft version of this article and for contributing the section on how the project unfolded.