What is Emergent Design?

Last week I had the opportunity to talk to a data science team about the problems associated with building a modern data capability in a corporate environment. The organisation’s Head of Data was particularly keen to hear about how the Emergent Design approach proposed in my recent book might help with some of the challenges they are facing. To help me frame my talk, he sent me a bunch of questions, which included the following two:

1. Can you walk us through the basic steps of emergent design in data science?
2. How does emergent design work with other data science methodologies, such as CRISP-DM or Agile?

On reading these questions, I realized I had a challenge on my hands: Emergent Design is about cultivating a mindset or disposition with which to approach problematic situations, rather than implementing a canned strategic framework or design methodology. Comparing it to other methodologies would be a category error – like comparing pizza to philosophy. (Incidentally, I think this is the same error that some (many?) Agile practitioners make when they mistake the rituals of Agile for the mindset required to do it right…but that’s a story for another time.)

The thing is this: the initial task of strategy or design work is about a) understanding what ought to be done, considering the context of the organisation and b) constructing an appropriate approach to doing it.  In other words, it is about framing the (strategic or any other) problem and then figuring out what to do about it, all the while keeping in mind the specific context of the organization. Emergent Design is a principled approach to doing this.

–x–

So, what is Emergent Design?

A good place to start is with the following passage from the PhD thesis of David Cavallo, the man who coined the term and developed the ideas behind it:

“The central thrust of this thesis is the presentation of a new strategy for educational intervention. The approach I describe here resembles that of architecture, not only in the diversity of the sources of knowledge it uses but in another aspect as well – the practice of letting the design emerge from an interaction with the client. The outcome is determined by the interplay between understanding the goals of the client; the expertise, experience, and aesthetics of the architect; and the environmental and situational constraints of the design space. Unlike architecture where the outcome is complete with the artifact, the design of [such initiatives] interventions is strengthened when it is applied iteratively. The basis for action and outcome is through the construction of understanding by the participants. I call this process Emergent Design.”

Applied to the context of building a data capability, Emergent Design involves:

1. Having conversations across the organization to understand the kinds of problems people are grappling with. The problems may or may not involve data. The ones that do not involve data give you valuable information about the things people worry about.  In general the conversations will cover a lot of territory and that is OK – the aim is to get a general feel for the issues that matter.
2. Following the above, framing a handful – two or three – concrete problems that you can solve using skills and resources available on hand. These proof-of-concept projects will help you gain allies and supporters for your broader strategic efforts.

While doing the above, you will notice that people across the organization have wildly different perspectives on what needs to be done and how. Moreover, they will have varying perspectives on what technologies should be used. As a technology strategist, your key challenge is around how to reconcile and synthesise these varied viewpoints. This is what makes the problem of strategy a wicked problem. For more on the wicked elements of building data capabilities, check out the first chapter of my book which is available for free here.

Put simply then, Emergent Design is about:

1. Determining a direction rather than constructing a roadmap
2. Letting the immediate next steps be determined by what adds the greatest value (for your stakeholders)

With that said for the “what,” I will now say a few words about the “how.”

–x–

In the book we set out eight guidelines or principles for practicing Emergent Design. I describe them briefly below:

Be a midwife rather than an expert: In practical terms, this involves having conversations with key people in business units across the organisation, with the aim of understanding their pressing problems and how data might help in solving them (we elaborate on this at length in Chapter 4 of the book). The objective at this early stage is to find out what kind of data science function your organisation needs. Rather than deep expertise in data science, this requires an ability to listen to experts in other fields, and translate what they say into meaningful problems that can potentially be solved by data science. In other words, this requires the strategist to be a midwife rather than an expert.

Use conversations to gain commitment: In their ground- breaking book on computers and cognition, Winograd and Flores observed that “organisations are networks of commitments.” between people who comprise the organisation. It is through conversations that commitments between different groups of stakeholders are established and subsequently acted on. In Chapter 3 of the book, we offer some tips on how to have such conversations. The basic idea in the above is to encourage people to say what they really think, rather than what they think you want them to say. It is crucial to keep in mind that people may be unwilling to engage with you because they do not understand the implications of the proposed changes and are fearful of what it might mean for them.

Understand and address concerns of stakeholders who are wary of the proposed change: In our book, The Heretic’s Guide to Management, Paul Culmsee and I offer advice on how to do this in specific contexts using what we call “management teddy bears”. These involve offering reassurance, advice, or opportunities that reduce anxiety, very much akin to how one might calm anxious children by offering them teddy bears or security blankets. Here are a few examples of such teddy bears:

• A common fear that people have is that the new capability might reduce the importance of their current roles. A good way to handle this is to offer these people a clear and workable path to be a part of the change. For example, one could demonstrate how the new capability (a) enriches their current role or (b) offers opportunities to learn new skills or (c) enhances their effectiveness. We could call this the “co- opt teddy bear”. In Chapter 7 of our data science strategy book, we offer concrete ways to involve the business in data science projects in ways that makes the projects theirs.
• It may also happen that some stakeholder groups are opposed to the change for political reasons. In this case, one can buy time by playing down the significance of the new capability. For example, one could frame the initiative as a “pilot” project run by the current data and reporting function. We could call this the “pilot teddy bear.” See the case study towards the end of Chapter 3 of the data science strategy book for an example of a situation in which I used this teddy bear.

Frame the current situation as an enabling constraint: In strategy development, it is usual to think of the current situation in negative terms, a situation that is undesirable and one that must be changed as soon as practicable. However, one can flip this around and look at the situation from the perspective of finding specific things that you can change with minimal political or financial cost. In other words, you reframe the current situation as an enabling constraint (see this paper by Kauffman and Garre for more on this). The current situation is well defined, but there are an infinite number of possible next steps. Although the actual next step cannot be predicted, one can make a good enough next step by thinking about the current situation creatively in order to explore what Kauffman calls the adjacent possible – the possible future states that are within reach, given the current state of the organisation (see this paper by Kauffman). You may have to test a few of the adjacent possible states before you figure out which one is the best. This is best done via small, safe- to- fail proof of con­cept projects (discussed at length in Chapter 4 of our data science strategy book).

Consider long- term and hidden consequences: It is a fact of life that when choosing between different approaches, people will tend to focus on short- term gains rather than long- term consequences. Indeed, one does not have to look far to see examples that have global implications (e.g., the financial crisis of 2008 and climate change). Valuing long- term results is difficult because the distant future is less salient than the present or the immediate future. A good way to look beyond immediate concerns (such as cost) is to use the “solution after next principle” proposed by Gerald Nadler and Shozo Hibino in their book entitled Breakthrough Thinking. The basic idea behind the principle is to get people to focus on the goals that lie beyond the immediate goal. The process of thinking about and articulating longer term goals can often provide insights into potential problems with the current goals and/ or how they are being achieved. We discuss this principle and another approach to surfacing hidden issues in Chapters 3 and 4 of the data science strategy book.

Create an environment that encourages learning: Emergent Design is a process of experimentation and learning. However, all learning other than that of the most trivial kind involves the possibility of error. So, for it to work, one needs to create an environment of psychological safety – i.e., an environment in which employees feel safe to take risks by trialling new ideas and processes, with the possibility of failure. A key feature of learning organisations is that when things go wrong, the focus is not on fixing blame but on fixing the underlying issue and, more importantly, learning from it so that one reduces the chances of recurrence. It is interesting to note that this focus on the system rather than the individual is also a feature of high reliability organisations such as emergency response agencies.

Beware of platitudinous goals: Strategies are often littered with buzzwords and platitudes – cliched phrases that sound impressive but are devoid of meaning. For example, two in- vogue platitudes at the time our book was being written are “digital transformation” and “artificial intelligence.” They are platitudes because they tell you little about what exactly they mean in the specific context of the organisation.

The best way to deconstruct a platitude is via an oblique approach that is best illustrated through an example. Say someone tells you that they want to implement “artificial intelligence” (or achieve a “digital transformation” or any other platitude!) in their organisation. How would you go about finding out what exactly they want? Asking them what they mean by “artificial intelligence” is not likely to be helpful because the answer you will get is likely to be couched in generalities such as data- driven decision making or automation, phrases that are world- class platitudes in their own right! Instead, it is better to ask them how artificial intelligence would make a difference to the organisation. This can help you steer the discussion towards a concrete business problem, thereby bringing the conversation down from platitude- land to concrete, measurable outcomes.

Act so as to increase your choices: This is perhaps the most important point in this list because it encapsulates all the other points. We have adapted it from Heinz von Foerster’s ethical imperative which states that one should always act so as to increase the number of choices in the future (see this lecture by von Foerster). Keeping this in mind as you design your data science strategy will help you avoid technology or intellectual lock in. As an example of the former, when you choose a product from a particular vendor, they will want you to use their offerings for the other components of your data stack. Designing each layer of the stack in a way that can work with other technologies ensures interoperability, an important feature of a robust data technology stack (discussed in detail in Chapter 6 of the strategy book). As an example of the latter, when hiring data scientists, hire not just for what they know now but also for evidence of their curiosity and proclivity to learn new things – a point we elaborate on in Chapter 5 of the data strategy book.

You might be wondering why von Foerster called this the “ethical imperative.”  An important aspect of this principle is that your actions should not constrain the choices of others. Since the predictions of your analytical models could well affect the choice of others (e.g. whether or not they are approved for a loan or screened out for a job), you should also cast an ethical eye on your work. We discuss ethics and privacy at length in Chapter 8 of the data strategy book.

–x–

These principles do not tell you what to do in specific situations. Rather, they are about cultivating a disposition that looks at technology through the mul­tiple, and often conflicting, perspectives of those whose work is affected by it. A technical capability is never purely technical; within the context of an  organisation it becomes a sociotechnical capability.

In closing: Emergent Design encourages practitioners to recognise that building and embedding a data capability (or any other technical capability such as program evaluation) is a process of organisational change that is best achieved in an evolutionary manner. This entails starting from where people are – in terms of capability, culture, technology, and governance – and enabling them to undertake a collective journey in an agreed direction.

…and, yes, all the while keeping in mind that it is the journey that matters, not the destination.

Preface to “Data Science and Analytics Strategy: An Emergent Design Approach”

As we were close to completing the book (sample available here), Harvard Business Review published an article entitled, Is Data Scientist Still the Sexiest Job of the 21st Century? The article revisits a claim made a decade ago, in a similarly titled piece about the attractiveness of the profession.2 In the recent article, the authors note that although data science is now a well- established function in the business world, setting up the function presents a number of traps for the unwary. In particular, they identify the following challenges:

• The diverse skills required to do data science in an organisational setting.
• A rapidly evolving technology landscape.
• Issues around managing data science projects; in particular, productionising data science models – i.e., deploying them for ongoing use in business decision- making.
• Putting in place the organisational structures/ processes and cultivating individual dispositions to ensure that data science is done in an ethical manner.

On reviewing our nearly completed manuscript, we saw that we have spoken about each of these issues, in nearly the same order that they are discussed in the article (see the titles of Chapters 5– 8). It appears that the issues we identified as pivotal are indeed the ones that organisations face when setting up a new data science function. That said, the approach we advocate to tackle these challenges is somewhat unusual and therefore merits a prefatory explanation.

The approach proposed in this book arose from the professional experiences of two very different individuals, whose thoughts on how to “do data” in organisational settings converged via innumerable conversations over the last five years. Prior to working on this book, we collaborated on developing and teaching an introductory postgraduate data science course to diverse audiences ranging from data analysts and IT professionals to sociologists and journalists. At the same time, we led very different professional lives, working on assorted data- related roles in multinational enterprises, government, higher education, not- for- profit organisations and start- ups. The main lesson we learned from our teaching and professional experiences is that, when building data capabilities, it is necessary to first understand where people are – in terms of current knowledge, past experience, and future plans – and grow the capability from there.

To summarise our approach in a line: data capabilities should be grown, not grafted.

This is the central theme of Emergent Design, which we introduce in Chapter 1 and elaborate in Chapter 3. The rest of the book is about building a data science capability using this approach.

Naturally, we were keen to sense- check our thinking with others. To this end, we interviewed a number of well- established data leaders and practitioners from diverse domains, asking them about their approach to setting up and maintaining data science capabilities. You will find their quotes scattered liberally across the second half of this book. When speaking with these individuals, we found that most of them tend to favour an evolutionary approach not unlike the one we advocate in the book. To be sure, organisations need formal structures and processes in place to ensure consistency, but many of the data leaders we spoke with emphasised the need to grow these in a gradual manner, taking into account the specific context of their organisations.

It seems to us that many who are successful in building data science and analytics capabilities tacitly use an emergent design approach, or at least some elements of it. Yet, there is very little discussion about this approach in the professional and academic literature. This book is our attempt at bridging this gap.

Although primarily written for business managers and senior data professionals who are interested in establishing modern data capabilities in their organisations, we are also speaking to a wider audience ranging from data science and business students to data professionals who would like to step into management roles. Last but not least, we hope the book will appeal to curious business professionals who would like to develop a solid understanding of the various components of a modern data capability. That said, regardless of their backgrounds and interests, we hope readers will find this book useful … and dare we say, an enjoyable read.

Note:

You can buy the book from the Routledge website. If you do, please use the code AFL01 for a 20% discount (code valid until June 2023). Note that the discount has already been applied in some countries.

Perceptions of change

Management, as it is taught in business schools, is rife with abstractions such as “strategic alignment” and “organizational culture”. The incident I’m about to relate happened about nine years ago, a few days after I had read Claudio Ciborra’s brilliant critique of strategic alignment and published an article about it.

I was at a company dinner where I happened to be sitting next to a senior executive from headquarters.  At that time the organization was in the throes of a large-scale IT transformation initiative aimed at “aligning IT with the business.” As might be expected, the conversation turned to the impending changes and how they would achieve “strategic alignment.”

Perhaps unwisely, I started talking about Ciborra’s critique and the gap between management abstractions and coalface reality. The conversation segued into the differences between management and employee perceptions of the changes we were going through, and at some point I said, “employee perceptions tend to become their reality.”

 The executive set his fork down on his plate. “You’ve got that wrong,” he said with a tight smile, “my perception is your reality.”

–x–

Management theorists invented the concept of organizational culture to deal with the “problem” of aligning employee values with those of the organization.  However, as noted in a classic paper by Hugh Willmott, the concept is inherently flawed, not to mention a shade Orwellian:

“[the notion of organizational culture is based on] an implicit understanding that the distinctive quality of human action, and of labour power, resides in the capacity of self-determination [of purpose and action]. This insight informs the understanding that corporate performance can be maximized only if this capacity is simultaneously respected and exploited…corporate culture invites employees to understand that identification with its values ensures their autonomy. That is the seductive doublethink of corporate culture: the simultaneous affirmation and negation of the conditions of autonomy.”

And then, a bit later in the piece:

“[Advocates of organisational culture] take it for granted that the objectives of the organization can be engineered to become consensual. Since every employee is assumed to share these objectives, and to benefit from their realization, there can be no moral objection to corporate cultural demands.”

However, such thinking is morally ambiguous:

“Instead of contributing to the development of a societal culture in which individuals learn to appreciate, and struggle with, the problematical experience and significance of indeterminacy, [organisational] culturism promotes what is, in effect, a totalitarian remedy for this existential problem. [Organisational] culturism directly exploits the feelings of insecurity and ‘irrationalism’ that are intensified by the capitalist process of commodification [of skills and labour].”

Employees tend to toe the corporate culture line because of the security it appears to offer. However, such buy-in is largely in letter, not spirit: although you might be able to control what people do, and may even get them to pledge allegiance to “organizational values,” you cannot control what they think. This is the point I was trying to get across to the executive.

I left the organization three years later, surprised that I lasted that long.

 –x–

Perceptions of change depend on where one sits in the organizational hierarchy.

Many years ago, I was part of a project team that was working on replacing a venerable Lotus Notes–based system with a newer customer relationship management (CRM) product. I was tasked with integrating data from the CRM with other syndicated and publicly available data sets. The requirements were complex, but the system design evolved through continual, often animated discussions between the development team and key business stakeholders in an environment characterized by openness and trust.

The system was delivered on schedule, with minimal rework required.

Five years later, I was invited to participate in a regional project at the same company. The objective, which was set by the corporate IT office located in Europe, was to build a data warehouse for subsidiaries across Asia.  Corporate’s rationale for the project was quite reasonable. The data landscape across Asia was messy, with each subsidiary taking a bespoke approach to data management in order to address their local reporting needs. From corporate’s perspective, this was a situation crying out for standardization. On the other hand, the subsidiaries were happy with their existing systems. They perceived the push for standardization as a corporate power play that would result in a loss of local autonomy over data. 

To top it all, there were cultural differences around how such conflicts should be resolved.

Predictably, the discussions aimed at reaching a consensus devolved into heated Skype exchanges between stakeholders, forcing regional IT to step in and call a meeting to resolve the issue.

The story of how we resolved differences between corporate and local perspectives is documented in this article and this paper so I won’t reproduce it here. The point I wish to make is that the larger the change, the greater the effort required to align perceptions. Moreover, success depends entirely on developing trust-based relationships between warring stakeholder groups. This involves surfacing key points of contention and developing consensus decisions about them in a way that is acceptable to all parties. This is a matter of communication, not culture.

–x–

That said, I don’t like the term “communication” much. It glosses over details of what one must do to resolve differences, and the details matter because they are not obvious.

The anthropologist and polymath, Gregory Bateson once noted that “what [we lack] is a theory of action within large complex systems, where the active agent is himself a part and a product of the system.”

In the very next line to the one quoted above, Bateson offered a hint as to where an answer might lie. He noted that Kant’s categorical imperative – “act so to treat humanity, whether in your own person or in another, always as an end and never as only a means – might provide a starting point for such a theory.”

He then went on to say something truly intriguing: “It seems also that good teachers and therapists avoid all direct attempts to influence the action of others and, instead, try to provide the settings or contexts in which some (usually imperfectly specified) change may occur.”

This line resonated deeply when I read it first because it spelt out something that I had learnt through experience but had not found the words to articulate: change is best achieved by framing (or creating) a context within which individuals will see things differently and change of their own accord.

–x–

“I can’t handle failure,” she said. “I’ve always been at the top of my class.”

She was being unduly hard on herself. With little programming experience or background in math, machine learning was always going to be hard going.  “Put that aside for now,” I replied. “Just focus on understanding and working your way through it, one step at a time. In four weeks, you’ll see the difference.”

“OK,” she said, “I’ll try.”

She did not sound convinced but to her credit, that’s exactly what she did. Two months later she completed the course with a distinction.

“You did it!” I said when I met her a few weeks after the grades were announced.

“I did,” she grinned. “Do you want to know what made the difference?”

Yes, I nodded.

“Thanks to your advice, I stopped treating it like a game I had to win,” she said, “and that took the pressure right off.  I then started to enjoy learning.”

–x–

The student reframed her thinking in a way that changed her perceptions of the task at hand. Instead of treating it as an obstacle or race, she began to view it as an opportunity to learn. Doing so enabled her to meet the academic requirements of the university. Paradoxically, she passed the course with ease when she stopped obsessing about doing well and focused on learning instead. This echoes Bateson’s advice about changing a system from within and is an example of what John Kay calls obliquity: the idea that certain goals are best achieved indirectly. 

You do not get people to change their perceptions by telling them to change. Instead, you reframe the situation in a way that enables them to see it in a different light. They might then choose to change of their own accord.

As a change agent, isn’t that what you really wish for?

–x–