As humans, we tend to believe that events and processes will unfold in the way we expect them to. Unfortunately things rarely go according to our expectations and we end up being, well… surprised! But it isn’t just individuals, entire organisations can be caught unawares by events. In this post I draw on a paper entitled, Clues, Cues and Complexity: Unpacking the Concept of Organisational Surprise, to elaborate on the different ways in which surprises can crop up in organisational settings and in particular, in projects. My focus is on the latter because the temporary and one-off nature of projects seems to render them particularly prone to surprises.
The authors define organizational surprise as, “any event that happens unexpectedly or any expected event that takes an unexpected shape.” One does not have to look far to see examples of organizational surprises. It is almost certain that any experienced project manager would have encountered a number of surprises in the course of his or her professional work. These could be unforeseen events (such as a project sponsor leaving for greener pastures) or unexpected twists and turns in what ought to have been a straightforward process (such as a software upgrade turning out to be more complicated than expected).
The ubiquity of organizational surprises begs the question as to why we are still “surprised by surprises.” The short answer is that this happens because we tend to overestimate our ability to control the future. The authors suggest that we would be better served by regarding surprise as an inherent property of all open systems (which includes entities such as organisations and projects). After living through the consequences of many over-optimistic managerial actions (both, my own and those of others), I would have to agree.
Classifying surprise – a typology of surprises
Nothing I have said so far would be surprising to readers: indeed, project management is largely about managing uncertainty…and all project managers know that. What might be new, however, is a classification of surprises proposed by the authors of the paper. I have hinted at the classification in the previous section where I gave one example each of a surprising event and a surprising process. It is now time to generalize this to a typology of surprises.
The authors classify surprises along two dimensions:
- Issue – An issue can occur in one of two ways:
- When something unusual happens.
- When something that usually happens does not happen.
It is important to note that although the term issue has negative connotations in project management parlance, it is used here in a neutral sense – i.e. issues can be either positive or negative.
- Process – a process is a chain of related events that unfolds in an unusual manner. For example, when an ATM cash withdrawal fails because the machine does not have enough money left to process the withdrawal.
From the perspective of surprise, issues and processes can be either expected or unexpected. This gives us the four categories illustrated in Fig 1.
Let’s take a tour of the four categories
- Expected issue and process: This is the zone of predictability where one-off events tend to go as foreseen. An example of an expected issue that was successfully dealt with through planning was the Y2K problem. Another example is provided by (successful) risk management activities that are triggered when a foreseen risk eventuates.
- Unexpected issue, expected process: This is where a surprising issue occurs, but the consequences follow are expected. An example this would be a chance occurrence (say, a project team member on a troubled project stumbles on a novel technique that saves development time), and this leads to the project being completed within time and budget (expected process following the event).
- Expected issue, unexpected process: This occurs when an expected event evolves in an unexpected way – i.e. leads to a surprising process. A common example of this in a project environment is when a front-end project decision unfolds in unexpected ways. Another common example is an organizational change that has unintended consequences.
- Unexpected issue, unexpected process: This is a situation where both the event and the processes around it are counter to conventional wisdom. In this case, those involved need to understand, or make sense of the situation and hence the term sensemaking crisis. An example of this is when project managers fail to anticipate factors that turn out to have a major influence on the way their projects evolve. One could argue that many high profile project failures were the result of such crises. The Denver Baggage Handling System and the Merck Vioxx affair are good examples. In both cases, the projects failed because those responsible failed to react to certain events that changed the trajectory of the projects irrevocably.
Let’s now take a brief look at the usefulness of this classification.
Coping with surprise
Managers expend a great deal of effort in attempting to predict surprises and hence corral them into the zone of predictability (Reminder: this is the bottom left quadrant in Figure 1). As mentioned earlier, this is difficult because organisations are open systems, and novelty is an inherent property of such systems. The main implication of this is that surprises in quadrants other than the zone of predictability cannot be foreseen. So, instead of worrying about predicting surprises, project and program managers would do better by focusing their efforts on creating an environment that enables team members to cope with nasty surprises and take advantage of good ones.
What might such an environment look like?
This question is best approached via a related question: what are the qualities displayed by project teams that are able to cope with surprises?
Here are some essential ones that are mentioned in the paper:
- Vigilance / problem sensing– a deep awareness of the project environment, with the ability to sense any changes in it.
- Resilience – the capacity to adjust to changes in the internal and external environment.
- Ability to improvise– the ability to respond to the unexpected by devising appropriate courses of action under pressure
The striking thing about these qualities is that they are impossible to create or engender by management fiat: teams will not improvise unless they feel empowered to, nor will they be resilient or vigilant unless they are intrinsically motivated to be so.These characteristics are emergent in the sense that they will be displayed spontaneously by teams that are in a frame of mind that comes out of being in the right environment.
The primary task of a project manager, or any manager for that matter, is to create such a holding environment that provides psychological safety to the team and encourages rational (or open) dialogue between all project stakeholders (yes, including project sponsors). I won’t elaborate on these terms here since they are dealt with at length in the articles that I have provided links to in the previous sentence.
Different types of surprise require different approaches
Having the right environment is the key to dealing with all four kinds of surprises. However, even within such an environment, it is important to note that different types of surprises have to be tackled in different ways. In particular:
- Predictable surprises are best tackled through traditional management approaches (as discussed in PMBOK, for example). In view of the prevalence of such approaches, I should perhaps emphasise again that they work only for a small subset of all possible surprises (only those that lie in the first quadrant)
- Surprising events and surprising processes are best dealt with by the people who are at the coalface of the problem since they are intimately familiar with the context and history of the problem.
- Sensemaking crises are best handled by collaborative problem solving approaches such as Dialogue Mapping.
The above yet again underscores the importance of the creating the right environment, for although predictable surprises can be tackled through traditional approaches to project management, those that lie in the other three quadrants cannot.
A fact of organizational life is that project managers are often caught unawares by unforeseen events and their dynamics. In this post, I have summarized a typology of organizational surprises and have elaborated on its relevance to project management. I have also briefly discussed the ways in which different types of surprises can be tackled, emphasising that the key to tackling surprise lies in creating an environment that provides psychological safety and encourages open dialogue.
In closing, I reiterate that projects and organisations are open systems, and surprises are characteristic of such systems. The biggest surprise, therefore, is that we are continually surprised by some of the events and processes that occur within them
Enterprise IT systems that are built or customised according to specifications sometimes fail because of lack of adoption by end-users. Typically this is treated as a problem of managing change, and is dealt with in the standard way: by involving key stakeholders in system design, keeping them informed of progress, generating and maintaining enthusiasm for the new system and, most important, spending enough time and effort in comprehensive training programmes.
Yet these efforts are sometimes in vain and the question, quite naturally, is: “Why?” In this post I offer an answer, drawing on a paper by Brian Pentland and Martha Feldman entitled, Designing Routines: On the Folly of Designing Artifacts While Hoping for Patterns of Action.
Setting the context
From a functional point of view, enterprise software embodies organisational routines - i.e. sequences of actions that have well-defined objectives, and are typically performed by business users in specific contexts. Pentland and Feldman argue that although enterprise IT systems tend to treat organisational routines as well defined processes (i.e. as objects), it is far from obvious that they are so. Indeed, the failure of business process “design” or “redesign” initiatives can often be traced back to a fundamental misunderstanding of what organisational routines are. This is a point that many system architects, designers and even managers / executives would do well to understand.
As Pentland and Feldman state:
… the frequent disconnect between [system or design] goals and results arises, at least in part, because people design artifacts when they want patterns of action…we believe that designing things while hoping for patterns of action is a mistake. The problem begins with a failure to understand the nature of organizational routines, which are the foundation of any work process that involves coordination among multiple actors… even today, organizational routines are widely misunderstood as rigid, mundane, mindless, and explicitly stored somewhere. With these mistaken assumptions as a starting point, designing new work routines would seem to be a simple matter of creating new checklists, rules, procedures, and software. Once in place, these material artifacts will determine patterns of action: software will be used, checklists will get checked, and rules will be followed.
The fundamental misunderstanding is that design artifacts, checklists, and rules and procedures encoded in software are mistaken for the routine instead of being seen for what they are: idealised representations of the routine. Many software projects fail because designers do not understand this. The authors describe a case study that highlights this point and then draw some general inferences from it. I describe the case study in the next section and then look into what can be learnt from it.
The case study deals with a packaged software implementation at a university. The university had two outreach programs that were quite similar, but were administered independently by two different departments using separate IT systems. Changes in the university IT infrastructure meant that one of the departments would lose the mainframe that hosted their system.
An evaluation was performed, and the general consensus was that it would be best for the department losing the mainframe to start using the system that the other department was using. However, this was not feasible because the system used by the other department was licensed only for a single user. It was therefore decided to upgrade to a groupware version of the product. Since the proposed system was intended to integrate the outreach-related work of the two departments, this also presented an opportunity to integrate some of the work processes of the two departments.
A project team was set-up, drawing on expertise from both departments. Requirements were gathered and a design was prepared based on the requirements. The system was customised and tested as per the design, and data from the two departments was imported from the old systems into the new one. Further, the project team knew the importance of support and training: additional support was organised as were training sessions for all users.
But just as everything seemed set to go, things started to unravel. In the author’s words:
As the launch date grew nearer, obstacles emerged. The implementation met resistance and stalled. After dragging their feet for weeks, [the department losing the mainframe] eventually moved their data from the mainframe to a stand-alone PC and avoided [the new system] entirely. The [other department] eventually moved over [to the new system], but used only a subset of the features. Neither group utilized any of the functionality for accounting or reporting, relying instead on their familiar routines (using standalone spreadsheets) for these parts of the work. The carefully designed vision of unified accounting and reporting did not materialize.
People in the department that was losing the mainframe worked around the new system by migrating their data to a standalone PC and using that to run their own system. People in the other department did migrate to the new system, but used only a small subset of its features.
All things considered, the project was a failure.
So what went wrong?
The authors emphasise that the software was more than capable of meeting the needs of both departments, so technical or even functional failure can be ruled out as a reason for non-adoption. On speaking with users, they found that the main objections to the system had to with the work patterns that were imposed by it. Specifically, people objected to having to give up control over their work practices and their identities (as members of specific) departments. There was a yawning gap between the technical design of the system and the work processes as they were understood and carried out by people in the two departments.
The important thing to note here is that people found ways to work around the system despite the fact that the system actually worked as per the requirements. The system failed despite being a technical success. This is a point that those who subscribe to the mainstream school of enterprise system design and architecture would do well to take note of.
Dead and live routines
The authors go further: to understand resistance to change, they invoke the notion of dead and live routines. Dead routines are those that have been represented in technical artifacts such as documentation, flowcharts, software etc. whereas live routines are those that are actually executed by people. The point is, live routines are literally living objects – they evolve in idiosyncratic ways because people inevitably tweak them, sometimes even without being conscious that they are making changes. As a consequence live routines often generate new patterns of action.
The crux of the problem is that software is capable of representing dead routines, not live ones.
Complementary aspects of routines
Routines are composed of people’s understandings of the routines (the authors call this the ostensive aspect) and the way in which they are actually performed or carried out (the authors call this the performative aspect). The two aspects, the routine as understood and the routine as performed, complement each other: new actions will modify one’s understanding of a routine, and the new understanding in turn modifies future actions.
Now, the interesting thing is that no artifact can represent all aspects of a routine – something is always left out. Even though certain artifacts such as written procedures may be intended to change both the ostensive and performative aspects of a routine (as they usually are), there is no guarantee that they will actually influence behaviour. Managers who encounter resistance to change will have had first-hand experience of this: it is near impossible to force change upon a group that does not want it.
This leads us to another set of complementary aspects of routines. On the one hand, technology puts in place structures that enable certain actions while constraining others – this is the standard technological or material view of processes as represented in systems. In contrast, according to the social or agency view, people are free to use technology in whatever way they like and sometimes even not use it at all. In the case study, the former view was the one that designers took whereas users focused on the latter one.
The main point to take away from the foregoing discussion is that designers/ engineers and users often have very different perspectives on processes. Software that is based on a designer/engineer perspective alone will invariably end up causing problems for end users.
Routines and reality
Traditional systems design proceeds according to the following steps:
- Gather requirements
- Analyse and encode them in rules
- Implement rules in a software system
- Provide people with incentives and training to follow the rules
- Rollout the system
Those responsible for the implementation of the system described in the case study followed the steps faithfully, yet the system failed because one department didn’t use it at all and the other used it selectively. The foregoing discussion tells us that the problem arises from confusing artifacts with actions, or as I like to put it, routines with reality.
As the authors write:
This failure to understand the difference between artifacts and actions is not new…the failure to distinguish the social (or human) from the technical is a “category mistake.” Mistakenly treating people like machines (rule-following automata) results in a wide range of undesirable, but not entirely surprising, outcomes. This category difference has been amply demonstrated for individuals, but it applies equally (if not more so) to organizational routines. This is because members’ embodied and cognitive understandings are often diverse, multiple, and conflicting.
The failure to understand the difference between routines and reality is not new, but it appears that the message is yet to get through: organisations continue to implement (new routines via) enterprise systems without putting in the effort to understand ground-level reality.
Some tips for designers
The problem of reconciling user and designer perspectives is not an easy one. Nevertheless, the authors describe an approach that may prove useful to some. The first concept is that of a narrative network – a collection of functional events that are related by the sequence in which they occur. A functional event is one in which two actants (or objects that participate in a routine) are linked by an action. The actants here could be human or non-human and the action represents a step in the process. Examples of functional events would be:
- The sales rep calls on the customer.
- The customer orders a product.
- The product is shipped to the customer.
The important thing to note is that the functional event does not specify how the process is carried out – that can be done in a myriad different ways. For example, the sales rep may meet the customer face-to-face or she may just make a phone call. The event only specifies the basic intent of the action, not the detail of how it is to be carried out. This leaves open the possibility for improvisation if the situation calls for it. More importantly it places the responsibility for making that decision on the person responsible for the carrying out the action (in the case of a human actant).
Then it is useful to classify a functional event based on the type of actants participating in the event – human or (non-human) artifact. There are four possibilities as shown in Figure 1.
The interesting thing to note is that from the perspective of system design, an artifact-artifact event is one over which the designer has the strongest control (actions involving only artifacts can be automated) whereas a designer’s influence is the weakest in human-human events (humans rarely, if ever, do exactly as they are told!). Decomposing a routine in this way can thus serve to focus designer efforts in areas where they are likely to pay off and, more importantly, help them understand where they are likely to get some push back.
Finally, there are some general points to consider when designing a system. These include:
- Reinforce the patterns of the routine: Much as practise makes an orchestra perform better, organisations have to invest in practicing the routines. Practice connects the abstract routine to the performed one.
- Consider every stakeholder group’s point of view: This is a point I have elaborated at length in my posts on dialogue mapping so I’ll say no more about it here.
- Understand the relationship between abstract patterns and actual performances: This involves understanding whether there are different ways to achieve the same goal. Also consider whether it is important to enforce a specific set of actions leading up to the goal, or whether the actions matter less than the goal.
- Encourage people to follow patterns of action that are important: This involves creating incentives to encourage people to carry out certain important actions in specified ways. It is not enough to design and implement a particular pattern within a routine. One has to make it worthwhile for people to follow it.
- Make it possible for users to become designers: Most routines involve decisions points where actors have to choose between alternate, but fixed, courses of action. At such points human actors have little room to improvise. Instead, it may be more fruitful to replace decision points with design points, where actors improvise their actions.
- Lock in actions that matter: Notwithstanding the previous point, there are certain actions that absolutely must be executed as designed. It is as important to ensure that these are executed as they should be as it is to allow people to improvise in other situations.
- Keepan open mind and invite engagement: Perhaps the most important point is to continually engage with people who work the routine and to be open to their input as to what’s working and what isn’t.
Most of the above points are not new; I’d even go so far as to say they are obvious. Nevertheless they are worth restating because they are often unheeded.
A paradox of enterprise systems is that they can fail even if built according to specification. When this happens it is usually because designers fail to appreciate the flexibility of the business process(es) in question. As the authors state, “…even today, organizational routines are widely misunderstood as rigid, mundane, mindless, and explicitly stored somewhere.” This is particularly true for processes that involve humans. As the authors put it, when automating processes that involve humans it is a mistake to design software artefacts while hoping for (fixed) patterns of action. The tragedy is that this is exactly how enterprise systems are often designed.
The hierarchical structure of many workplaces tends to constrain or even stifle open exchange of ideas and information. This is particularly apparent in communication between employees who are at different levels in a hierarchy: people are generally reluctant to speak their minds in front of their managers, even when assured that it is perfectly OK to do so. There is good reason for this: managers often “talk the talk” about being open to other points of view but contradict their words subsequently (see my article entitled, the paradox of the learning organization, for an example of this).
In this post I draw on this paper by Max Visser to describe some of the tactics or patterns of miscommunication which managers employ to sideline, devalue or even completely dismiss employee viewpoints.
Those who toil in the lower echelons of an organisation’s hierarchy can easily sense the gap between managerial talk and intent. One setting in which this gap becomes particularly evident is in group meetings, where a manager’s words may say, “speak freely” but his body language or responses may append an unspoken “be aware of the consequences” clause.
As I have discussed in this post, communication is just as much about context (e.g. manager-subordinate relationship within an organisational setting) as it is about content. This point of view is central to the interactional view of communication that originated in the work of Gregory Bateson and Paul Watzlawick. According to the interactional view, communication operates at two levels: the spoken or written meaning (content) and the situation/relationship (context). Among other things, this view focuses on the ways in which the content of a message – such as “speak freely” – may be rendered ambiguous by signals that appear to contradict it. In the remainder of this post we’ll look at a few ways in which managers do this via verbal communication. We’ll also take a brief look at the different ways in which employees respond to such behaviour.
Patterns of miscommunication
The best way to describe these patterns is through an example. Consider the following situation:
An employee presents a business case for a new CRM system to his manager. In the presentation, the employee describes the rationale for implementing a new system and then evaluates a few products based on agreed financial, technical and other criteria. Finally, he recommends a particular product, System X, based on the evaluation and then seeks feedback from his manager.
The manager, who does not want to commit to a course of action may choose one of the following strategies to devalue the employee’s work:
In this case the manager makes a statement that acknowledges the employee’s message but ignores its content and intent by saying something like:
“So how long have you been working on this?”
By going off on a tangent, the manager avoids giving a relevant response.
There are four types of disqualification
This occurs when a manager avoids giving a response by changing the topic. For example, the manager might glance at his watch and saying:
“Oh is that the time? I have to go, I’m late for a meeting with my boss.”
The difference between tangentialisation and evasion is that in the latter, the manager does not even acknowledge the message.
Sleight of hand
Here the manager appears to acknowledge the message, but then switches the topic. An example of this would be a response along the lines of:
“Yes, you enough data for a Phd thesis here [laughs]. I think we’re drowning in data.“
The point here is that the manager initiates a discussion about a side issue – the volume of information presented in the business case rather than its relevance or veracity. Moreover this is done in an apparently light-hearted, yet somewhat demeaning way. Thus although the manager avoids giving direct feedback, he still makes it clear he does not think that the employee’s work is up to scratch.
Here the manager switches the focus from the message to the messenger. Usually status disqualification is accompanied by insinuations regarding the messenger’s competence. A typical example of this would be a comment like:
“It’s clear you have not done these kinds of presentations before!”
Without saying it explicitly, the manager is implying that the employee has not done a good job and therefore no further discussion is necessary.
This is where the manager lobs the ball back in the employee’s court by asking a question that implicitly challenges the employee’s conclusions. An example would be:
“[smiles knowingly] I see, but does your data justify your choice of System X?”
Such a question signals the manager is not convinced, but without explicit disagreement. The onus is now on the employee to justify his conclusions.
Here the manager changes the context of the discussion altogether by saying something like:
“Let me tell you something about CRM systems.”
Here the manager changes the frame of the discussion – it is now about educating the employee rather than evaluating the product. Of course, in doing so he also insinuates that the employee’s analysis is not worthy of a response.
Employee responses to managerial miscommunication
When faced with any of the above tactics, the employee can respond in one of the following ways:
- Meta-communication: Here the employee understands the manager’s tactics and attempts to point out the inconsistency and double speak in the manager’s response. This is a risky course of action because the manager may view it as a direct challenge to his or her authority. However, if done right, the manager may actually become aware of the incongruence of his/her response and change behaviour accordingly.
- Evasion: Here the employee withdraws from the conversation by ignoring the manager’s message altogether. One way to do this is to offer no response at all, but this might not be possible as the manager may well insist on a response.
- Acceptance: In this case the employee accepts the content of the manager’s response, but ignores the non-verbal signals (derogatory tone, looking at watch etc.). In doing so, the employee effectively accepts the manager’s criticisms.
- Countering: Here the employee counters the manager’s message by using one of the tactics of the previous section. This generally leads to a verbal escalation as the manager will view such a response as a direct challenge to his authority and thus respond in kind.
Because of the nature of the manager-employee relationship and the fear of challenging authority, I would hazard a guess that majority of employees would respond by acceptance or (more infrequently) by evasion. In an ideal organisation, of course, they would respond by meta-communicating.
In this post I have described some common patterns of miscommunication between managers and the managed in organisation-land. The common element in all the patterns is that the manager acknowledges the message at one level but responds in such a way as to leave the employee confused about how the response should be interpreted. In effect, the miscommunicating manager avoids giving a response.
The interactional view of communication tells us that context and relationship are more important than the content of a message because what is not said is often more significant than what is. The patterns listed above make this amply clear: managers who miscommunicate are asserting their positional authority rather than saying anything of substance or value.
The expenses application crashed just as Tina had finished entering the last line. She wasn’t duly alarmed; this had happened to her a couple of times before, but Nathan in IT was able to sort it out without her having to reenter her expenses.
She dialled his number, he answered in a couple of rings. After a brief exchange of pleasantries, she described the problem.
To her surprise, he replied, “I’m sorry Tina, I can’t help you. You will have to call the service desk.”
“The service desk?” She asked, “What’s that?”
“We have streamlined our IT service procedures to comply with the ABSERD standard – which stands for Absolutely Brilliant SERvice Desks. It is an ABSERD requirement that all service calls must be routed through a centralised service desk.” explained Nate. “The procedures and the numbers you need to call were in the email that was sent out to everyone last week.”
“Yes, I read it, but I didn’t think the ABSERD procedures applied to something like the expenses app.,” said Tina, somewhat bemused.
“I’m afraid it applies to all services that IT offers,” said Nate.
“But isn’t the service desk located elsewhere? Will they even know what the expenses app is let alone how to fix it?”
“Ummm…they’ll fix it if they can and escalate it to the next level if they can’t,” replied Nate. “The ABSERD processes are detailed in the email,” he explained again helpfully.
“You know what the problem is. Tell me honestly: do you think they’ll be able to fix it?”
“Probably not,” admitted Nate.
“So they’ll escalate it. How long will that take?”
“The ABSERD service level agreement specifies that all non-critical issues will be responded to within 48 hours. I’m afraid the expenses app is classified as non-critical.”
“So that’s 48 hours to fix an issue that you could sort out in minutes,” stated Tina in a matter of fact tone.
“Ummm…no, it’s 48 hours to respond. That’s the time frame in which they will fix the issue if they can or escalate it if they can’t fix it. As I mentioned, in this case they’ll probably have to escalate” clarified Nate.
“You mean they’ll take 48 hours to figure out they can’t do it. Now, that is truly absurd!” Tina was seriously annoyed now.
“Well, the service desk deals with calls from the entire organisation. They have to prioritise them somehow and this is the fairest way to do it,” said Nate defensively. “Moreover, the service level agreement specifies 48 hours, but there’s a good chance you’ll get a response within a day,” he added in an attempt to placate her.
“And who will they escalate the call to after 48 (or 24) hours if they can’t fix it?” asked Tina exasperatedly.
“Ummm….that would be me,” said Nate sheepishly.
“I’m sorry, but I’m totally lost now. By your own admission, you’ll probably be the one to fix this problem. So why can’t you just do it for me?”
“I’d love to, Tina” said Nate, “but I can’t. Jim will have my hide if he knows that I have bypassed the ABSERD process. I’m sorry, you’re just going to have to call or email the service desk. I can’t do anything about it”
“Why are we suddenly following this ABSERD process anyway? What’s the aim of it all?” asked Tina.
“Well, our aim is to improve the quality of our service. The ABSERD standard is a best practice for IT service providers…,” he trailed off, realising that he sounded like a commercial for ABSERDity.
“You do agree that it actually increases the service time for me. You could have fixed the issue for me in the time we’ve had this conversation but I’m going to have to wait at least 24 hours. I fail to see what has “improved” here.”
“Look, this is the new process. I’m sorry can’t do anything about it,” said Nate lamely.
“OK, I’ll log the call.” she said resignedly.
“I’m sorry, Tina. I really am.”
“It’s not your fault,” she said in a gentler tone, “but I’m probably going to miss the deadline for getting my expenses in this month.”
“Tell you what,” said Nate, as the obvious solution dawned on him, “I’ll fix the problem now… but please log the call just in case someone checks.”
“Are you sure you can do that?” asked Tina. “It would be nice to get reimbursed this month, but I do not want you to get into trouble.”
“It shouldn’t be a problem as long as you don’t tell anyone about it…I wouldn’t want to make it known that I bypass the ABSERD procedures as a matter of course.”
“My lips are sealed,” said Tina. “Thanks Nate, I really appreciate your help with this.”
“No worries Tina. I’ll call you when it’s done,” he said as he ended the call.
Dark clouds, silver linings: a transaction cost perspective on the outsourcing of information systems
Some years ago, I wrote a post applying ideas of transaction cost economics to the question of outsourcing IT development work. In that post I argued that evaluating costs on the basis of vendor quotes alone is highly misleading. One has to also factor in transaction costs – i.e. costs relating to things such as search, bargaining and contract enforcement.
In the present post I discuss transaction cost theory as it applies to the question of whether or not organisations should outsource their enterprise systems (such as ERP and CRM applications) to Software as a Service (SaaS) vendor. With the increasing number of offerings on the market, this issue is one that is high on the agenda of IT decision makers.
To be sure, there are many well-known (and massively hyped!) benefits of cloud-based solutions. To name just one: organisations that take the SaaS route do not have to worry about maintenance, upgrades etc. as these are handled by the vendor. As we shall see, however, when it comes to costs, things are not so clear.
Setting the context
Today’s IT landscape is considerably more complex than that of a couple of decades ago. Improvements in infrastructural technology now offer the IT decision maker choices that were simply not available then. One of the basic choices a decision maker is faced when implementing a new enterprise system is whether to develop (or customize), host and support it in-house or opt for a cloud-based offering. Most often decisions on these matters are made on the basis of vendor-quoted costs alone. A typical discussion between a supporter and skeptic of outsourcing may go as follows:
Supporter: We should outsource our CRM system because we will save costs. We can save X million dollars on licensing and hardware…and then there are potential savings on personnel costs in the longer term.
Skeptic: Yes, but we lose flexibility to modify the application to suit our needs.
Supporter: Flexibility is overrated. We have done a gap-fit analysis and have found that most of core business needs can be satisfied by the three shortlisted cloud offerings. We are not a complex business: we call on customers, sell them our product and then follow-up from time to time to see how they are going and whether there are potential opportunities to sell them upgrades. It’s pretty much what everyone else in the business does.
Skeptic: What about vendor lock-in?
Supporter: There is no lock in we pay as we go; per user per month.
….and so the conversation goes. The supporter seems to have an answer for every question so the skeptic may eventually concede. However, the latter may still be left with a vague sense of unease that something’s been overlooked, and indeed something has…which brings us to the next topic.
A firm has two choices for any economic activity: performing the activity in-house or going to market. In either case, the cost of the activity can be decomposed into:
- Production costs, which are direct (easily quantifiable) costs of producing the good or service
- Transaction costs, which are other (indirect) costs incurred in performing the economic activity.
Production costs include things such as per user cost etc. These are typically quoted upfront and are easy to contractualise (i.e. put into a contract). Things are more complicated when it comes to transactions costs. To see this, let’s take a quick look at the different types of transaction costs:
- Search /selection costs: These are the costs associated with searching for and shortlisting vendors.
- Bargaining costs: These are costs associated with negotiations for a mutually acceptable contract.
- Maintenance costs: These are expected costs (i.e. those that were foreseen when the contract drawn up) associated with ongoing support, enhancements or upgrades.
- Costs of enforcement and change: These are costs associated with enforcing the terms of the contract and those associated with change.
These costs are typically hard to estimate upfront, and nearly impossible to contractualise. Moreover, in most cases, they are largely borne by the client.
The difficulties associated with contractualising transaction costs arise from the following:
- Unexpected events: Unforeseen and unforeseeable changes in the customer / vendor organisations or in the business environment may entail major changes in the application functionality or even the outsourcing arrangement.
- Bounded rationality: Business environments are complex and it is impossible for the human mind to anticipate everything that can possibly occur. As a consequence, every contract is necessarily incomplete; there is bound to be something that is overlooked. It is impossible contractualise every eventuality.
- Strategic / opportunistic behaviour: the vendor or customer may deliberately withold certain information from the other party at the outset in order to secure the contract at a favourable rate. Typical vendor behaviour may include not revealing certain limitations of the software on the other hand, the customer may attempt to include unfair penalty clauses into the contract or squeeze the vendor on margins.
Now before I’m accused of being unduly alarmist I should mention that certain kinds of enterprise systems are not subject to the above difficulties, or at least not significantly. Typically these are infrastructural services such as servers, operating systems, databases and even simple, context-independent applications such as email. These can be outsourced successfully without any problems because the services to be provided can be specified accurately upfront and thus contractualised unambiguously.
However, for enterprise applications such as ERP and CRM the situation is different because these systems are more or less unique to a given organisation – in other words, they are context-dependent. Their uniqueness renders them vulnerable to the points mentioned above because it is impossible to foresee all possibilities. As a consequence it is inevitable that any outsourcing deal involving such systems will eventually be affected by one or more of the above uncertainties.
None of the above points is new: both vendors and customers are at least somewhat aware that application outsourcing deals have many unknowns. Consequently, both parties try to minimise their exposure to uncertainty. Unfortunately, they usually go about this in exactly the wrong way: they attempt to build safety for themselves at the cost of the other party. They do this by attempting to contractualise all possible things that can go wrong from their point of view. This is impossible because the future cannot be predicted. Contracts, however detailed, will necessarily be incomplete.
The way out is simple. As Oliver Williamson, winner of the 2009 Economics Nobel Prize, tells us:
…important to the transaction-cost economics enterprise is the assumption that contracts, albeit incomplete, are interpreted in a farsighted manner, according to which economic actors look ahead, perceive potential hazards and embed transactions in governance structures that have hazard-mitigating purpose and effect. Also, most of the governance action works through private ordering with courts being reserved for purposes of ultimate appeal.
As he tells us, contracts need to be interpreted in a farsighted manner and governance action should work through private ordering (directly between the customer and vendor). Essentially, the success of an outsourcing deal is to a large extent the joint responsibility of the customer and vendor. Above all, it calls for a relationship based on old-fashioned notion of trust.
Interestingly, Elinor Ostrom, who jointly won the 2009 Economics Nobel with Williamson, had much to say about informal contracts, private ordering and trust. Her extensive studies on collectives lead her to conclude that successful collective endeavours have the following two elements in common:
- High levels of face-to-face communication
- Innovative governance structures that are designed and enforced by the participants rather than external authorities.
The relevance of these to an outsourcing arrangement is clear: face to face communication builds trust, and innovative internal governance structures that are enforceable without legal recourse will discourage opportunistic behaviour from both parties. Although the latter may sound a bit utopian there are proven ways to set up such governance structures (see this paper for an example).
In this post I have argued that a proper analysis of SaaS deals must include transaction costs. Although these are difficult to quantify, a consideration of the different categories of transaction costs will at least lead to a more realistic appraisal of such arrangements. It is my belief that many outsourcing deals go sour because transaction costs are overlooked. Given that it is impossible to foresee the future, the best course of action is to develop a business relationship that is based on trust.
To sum up: most important factor in enterprise application outsourcing is not cost, but trust - an ineffable element that can neither be quantified nor contractualised.
Salviati: Hello Simplicio, I haven’t seen you for a while. Where have you been?
Simplicio: Salviati! It is good to see you. I’ve been busy learning about project management.
Salviati: That is good news indeed, Simplicio. How are you learning? Are you working on a project?
Simplicio: No, of course not; I’m still learning. I don’t think my boss would let me work on a real project until I’ve completed my certification.
Salviati: Certification? Now I’m really curious.
Simplicio: Oh yes, and as a part of it I’m reading this wonderful book that is the authoritative guide to project management. I’m also attending an evening discussion group twice a week where I get to explore the finer details of “The Book”. It’s great! We have some experienced project managers in the group who tell us stories.
Salviati: That’s good, and I would pay more attention to their stories than the tools and techniques in “The Book”.
Simplicio: Really? Why would I want to listen to a bunch of stories about old projects? Surely it’s the tools and techniques that are more important. Stories are….well, just stories. Half of them are probably embellished anyway.
Salviati: May be so, but the fact is, project managers often make sense of their work by constructing stories about it.
Simplicio: What do you mean “make sense of their work?”
Salviati: Well, despite project managers’ best efforts, things never quite go as planned: team members fall sick or leave the company; vendors do not deliver on time; users change their requirements on a daily basis…I could go on. When these things happen, project managers need to understand what has happened so that they can devise appropriate responses. They often do this by building narratives of what happened or, in simple terms, by telling stories.
Simplicio: To be perfectly honest I think the real reason things go wrong is that people do not follow processes properly. It seems to me that storytelling is just a means to cover up the truth, a rationalisation.
Salviati: Ah, truth. You see, Simplicio, truth in such situations is often a matter of opinion. Different stakeholders will have different views on what happened. Say a vendor is late in delivering something – the customer may see it as gross incompetence whereas the vendor will, no doubt, have a perfectly reasonable explanation. So, whose truth is the truth? And even if you were able to answer that, does it really matter? As a project manager, you’re on the spot; you have to move ahead despite the setback. The truth doesn’t help you here, and neither does process. In fact trying to get to the truth and insisting on process may only end up exacerbating the problem.
Simplicio: Hmm, OK, you may have a point there, but are you suggesting there is nothing of value in “The Book?” Is it all just impractical theory?
Salviati: Oh don’t get me wrong, it is necessary to know the stuff in that’s in “The Book”. But it is also important to remember that there is a gap between theory and practice.
Simplicio: Gap between theory and practice?
Salviati: Yes there is a significant gap between what is taught in business schools (or written in “The Book”) and the way managers actually do their jobs. The former is called espoused theory and the latter, theory in use (Editor’s note: see this article for more on espoused theory vs theory in use). Espoused theory works in an ideal world where cause-effect relationships are unambiguous, and uncertainty can be predicted and planned for. This is the sort of world that is depicted in those pretty process diagrams that people draw on a whiteboard. In the real world, however, causes aren’t always apparent and best laid plans often go awry. Managers have to deal with this. When doing so, they often improvise on what they have learnt through experience. What books and project theorists tend to overlook is that planning and improvisation are complementary facets of project work. Indeed the most compelling project management stories are about improvisation; about what people did when theory or process was no help at all.
Simplicio: So you’re saying that theory is incomplete…
Salviati: Absolutely! Theory cannot teach you what experience does. You see, many project management skills are tacit, they can only be learned by doing. Would you pick up a book about guitar and music theory and expect to play like a virtuoso in an afternoon… or even a month or a year? . So it is with project management. But, look, tacitness is not the only issue. Another major shortcoming of project management, as it is taught, is that it overlooks the fact that every project is invariably part of a larger system: namely, the hosting organisation and its environment. Understanding this is critical to the success of a project.
Simplicio: I’m not sure I understand what you mean by “a larger system”.
Salviati: Consider the question of project failure. Many experts will tell you that the top causes of project failure are things like “lack of executive support” or “lack of user input” or even “incomplete requirements.” What these experts do not understand is that these are symptoms rather than causes. The true causes of failure invariably lie in the hosting organisation, not the project. For example, “lack of user input” often occurs because users typically work on projects in addition to their normal duties. It is but natural that they will therefore view projects as burdens rather than initiatives that might benefit them in the future. The fault here lies beyond the project. These kinds of issues need to be negotiated through open dialogue between all affected stakeholders rather than via top-down decrees .
Simplicio: OK, I understand the importance of taking a system-based view, but what is “open dialogue”?
Salviati: Ever worked for a team or organization where there are some things that can never be discussed? Ever had bosses who only want to know the good news? Most projects have many different stakeholder groups, each with their own view of the project and motivations. Sponsors, managers, project teams and users – all have their own view on a project’s objectives. As strange as it may sound, these viewpoints are often divergent, but are never reconciled until its too late. …
Simplicio: [interrupting] That’s crazy! Why would project managers allow themselves to get into a situation where they are managing projects in which stakeholders hold different views on things like scope? That is completely against what “The Book” says! According to it, things such as scope issues should not be ambiguous at all.
Salviati: Ah, now we get to the heart of the matter! Yes, it is crazy when you think about it, but we are dealing with office hierarchy and politics, as well individual rationality. Many organisations have a blame culture – and as a result, people tend to position themselves for blame avoidance. This creates all sorts of dysfunctional behaviours, and makes it very difficult to discuss things openly. The trick – and why you need to listed to the stories – is to break down these barriers so that a group can engage in open dialogue that will bring such issues out into the open. There are ways to do this, a couple of guys have even written a book on it. (Editor’s note: Perhaps he’s referring to this book?)
Simplicio: OK, I see your point, but what about the unknown unknowns – issues that no one can foresee at the start.
Salviati: That’s where trust comes in. The point is, if key stakeholders have a relationship based on trust, they will feel comfortable about informing each other of potential uncertainties as they emerge. They can then work together to address the uncertainty without the usual finger pointing and blame shifting that typically occurs in organisations. They will be no better than anyone else at predicting the future, but they will be able to deal with whatever comes up because they will face it as a group.
Simplicio: Sounds good, but how does one get stakeholders to trust one another and discuss issues openly?
Salviati: Well, as I mentioned earlier, much of present-day project management practice operates within a cause and effect paradigm…do this and that will happen. Instead the focus ought to be on creating the right conditions or environment in which a group of people can collaborate and work together as a genuine team. There’s a ton of interesting work on this – some of it dating back to the 1950s
Simplicio: Why hasn’t anyone mentioned this in our discussion group? This is really important!
Salviati: The conditions over causes argument is yet to make an impact on mainstream practice – particularly in project management. Unfortunately, those who wrote the “The Book” (and those who update it) seem to be unaware that conditions are more important than causes. It is a completely different way of looking at projects, so it may take a while for aficionados of “The Book” to make the change. That said, I’m an optimist so I believe that it will eventually catch on; it is just a matter of time …
[ Salviati’s watch alarm goes off, cutting him off mid-sentence. He glances at it]
Speaking of time, we’re all prisoners of time, it seems. I’ve got to go; I’m late for a meeting. We’ll continue our conversation later.
Simplicio: Thanks Salviati. I’d very much like that as I’m curious to hear more about your thoughts on this.
Salviati (turning to leave): Sure, I’ll be delighted to chat about it. Let’s meet on the weekend. Catch you later.
Simplicio: See you later.
[The two depart, going their separate ways]
A metalogue is a real or imaginary conversation whose structure resembles the topic being discussed. This piece is inspired by Gregory Bateson’s metalogues in Part 1 of his book, Steps To an Ecology of Mind.
The characters in the above metalogue are borrowed from Galileo’s Dialogue Concerning The Two Chief World Systems in which the character Salviati is a proponent of the Copernican “heresy” that the Earth is not at the centre of the universe whereas Simplicio favours the Geocentric view proposed by the Greek philosopher Ptolemy.
This post is a part of the first ever #PMFlashBlog initiative which involves over 70 bloggers from Australia, Canada, Colombia, Denmark, France, Italy, Mexico, Poland, Portugal, Singapore, South Africa, Spain, UK and the USA, all posting a piece on “What Project Means to Me” on their blogs @ 01:00 GMT on 25th September 2013. A complete list of participants can be found here
My thanks go out to Shim Marom for coming up with the wonderful idea of a project management flashblog and for the opportunity to participate in it.
I’m indebted to Paul Culmsee for feedback on a draft version of this post and for countless conversations over the years on the philosophical and practical aspects of projects, organisations and systems. Be sure to check out his blog, in particular his PMFlashBlog post which provides a practical (and very entertaining!) perspective on the “conditions over causes” principle mentioned in this metalogue.
In his wonderful book on obliquity, John Kay tells of a famous letter in which Benjamin Franklin describes a decision-making method. Here is a description of Franklin’s method, excerpted from his letter:
…my Way is, to divide half a Sheet of Paper by a Line into two Columns, writing over the one Pro, and over the other Con. Then during three or four Days Consideration I put down under the different Heads short Hints of the different Motives that at different Times occur to me for or against the Measure. When I have thus got them all together in one View, I endeavour to estimate their respective Weights; and where I find two, one on each side, that seem equal, I strike them both out: If I find a Reason pro equal to some two Reasons con, I strike out the three. If I judge some two Reasons con equal to some three Reasons pro, I strike out the five; and thus proceeding I find at length where the Ballance lies; and if after a Day or two of farther Consideration nothing new that is of Importance occurs on either side, I come to a Determination accordingly.
And tho’ the Weight of Reasons cannot be taken with the Precision of Algebraic Quantities, yet when each is thus considered separately and comparatively, and the whole lies before me, I think I can judge better, and am less likely to take a rash Step; and in fact I have found great Advantage from this kind of Equation, in what may be called Moral or Prudential Algebra.
Modern decision making techniques often claim to do better than Franklin because they use quantitative measures to rate decision options However, as I have pointed out in this post, measures are often misleading. There are those who claim that this can be fixed by “doing it correctly,” but this is a simplistic view for reasons I have discussed at length in this post. So, despite all the so-called “advances” in decision making, it is still pretty much as Franklin wrote: “the Weight of Reasons cannot be taken with the Precision of Algebraic Quantities” .
With that for background, I can now get to the main point of this post. The reader may have wondered about my use of the word gambit rather than technique (or any of its synonyms) in the title of this post. A quick look at this online dictionary tells us that the two words are very different:
Technique (noun): the body of specialized procedures and methods used in any specific field, especially in an area of applied science.
Gambit (noun): a manoeuvre by which one seeks to gain advantage.
Indeed, as Kay mentions in his book, Franklin’s method is often used to justify decisions that are already made - he calls this Franklin’s Gambit.
Think back to some of the recent decisions you have made: did you make the decision first and then find reasons for it or did you weigh up the pros and cons of each option before reaching your decision? If I’m honest, I would have to admit that I have often done the former. This is understandable, even defensible. When we make a decision, we have to make several assumptions regarding the future and how it will unfold. Since this is based on (some times educated) guesswork, it is only natural that we will show a preference for a choice that we are comfortable with. Once we have settled on an option, we seek reasons that would enable us to justify our decision to to others; we would not want them to think we have made a decision based on gut-feel or personal preferences.
This not necessarily bad thing. When decisions cannot be rated meaningfully, any choice that is justifiable is a reasonable one….providing one can convince others affected that it is so. What one should guard against is the mindless use of data and so-called rational methods to back decisions that have no buy in.
Finally, as we all know well from experience, it is never a problem to convince ourself of the rightness of our decision. In fact, Mr. Franklin, despite his pronouncements on Moral Algebra understood this. For, as he once wrote:
…so convenient a thing is it to be a reasonable creature, since it enables one to find or make a reason for everything one had a mind to do.
Indeed, “reasonable” creatures that we are, we will desperately seek reasons for the things we wish to do. The difficulty, as always, lies in convincing other reasonable creatures of our reasonableness.