Eight to Late

Sensemaking and Analytics for Organizations

Archive for April 2014

Sherlock Holmes and the case of the failed projects

with 5 comments

….as narrated by Dr. John H. Watson M. D.

Foreword

Of all the problems which had been submitted to my friend, Mr. Sherlock Holmes, for consideration during the years of our friendship, there has been one that stands out for the sheer simplicity of its resolution.  I have (until now) been loath to disclose details of the case as I felt the resolution to be so trivial as to not merit mention.

So why bring it up after all these years?

Truth be told, I am increasingly of the mind that Holmes’ diagnosis in the Case of the Failed Projects (as I have chosen to call this narrative), though absolutely correct, has been widely ignored. Indeed, the writings of Lord Standish and others from the business press have convinced me that the real lesson from his diagnosis is yet to be learnt by those who really matter:  i.e. executives and managers.

As Holmes might have said, this is symptomatic of a larger malaise: that of a widespread ignorance of elementary logic and causality.

A final word before I get into the story. As most readers know, my friend is better known for his work on criminal cases. The present case, though far more mundane in its details, is in my opinion perhaps his most important because of its remarkable implications. The story has, I believe, been told at least once before but, like all such narratives, its effect is much less striking when set forth en bloc in a single half-column of print than when the facts slowly emerge before one’s own eyes.

So, without further ado, then, here is the tale…

The narrative

Holmes was going through a lean patch that summer, and it seemed that the only cases that came his way had to do with pilfered pets or suspicious spouses.   Such work, if one can call it that, held little allure for him.

He was fed up to the point that he was contemplating a foray into management consulting.  Indeed, he was certain he could do as well, if not better than the likes of Baron McKinsey and Lord Gartner (who seemed to be doing well enough). Moreover his success with the case of the terminated PMO  had given him some credibility in management circles.   As it turned out, it was that very case that led Mr. Bryant (not his real name) to invite us to his office that April morning.

As you may have surmised, Holmes accepted the invitation with alacrity.

The basic facts of the issue, as related by Bryant, were simple enough: his organization, which I shall call Big Enterprise, was suffering from an unduly high rate of project failure. I do not recall the exact number but offhand, it was around 70%.

Yes, that’s right: 7 out of every 10 projects that Big Enterprise undertook were over-budget, late or did not fulfil business expectations!

Shocking, you say… yet entirely consistent with the figures presented by Lord Standish and others.

Upon hearing the facts and figures, Holmes asked the obvious question about what Big Enterprise had done to figure out why the failure rate was so high.

“I was coming to that,” said Bryant, “typically after every project we hold a post-mortem.  The PMO  (which, as you know,I manage)  requires this. As a result, we have a pretty comprehensive record of ‘things that went well’ on our projects and things that didn’t.  We analysed the data from failed projects and found that there were three main reasons for failure: lack of adequate user input, incomplete or changing user requirements and inadequate executive support.”

“….but these aren’t the root cause,” said Holmes.

“You’re right, they aren’t” said Bryant, somewhat surprised at Holmes’ interjection. “Indeed, we did an exhaustive analysis of each of the projects and even interviewed some of the key team members. We concluded that the root cause of the failures was inadequate governance on the PMO’s  part,” said Bryant.

“I don’t understand.  Hadn’t you established governance processes prior to the problem? That is after all the raison d’etre of a PMO…”

“Yes we had, but our diagnosis implied those processes weren’t working. They needed to be tightened up.”

“I see,” said Holmes shortly. “I’ll return to that in due course. Please do go on and tell me what you did to address the issue of poor…or inadequate governance, as you put it.”

“Yes, so we put in place processes to address these problems. Specifically, we took the following actions. For the lack of user input, we recommended getting a sign-off from business managers as to how much time their people would commit to the project. For the second issue – incomplete or changing requirements – we recommended that in the short term, more attention be paid to initial requirement gathering, and that this be supported by a stricter change management regime. In the longer term, we recommended that the organization look into the possibility of implementing Agile approaches. For the third point, lack of executive support, we suggested that the problem be presented to the management board and CEO, requesting that they reinforce the importance of supporting project work to senior and middle management.”

Done with his explanation, he looked at the two of us to check if we needed any clarification. “Does this make sense?” he enquired, after a brief pause.

Holmes shook his head, “No Mr. Bryant the actions don’t make sense at all.  When faced with problems, the kneejerk reaction is to resort to more control. I submit that your focus on control misled you.”

“Misled? What do you mean?”

“Well, it didn’t work did it? Projects in Big Enterprise continue to fail, which is why we are having this meeting today.  The reason your prescription did not work is that you misdiagnosed the issue. The problem is not governance, but something deeper.”

Bryant wore a thoughtful expression as he attempted to digest this. “I do not understand, Mr. Holmes,” he said after a brief pause. “Why don’t you just tell me what the problem is and how can I fix it? Management is breathing down my neck and I have to do something about it soon.”

“To be honest, the diagnosis is obvious, and I am rather surprised you missed it,” said Holmes, “I shall give you a hint: it is bigger, much bigger, than the PMO and its governance processes.”

“I’m lost, Mr. Holmes.  I have thought about it long enough but have not been able to come up with anything. You will have to tell me,” said Bryant with a tone that conveyed both irritation and desperation.

“It is elementary, Mr. Bryant, when one has eliminated the other causes, whatever remains, however improbable, must be the truth. Your prior actions have all but established that the problem is not the PMO, but something bigger. So let me ask the simple question: what is the PMO a part of?”

“That’s obvious,” said Bryant, “it’s the organization, of course.”

“Exactly, Mr. Bryant: the problem lies in Big Enterprise’s organisational structures, rules and norms. It’s the entire system that’s the problem, not the PMO per se.”

Bryant looked at him dubiously.  “I do not understand how  the three points I made earlier – inadequate user involvement, changing requirements and lack executive sponsorship – are due to Big Enterprise’s structures, rules and norms. “

“It’s obvious,” said Holmes, as he proceeded to elaborate how lack of input was a consequence of users having to juggle their involvement in projects with their regular responsibilities. Changes in scope and incomplete requirements were but a manifestation of  the fact that users’ regular work pressures permitted only limited opportunities for interaction between users and the project team – and that it was impossible to gather all requirements…or build trust through infrequent interactions between the two parties. And as for lack of executive sponsorship – that was simply a reflection of the fact that the executives could not stay focused on a small number of tasks because they had a number of things that competed for their attention…and these often changed from day to day. This resulted in a reactive management style rather than a proactive or interactive one.  Each of these issues was an organizational problem that was well beyond the PMO.

“I see,” said Bryant, somewhat overwhelmed as he realized the magnitude of the problem, “…but this is so much bigger than me. How do I even begin to address it?”

“Well, you are the Head of the PMO, aren’t you?  It behooves you to explain this to your management.”

“I can’t do that!” exclaimed Bryant. “I could lose my job for stating these sorts of things, Mr. Holmes – however true they may be. Moreover, I would need incontrovertible evidence…facts demonstrating exactly how each failure was a consequence of organizational structures and norms, and was therefore out of the PMO’s control.”

Holmes chuckled sardonically. “I don’t think facts or ‘incontrovertible proof’ will help you Mr. Bryant. Whatever you say would be refuted using specious arguments…or simply laughed off.  In the end, I don’t know what to tell you except that it is a matter for your conscience;  you must do as you see fit.”

We left it at that; there wasn’t much else to say. I felt sorry for Bryant. He had come to Holmes for a solution, only to find that solving the problem might involve unacceptable sacrifices.

We bid him farewell, leaving him to ponder his difficult choices.

—-

Afterword

Shortly after our meeting with him, I heard that Bryant had left Big Enterprise. I don’t know what prompted his departure, but I can’t help but wonder if our conversation and his subsequent actions had something to do with it.

…and I think it is pretty clear why Lord Standish and others of his ilk still bemoan the unduly high rate of project failure.

 Notes

  1. Sherlock Holmes aficionados may have noted that the foreword to this story bears some resemblance to the first paragraph of the Conan Doyle classic, The Adventure of the Engineer’s Thumb.
  2. See my post entitled Symptoms not causes, a systems perspective on project failure for a more detailed version of the argument outlined in this story.
  3. For insight into the vexed question of governance, check out this post by Paul Culmsee and the book I co-authored with him.

Written by K

April 15, 2014 at 8:28 pm

Six heresies for business intelligence

with 10 comments

What is business intelligence?

I recently asked a few acquaintances to answer this question without referring to that great single point of truth in the cloud.  They duly came up with a variety of  responses ranging from data warehousing and the names of specific business intelligence tools to particular functions such as reporting or decision support.

After receiving their responses, I did what I asked my respondents not to: I googled the term.  Here are a few samples of what I found:

According to CIO magazine, Business intelligence is an umbrella term that refers to a variety of software applications used to analyze an organization’s raw data.

Wikipedia, on the other hand, tells us that BI is a set of theories, methodologies, architectures, and technologies that transform raw data into meaningful and useful information for business purposes.

Finally, Webopedia, tell us that BI [refers to] the tools and systems that play a key role in the strategic planning process of the corporation.

What’s interesting about the above responses and definitions is that they focus largely on processes and methodologies or tools and techniques. Now, without downplaying the importance of either, I think that many of the problems of business intelligence practice come from taking a perspective that is overly focused on methodology and technique.  In this post, I attempt to broaden this perspective by making some potentially controversial statements –or heresies – that challenge this view. My aim is not so much to criticize current practice as to encourage – or provoke – business intelligence professionals to take a closer look at some of the assumptions underlie their practices.

The heresies

Without further ado, here are my six heresies for business intelligence practice (in no particular order).

A single point of truth is a mirage

Many organisations embark on ambitious programs to build enterprise data warehouses – unified data repositories that serve as a single source of truth for all business-relevant data.  Leaving aside the technical  and business issues associated with establishing definitive data sources and harmonizing data, there is the more fundamental question of what is meant by truth.

The most commonly accepted notion of truth is that information (or data in a particular context) is true if it describes something as it actually is. A major issue with this viewpoint is that data (or information) can never fully describe a real-world object or event. For example, when a sales rep records a customer call, he or she notes down only what is required by the customer management system. Other data that may well be more important is not captured or is relegated to a “Notes” or “Comments” field that is rarely if ever searched or accessed. Indeed, data represents only a fraction of the truth, however one chooses to define it – more on this below.

Some might say that it is naïve to expect our databases to capture all aspects of reality, and that what is needed is a broad consensus between all relevant stakeholders as to what constitutes the truth. The problem with this is that such a consensus is often achieved by means that are not democratic. For example, a KPI definition chosen by a manager may be hotly contested by an employee.  Nevertheless, the employee has to accept it because that is the way (many) organisations work. Another significant issue is that the notion of relevant stakeholders is itself problematic because it is often difficult to come up with clear criterion by which to define relevance.

There are other ways to approach the notion of truth: for example, one might say that a piece of data is true as long as it is practically useful to deem it so. Such a viewpoint, though common, is flawed because utility is in the eye of the beholder: a sales manager may think it useful to believe a particular KPI whereas a sales rep might disagree (particularly if the KPI portrays the rep in a bad light!).

These varied interpretations of what constitute a truth have implications for the notion of a single point of truth. For one, the various interpretations are incommensurate – they cannot be judged by the same standard.  Further, different people may interpret the same piece of data differently. This is something that BI professionals have likely come across – say when attempting to come up with a harmonized definition for a customer record.

In short: the notion of a single point of truth is problematic because there is a great deal of ambiguity about what constitutes a truth.

There is no such thing as raw data

In his book, Memory Practices in the Sciences, Geoffrey Bowker wrote, “Raw data is both an oxymoron and a bad idea; to the contrary, data should be cooked with care.”  I love this quote because it tells a great truth (!) about so-called “raw” data.

To elaborate: raw data is never unprocessed. Firstly, the data collector always makes a choice as to what data will be collected and what will not. So in this sense, data already has meaning imposed on it. Second, and perhaps more important, the method of collection affects the data. For example, responses to a survey depend on how the questions are framed and how the survey itself is carried out (anonymous, face-to-face etc.).   This is also true for more “objective” data such as costs and expenses. In both cases, the actual numbers depend on specific accounting practices used in the organization. So, raw data is an oxymoron because data is never raw, and as Bowker tells us, we need to ensure that the filters we apply and the methods of collection we use are such that the resulting data is “cooked with care.”

In short: data is never raw, it is always “cooked.”

There are no best practices for business intelligence, only appropriate ones

Many software shops and consultancies devise frameworks and methodologies for business intelligence which they claim are based on best or proven practices. However, those who swallow that line and attempt to implement the practices often find that the results obtained are far from best.

I have discussed the shortcomings of best practices in a general context in an earlier article, and (at greater length) in my book. A problem with best practice approaches is that they assume a universal yardstick of what is best.  As a corollary, this also suggest that practices can be transplanted from one organization to another in a wholesale manner, without extensive customisation. This overlooks the fact that organisations are unique, and what works in one may not work in another.

A deeper issue is that much of the knowledge pertaining to best practices is tacit – that is, it cannot be codified in written form. Indeed, what differentiates good business intelligence developers or architects from great ones is not what they learnt from a textbook (or in a training course), but how they actually practice their craft.  These consist of things that they do instinctively and would find hard to put into words.

So, instead of looking to import best practices from your favourite vendor, it is better to focus on understanding what goes on in your environment. A critical examination of your environment and processes will reveal opportunities for improvement. These incremental improvements will cumulatively add up to your very own, customized “best practices.”

In short: develop your own business intelligence best practices rather than copying those peddled by “experts.”

Business intelligence does not support strategic decision-making

One of the stated aims of business intelligence systems is to support better business decision making in organisations (see the Wikipedia article, for example). It is true that business intelligence systems are perfectly adequate – even indispensable – for certain decision-making situations. Examples of these include, financial reporting (when done right!) and other operational reporting (inventory, logistics etc).  These generally tend to be routine situations with clear cut decision criteria and well-defined processes – i.e. decisions that can be programmed.

In contrast, decisions pertaining to strategic matters cannot be programmed. Examples of such decisions include: dealing with an uncertain business environment, responding to a new competitor etc. The reason such decisions cannot be programmed is that they depend on a host of factors other than data and are generally made in situations that are ambiguous.  Typically people use deliberative methods – i.e. methods based on argumentation – to arrive at decisions on such matters.  The sad fact is that all the major business tools in the market lack support for deliberative decision-making. Check out this post for more on what can be done about this.

In short: business intelligence does not support strategic decision-making .

Big data is not the panacea it is trumpeted to be

One of the more recent trends in business intelligence is the move towards analyzing increasingly large, diverse, rapidly changing datasets – what goes under the umbrella term big data.  Analysing these datasets entails the use of new technologies (e.g. Hadoop and NoSQL)  as well as statistical techniques that are not familiar to many mainstream business intelligence professionals.

Much has been claimed for big data; in fact, one might say too much.  In this article Tim Harford (aka the Undercover Economist) summarises the four main claims of “big data cheerleaders” as follows (the four phrases below are quoted directly from the article):

  1. Data analysis produces uncannily accurate results.
  2. Every single data point can be captured, making old statistical sampling techniques obsolete.
  3. It is passé to fret about what causes what, because statistical correlation tells us what we need to know.
  4. Scientific or statistical models aren’t needed.

The problem, as Harford points out, is that all of these claims are incorrect.

Firstly, the accuracy of the results that come out of a big data analysis depend critically on how the analysis is formulated. However, even analyses based on well-founded assumptions can get it wrong, as is illustrated in this article about Google Flu Trends.

Secondly, it is pretty obvious that it is impossible to capture every single data point (also relevant here is the discussion on raw data above – i.e. how data is selected for inclusion).

The third claim is simply absurd. The fact is detecting a correlation is not the same as  understanding what is going on a point made rather nicely by Dilbert.  Enough said, I think.

Fourthly, the claim that scientific or statistical models aren’t needed is simply ill-informed. As any big data practitioner will tell you, big data analysis relies on statistics. Moreover, as mentioned earlier, a correlation-based understanding is no understanding at all –  it cannot be reliably extrapolated to related situations without the help of hypotheses and (possibly tentative)  models of how the phenomenon under study works.

Finally, as Danah Boyd and Kate Crawford point out in this paper , big data changes the meaning of what it means to know something….and it is highly debatable as to whether these changes are for the better. See the paper for more on this point. (Acknowledgement: the title of this post is inspired by the title of the Boyd-Crawford paper).

In short:  business intelligence practitioners should not uncritically accept the pronouncements of big data evangelists and vendors.

Business intelligence has ethical implications

This heresy applies to much more than business intelligence: any human activity that affects other people has an ethical dimension. Many IT professionals tend to overlook this facet of their work because they are unaware of it – and sometimes prefer to remain so. Fact is, the decisions business intelligence professionals make with respect to usability, display, testing etc. have a potential impact on the people who use their applications. The impact may be as trivial as having to click a button or filter too many before they get their report, to something more significant, like a data error that leads to a poor business decision.

In short: business intelligence professionals ought to consider how their artefacts and applications affect their users.

In closing

This brings me to the end of my heresies for business intelligence. I suspect there will be a few practitioners who agree with me and (possibly many) others who don’t…and some of the latter may even find specific statements provocative. If so, I consider my job done, for my intent was to get business intelligence practitioners to question a few unquestioned tenets of their profession.

Written by K

April 3, 2014 at 9:29 pm

%d bloggers like this: