Eight to Late

Sensemaking and Analytics for Organizations

On the limitations of business intelligence systems

with 7 comments

Introduction

One of the main uses of business intelligence  (BI) systems is to support decision making in organisations.  Indeed, the old term Decision Support Systems is more descriptive of such applications than the term BI systems (although the latter does have more pizzazz).   However, as Tim Van Gelder pointed out in an insightful post,  most BI tools available in the market do not offer a means to clarify the rationale behind decisions.   As he stated, “[what] business intelligence suites (and knowledge management systems) seem to lack is any way to make the thinking behind core decision processes more explicit.”

Van Gelder is absolutely right:  BI tools do not support the process of decision-making directly, all they do is present data or information on which a decision can be based.  But there is more:  BI systems are based on  the view that data should be the primary consideration when making decisions.   In this post I explore some of the (largely tacit) assumptions that flow from such a data-centric view. My discussion builds on some points made by Terry Winograd and Fernando Flores in their wonderful book, Understanding Computers and Cognition.

As we will see, the assumptions regarding the centrality of data are questionable, particularly when dealing with complex decisions. Moreover, since these assumptions are implicit in all BI systems, they highlight the limitations of using BI systems for making business decisions.

An example

To keep the discussion grounded, I’ll use a scenario to illustrate how assumptions of data-centrism can sneak into decision making. Consider a sales manager who creates sales action plans for representatives based on reports extracted from his organisation’s BI system. In doing this, he makes a number of tacit assumptions. They are:

  1. The sales action plans should be based on the data provided by the BI system.
  2. The data available in the system is relevant to the sales action plan.
  3. The information provided by the system is objectively correct.
  4. The  side-effects of basing decisions (primarily) on data are negligible.

The assumptions and why they are incorrect

Below I state some of the key assumptions of the data-centric paradigm of BI and discuss their limitations using the example of the previous section.

Decisions should be based on data alone:    BI systems promote the view that decisions can be made based on data alone.  The danger in such a view is that it overlooks social, emotional, intuitive and qualitative factors that can and should influence decisions.  For example, a sales representative may have qualitative information regarding sales prospects that cannot be inferred from the data. Such information should be factored into the sales action plan providing the representative can justify it or is willing to stand by it.

The available data is relevant to the decision being made: Another tacit assumption made by users of BI systems is that the information provided is relevant to the decisions they have to make. However, most BI systems are designed to answer specific, predetermined questions. In general these cannot cover all possible questions that managers may ask in the future.

More important is the fact that the data itself may be based on assumptions that are not known to users. For example, our sales manager may be tempted to incorporate market forecasts simply because they are available in the BI system.  However, if he chooses to use the forecasts, he will likely not take the trouble to check the assumptions behind the models that generated the forecasts.

The available data is objectively correct:  Users of BI systems tend to look upon them as a source of objective truth. One of the reasons for this is that quantitative data tends to be viewed as being more reliable than qualitative data.  However, consider the following:

  1. In many cases it is impossible to establish the veracity of quantitative data, let alone its accuracy. In extreme cases, data can be deliberately distorted or fabricated (over the last few years there have been some high profile cases of this that need no elaboration…).
  2. The imposition of arbitrary quantitative scales on qualitative data can lead to meaningless numerical measures. See my post on the limitations of scoring methods in risk analysis for a deeper discussion of this point.
  3. The information that a BI system holds is based the subjective choices (and biases) of its designers.

In short, the data in a BI system does not represent an objective truth. It is based on subjective choices of users and designers, and thus may not be an accurate reflection of the reality it allegedly represents. (Note added on 16 Feb 2013:  See my essay on data, information and truth in organisations for more on this point).

Side-effects of data-based decisions are negligible:  When basing decisions on data, side-effects are often ignored. Although this point is closely related to the first one, it is worth making separately.  For example, judging a sales representative’s performance on sales figures alone may motivate the representative to push sales at the cost of building sustainable relationships with customers.  Another example of such behaviour is observed in call centers where employees are measured by number of calls rather than call quality (which is much harder to measure). The former metric incentivizes employees to complete calls rather than resolve issues that are raised in them. See my post entitled, measuring the unmeasurable, for a more detailed discussion of this point.

Although I have used a scenario to highlight problems of the above assumptions, they are independent of the specifics of any particular decision or system. In short, they are inherent in BI systems that are based on data – which includes most systems in operation.

Programmable and non-programmable decisions

Of course, BI systems are perfectly adequate – even indispensable –  for certain situations. Examples of these include, financial reporting (when done right!) and other operational reporting (inventory, logistics etc).  These generally tend to be routine situations with clear cut decision criteria and well-defined processes. Simply put, they are the kinds of decisions that can be programmed.

On the other hand, many decisions cannot be programmed: they have to be made based on incomplete and/or ambiguous information that can be interpreted in a variety of ways. Examples include issues such as what an organization should do in response to increased competition or formulating a sales action plan in a rapidly changing business environment. These issues are wicked: among other things, there is a diversity of viewpoints on how they should be resolved. A business manager and a sales representative are likely to have different views on how sales action plans should be adjusted in response to a changing business environment. The shortcomings of BI systems become particularly obvious when dealing with such problems.

Some may argue that it is naïve to expect BI systems to be able to handle such problems. I agree entirely. However, it is easy to overlook over the limitations of these systems, particularly when called upon to make snap decisions on complex matters. Moreover, any critical reflection regarding what BI ought to be is drowned in a deluge of vendor propaganda and advertisements masquerading as independent advice in the pages of BI trade journals.

Conclusion

In this article I have argued that BI systems have some inherent limitations as decision support tools because they focus attention on data to the exclusion of other, equally important factors.  Although the data-centric paradigm promoted by these systems is adequate for routine matters, it falls short when applied to complex decision problems.

Written by K

November 24, 2011 at 6:20 am

7 Responses

Subscribe to comments with RSS.

  1. Some interesting points here though it seems to suggest that because BI systems are basic data presentation tools everyone will take it literally and only use that for their decision making. In my own experience, there is significant intervention in business decisions which are not based on data from BI systems. In fact, from what I’ve found, if the numbers don’t align with how people think they should be, are re-evaluated based on the perception of the situations and parties involved. Imagine that many companies are like this but would vary depending on industry and culture. Share listed companies are more likely to be driven by numbers as the flow on effects of public reporting requirements, profitability, share value etc.

    In terms of quantity vs. quality example of the call centre, the company I have most experience with has focused very heavily on product knowledge training and coaching. It does seem that many companies focus much more on quantity and cost. Personally I feel businesses generally focus too much on cost and not enough on efficiency or innovative ideas that lead to greater profitability and revenue.

    Definitely agree with the observation that BI systems are not the way for complex decision making though larger organisations may well be more reliant on this due to scale. They are merely tools to track indicators which the business believes is important (or is required to from an operational/regulatory point of view). Seems that companies which come from nothing to being really big/prominent have done so in non-traditional ways….but once they in turn become larger, resort to traditional methods to shore up their gains and reduce risk. Ultimately they are likely to plateau and then other innovative companies start up😉

    These are only my opinions of course, just like that of the articles author.

    Like

    A

    December 8, 2011 at 1:55 pm

    • A,

      Thanks for your comments.

      I would argue that a great many decisions in organisations are indeed taken on the basis of simple data presentation tools. Sure, people will follow up when they think the numbers are wrong but otherwise – if numbers look OK- people are happy to use them uncritically to make or justify decisions.

      I agree entirely that BI systems are not of much help in making complex decisions. For such decisions, it is best to debate options with people who represent all possible (or at least the major) options available. This is a theme I have discussed at length on this blog (see this post, for example). BI systems may be used to support the process, but decision itself does not come from the data.

      Finally, the point about BI systems being used to monitor compliance is an excellent one and a perfect example of the usefulness of BI systems for routine (or “programmable”) decisions.

      Many thanks again for taking the time to read and comment.

      Regards,

      K.

      Like

      K

      December 9, 2011 at 8:28 pm

  2. K,

    May I add to the downfalls and to your conclusion?

    Downfalls:
    Time lag – BI systems are often used to guide future actions and are based on old data.
    o Usually without being able to account for the importance of recency (more weight is placed on data over time – so increasing the lag)This applies even in real time data collection systems.
    o Forecasting or even extrapolation of trends is not accounted for on the input data – even if some separate forecasting may be done to validate the output averages.
    o Therefore, the BI system may be giving you the best advice on what you should have done last year … last month etc…

    To the conclusion:
    • Who cares? (about the downfalls of BI systems) … because the alternatives are often disastrous
    o Without a BI system, the alternatives are likely to be expert opinions. Try an find an expert in the area who does not have either a vested interest, or expertise in only a segment of the required area (i.e. a Victorian manager making National judgments).
    o or even worse … multiple mini BI systems which are different enough to ensure that there can be no centralised control and no one person who understands why what is planned was planned.
    o Therefore, a BI system needs (like any other function) some amount of accountability to be placed on it to match the responsibility vested in it …
     Who brings the system to account and how may be forgotten though …?

    Like

    Sean

    December 9, 2011 at 11:46 am

  3. Sean,

    Thanks for making some excellent points!

    As you point out, the question regarding the use of historical data in making decisions about the future is a really interesting and complex one. Extrapolations based on history assume that the future will be much like the past or more correctly, will trend like the past. This assumption is moot, especially in a fast-changing business environment.

    For complex matters, I think the best decisions come through open debate between diverse stakeholders whose views cover the widest possible spectrum of options. This can be difficult in organisations because hierarchical management structures work against genuine bottom-up decision-making. Moreover, vested interests and hidden agendas can also trip things up. There are ways to deal with these issues through innovative governance structures supplemented by problem structuring and argument mapping techniques. Paul Culmsee and I discuss some of these in detail in our book. Although far from perfect, these address many of the shortcomings of purely data-centric or expert-reliant decision making.

    Regards,

    K.

    Like

    K

    December 9, 2011 at 9:59 pm

  4. If you look at the quantitative data in a BI system, as Sean mentioned, it is “old data”. I think old data is still valuable if you have context around that data. It’s a direct representation of decisions you made in the past.

    Take the approach of a scientist: They will make a hypothesis, create an experiment to test the hypothesis, record the findings, and see where the hypothesis stands with this “old data” at hand. The data only makes sense if you know how the experiment was set up.

    If a business insists on a data-centric model, then BI could be more valuable if context could also be captured, which of course is a whole other problem. In the mean time, you can take the scientific approach and use it as a prior decisions measure of success. Then at the very least, over time you’ll have the benefit of empirical evidence to help you make future decisions.

    Like

    EKougi

    March 1, 2013 at 3:55 pm

    • Thanks for the comment.

      Using (the outcomes of) prior decisions to gauge the usefulness of data is an interesting idea. The difficulty I see is that the link between a decision and the data supporting it may be hard to figure out. This is especially so for complex decisions. For example, an investment decision may be based on a complicated mix of criteria many of which do not involve hard data. On the other hand, it is relatively easy to draw the connections between data and operational decisions like, say, deciding that a project needs attention because it has gone over budget.

      In brief: you are right about data, but I think it is important to make a distinction between its utility for the two kinds of decision problems.

      Thanks again for taking the time to read this post and write a comment: not only have you made an interesting point, you’ve also given me an idea for another post on this topic🙂

      Regards,

      Kailash.

      Like

      K

      March 1, 2013 at 8:45 pm

  5. […] in organisations (see the Wikipedia article on BI, for example). However, as I have discussed in an earlier post, the usefulness of BI systems in making decisions regarding complex or ambiguous matters is moot. […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: