Eight to Late

Sensemaking and Analytics for Organizations

Archive for the ‘Business Intelligence’ Category

On the disconnect between business intelligence and strategic decision-making

with 4 comments

One of the stated aims of business intelligence (BI) systems is to support better business decision making in organisations (see the Wikipedia article on BI, for example). However, as I have discussed in an earlier post, the usefulness of BI systems in making decisions regarding complex or ambiguous matters is moot. Quoting from that post:

…many decisions [in organisations] have to be made based on incomplete and/or ambiguous information that can be interpreted in a variety of ways. Examples include issues such as what an organization should do in response to increased competition or formulating a sales action plan in a rapidly changing business environment. These issues are wicked; among other things, there is a diversity of viewpoints on how they should be resolved. A business manager and a sales representative are likely to have different views on how sales action plans should be adjusted in response to a changing business environment. The shortcomings of BI systems become particularly obvious when dealing with such problems.

This brings up the question as to how is BI actually used in organisations.

Quoting again from my earlier article:

BI systems are perfectly adequate – even indispensable – for certain situations. Examples of these include, financial reporting (when done right!) and other operational reporting (inventory, logistics etc).  These generally tend to be routine situations with clear cut decision criteria and well-defined processes. Simply put, they are the kinds of decisions that can be programmed.

Typically programmed decisions are made when checking on or monitoring business activities.  I would hazard a guess that BI applications are generally used to carry out such routine monitoring of business processes (and take rule-based corrective action, if necessary) rather than in making complex decisions.  To use a phrase coined by James March, BI applications are used in surveillance mode rather than decision mode.

Unfortunately most BI vendors are yet to address this gap. Most new features that vendors come up with operate in surveillance mode rather than decision mode – that is, they help organisations track (and correct) performance rather than decide on complex/uncertain matters. Thus, despite vendor claims to the contrary, BI is still used as a means to measure and manage  operational matters rather than to make strategic decisions.

Incidentally, this is also true of over-hyped new areas such as Big Data and Predictive Analytics.

Big data refers to a set of technologies and techniques that are useful when analysing large volumes of fast-changing, unstructured data to make operational decisions. For example, commercially available big data products such as splunk can monitor  vast numbers  of unstructured server logs in real time and tell you what corrective actions need to be taken (an operational decision) but they cannot tell you what IT investments you should make over the next five years (a strategic decision).

Predictive analytics refers to a wide range of techniques that are used to identify patterns in past data in order to make predictions about the future events. However, the predictions made using such techniques can only be as good as the underlying mathematical models. Consequently, success in predictive analytics depends crucially on knowing the key variables that govern the phenomena of interest. Identifying these variables can be difficult, if not impossible, in the case of business decisions because of human factors (intentions, motivations etc.). As  Gregory Piatetski-Shapiro puts it in this article, “Predictive analytics can figure out how to land on Mars, but not who will buy a Mars bar.”

So, the question arises:  what do BI vendors need to do in order to facilitate decision-making on complex matters?

To answer this we need to take a brief look at the process of decision-making. The traditional  view is that the decisions are made by working through the following steps:

  1. Identifying available options
  2. Understanding the consequences of each option.
  3. Rating options based on preferences for those consequences
  4. Selecting an option (based on rules and ratings)

However, as I have discussed in a post on the nature of decision making in organisations, in the case of complex decisions not only is it hard to identify all options and their consequences, even preferences and/or selection rules may change as one’s knowledge of the options improves.  As a consequence, such decisions necessarily involve informal reasoning  – a deliberative process that takes into account partipants’ values and beliefs,  in addition to logic and  “hard facts”.  The important point, as Tim van Gelder notes in a brilliant post entitled, The missing “I” in BI, is that none of the BI suites in the market support informal reasoning.  The lack of support is especially strange because there are well-known techniques such as Issue Based Information System (IBIS) and Argument Mapping  that can be used to facilitate and capture such reasoning.

This gap does not matter in the case of operational decisions as the choice is made on the basis of straightforward (or programmable) rules, as in steps 1 through 4 above. However, the situation is different in case of complex or non-programmable decisions such as those that are made in the face of uncertainty. In these cases the lack of support for facilitating, capturing and storing decision rationale becomes a huge handicap.

In summary:  Currently available BI tools are good for operational rather than strategic decision-making because they do not offer any support for the deliberative process that is needed to make complex decisions in the face of ambiguity or uncertainty. The adage, “data doesn’t make decisions, people do”  is particularly true for strategic decisions, but it appears  BI vendors are yet to recognise this.

Acknowledgement:

This post was inspired by a recent comment on one of my earlier posts on business intelligence.

Written by K

March 7, 2013 at 8:58 pm

Out damn’d SPOT: an essay on data, information and truth in organisations

with 4 comments

Introduction

Jack: My report tells me that we are on track to make budget this year.

Jill: That’s strange, my report tells me otherwise

Jack: That can’t be. Have you used the right filters?

Jill: Yes – the one’s you sent me yesterday.

Jack: There must be something else…my figures must be right, they come from the ERP system.

Jill: Oh, that must be it then…mine are from the reporting system.

Conversations such as the one above occur quite often in organisation-land.  It is one of the reasons why organisations chase the holy grail of a single point of truth (SPOT): an organisation-wide repository that holds the officially endorsed true version of data, regardless of where it originates from. Such a repository is often known as an Enterprise Data Warehouse (EDW).

Like all holy grails, however, the EDW, is a mythical object that exists in only in the pages of textbooks (and vendor brochures…). It is at best an ideal to strive towards. But, like chasing the end of a rainbow it is an exercise that may prove exhausting and ultimately, futile.

Regardless of whether or not organisations can get to that mythical end of the rainbow – and there are those who claim to have got there – there is a deeper issue with the standard view of data and information that hold sway in organisation-land.   In this post I examine these standard conceptions of data and information and truth, drawing largely on this paper by Bernd Carsten Stahl and a number of secondary sources.

Some truths about data and information

As Stahl observes in his introduction:

Many assume that information is central to managerial decision making and that more and higher quality information will lead to better outcomes. This assumption persists even though Russell Ackoff argued over 40 years ago that it is misleading

The reason for the remarkable persistence of this incorrect assumption is that there is a lack of clarity as to what data and information actually are.

To begin with let’s take a look at what these terms mean in the sense in which they are commonly used in organisations. Data typically refers to raw, unprocessed facts or the results of measurements. Information is data that is imbued with meaning and relevance because it is referred to in a context of interest. For example, a piece of numerical data by itself has no meaning – it is just a number. However, its meaning becomes clear once we are provided a context – for example, that the number is the price of a particular product.

The above seems straightforward enough and embodies the standard view of data and information in organisations. However, a closer look reveals some serious problems. For example, what we call raw data is not unprocessed – the data collector always makes a choice as to what data will be collected and what will not. So in this sense, data already has meaning imposed on it. Further, there is no guarantee that what has been excluded is irrelevant. As another example, decision makers will often use data (relevant or not) just because it is available. This is a particularly common practice when defining business KPIs – people often use data that can be obtained easily rather than attempting to measure metrics that are relevant.

Four perspectives on truth

One of the tacit assumptions that managers make about the information available to them is that it is true.  But what exactly does this mean?  Let’s answer this question by taking a whirlwind tour of some theories of truth.

The most commonly accepted notion of truth is that of correspondence, that a statement is true if it describes something as it actually is.  This is pretty much how truth is perceived in business intelligence: data/information is true or valid if it describes something – a customer, an order or whatever – as it actually is.

More generally, the term correspondence theory of truth refers to a family of theories that trace their origins back to antiquity. According to Wikipedia:

Correspondence theories claim that true beliefs and true statements correspond to the actual state of affairs. This type of theory attempts to posit a relationship between thoughts or statements on one hand, and things or facts on the other. It is a traditional model which goes back at least to some of the classical Greek philosophers such as Socrates, Plato, and Aristotle. This class of theories holds that the truth or the falsity of a representation is determined solely by how it relates to a reality; that is, by whether it accurately describes that reality.

One of the problems with correspondence theories is that they require the existence of an objective reality that can be perceived in the same way by everyone. This assumption is clearly problematic, especially for issues that have a social dimension. Such issues are perceived differently by different stakeholders, and each of these will legitimately seek data that supports their point of view. The problem is that there is often no way to determine which data is “objectively right.” More to the point, in such situations the very notion of “objective rightness” can be legitimately questioned.

Another issue with correspondence theories is that a piece of data can at best be an abstraction of a real-world object or event.  This is a serious issue with correspondence theories in the context of data in organisations. For example, when a sales rep records a customer call, he or she notes down only what is required by the customer management system. Other data that may well be more important is not captured or is relegated to a “Notes” or “Comments” field that is rarely if ever searched or accessed.

Another perspective is offered by the so called consensus theories of truth which assert that true statements are those that are agreed to by the relevant group of people. This is often the way truth is established in organisations. For example, managers may choose to calculate Key Performance Indicators (KPIs )using certain pieces of data that are deemed to be true.  The problem with this is that consensus can be achieved by means that are not necessarily democratic. For example, a KPI definition chosen by a manager may be hotly contested by an employee.  Nevertheless, the employee has to accept it because organisations are typically not democratic. A more significant issue is that  the notion of “relevant group” is problematic because there is no clear criterion by which to define relevance.

Pragmatic theories of truth assert that truth is a function of utility – i.e. a statement is true if it is useful to believe it is so. In other words, the truth of a statement is to be judged by the payoff obtained by believing it to be true.  One of the problems with these theories is that it may be useful for some people to believe in a particular statement while is useful for others to disbelieve it. A good example of such a statement is: there is an objective reality. Scientists may find it useful to believe this whereas social constructionists may not. Closer home, it may be useful for a manager to believe that a particular customer is a good prospect (based on market intelligence, say), but a sales rep who knows the customer is unlikely to switch brands may think it useful to believe otherwise.

Finally, coherence theories of truth tell us that statements that are true must be consistent with a wider set of beliefs. In organisational terms, a piece of information or data that is true only if it does not contradict things that others in the organisation believe to be true. Coherence theories emphasise that the truth of statements cannot be established in isolation but must be evaluated as part of a larger system of statements (or beliefs). For example, managers may believe certain KPIs to be true because they fit in with other things they know about their business.

…And so to conclude

The truth is a slippery beast: what is true and what is not depends on what exactly one means by the truth and, as we have seen, there are several different conceptions of truth.

One may well ask if this matters from a practical point of view.  To put it plainly: should executives, middle managers and frontline employees (not to mention business intelligence analysts and data warehouse designers) worry about philosophical theories of truth?  My contention is that they should, if only to understand that the criteria they use for determining the validity of their data and information are little more than conventions that are easily overturned by taking other, equally legitimate, points of view.

Written by K

October 17, 2012 at 9:11 pm

On the limitations of business intelligence systems

with 7 comments

Introduction

One of the main uses of business intelligence  (BI) systems is to support decision making in organisations.  Indeed, the old term Decision Support Systems is more descriptive of such applications than the term BI systems (although the latter does have more pizzazz).   However, as Tim Van Gelder pointed out in an insightful post,  most BI tools available in the market do not offer a means to clarify the rationale behind decisions.   As he stated, “[what] business intelligence suites (and knowledge management systems) seem to lack is any way to make the thinking behind core decision processes more explicit.”

Van Gelder is absolutely right:  BI tools do not support the process of decision-making directly, all they do is present data or information on which a decision can be based.  But there is more:  BI systems are based on  the view that data should be the primary consideration when making decisions.   In this post I explore some of the (largely tacit) assumptions that flow from such a data-centric view. My discussion builds on some points made by Terry Winograd and Fernando Flores in their wonderful book, Understanding Computers and Cognition.

As we will see, the assumptions regarding the centrality of data are questionable, particularly when dealing with complex decisions. Moreover, since these assumptions are implicit in all BI systems, they highlight the limitations of using BI systems for making business decisions.

An example

To keep the discussion grounded, I’ll use a scenario to illustrate how assumptions of data-centrism can sneak into decision making. Consider a sales manager who creates sales action plans for representatives based on reports extracted from his organisation’s BI system. In doing this, he makes a number of tacit assumptions. They are:

  1. The sales action plans should be based on the data provided by the BI system.
  2. The data available in the system is relevant to the sales action plan.
  3. The information provided by the system is objectively correct.
  4. The  side-effects of basing decisions (primarily) on data are negligible.

The assumptions and why they are incorrect

Below I state some of the key assumptions of the data-centric paradigm of BI and discuss their limitations using the example of the previous section.

Decisions should be based on data alone:    BI systems promote the view that decisions can be made based on data alone.  The danger in such a view is that it overlooks social, emotional, intuitive and qualitative factors that can and should influence decisions.  For example, a sales representative may have qualitative information regarding sales prospects that cannot be inferred from the data. Such information should be factored into the sales action plan providing the representative can justify it or is willing to stand by it.

The available data is relevant to the decision being made: Another tacit assumption made by users of BI systems is that the information provided is relevant to the decisions they have to make. However, most BI systems are designed to answer specific, predetermined questions. In general these cannot cover all possible questions that managers may ask in the future.

More important is the fact that the data itself may be based on assumptions that are not known to users. For example, our sales manager may be tempted to incorporate market forecasts simply because they are available in the BI system.  However, if he chooses to use the forecasts, he will likely not take the trouble to check the assumptions behind the models that generated the forecasts.

The available data is objectively correct:  Users of BI systems tend to look upon them as a source of objective truth. One of the reasons for this is that quantitative data tends to be viewed as being more reliable than qualitative data.  However, consider the following:

  1. In many cases it is impossible to establish the veracity of quantitative data, let alone its accuracy. In extreme cases, data can be deliberately distorted or fabricated (over the last few years there have been some high profile cases of this that need no elaboration…).
  2. The imposition of arbitrary quantitative scales on qualitative data can lead to meaningless numerical measures. See my post on the limitations of scoring methods in risk analysis for a deeper discussion of this point.
  3. The information that a BI system holds is based the subjective choices (and biases) of its designers.

In short, the data in a BI system does not represent an objective truth. It is based on subjective choices of users and designers, and thus may not be an accurate reflection of the reality it allegedly represents. (Note added on 16 Feb 2013:  See my essay on data, information and truth in organisations for more on this point).

Side-effects of data-based decisions are negligible:  When basing decisions on data, side-effects are often ignored. Although this point is closely related to the first one, it is worth making separately.  For example, judging a sales representative’s performance on sales figures alone may motivate the representative to push sales at the cost of building sustainable relationships with customers.  Another example of such behaviour is observed in call centers where employees are measured by number of calls rather than call quality (which is much harder to measure). The former metric incentivizes employees to complete calls rather than resolve issues that are raised in them. See my post entitled, measuring the unmeasurable, for a more detailed discussion of this point.

Although I have used a scenario to highlight problems of the above assumptions, they are independent of the specifics of any particular decision or system. In short, they are inherent in BI systems that are based on data – which includes most systems in operation.

Programmable and non-programmable decisions

Of course, BI systems are perfectly adequate – even indispensable –  for certain situations. Examples of these include, financial reporting (when done right!) and other operational reporting (inventory, logistics etc).  These generally tend to be routine situations with clear cut decision criteria and well-defined processes. Simply put, they are the kinds of decisions that can be programmed.

On the other hand, many decisions cannot be programmed: they have to be made based on incomplete and/or ambiguous information that can be interpreted in a variety of ways. Examples include issues such as what an organization should do in response to increased competition or formulating a sales action plan in a rapidly changing business environment. These issues are wicked: among other things, there is a diversity of viewpoints on how they should be resolved. A business manager and a sales representative are likely to have different views on how sales action plans should be adjusted in response to a changing business environment. The shortcomings of BI systems become particularly obvious when dealing with such problems.

Some may argue that it is naïve to expect BI systems to be able to handle such problems. I agree entirely. However, it is easy to overlook over the limitations of these systems, particularly when called upon to make snap decisions on complex matters. Moreover, any critical reflection regarding what BI ought to be is drowned in a deluge of vendor propaganda and advertisements masquerading as independent advice in the pages of BI trade journals.

Conclusion

In this article I have argued that BI systems have some inherent limitations as decision support tools because they focus attention on data to the exclusion of other, equally important factors.  Although the data-centric paradigm promoted by these systems is adequate for routine matters, it falls short when applied to complex decision problems.

Written by K

November 24, 2011 at 6:20 am

%d bloggers like this: