Eight to Late

Sensemaking and Analytics for Organizations

Out damn’d SPOT: an essay on data, information and truth in organisations

with 4 comments

Introduction

Jack: My report tells me that we are on track to make budget this year.

Jill: That’s strange, my report tells me otherwise

Jack: That can’t be. Have you used the right filters?

Jill: Yes – the one’s you sent me yesterday.

Jack: There must be something else…my figures must be right, they come from the ERP system.

Jill: Oh, that must be it then…mine are from the reporting system.

Conversations such as the one above occur quite often in organisation-land.  It is one of the reasons why organisations chase the holy grail of a single point of truth (SPOT): an organisation-wide repository that holds the officially endorsed true version of data, regardless of where it originates from. Such a repository is often known as an Enterprise Data Warehouse (EDW).

Like all holy grails, however, the EDW, is a mythical object that exists in only in the pages of textbooks (and vendor brochures…). It is at best an ideal to strive towards. But, like chasing the end of a rainbow it is an exercise that may prove exhausting and ultimately, futile.

Regardless of whether or not organisations can get to that mythical end of the rainbow – and there are those who claim to have got there – there is a deeper issue with the standard view of data and information that hold sway in organisation-land.   In this post I examine these standard conceptions of data and information and truth, drawing largely on this paper by Bernd Carsten Stahl and a number of secondary sources.

Some truths about data and information

As Stahl observes in his introduction:

Many assume that information is central to managerial decision making and that more and higher quality information will lead to better outcomes. This assumption persists even though Russell Ackoff argued over 40 years ago that it is misleading

The reason for the remarkable persistence of this incorrect assumption is that there is a lack of clarity as to what data and information actually are.

To begin with let’s take a look at what these terms mean in the sense in which they are commonly used in organisations. Data typically refers to raw, unprocessed facts or the results of measurements. Information is data that is imbued with meaning and relevance because it is referred to in a context of interest. For example, a piece of numerical data by itself has no meaning – it is just a number. However, its meaning becomes clear once we are provided a context – for example, that the number is the price of a particular product.

The above seems straightforward enough and embodies the standard view of data and information in organisations. However, a closer look reveals some serious problems. For example, what we call raw data is not unprocessed – the data collector always makes a choice as to what data will be collected and what will not. So in this sense, data already has meaning imposed on it. Further, there is no guarantee that what has been excluded is irrelevant. As another example, decision makers will often use data (relevant or not) just because it is available. This is a particularly common practice when defining business KPIs – people often use data that can be obtained easily rather than attempting to measure metrics that are relevant.

Four perspectives on truth

One of the tacit assumptions that managers make about the information available to them is that it is true.  But what exactly does this mean?  Let’s answer this question by taking a whirlwind tour of some theories of truth.

The most commonly accepted notion of truth is that of correspondence, that a statement is true if it describes something as it actually is.  This is pretty much how truth is perceived in business intelligence: data/information is true or valid if it describes something – a customer, an order or whatever – as it actually is.

More generally, the term correspondence theory of truth refers to a family of theories that trace their origins back to antiquity. According to Wikipedia:

Correspondence theories claim that true beliefs and true statements correspond to the actual state of affairs. This type of theory attempts to posit a relationship between thoughts or statements on one hand, and things or facts on the other. It is a traditional model which goes back at least to some of the classical Greek philosophers such as Socrates, Plato, and Aristotle. This class of theories holds that the truth or the falsity of a representation is determined solely by how it relates to a reality; that is, by whether it accurately describes that reality.

One of the problems with correspondence theories is that they require the existence of an objective reality that can be perceived in the same way by everyone. This assumption is clearly problematic, especially for issues that have a social dimension. Such issues are perceived differently by different stakeholders, and each of these will legitimately seek data that supports their point of view. The problem is that there is often no way to determine which data is “objectively right.” More to the point, in such situations the very notion of “objective rightness” can be legitimately questioned.

Another issue with correspondence theories is that a piece of data can at best be an abstraction of a real-world object or event.  This is a serious issue with correspondence theories in the context of data in organisations. For example, when a sales rep records a customer call, he or she notes down only what is required by the customer management system. Other data that may well be more important is not captured or is relegated to a “Notes” or “Comments” field that is rarely if ever searched or accessed.

Another perspective is offered by the so called consensus theories of truth which assert that true statements are those that are agreed to by the relevant group of people. This is often the way truth is established in organisations. For example, managers may choose to calculate Key Performance Indicators (KPIs )using certain pieces of data that are deemed to be true.  The problem with this is that consensus can be achieved by means that are not necessarily democratic. For example, a KPI definition chosen by a manager may be hotly contested by an employee.  Nevertheless, the employee has to accept it because organisations are typically not democratic. A more significant issue is that  the notion of “relevant group” is problematic because there is no clear criterion by which to define relevance.

Pragmatic theories of truth assert that truth is a function of utility – i.e. a statement is true if it is useful to believe it is so. In other words, the truth of a statement is to be judged by the payoff obtained by believing it to be true.  One of the problems with these theories is that it may be useful for some people to believe in a particular statement while is useful for others to disbelieve it. A good example of such a statement is: there is an objective reality. Scientists may find it useful to believe this whereas social constructionists may not. Closer home, it may be useful for a manager to believe that a particular customer is a good prospect (based on market intelligence, say), but a sales rep who knows the customer is unlikely to switch brands may think it useful to believe otherwise.

Finally, coherence theories of truth tell us that statements that are true must be consistent with a wider set of beliefs. In organisational terms, a piece of information or data that is true only if it does not contradict things that others in the organisation believe to be true. Coherence theories emphasise that the truth of statements cannot be established in isolation but must be evaluated as part of a larger system of statements (or beliefs). For example, managers may believe certain KPIs to be true because they fit in with other things they know about their business.

…And so to conclude

The truth is a slippery beast: what is true and what is not depends on what exactly one means by the truth and, as we have seen, there are several different conceptions of truth.

One may well ask if this matters from a practical point of view.  To put it plainly: should executives, middle managers and frontline employees (not to mention business intelligence analysts and data warehouse designers) worry about philosophical theories of truth?  My contention is that they should, if only to understand that the criteria they use for determining the validity of their data and information are little more than conventions that are easily overturned by taking other, equally legitimate, points of view.

Written by K

October 17, 2012 at 9:11 pm

4 Responses

Subscribe to comments with RSS.

  1. Brilliant again and spot on. Allow me to riff…Truth is influenced by belief, not only in organizations but anywhere you discover a human being. In organizations, or groups, there is often a shared truth that is tacitly agreed to in those things we call vision, mission, objective, strategy, etc., nevertheless:

    We should all question the data we see and get, employ some critical thinking skills, and understand that quite often we are not provided with anything definitive, just a slice of something that is entirely dependent on the questions that are asked of the data and whether the findings reveal what’s necessary and informative or simply what’s expedient. We know this — we learned it in high school — how statistics can say anything you want them to…and I was reminded of this in my own experience too:

    Once upon a time ago in a large organization there was a set of data used to inform executive and board decision-making and those decisions in turn affected the day-to-day of hundreds of staff and thousands and thousands of people outside the organization. This was a set of rolling quarterly data, and had been in place for about four years. In the course of budget forecasting for my own line of business and accountability, I had to draw on the same data, but it wasn’t deep enough data to provide the level of rationale insisted upon by the Finance department. I tried to extract more information from the raw data, only to discover that the data set was exceptionally limited in its scope and did not provide an accurate status of the key business objectives and performance indicators, of the costs, of the timing and impacts of process, so I expanded the parameters to find answers to those questions and what I got from the analysts was devastating. I didn’t believe the findings at first. I thought I’d done it wrong, so I pulled a team of smart, cheeky and reasonably objective staff together and we redid the work three times. Same result.

    There was a lot riding on the data, including public trust. It was not fun to be the bearer of extremely bad news. The COO did not believe it at first and it took six months for him COO to even consider the possibility that there was another version of the truth which happened only after he had his own team review both sets of data and findings (the first, incorrect set, and my team’s set) and was none too pleased to learn that for three years, based on wrong data, he had informed his colleagues and the board and the public using patently incorrect data and that they spent money, made decisions and negatively affected a huge population of people with the wrong information that they all believed was the truth — but wasn’t and if the COO, the other execs and the board had questioned what was in front of them it would have been evident. The wrong data set told them all that things were on track, on target, on time, and objectives were being met. Once the broader picture was finally accepted — which took 18 months — huge organizational unlearning had too happen, unlearning old business process and learning of new ones. (I won’t say change management, because change management is an oxymoron, or perhaps a paradoxical pairing of words)

    Data in and of itself is neither true nor untrue — it’s what we do with it (Haven’t I heard that argument before?). Data gathering is interesting too, and also influenced by individual perceptions of the truth: the parameters can be so narrow, the questions the data seek to answer can be crafted so finely as to guarantee a desired finding. Doesn’t mean it’s untrue, just not terribly useful in the overall scheme of things.

    Organizations do not yet run themselves: they are still run by people and people are fallible and want to believe that what they are doing is right and because it is right, then it must be true and if it is true, there’s no need to question it or look at anything from any other perspective. There isn’t time or money for that anyway. Building in a process, and helping people take time to ensure greater objectivity, a more wholistic view of the truth might just be a bit too expensive in the current business and management environment.

    *end of riff* 😉

    Like

    FS

    October 17, 2012 at 10:56 pm

  2. Hi FS,

    Thanks so much for your very informative and entertaining riff. I really enjoyed it and can relate to your story about how organisations tend to read too much into the limited data they have at their disposal.

    Business school curricula emphasise the scientific-rational tradition wherein data and information are king, and all else mere opinion. This is unfortunate because even a cursory reading (and understanding) of philosophy reveals the tenuous nature of (what we take to be) reality.

    After years of schooling in the hard sciences followed by a good many more of working as a physicist/engineer, the last few years of reading and thinking about the social dimensions of the organisational problems have been a revelation: our reality is socially constructed…and this includes the data, information and truths we hold so dear. A point brilliantly illustrated by your story.

    Thanks again for the great thoughts!

    Regards,

    Kailash.

    Like

    K

    October 18, 2012 at 9:12 pm

  3. […] an accurate reflection of the reality it allegedly represents. (Note added on 16 Feb 2013:  See my essay on data, information and truth in organisations for more on this […]

    Like

  4. […] other well known theories of truth but it would take me too far afield to discuss them here. See my post on data, information and truth if you are interested in finding out […]

    Like


Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.