How to be wrong when you know you're right, and can prove it
In Chapter 1 of Reinventing the C-Suite I talk about the McNamara Fallacy, and I thought I would look at some examples and discuss it here.
The McNamara Fallacy deals with the use of measurements as a fallacious way of setting goals. There are four steps in the McNamara process, but Charles Handy added another – it’s Step 2 here:
Step 1: Measure whatever you can readily measure.
Step 2: – Assume that what you’re measuring will result in the outcomes you want.
Step 3: Ignore what you can’t measure
Step 4: Think that if you can’t measure it, it can’t be important.
Step 5: Consider that if it’s not measured, it doesn't exist.
Already you can see that it’s a fallacy. In a nutshell, the mistake is that quantitative measures are real and qualitative factors are unimportant, therefore don’t exist. There are numerous qualitative factors that companies, schools, and governments don’t measure, but they assuredly are real and are clearly more important than what is being measured and reported.
Here are some examples:
The most obvious one for companies is spreadsheets. People believe them, yet Raymond Panko studied spreadsheet errors and found that more than 80% have critical errors in them. Moreover, the average number of references to other spreadsheets is 15 (which compounds the error rate). Yet companies budget, quote, analyze and act on spreadsheets as if they were correct. Madness. Performance reviews, balanced scorecards, business cases (which primarily measure the value of a project by ROI and risk) are all examples of the McNamara Fallacy in action.
Schools measure the grades of students, as if that, and not education, was the goal. Even this grading is faulty: They use standardized testing, which uses a standard set of questions, and group scores are normalized, both of which result in relative not absolute scores. Tests are indeed a proxy for student performance, as a scholar may be performing well, but be bad at tests. None of this indicates how well educated a student, how fit for society they are, how curious or innovative they are, how successful they could be, or even how happy they are – which are undoubtedly the goals of the schooling system, rather than grades.
The ultimate government fallacy is voting numbers. A vote is no measure of the voter’s commitment or understanding of the issues, or indeed if the voter is even voting for the party (they may be voting against another party). We won’t go into qualified franchise as that is yet another skewing of voting numbers.
The McNamara Fallacy started with a government mis-measure of success when Robert McNamara was the US Secretary of Defense during the Vietnam war. The metric used by the US government was “body-count,” which to me is a chilling measure in the first place. However, they didn't measure chaos, destruction, economic impact, misery, and importantly they weren’t looking at the mood of their own people. All of the above (except body-count) were what ended the war.
One further fallacy that is most often discussed is in the treatment of cancer. The measure of success is “progression-free survival” which is the time that the patient survives during and after treatment while a tumor is still present, and “disease-free survival” which measures how long people live after there is no trace of cancer. It’s easy to measure, but that’s all. It doesn’t measure the quality of life, the cost of treatment, or the effect of the bits that they’ve cut out on survival. Moreover, “survival” is such a negatively charged word.
The McNamara Fallacy is also sometimes called the quantitative fallacy, which makes it weirdly recursively circular – the term “quantitative fallacy” invites the McNamara Fallacy right in. Using the word “quantitative” suggests that only things that can be counted can be measured, which further suggests that it’s a Boolean argument about quantitative and qualitative, which gets us firmly into Step 2 and beyond.
Now that I’m aware of the McNamara Fallacy, I see it everywhere – but mostly in the speeches of politicians.