The Cost of Confidence

by Roger Bourke White Jr., copyright Apr 2011


"Doubt is an unpleasant condition. But certainty is absurd." -- Voltaire



What do the following have in common?

o Multi-million or billion dollar cost overruns on complex projects

o The popularity of personalities such as Michelle Bachmann, Donald Trump, Michael Moore and Glenn Back

o The popularity of tech support people who always have a snap answer for your problem

o Bernard Madoff



These are all circumstances in which the public has proved ready, willing and able to pay a lot extra for displays of confidence.


Confidence and Competence

Confidence may not be called one, but it is treated as a virtue by many people. People love watching other people who are confident and they instinctively trust other people who are confident.

Trusting in confidence is an instinct because in many conditions it pays off well, and it has done so for generations. If a person is well practiced at a task, they can do it quickly, easily and reliably. The person doing the task looks... confident, and the results of their efforts are reliably positive. Confidence is an indicator of competence.

Where we get into trouble is when there is confidence being displayed, but no competence backing that display up.

Historically, this condition of confidence but no competence is common enough that it is a common story element -- it is something to be cautious about.

But we don't live in the Stone Age anymore, and one of the tools of modern technology is diverse ways of communicating with many people. One way of gaming this new technology is to use it to build the appearance of confidence and divorce the appearance from the competence it is supposed to be linked to.


The cost of confidence

The cost of confidence may be painfully obvious as it is in the case of Bernard Madoff. In other cases it's not quite so obvious, but the cost is the same: it is spending resource in the wrong place so the resource is wasted.

If a contractor gets a bid because he or she is confident, not because they have presented a realistic bid, then the purchaser is paying extra.

If people are electing a confident person who is not competent they are not going to get good choices on voting or political negotiations. If they are following advice coming from a confident person who is not competent, they are going to waste their time and money.

If people are getting snap answers from technical support, they will be getting answers that don't solve problems as often as those that do.


Confidence versus Research

The alternative to confidence is research: Investigating the facts and doing analysis. This can mean doing more investigating when a project is presented, or doing more checking to find out if what the confident person is saying is true.

Research is expensive, and it can be hard for a person to do. It is analytic thinking, the hard kind of thinking. This is why the confidence short-cut is emotionally attractive.

The other instinctively hard part is believing an informed person when they say, "You know... I don't know about that." and then not dismissing that person out-of-hand because they are not confident. But, the real world, especially the civilized real world, is a complex place. It is filled with many mysteries and snap-answer confidence will not correctly answer all the questions the real world brings up.



Waste is the cost of following confidence that is not backed with competence. This is the Cost of Confidence, and it can be a big cost. The alternative to following confidence is to do independent research and analysis to find out if the confidence another person displays is merited.


Update: Here is a video on a closely related topic, Kathryn Schulz: On Being Wrong. Talking in an entertaining way about the mental and emotional processes surrounding being wrong.

Update: A couple of ideas brought to me by Dan Lufkin, a fellow MIT alum:

The Dunning-Kruger effect is a science study on this phenomenon. It is the science version of the proverb, "A little knowledge is dangerous."

The actual quotation is:

A little learning is a dangerous thing;
drink deep, or taste not the Pierian spring:
there shallow draughts intoxicate the brain,
and drinking largely sobers us again.

Alexander Pope, An essay on Criticism English poet & satirist (1688 - 1744)

There's a subtle difference between "learning" and "knowledge".

The best poetic reference to D-K, in my opinion, is from Yeats's "The Second Coming".

The best lack all conviction, while the worst
Are full of passionate intensity

This is where I first realized that the current political scene is pure D-K. Considering that Yeats wrote the poem just after WW I, I think he was remarkably prescient.

Update: This Reason magazine July 2011 article, It’s Hard to Make Predictions, Especially About the Future by Ronald Bailey, describes research done on the dismal accuracy of pundit predictions. The theme of the article, "Why dart-throwing chimps are better than the experts."

Update: This TED video was brought to my attention by Tin Klanjscek, a fellow MIT alumnus, Tim Harford: Trial, error and the God complex. It is covering this same issue. Tim calls being confident in your solution in the face of a problem of enormous complexity the God complex, and recommends a lot of trial and error to see if that confidence is truly justified.

Update: A 28 Jan 12 Economist article on this topic, Fleecing the Flock.


-- The End --