Too good to fail? The surprising way a top-performing system can hurt you.

Imagine two (human) workers:

  • Chris Careless is a constant disappointment to you, performing your task well 70% of the time and producing an absolute cringe the rest of the time. Watching Chris make 10 attempts is more than enough to provoke an “oh, dear” response from you.
  • Ronnie Reliable is another story. You’ve seen Ronnie in action over a hundred times and you’ve been consistently impressed.

Here comes the million-dollar question. Which worker is more dangerous to your business?

On a high-stakes task, the answer could be Ronnie Reliable… but perhaps not for the first reason that comes to mind.

This isn’t about bad projects

In another article, I’ve pointed out that ultra-reliable workers can be dangerous when the decision-maker is deranged. They “just follow orders” even if those orders are terrible, so they can amplify incompetence (or malice). But that’s not the logic I’ll present here, since you’ve already heard me make that argument. Let’s squint at this from a different angle.

Supposing that the project is a wonderful idea that will make the world a better place if it’s done right, is Ronnie Reliable still the better choice?

When you know better than to trust

The thing is, you know you shouldn’t trust Chris Careless. It’s obvious to you. You expect failure… which is why you’re not going to bet the house on Chris. (Right?) You’re not going to let incompetence take you by surprise, so you’ll design around it. You’ll be wise enough to put failsafes in place for the inevitable booboo.

You’ll also make sure you’re keeping an eye on things, so you’re monitoring Chris Careless thoroughly. But Ronnie? You trust Ronnie Reliable. Why check up or build safety nets? Ronnie’s flawless, right?

Beware the reliable worker

Ronnie’s not flawless. You just haven’t seen the failure yet — it takes more data to observe the breaking point. The fact of the matter is that you haven’t had a chance to properly evaluate just how catastrophic Ronnie Reliable’s meltdown could be.

Too much trust is a problem. When a system is obviously flawed, you plan around its mistakes. You don’t rely on perfect execution.

By failing to understand that there’s a crucial difference between good and perfect, leaders can turn the blessing of a good worker into the curse of the high performer.

At scale, nothing is perfect

The problem is that you think you’ve tested Ronnie thoroughly, but you haven’t. It takes far more than 100 tries to see what a screw up looks like. Scale up operations and you’re in for a nasty sort of treat.

While the advice in this article holds for human workers, it’s even more urgent for AI systems and other scalable solutions. One of the most dangerous things about solutions based on math and data is that non-experts trust them too much. Don’t be the sort of sitting duck that trusts in perfection on complex tasks.

When you increase the scale, you’ll meet the long tail.

It’s best to assume that nothing is perfect. Even the safest systems can fail… especially when you give them plenty of opportunities.

As site reliability engineers love to say, “When you increase the scale, you’ll meet the long tail.”

Rounding up the long tail

Even if your system has been tested thoroughly and found to be 99.99% good, that still doesn’t mean it’s perfect. Unfortunately, if you’re not careful, you might mentally round that up to 100%. In other words, you’ll dismiss the possibility of mistakes because their probability is low. That’s another way in which the high-performing system can be more dangerous than a low-performing one… unless you do something about it.

Don’t dismiss the *possibility* of mistakes when their *probability* is low.
The “unsinkable” Titanic is an example of mental rounding up. According to NBC, ‘The phrase was originally “practically unsinkable” and was from an obscure engineering journal, but after a while it didn’t matter. On top of that, someone claims to have heard ship Capt. Edward John Smith say “Even God himself couldn’t sink this ship.”’ Image: SOURCE.

Staying safe

What makes Ronnie Reliable dangerous here is not superlative performance. The threat is excessive trust.

So, what’s the solution? How do you reap all the benefits of excellence without exposing yourself to the risks? Simple! Build safety nets for Ronnie as if you’re dealing with Chris Careless. Then you get the best of all worlds.

Just because you haven’t seen a mistake yet doesn’t mean your system is perfect. Plan for failure and build safety nets!

Whether the task is handled by humans or machines, never underestimate the importance of safety nets. Allowing yourself to be lulled into a false sense of security by seemingly-flawless performance is bad leadership.

Trusting in perfection is dangerous. Think of perfection as a nice bonus, but never rely on it.

Instead, ask yourself unpalatable what-if questions. What if your top surgeon is ill on the job? What if the machine monitoring a patient’s vitals malfunctions? What if the driver is too tired to pay attention to the road? What if an automated border control’s facial recognition system misidentifies someone? What if the human checking a passport makes a mistake? What happens next?

“What happens next?”

Whenever I see stomach-churningly ill-fated AI applications, the part that makes my hair stand on end is rarely the automation itself. It’s the builders’ blissful ignorance about mistakes. Occasionally this kind of ignorance borders on criminal.

Mistakes *will* happen.

The question to ask about mistakes isn’t, “Will they happen?” They will. Instead, ask:

  • What safety nets are in place to protect people from the consequences of those mistakes?
  • If the whole system fails — safety nets and all — what is the plan for making things right?

If there’s no plan to prevent and remedy harm, buckle up for disaster. Whoever’s in charge of such a project is worse than incompetent. They’re a menace to society. Don’t be that person.

Humans versus machines

If a mistake is so catastrophic that failure is intolerable, then don’t automate the task and don’t let human workers do it either. Or, if there’s something in your ethics that says it’s more okay for the failure to come from a human worker than from a machine worker (the crux of many autonomous vehicle debates), then use a human-in-the-loop approach.

Better is not the same as perfect.

But whatever you do, remember that mistakes are possible. Humans make mistakes, AI systems make mistakes. Even if the AI system you deploy will make fewer mistakes than the human alternative, remember that fewer is not the same as none. Better is not the same as perfect.

Whenever tasks are complex or the inputs are varied, mistakes will happen.

Believing in the myth of perfection can have dire consequences, so don’t let mathemagical thinking get in the way of common sense. Whenever tasks are complex or the inputs are varied, mistakes will happen.

In summary

If there’s no plan for how to deal with a mistake, the result might be catastrophic! It might hit you much harder than a mistake from a poor performer for the exact reason that you forgot to plan for it.

So, if you’re wise, you’ll opt for the best system but build safety nets as if it’s the worst system.

Original article can be found here