The great question of the 21st century: Whose black box do you trust? - O'Reilly Media

Some years ago, John Mattison, the chief medical information officer of Kaiser Permanente, the large integrated health provider, said, "The great question of the 21st century is going to be 'Whose black box do you trust?'"

Mattison was talking about the growing importance of algorithms in medicine, but his point, more broadly, was that we increasingly place our trust in systems whose methods for making decisions we do not understand. (A black box, by definition, is a system whose inputs and outputs are known, but the system by which one is transformed to the other is unknown.)

A lot of attention has been paid to the role of algorithms in shaping the experience of consumers. Much less attention has been paid to the role of algorithms in shaping the incentives for business decision-making.

Eli Pariser warned of a "filter bubble," in which the algorithm takes account of our preferences and continues to feed us more of what we already want to hear, rather than exposing us to other points of view. This is a real risk—though one that search engines and social media companies are making efforts to overcome.

But there's a deeper, more pervasive risk that came out in a conversation I had recently with Chris O'Brien of VentureBeat.  

As Warren Buffet is reputed to have said, "It takes 20 years to build a reputation and five minutes to ruin it. If you think about that, you'll do things differently."

Because many of the algorithms that shape our society are black boxes—either for reasons like those cited by Facebook, or because they are, in the world of deep learning, inscrutable even to their creators—that question of trust is key.

Understanding how to evaluate algorithms without knowing the exact rules they follow is a key discipline in today's world. And it is possible. Here are four rules for evaluating whether you can trust an algorithm:

1. Its creators have made clear what outcome they are seeking, and it is possible for external observers to verify that outcome.
2. Success is measurable.
3. The goals of the algorithm's creators are aligned with the goals of the algorithm's consumers.
4. Does the algorithm lead its creators and its users to make better longer term decisions?