(Article Originally Appeared On Forbes.com, April 2, 2017. Author: Bryce Hoffman)
There is nothing surprising about BlackRock’s decision to begin replacing human portfolio managers with artificial intelligences, at least not to anyone who has been following the financial industry. And unless you are one of the dozens of human portfolio managers whose jobs are being replaced by AIs, there is nothing wrong with it either — as long as BlackRock remembers what happened to Knight Capital.
In 2012, Knight Capital almost went bankrupt after its new automated trading algorithm made $440 million in bad trades in just 45 minutes.
The Knight Capital story is a cautionary tale about what can happen when companies defer too much of their decision making to automated systems, and a textbook example of what cognitive psychologists call automation bias.
Automation bias refers to our tendency to stop questioning the results generated by such automated systems once we begin to rely on them.
It is a big problem in the airline industry, where increased cockpit automation has made pilots dangerously dependent on such technology. Study after study in flight simulators has shown that even experienced pilots will often disregard important information when automated systems fail to alert them to it or, worse still, make dangerous mistakes when those same systems provide them with erroneous data. Automation bias has been cited as a major factor in several real-world crashes, too, including the loss of Air France 447 in 2009.
The problem is that most people, while initially skeptical of automated systems, come to view them as infallible after they get used to working with them. And that is as true for executives at investment firms as it is for airline pilots.
BlackRock’s robots won’t cost anyone their lives, but the company’s increased reliance on AI fund managers does pose new risks for investors, as the Knight Capital story illustrates.
Of course, so does not using such systems.
Researchers have also shown that most human stock pickers can’t match the performance of the market as a whole. In fact, a stunning three-quarters of all U.S. stock mutual funds failed to beat the market over the past decade.
AIs not only have a better track record when it comes to picking the right investments, but they do their work at a fraction of the cost of their human rivals. According to Opimas, switching to AIs will lower BlackRock’s ratio of costs to profits by about 28%.
Further automation of the investment industry — and indeed everyindustry — is inevitable. But BlackRock — and indeed any company that increases its reliance on such automated systems — needs to ensure it also has human systems in place to check the work of its AIs and make sure the choices they are making are sound.
Red teaming, a system developed by the military and intelligence agencies to help organizations make better decisions, is one way of doing that. Red teaming requires that a sample of the recommendations made by automated systems be regularly subjected to an independent, critical review.
This sort of backstopping will become increasingly important as more and more companies follow BlackRock’s lead.
Let us help you stay two steps ahead of the competition