1.If we take a step back from the specifics, what would you say are the deeper causes of this event?2. Could this problem have been prevented by better management? What different procedures for change control, event response, etc. should have been in place that were not?
Unformatted Attachment Preview
For the exclusive use of G. Davis, 2017.
Professors Robert D. Austin and Darren Meister wrote this case solely to provide material for class discussion. The authors do
not intend to illustrate either effective or ineffective handling of a managerial situation. The authors may have disguised certain
names and other identifying information to protect confidentiality.
This publication may not be transmitted, photocopied, digitized or otherwise reproduced in any form or by any means without
the permission of the copyright holder. Reproduction of this material is not covered under authorization by any reproduction
rights organization. To order copies or request permission to reproduce materials, contact Ivey Publishing, Ivey Business
School, Western University, London, Ontario, Canada, N6G 0N1; (t) 519.661.3208; (e) email@example.com; www.iveycases.com.
Copyright © 2015, Richard Ivey School of Business Foundation
We’re pleased to report that total revenues rose 22.2 per cent and pre-tax earnings rose 24.8
per cent compared to 2010 . . . In 2011, we again led all securities firms in shares traded of
NYSE- and Nasdaq-listed stocks, ETFs [exchange traded funds], and OTC [over-the-counter]
Bulletin Board and OTC Market Securities. In recognition of our exceptional technology,
Knight was ranked 1st in the banking and financial services category of the InformationWeek
500, and 14th overall . . .
— Thomas M. Joyce, Knight’s chairman and chief executive officer (CEO) 2
On Wednesday, August 1, 2012, at 8:01 a.m., the computer systems of Knight Capital Americas LLC
(Knight) started to send emails to employees referencing “SMARS,” the company’s automated system
that sent equity orders to the market for execution. The would-be recipients were being notified about a
“Power Peg disabled error.” Over the next hour and a half, 97 emails were sent, but most went unread.
None prompted significant action. 3
A little before 9:30 a.m., Knight systems began processing 212 orders from broker-dealers who had
been pre-qualified to participate in a New York Stock Exchange (NYSE) program scheduled to start on
that day, the “Retail Liquidity Program” (RLP). Knight’s information technology (IT) staff had
deployed new software in the previous four days to prepare for the RLP launch. As stock markets
opened, the new software began to operate. 4
But there was something wrong. One of eight servers that ran SMARS software was somehow
different from the rest. This “rogue server” began sending trade orders to the market without keeping
track of whether previous orders had been filled. 5 In large volume and at a dizzying speed, buy and sell
orders streamed from Knight Capital into the markets.
These orders produced noticeable effects. The prices of 150 stocks of companies large and small began
to swing wildly, some rising, some falling, many by more than 10 per cent in mere minutes. 6 A trading
frenzy commenced as others joined in. Wells Fargo (WFC), for example, traded 22.6 million shares,
more than it had traded in the previous two days, in just a few minutes. 7
Knight continued this trading for 45 minutes. Realizing that something was wrong, the company then
notified clients “to route listed orders away” and shut down trading. The frenzy that Knight had started
continued, however, and at 11 a.m., the NYSE halted trading in six stocks, as so-called “circuit
breakers” triggered; later that afternoon, the NYSE would cancel trades in a handful of stocks that had
traded during the morning frenzy at 30 per cent or more higher or lower than their opening prices.
This document is authorized for use only by Gavin Davis in 2017.
For the exclusive use of G. Davis, 2017.
Market observers concluded that Knight had likely been the cause of the market disruptions, and the
company’s stock began to trend downward. By the end of the day, Knight’s stock price would drop 33
per cent. 8
Less than an hour after the markets’ opening and struggling to understand what had happened,
managers at Knight were stunned to discover that the company had, in a mere 45 minutes, sent millions
of orders, resulting in four million trade executions in 154 stocks for more than 397 million shares.
Knight had assumed an approximately $3.5 billion net long position in 80 stocks and an approximately
$3.15 billion net short position in 74 stocks. 9
Later in the day, Knight announced that “a technology issue” had resulted in trading losses of
approximately $440 million, an amount comparable to its total market capitalization at market close on
Wednesday, that is, $681 million. 10 As one writer put it, the company had shown the world “how to
lose $172,222 per second for 45 minutes.” 11
CEO Joyce had been out of the office on Tuesday for knee surgery. On Wednesday morning, he
limped back into a company deep in crisis. The stock fell 63 per cent on Thursday, as the company
sought an investor or a buyer. Worth more than $1 billion at the beginning of the day on Tuesday,
Knight had shrunk to $250 million by Thursday night. Large customers, such as the Vanguard Group
and TD Ameritrade, announced that they had stopped doing business with Knight. Analysts suggested
that without rescue by an external party within two or three days, the company would not survive. 12
On Friday morning, Knight announced that it had secured a line of credit from a group of investors that
included TD Ameritrade and Blackstone in an arrangement that involved selling the new backers $400
million in convertible shares. The deal was painful for shareholders because it heavily diluted their
shares, but it was better than a Knight bankruptcy, the only apparent alternative. TD Ameritrade and
other firms promptly resumed business with Knight, and the stock began to recover. By the time
markets closed for the weekend, the stock was up 57 per cent from its low point.13 The company had
survived a near-death experience — but not without incurring serious damage to its reputation and
In September, Knight hired IBM to conduct a review of its internal systems development process. 14 In
October, the company announced its quarterly results — a net loss of $389.9 million on negative
revenues of $189.8 million — and disclosed that losses from the August 1 event had increased to $461
million. It also announced the creation of a board-level risk committee. 15
In December, Getco, a high speed trading firm that had been part of the investor group, agreed to
acquire Knight in a $1.4 billion deal. Upon completion of the deal in July 2013, Joyce announced his
Founded in 1995, Knight was “built on the idea that the self-directed retail investor would desire a
better, faster and more reliable way to access the market.” The company was a market maker, a brokerdealer that handled and submitted proposed trades on behalf of others (e.g., retail investors); it was the
largest in the United States in terms of volume on major stock markets (NYSE, NASDAQ). Knight
clients included more than 5,000 of the world’s largest institutions and financial services firms to
whom it provided ”superior trade executions in a cost effective way for a wide spectrum of clients in
multiple asset classes, including: equities (domestic and foreign securities), fixed income securities,
derivatives, and currencies.”
This document is authorized for use only by Gavin Davis in 2017.
For the exclusive use of G. Davis, 2017.
By 2012, on active days Knight executed more than 10 million trades, with volume exceeding 20
billion shares. The company maintained offices in the United States, Europe and Asia and employed
more than 1,500 people. In 2011, Knight:
Made markets in or traded approximately 19,000 securities.
Executed trades of more than one trillion shares (approximately 4 billion per day) in U.S. equities.
Executed more than 900 million equity trades (approximately 4 million per day).
Traded more than $6.4 trillion in notional value (over $24 billion per day). ͒
Accounted for approximately 10 per cent of all trades in listed U.S. equity securities.
The majority of trades executed by Knight originated with retail investors and came to Knight via retail
brokers. But institutional clients also provided orders to Knight on behalf of mutual funds and pension
Knight enjoyed particular renown for its use of technology. The company worked actively to promote
its reputation as a leader in the use of trading technology. In June 2012, five weeks before the problem
on August 1, Joyce gave testimony to the U.S. Congress, advising it on how to promote the efficiency
of stock markets. In the process, he advertised the unique merits of Knight’s IT capabilities:
Our data centers are some of the largest and most reliable in the industry. We spend tens of
millions of dollars every year making our technology platform better, faster and more reliable.
Today, we have the capacity to process 20 million trades per day. We have connectivity to
nearly every source of liquidity in the equities market, and our trade response times are
measured in milliseconds. Our years of research and development, technology platform
enhancements, and connectivity to liquidity wherever it resides is all brought to bear in our
endeavour to secure best execution on behalf of our customers . . . As a result, we believe that
Knight is uniquely qualified to comment on the market structure issues which are the focus of
In 2011, Knight’s revenues reached a record $1.36 billion, while earnings increased to $115.6 million.
At the end of 2011, Knight had $7.2 billion in assets, $4.4 billion of which primarily consisted of cash
or assets readily convertible to cash. During the time Joyce had been CEO, revenues had multiplied
nine times, and share price had climbed 85 per cent. 18
$ 9(5< %5,() +,6725< 2) ),1$1&,$/ 0$5.(7 ',65837,216 3266,%/< &$86(' %< &20387(56<67(06 On Monday, October 19, 1987, stock markets around the world crashed suddenly and without warning. By the end of October, stock markets in Hong Kong had fallen 45.5 per cent, Australia 41.8 per cent, Spain 31 per cent, the United Kingdom 26.45 per cent, the United States 22.68 per cent, Canada 22.5 per cent and New Zealand about 60 per cent. Most markets recovered relatively quickly. The U.S. Dow Jones Industrial Average (DJIA) closed that year up, but it did not recover pre-crash highs for two years. There has never been conclusive agreement on what caused the crash, but many pointed to automated or “program” trading. Having expanded dramatically at Wall Street firms in the previous few years, this form of trading involved letting computers make very fast decisions about buying and selling in response to real-time data about market prices and conditions, without the intervention of human operators. Many theories about the cause of the “Black Monday” crash featured chain reactions of program trading, in which programmed sell-offs triggered additional sell-offs, which triggered additional sell-offs, and so on. This document is authorized for use only by Gavin Davis in 2017. For the exclusive use of G. Davis, 2017. 3DJH %( In 2005, the SEC issued a new regulation, Reg NMS, intended to prevent a particular kind of unfair trading by human traders, called “frontrunning.” The new regulation created a precise legal definition of “best price” and obliged brokers to abide by this new standard, which was implemented technically inside a computer system, the Securities Information Processor (or “SIP”). With best price implemented inside a computer, the mechanics of trading of all kinds moved rapidly into computers also. By about 2007, the classic imagery of traders in colour-coded jackets shouting out orders in trading pits had become obsolete. Trading had moved almost completely “inside the machine.” Traders sat at computer terminals to input orders, but no person was involved in execution of the trades. 19 On May 6, 2010, U.S. stock markets experienced the so-called “Flash Crash” (also known as the “Crash of 2:45”). Without warning, the DJIA fell 1,000 points (9 per cent) and then recovered in minutes. It was the biggest single day point decline, as well as the second largest point swing, in the history of the Average. One trillion dollars in market value vanished, albeit temporarily. Stocks of eight major companies, including Accenture, CenterPoint Energy and Exelon, fell (temporarily) to one cent, while Apple, Hewlet-Packard and Sotheby’s briefly exceeded $100,000 per share. 20 As with the Black Monday crash, most explanations involved either automated high frequency trading or technical glitches (or both). Concerns about the efficient functioning of the markets, and their possible disruption, whatever the cause, prompted Congressional hearings in the aftermath of this event. On May 18, 2012, the Facebook Initial Public Offering (IPO) was beset by a series of technical problems at NASDAQ. First, trading was delayed, then technical problems, including systems that went into a “loop” for awhile, resulted in traders believing their trades were not completed or that they were completed at prices different from trade prices. Knight participated in this trading and believed that it had lost millions due to the NASDAQ technical problems. 21 CEO Joyce, speaking with authority because he headed a firm regarded as a technology leader, was especially critical of the problems at NASDAQ. In a May 21 interview on CNBC, he said: I want to point out that this wasn’t in any way, shape or form an industry failure. This is not a systemic issue. All of the financial services firms that were out there handling client flow handled it perfectly . . . The failure was NASDAQ’s . . . this was the worst performance by an exchange on an IPO ever . . . They knew they had technical issues . . . They proceeded to continue . . . we did everything right. We are now sitting with a loss . . . This was like a server going down, except on a massive scale, and instead of stepping back and rebooting, they kept ploughing ahead. . . . NASDAQ, to the conference call on Sunday, said, “we tested our system, and tested over and over and over again, and we are ready to go.” Obviously, they weren’t ready to go. They didn’t test it enough. 22 See Exhibit 1 for a sampling of computer related disruptions of U.S. stock markets during 2012. :+$7+$33(1('$7.1,*+721$8*867 Technology breaks. It ain’t good. We don’t look forward to it. — Joyce 24 To allow its customers to participate in the RLP at the NYSE, Knight made a number of changes to its order handling systems and software. These changes involved developing and deploying new software components for SMARS. A core function of SMARS was to receive orders passed from other components of Knight’s trading platform and then to send one or more corresponding orders to external venues for execution. Among the changes made every day to Knight’s computer systems, amid the fast-moving world of automated trading, changes made for the RLP launch were relatively This document is authorized for use only by Gavin Davis in 2017. For the exclusive use of G. Davis, 2017. 3DJH %( commonplace, neither particularly major nor unusual. They were much more like routine maintenance than a major new release and were likely carried out by the IT staff that supported the affected systems from day to day; their work probably received no special management attention. In the process of rolling out the new RLP software, IT staff planned to delete some very old “dead code” — software that was no longer ever “called” (asked to run) by SMARS. Until 2003, this dead code had been used to accomplish functionality called “Power Peg.” But in 2003, Power Peg functionality was replaced; orders began to be routed through new code, and no orders ever went to Power Peg. It was as if a new road opened and an old one closed, and all traffic was diverted to the new road. In 2012, this Power Peg code remained dead — the old road remained closed — but it was still physically present within SMARS and therefore theoretically “callable” — like a closed road that could, theoretically, still be driven on. Knight IT staff members knew they needed to delete the dead Power Peg code as part of the RLP rollout because the new RLP software used a “flag” — a software switch — that had once been used to switch on the Power Peg functionality. For convenience, Knight’s developers had “re-purposed” the flag, putting it to new use in the RLP software. This meant they needed to delete the Power Peg code to avoid inadvertently activating it when they used the flag within the RLP software. In addition, in the old days when Knight had used the Power Peg code, a cumulative quantity function counted the number of shares executed when an order was passed to an external venue. This feature instructed the code to stop sending orders when the internal order had been filled. But in 2005, Knight developers repositioned this “stop ordering” functionality to a point earlier in the SMARS software execution sequence. The “stop ordering” functionality, then, was no longer located within the Power Peg code component. This mattered little in 2005 because the Power Peg code was no longer being used. It was as if an important safety feature had been removed from a closed road, which did not matter because the road was closed. Beginning on July 27, 2012, Knight rolled out the new RLP code in stages. It did not install new code on all of the eight servers that ran SMARS but rather began on a limited number of servers in order to perform pilot testing of the new software. When they established to their satisfaction that the new code was functional on a small number of servers, Knight’s staff set about deploying the new software to all the servers. But they missed one. The eighth server never had the new software installed on it. Somehow, they also forgot to delete the dead Power Peg code from the eight servers. No one at Knight realized these mistakes, and the company had no written procedures that required them to double check. On August 1, when Knight began to participate in the RLP, seven servers processed orders correctly. Most traffic followed the “new road.” However, orders sent with the repurposed flag to the eighth server awakened the long slumbering Power Peg code. This rogue server began sending orders to trading centres for execution. But because the cumulative quantity function that issued a “stop ordering” command when an order had been filled had been moved out of the Power Peg code, this server did not stop ordering. It kept ordering and ordering and ordering. Another part of Knight’s order handling system recognized that the orders had been filled, but it had no way of stopping the Power Peg code from ordering. It was as if some traffic had suddenly begun using the closed road and crashing because an old safety feature had been removed — and no one could stop it. Knight had controls in place for the overall SMARS system, but none of them had “hooks” into the old Power Peg code, which, having been dead for so long, was no longer integrated with the rest of SMARS. A risk-monitoring tool called “PMON” showed a large volume of positions accruing to Knight accounts, but the system relied on humans to know when to shut down trading. It did not generate alerts that told observers when limits were exceeded. And there were other, reasonable, This document is authorized for use only by Gavin Davis in 2017. For the exclusive use of G. Davis, 2017. 3DJH %( possible explanations for such accumulations. People watching, ... Purchase answer to see full attachment