What Hedge Fund Tech Can Learn from the Delta Airlines IT Outage

By Mary Beth Hamilton | Thursday, August 11th, 2016

Earlier this week Delta Airlines suffered a major system outage that resulted in more than 740 flight cancellations and thousands of flight delays.

Delta’s Chief Operating Officer Gil West explained that “Monday morning a critical power control module at [Delta’s] Technology Command Center malfunctioned, causing a surge to the transformer and a loss of power. The universal power was stabilized and power was restored quickly. But when this happened, critical systems and network equipment didn’t switch over to backups. Other systems did. [As a result, Delta saw] instability in these systems.”

As with any major “uh oh” moment, there are lessons that can be learned. So let’s take a look at what hedge funds can learn from Delta’s IT mishap.

1. Outdated technology can hurt in a big way. Airlines are saddled with legacy IT systems, complicated by mergers and acquisitions requiring complex integrations. Unlike airlines however, most asset management firms are not relying on technology from 80s or 90s. But that doesn’t give firms a pass when it comes to staying current with technology.

Outdated IT systems insert instability into a firm’s operations and provide holes for cyber hackers to exploit. The reality is that outdated systems will only continue to fall behind in the race of technology, trouble shooting will take longer, future applications will fail to run, or crash the server altogether, and the cost to migrate increases concurrently as the pool of experts shrinks.

2. You can’t ignore the IT industry’s transition to cloud computing. As noted in a ZDNet article, “the big question is why in 2016 airlines are being brought down by single points of failure when cloud services offer resiliency zones, backup options, and redundancy to keep critical systems running.”

Enterprise-grade clouds deliver significant resiliency in both the hardware and data centers, with cloud infrastructures spanning geographically diverse facilities. Beyond hardware, top tier cloud providers (Eze!) have teams of senior engineers managing and monitoring the infrastructure. Additionally systems are upgraded on a regular frequency.

In the investment management industry, it is common to hear investors state they are more comfortable with fund managers utilizing a private cloud rather than keeping IT on premise. At larger funds, the prevalence of cloud-based solutions provides Chief Technology Officers (CTOs) the opportunity to execute more strategic technology initiatives and focus on risk mitigation.

3. Disaster recovery tests must be real and comprehensive. Delta’s backup and disaster recovery plans did not work properly. We can’t speak to Delta’s DR test strategy, but we can use this as a reminder that firms must prepare and test for the worst case scenarios.

DR testing helps to ensure a firm’s DR site meets current business requirements. It is common for firms to grow, evolve, and change their production environments as they develop and expand their businesses. In turn, the DR site needs to evolve and change simultaneously in order to meet business requirements. Regularly DR testing can ensure employees understand procedures and that production and secondary systems are aligned.

4. Downtime (or a security breach) can tarnish your reputation. The long-term impact of Delta’s IT outage is unknown. At a minimum we can assume they have a number of unhappy customers, who they are seeking to placate with vouchers.

Downtime for most hedge funds and asset management firms is not an option. Firms must take the time to assess their IT operations and ensure the appropriate safeguards are in place.