Securing an Infrastructure Too Complex to Understand

At approximately 3:27 PM on Thursday, September 8, a high voltage transmission line at the North Gila power substation in Arizona lost power during routine maintenance.[1] Within 11 minutes the power disturbance cascaded across five different power grids.[2] Over a dozen generating stations in the region went offline and millions of people in the southwestern United States and northwestern Mexico were left without power, in some cases for many hours.[3]

The combination of rapidly emptying offices and non-functioning traffic signals led to nearly immediate gridlock on the roads in San Diego. Inland, where temperatures in many locations were well over one hundred degrees, air conditioners stopped working. Hundreds of cars were abandoned when their drivers were unable to purchase gas at electricity-dependent gas stations. ATMs stopped dispensing cash, dozens of flights were cancelled due to inoperative airport security screening equipment, advisories to boil water were issued,[4] and backup generators at several hospitals failed.[5] Once again, we were collectively reminded of our dependence on the infrastructure and of how quickly the slide to chaos begins when it fails.

We have all seen this movie before. In the aftermath of the blackout, studies will be performed, reports will be issued, hearings will be held, blame will be assigned and then disputed or diluted, and measures will be adopted to ensure that it won’t happen again. But, of course, it will.  Somewhere, a power plant employee will make an error, a tree branch will fall on a power line, a key piece of equipment will fail, or, most disturbingly of all, a terrorist or intelligence operative from a hostile state will enter a few keystrokes on a computer keyboard, and economic activity will once again grind to a halt for millions of people.

Events like the September 8 blackout and the May 6, 2010 “flash crash” in the financial markets are always explained in hindsight by highly pedigreed experts in language brimming with logic and confidence. But no one issued a blackout warning on September 7, 2011 or a flash crash warning on May 5, 2010. In stark and irrefutable terms, these events telegraph to the world that there are destabilizing dependencies in the critical infrastructure that we do not understand.

Concerns about infrastructure vulnerability have long been recognized. For example, the U.S. Department of Homeland Security has developed a comprehensive National Infrastructure Protection Plan[6] that includes partnerships across a wide range of infrastructure sectors. Initiatives are also in place in the United Kingdom,[7] the Netherlands,[8] and many other countries.[9]

However, when systems have so many inputs, outputs, and interconnections that it becomes impractical or impossible to accurately foresee the extremes of behavior they can exhibit, the best intentions of industries and governments can run up against the mathematics of complexity. It is a matchup in which complexity has the advantage. Identifying all of the possible interactions and dependencies among the hundreds of millions of components in today’s finance, communications, and energy distribution networks may be no more feasible than predicting when and where the strongest hurricanes of three years from now will make landfall.

It is tempting to take comfort in the prospect that continuing improvements in the computers and methods used to model complex systems will in time close the gap between what we can create and what we can understand. But that is a false comfort.

The electronic chips that are the brains in computers tend to double in capacity about every two years. While that is a high rate of improvement, it pales next to the rate of complexity growth in the increasingly interconnected systems at the core of many critical infrastructure sectors. In the future, it will get harder, not easier, to model the behavior of these systems.

Can an infrastructure that is both indispensible and too complex to fully understand be secured? Not completely. But security is always an endeavor involving odds and tradeoffs as opposed to certainties. When the nature of the infrastructure itself is uncertain, the methods for identifying, quantifying, and mitigating vulnerabilities need to be updated accordingly. As illustrated by the uncomfortably high number of proverbial hundred-year floods that have occurred in the past several years in various critical infrastructure sectors, legacy approaches to understanding risk are giving us the wrong answers.

Though it is anathema to many industry participants, one component of the solution lies in appropriate – and in some cases, increased – government oversight. Many of the current regulatory approaches were drafted in an era before the introduction of high speed trading in the financial markets, before the “smart” grid, before the spread of wireless communications networks comprising billions of devices, and before terrorism became a daily feature of the headlines. The government is uniquely positioned to facilitate the gathering and integration of information regarding the large-scale dynamics of complex systems and the threats they face, and to work with industry to identify solutions that increase reliability and security.

In addition, it is necessary to strike a more conservative balance in the tradeoff between the efficiency of complex systems and their robustness to accident or attack. Turning the dial too far in favor of efficiency at the cost of robustness has created an environment ripe for unforeseen consequences. A power grid design enabling a single tree to knock out power to 50 million people (as occurred in 2003 in the northeastern U.S. and Canada)[10] may have been efficient, but it certainly wasn’t robust. The unforeseen consequences that will almost certainly accompany the massive rollout of grid-connected smart meters over the next few years are sobering to contemplate.

Finally, a dose of humility and a more realistic acknowledgment of the complexity of the infrastructure security challenges we face will make those challenges easier to address. The California Independent System Operator, which is an electricity authority in the region impacted by the September 8 blackout, boasts on its web site that it oversees “one of the largest and most modern power grids in the world.”[11] A true statement, no doubt, but one that is also more than a little ironic if a mishap at a power facility in Arizona can cause a $100 million economic loss in San Diego.[12]

Though our infrastructure may be modern, our approach to managing and securing it is clearly not modern enough. Until we address that reality, we will all continue to be guinea pigs in a never-ending series of experiments that expose previously unknown flaws in the increasingly complex infrastructure on which we all depend.

[1] “Arizona power company baffled by events that led to outage,” Los Angeles Times, http://latimesblogs.latimes.com/lanow/2011/09/blackout-san-diego-arizona.html, retrieved September 18, 2011.

[2] “Power Outage Worsened by Plant Shutdown,” NBC Los Angeles, http://www.nbclosangeles.com/news/local/Power-Outage-Exacerbated-by-Plant-Shut-Down-130017958.html, retrieved September 18, 2011.

[3] “Difference Engine: Disaster waiting to happen,” The Economist, September 16.^, 2011, http://www.economist.com/blogs/babbage/2011/09/reliability-grid, retrieved September 17, 2011.

[4] “Outage prompts sewage spill, boil water advisory,” The Sacramento Bee, September 8, 2011, http://www.sacbee.com/2011/09/09/3896891/power-outage-caused-nearly-2m.html, retrieved September 18, 2011.

[5] D. Baker and K. Kucher, “Effects of Power Outage Linger.” Sign On San Diego (the web site of the San Diego Union Tribune), September 9, 2011, http://www.signonsandiego.com/news/2011/sep/09/effect-of-power-outage-linger/, retrieved September 18, 2011.

[6] See, for example, the 2009 National Infrastructure Protection Plan,  http://www.dhs.gov/files/programs/editorial_0827.shtm, retrieved September 18, 2011.

[7] Centre for the Protection of National Infrastructure, http://www.cpni.gov.uk/, retrieved September 18, 2011.

[8] E. A. M. Luiijf, H. H. Burger, and M.H .A. Klaver, “Critical (information) Infrastructure Protection in The Netherlands.” GI Jahrestagung (Schwerpunkt “Sicherheit – Schutz und Zuverlässigkeit”), pp. 9-19, 2003, http://subs.emis.de/LNI/Proceedings/Proceedings36/GI-Proceedings.36-1.pdf, retrieved September 18, 2011.

[9] “Protection of ‘Critical Infrastructure’ and the Role of Investment Policies Relating to National Security.” Organization for Economic Cooperation and Development (OECD), May 2008. http://www.oecd.org/dataoecd/2/41/40700392.pdf, retrieved September 18, 2011.

[10] B. Walsh, “Can We Prevent Another Blackout?,” Time, August 11, 2008, http://www.time.com/time/health/article/0,8599,1831346,00.html, retrieved August 19, 2011.

[11] http://www.caiso.com/about/Pages/default.aspx, retrieved September 18, 2011.

[12] R. Marosi and S. Allen, “Blackout losses could top $100 million,” Los Angeles Times, September 9, 2011, http://latimesblogs.latimes.com/lanow/2011/09/blackout-losses-could-top-100-million.html, retrieved September 17, 2011.