2011 Southwest blackout

States with areas largely affected by the Southwest Blackout of 2011

The 2011 Southwest blackout, also known as the Great Blackout of 2011, was a widespread power outage that affected the San Diego–Tijuana area, southern Orange County, Imperial Valley, Mexicali Valley, Coachella Valley, and parts of Arizona. It occurred on Thursday, September 8, 2011, beginning at about 3:38pm PDT, and was the largest power failure in California history.

Background

The Southwest Area Corridor, as of 2007

At the time of the 2011 blackout, three separate, parallel transmission pathways connected Southern California to generating stations in Arizona. Excess power was then sold on to Baja California, which had disabled 600 MW worth of generation equipment for maintenance.

The Southwest Power Link, a single circuit 500 kV transmission line, connected San Diego Gas & Electric (SDG&E, a subsidiary of Sempra Energy) to the Arizona Public Service (APS) system via the North Gila substation near Yuma, AZ No other 500 kV pathways connect SDG&E and adjacent utilities, but Path 44, a set of five separate 230 kV transmission lines, connected SDG&E to Southern California Edison (SCE) through the switchyard of San Onofre Nuclear Generating Station. SCE then connects to the APS system via another 500 kV pathway called Path 46.

In an overload event, the San Onofre switchyard was designed to disconnect the offending lines, but such disconnection had never occurred before. Neither the Western Electricity Coordinating Council (WECC), in charge of the system's reliability; nor the California Independent System Operator (CAISO), which coordinates the state's power generation, had either studied the effects of a disconnection or established alarms to indicate that the breakers were at risk of operating. Consequently, CAISO incorrectly believed that it would have 30 mins to react in the event of a Path 44 overload.

Imperial Irrigation District (IID), which manages a 92 kV north-south sub-transmission system in the Imperial Valley, typically imported power through a 230 kV circuit connected to SDG&E and another to SCE near Palm Springs. Western Area — Lower Colorado (WALC) typically exported power to each through analogous 161 kV lines to the Hoover Dam. In unusual grid conditions, SDG&E could (and would) circuitously import power through these lower-voltage ties, but the IID and WALC systems were generally considered too small-scale to incorporate in grid stability analyses.

The day of the outage was a hot late-summer/early-fall day, with California importing close to the maximum power that the transmission system could sustain. Across the border, Comisión Federal de Electricidad (CFE)'s Baja California system (connected to SDG&E, but isolated from the main Mexican electrical grid) was already operating at maximum generation capacity.

Grid managers maintain stability through the "n-1" criterion: the system should remain within operating limits even if a single component fails. Unless many components are run close to their operating limits, this practice typically ensures grid operators have enough time to react to failure. Prior to the outage, IID's system state violated the n-1 criterion: a transformer failure in the Coachella Valley would induce further transformer failures. Because IID did not require active human scrutiny of the n-1 condition at the time, IID did not attempt to redistribute power and reduce the risk.

Events

The outage was the result of 23 distinct events that occurred on 5 separate power grids in a span of 11 minutes.

Around 2pm, a capacitor bank at APS' North Gila substation failed. The technician sent to disconnect the capacitor bank accidentally skipped a critical step which involved bypassing the transmission circuit around the capacitors. Attempting to disconnect the capacitor bank under load resulted in an arc forming across the switches as they were being opened. At 3:27 pm, the arcs from two phases crossed, resulting in a phase-to phase short which caused an upstream oil circuit breaker to trip, protecting the substation from damage. However, a large transfer of charge that occurred when the line opened caused one of the transmission lines to shift out of phase, inhibiting reconnection.

The line's out-of-phase state was not visible to controllers until reenergization the following day. When WECC immediately called APS about the line's failure, APS assured them that the line could return to service within 5–10 min. Because the subsequent collapse took place over the next 11 minutes, the WECC took no action during the event to develop alternate transmission paths, generation schemes, or load distributions.

With the Southwest Power Link disconnected, the electricity flow from Arizona reconfigured itself in accordance with Kirchhoff's circuit laws. The transient currents from this reconfiguration may have caused one of CFE's generators to trip offline; CFE began importing power from southern California to make up the difference.

Of the power that the Southwest Power Link previously carried, 3⁄4 now traveled through SCE's transmission system through Riverside and Orange County, then south via San Onofre. The remaining 1⁄4 flowed through the lower-voltage IID and WALC sub-transmission systems. This current overloaded two transformers at IID's Coachella Valley substation, which prepared to trip offline to prevent damage. The IID control center's monitoring equipment had pegged off-scale, and so the controller did not react during the preprogrammed 40 s delay. Four minutes later a 3:32 pm, another transformer at IID's Ramon substation, which was the only other interconnection with SCE to the north, also tripped offline, and IID began automatically shedding load.

At 3:35 pm, WALC's system also began to struggle, first disconnecting from the Southwest Power Link at Gila, and then manually shedding load to the north at Parker. After these events, WALC's system stabilized, and would continue to supply power to Blythe in the Palo Verde Valley following the blackout.

Meanwhile, CAISO recognized that Southern California's grid was at risk of collapse. Because of its incorrect estimate of 30 min before failure, it did not inform SDG&E of the need to shed load. Instead, it ordered the Kearny Mesa and Wildflower Larkspur Energy Facility (near Otay Mesa) gas turbine plants into operation, which would take (respectively) 10 and 20 min to start.

Following the WALC disconnection at Gila, IID's system disconnected the only remaining north-south tie in an excess of caution at 3:37 pm. Thereafter, all imported power for San Diego; the Imperial Valley; Yuma, AZ; and Baja California could only travel via the Path 44 ties through San Onofre. At this point, a blackout was inevitable: the Path 44 lines could not sustainably carry so much current, and there was no time to tell the southern utilities to reduce load.

Seconds after IID disconnected from the north, it disconnected from SDG&E as well, but not before automatically tripping two generating plants in La Rosita, Mexico offline. The IID island would continue to shed load and black out.

Without the La Rosita generators, current along Path 44 would exceed the San Onofre switchyard's automatic relay settings. Within seconds, San Onofre disconnected San Diego, Baja California, and Yuma from the rest of the Western Interconnection, which had provided them with most of their generating capacity.

Instantly, SDG&E, CFE, and APS began automatically shedding load. However, CFE's generator protection schemes also reduced generation capacity comparable to the lost load; CFE continued to import power from SDG&E until the interconnections automatically opened 2 s later. At that same time, Yuma also disconnected from SDG&E, creating a pocket that the Yucca generating plant continued to power for another 15 seconds. By the time SDG&E had isolated itself, frequency had already declined to the point that San Diego could not recover without damage to the generators.

Analysis

Following the blackout, federal, regional and local officials investigated the outage. According to investigators at the US Federal Energy Regulatory Commission and the North American Electric Reliability Corporation, problems included:

• Transmission operators did not always update emergency plans to account for daily maintenance work. When they did create emergency plans, they failed to coordinate with neighboring utilities, analyze the effects of low-voltage distribution systems (like much of IID's network), or use fully up-to-date information about scheduled power transfers.
• WECC performed seasonal contingency analysis only within specified regions and subregions, without analyzing possible interregional effects, or unseasonal weather.
• Local transmission operators' contingency analyses generally did not analyze the full range of possible power dispatch scenarios.
• Many transmission operators did not benchmark their own or WECC's models against the system's real-life behavior, and consequently had unrealistic expectations of their system in a crisis.
• Transmission operators had limited-to-no ability to observe conditions internal to a neighbor's transmission network, even though those internal conditions can and do cause large external flows.
• Multiple transmission operators (not just IID) did not have audible or visual alarms in the event that system was not in a (n-1)-stable state. IID did not even attempt to maintain its system in a (n-1)-stable state. When it lost the ability to monitor its own state, WALC did not ask for assistance. Lastly, WECC did not consider interconnection failures when operationalizing the n-1 criterion.
• SDG&E, CFE, and IID all relied on the "S line" between La Rosita and El Centro, but did not coordinate on the line's protection settings. If the La Rosita generators had remained online, the blackout would likely have remained isolated to the IID system.
• SCE did not analyze the effects of the settings at San Onofre, which appeared to federal investigators to be unnecessary anyways.
• WECC appeared to misclassify a number of automatic emergency protocols, thereby shirking analyses that national standards required it to perform.
• In several cases, transmission operators set protection relays below emergency ratings, or insufficiently high above them to give operators time to react.
• APS could not monitor the phases of nonenergized power lines.
• During the blackout recovery process, the WECC failed to adequately coordinate between neighboring utilities.

Effect

Five utilities were affected: SDG&E, serving San Diego County and parts of southern Orange County and Riverside County; Imperial Irrigation District, serving the Imperial Valley; the portion of Comisión Federal de Electricidad (CFE), Mexico's electric utility, serving Baja California; Arizona Public Service (APS); and the Western Area Power Administration's Lower Colorado system (WALC). The blackout left nearly seven million people without power, including 1.4 million customers in San Diego County and 1.1 million customers in Mexico.

The hardest-hit region of the blackout, the San Diego-Tijuana metropolitan area, was essentially brought to a standstill. Surface streets became gridlocked due to the loss of traffic signals, and the San Diego skyline went dark. The San Diego Trolley system was shut down as there was no power to operate trains and related functions. Citizens in Tijuana and in inland areas like the Coachella Valley stayed outdoors late into the evening to escape the heat. Freeways in the region experienced extreme traffic congestion, especially on the I-5 and I-15 corridors between southeastern Greater Los Angeles and the San Diego area's North County. One hospital was left without power for two hours when its backup generator failed.

The outage caused significant losses to restaurants and grocery stores, which were forced to discard large quantities of spoiled food; perishable food losses at grocery stores, eating establishments and households were estimated at $12–18 million. The outage also caused some sewage pumping stations to fail, resulting in contaminated beaches and potentially unsafe water supplies in several areas. As a precaution, residents in some neighborhoods were advised to boil their water or use bottled water for several days after the outage. Due to the failure at the sewage pumping stations, diesel generators were installed at five pumping stations.

Both units 2 and 3 at San Onofre Nuclear Generating Station also automatically tripped offline following the formation of the San Diego-Baja-Yuma island, as the remaining Western Interconnection had less need for power generation.

Recovery

Eleven hours after the outage began, power was restored to 694,000 of the affected customers, and by 4:30 am on September 9, power was restored to all customers, although the system was still described as "fragile". As a precaution, all public schools in San Diego County and the Capistrano Unified School District in southern Orange County were closed on September 9. Most major universities and community colleges, as well as all federal courts in San Diego, closed for the day as well.

FERC citations

FERC cited six entities for alleged standards violations: the Arizona Public Service, California Independent System Operator, Imperial Irrigation District, Southern California Edison, Western Area Power Administration, and Western Electricity Coordinating Council.

Terrorism fears

The outage occurred days before the tenth anniversary of the September 11 attacks, and, hours before, the United States Department of Homeland Security warned of a potential terrorist attack leading up to the anniversary. Consequently, a first reaction to the blackout was to wonder if the blackout might be the result of an attack. However, the Federal Bureau of Investigation and SDG&E ruled out terrorism early in their investigation, nor did subsequent evidence suggest an intentional cause for the outage.

See also

• 1996 Western North America blackouts
• California electricity crisis
• Northeast blackout of 2003
• Path 46, also called West of Colorado River, Arizona-California West-of-the-River Path (WOR)