N.Y. PSC Issues Order Involving Electric Emergency Response Plan Review

ALBANY, New York, July 16 (TNSsro) -- The New York State Public Service Commission issued the following order (Case No. 20-E-0618):

In the Matter of December 15, 2020 Electric Emergency Response Plan Review (2021 Plan).

ORDER APPROVING, SUBJECT TO MODIFICATIONS, THE AMENDED ELECTRIC EMERGENCY RESPONSE PLANS

BY THE COMMISSION:

INTRODUCTION

Each Investor-owned electric utility under the Commission's jurisdiction is required to develop, update, and file with the Commission a detailed Emergency Response Plan (ERP), detailing the utility's response plans for emergencies that comply with Section 66 of the Public Service Law (PSL) and Title 16 New York Code, Rules, and Regulations (NYCRR) Part 105./1

The electric utility's ERPs contain essential processes and procedures necessary to engage and respond to a wide array of emergencies. External and internal communication practices, personnel roles and responsibilities, and procedures implemented before, during, and following emergencies, represent just a fraction of the numerous measures that are required to be in each electric utility's ERP.

This year's review of utility ERPs reflects the actions taken following Tropical Storm Isaias, including outcomes from the notice of apparent violations and immediate corrective action letters sent to Con Edison, Orange & Rockland, and Central Hudson, as well as the immediate corrective action letters sent to NYSEG, RG&E, National Grid, and PSEG LI on August 19, 2020 (collectively, Corrective Action Letters)./2

The ERPs also incorporate findings from the Interim Investigation Report on Tropical Storm Isaias (Report) issued on November 19, 2020./3

Many of the improvements made to the emergency response plans discussed in this Order relate to preventing inadequate storm responses, such as that of Con Edison, Orange & Rockland, Central Hudson, and PSEG LI during Tropical Storm Isaias. Several modifications were also made to all ERPs, including PSEG LI, to take advantage of best practices as well as to provide consistency across the ERPs./4

The Commission, pursuant to PSL Sec.66(21), has a statutory obligation to review each investor-owned electric utility's emergency plans annually and 16 NYCRR Sec.105.5 also requires annual review and approval of emergency plans which are designed to ensure improved preparedness for future outage events. PSL Sec.66(21) requires all electric ERPs to be updated annually and submitted to the Commission for approval on, or before, December 15 of each year. Accordingly, the ERPs, having been reviewed by the Commission are approved with the modifications discussed herein.

BACKGROUND

On August 4, 2020, Tropical Storm Isaias made landfall on the east coast, continued tracking north and passing through eastern New York State. The storm caused severe and extensive damage throughout the Mid-Hudson, New York City, and Long Island Regions./5 PSEG LI experienced the highest winds, which included recorded gusts between 70-78 miles per hour (mph). Gusts up to 70 mph were reported in the service territory of Con Edison, while Central Hudson, NYSEG, and O&R service territories experienced 45-60 mph gusts. The storm caused widespread damage to overhead electric infrastructure, leaving significant numbers of New Yorkers without power. Peak outages in New York State due to Tropical Storm Isaias reached approximately one million customers, and roughly 1.5 million New York customers experienced power outages during this event. On August 9, 2020, 90 percent of the utility customers who had lost power had been restored, with full restoration occurring on August 12.

The 16 NYCRR Part 105 storm reports filed by each utility, as required by the Department's regulations, identified approximate peak outages of 380,000/6 for PSEG LI, 290,000 for Con Edison, 138,000 for Orange & Rockland, 110,000 for Central Hudson, and 95,000 for NYSEG. The reports also noted total customer impacts of approximately 645,000 for PSEG LI, 330,000 for Con Edison, 189,000 for O&R, 116,000 for Central Hudson, and 183,000 for NYSEG./7

Approximately 90 percent of customers in the PSEG LI service territory were restored on August 10. PSEG LI restored all customers on August 12. Ninety percent of Con Edison customers were restored by August 9 and all customers were fully restored by August 12. Orange & Rockland restored 90 percent of its customers the morning of August 8, with full restoration on August 11. Central Hudson had 90 percent of customers restored by August 7 and fully restored all customers impacted by the storm by the evening of August 8. The majority of outages experienced by NYSEG were in its Brewster Division, which serves customers in Dutchess, Putnam, and Westchester Counties. By 5:00 pm on August 8, more than 90 percent of customers in NYSEG's Brewster Division were restored with full restoration just after 10:00 am on August 10.

As a result of the extended restoration, each utility was directed in the Corrective Action Letters to undertake immediate action to add crewing capacity via retainer contracts as well as develop plans to secure utility crews in addition to private contractor and mutual assistance beyond the North Atlantic Mutual Assistance Group (NAMAG) process. In addition, the utilities were directed to conduct stress tests on their respective information technology systems, refine coordination plans with municipalities tailored to each county, e.g., road clearing and local liaisons, and to update Life Support Equipment and critical infrastructure lists to remove or add customers as necessary. Each of these requirements are addressed in the ERPs before the Commission.

NOTICE OF PROPOSED RULE MAKING

Pursuant to the State Administrative Procedure Act (SAPA) Sec.202(1), a Notice of Proposed Rulemaking was published in the State Register on February 3, 2021 [SAPA No. 20-E-0618SP1]. The time for submission of comments pursuant to the Notice expired on April 5, 2021. No comments were received.

DISCUSSION

A thorough review was performed for each of the utility's electric emergency response plans filed in December 2020 and those refiled in June 2021. Department of Public Service staff (Staff) had numerous meetings, conference calls, and e-mail exchanges with each electric utility to attempt to resolve areas where additional information was necessary. As a result of this process, each utility filed an amended emergency response plan in June 2021 that addressed either in whole or in part, modifications discussed during the interactions with Staff. The following sections summarize key areas that were reviewed and modified to enable appropriate responses.

Resource Acquisition and Mutual Assistance

The ability to effectively restore power in a timely manner after any storm event depends on how quickly electric infrastructure such as poles and wires can be repaired. The time it takes to repair electric infrastructure is dependent on the number of internal and external line resources a utility has available. When an electric utility has service outages resulting from a minor storm event, it normally relies on its own resources, such as internal line workers and on-property contractors to restore service. During larger storm events, the utility typically will secure third party contractors that supply line workers and restoration equipment or other skilled workers such as tree crews and damage assessors. The standard method to track and report resources is Full-Time Equivalents, or FTEs.

When a utility's total internal resources and on-property contractor resources are not expected to or do not meet the estimated resources needed to restore service, the utility will then contact external contractor resources and request resources through the North Atlantic Mutual Assistance Group (NAMAG). Utilities also have agreements with multiple contractor resources and have contact lists to expedite securing third-party contractor line, service, and tree resources. In compliance with the Corrective Action Letters directing that the utilities immediately begin the process of securing additional resources in addition to private contractor and mutual assistance provided by the NAMAG before and after storms, the utilities immediately began the process of soliciting externally to augment their respective crewing levels, beyond the NAMAG process, through extensive canvassing to identify available resources and to enter into contractor retainer contracts and right of first refusal agreements. Currently, the utilities either have contracts or agreements in place, are in negotiations with third parties, have contracts or agreements pending resolution of cost recovery, e.g., rate cases, or are reviewing contractor proposals to comply with the Corrective Action Letter's direction to secure resources in place to expand their access to resources when needed. Through these endeavors, the utilities are positioned to leverage additional resources in future outage events and have done so on numerous occasions, since Tropical Storm Isaias Corrective Action Letters were issued. To that end, we direct the utilities to continue this process through execution of these contracts and/or agreements. It is exceedingly important for the utilities to exhaust every option available to them in order to obtain the supplementary resources needed to restore customers as rapidly and safely as possible.

Estimated Times of Restoration

The Estimated Time of Restoration (ETR) is the approximate date and time when the utility expects service will be restored after a power outage. Customers depend on ETRs to make health and safety decisions, including the need for alternative accommodations. Therefore, ETRs must be timely, accurate, and widely accessible. In August 2013, the Commission adopted ETR Protocols to help ensure that ETRs meet these three essential requirements./8

Last year, ETR Protocols were revised and strengthened based on recommendations from the 2018 Winter and Spring Storm Report./9

To balance the customer's need for information with available damage information and available crews, regional and local ETRs apply to 95 percent of customers instead of 90 percent. During Tropical Storm Isaias, customers and government officials were misinformed as a result of inaccurate ETRs, untimely issuance of revised ETRs, or the lack of clarity on who was covered by the ETR. As a result, customers and public officials were frustrated by the inaccurate, inconsistent, and insufficient messaging.

The ETR Protocols clearly outline extended time requirements for issuing local ETRs to enable utilities to provide more detailed and targeted information. All of the plans now identify individuals responsible for developing, issuing, and revising ETRs. It is imperative that utilities utilize all means of communication to inform the customers and government officials of ETR changes, including proactively reaching out to customers affected by changes and modifying their websites.

Coordinating with Counties and Municipalities

Following major storms, counties, municipalities, and utilities have a make-safe period that focuses on the elimination of hazards to the public. Public hazards include exposure to potential shock risks, which are hazards that can trap residents in their homes and can prevent emergency resources from responding to requests for assistance or fires. Strong and collaborative communications are critical when coordinating these efforts to develop daily work plans.

Additionally, regional municipal calls provide a venue for more detailed discussion of the municipalities' priorities and the presentation of updated information to the municipalities that is not available publicly. There have also been improvements in how utilities and municipalities interact, such as using virtual communications and through the use of municipal dashboards where municipalities can go online to report hazardous conditions, outages, blocked roads, and to track outages in their areas.

This information can also be obtained through utility liaisons, who may be co-located in municipal command centers, county EOCs or available around the clock to the municipalities by telephone. The utilities, however, need to understand that liaisons are only as good as the information that is provided to them. The ERPs contain language that acknowledges municipalities expect their liaisons to do more than reiterate publicly available information, e.g., locations and number of crews, which circuits are being worked, and the ETR status of their residents and/or constituents. Lastly, the ERPs include additional language for the utilities to request input from the counties in their service territories on dry ice distribution locations, however if counties are unresponsive, the utilities will still take action to ensure that customers get dry ice and will endeavor to accommodate reasonable requests to strategically locate distribution sites based on customer impacts.

Testing Critical Information Technology Systems

Customer reported outages are key to identifying the full impact of a storm event on an electric system after a major outage event. Therefore, outage reports from multiple sources, including the call center, Interactive Voice Response (IVR), utility website, and text messaging, are electronically posted to outage management systems (OMS) which are core applications used by utilities to manage outage events. OMS systems process reported customer outage information received through various means, predict outages to capture the full extent of customers impacted, create, prioritize, and manage jobs, and interface with various applications to provide consistent and updated outage information to utility personnel and the public during normal and emergency operations.

Two months prior to Tropical Storm Isaias, PSEG LI had gone into production with an updated version of its OMS without adequately performing realistic stress testing on the new version. Shortly after Tropical Storm Isaias hit its service territory, various failures in the OMS were identified, causing significant challenges not only for the utility but, more importantly, for the public. PSEG LI's OMS suffered a catastrophic failure that led to the Company's inability to provide accurate outage and job numbers, locations of reported outages, ETR information, as well as efficient and effective dispatching of resources.

As a result of the issues experienced by PSEG LI, each electric utility was required to stress test its respective OMS systems more frequently using more stringent parameters including, as part of the corrective actions following Tropical Storm Isaias, an outage impacting 90% or more of customers in the Company's service territory over a 24-hour period. These conditions will ensure the testing is in line with a realistic outage event and that the OMS will perform adequately in a major outage event.

To ensure this practice continues, the ERPs now include provisions that require OMS testing under the parameters listed above. The tests will be performed semi-annually with the detailed results provided to the Director of the Office of Resilience and Emergency Preparedness following the completion of each test. In addition, reporting requirements have been established that require reporting of detailed test results, as well as require disclosure of interim measures the utilities will use should the OMS stress test fail, until such time as permanent resolutions have been implemented. These measures will ensure stress testing processes and parameters are consistent, better defined and provide for worse case scenarios, including increased real-time monitoring of the systems during major outage events, and verifying the interaction with automated inputs when the system is experiencing high usage. They also provide a communication strategy to address multiple scenarios concerning failure of critical information technology (IT) systems.

Additionally, if a utility OMS system fails a test, the utility is required to retest, after taking corrective actions, to ensure that the failure is corrected. A failed test does not count towards the semi-annual requirement. Each electric utility ERP also identifies how Advanced Metering Infrastructure (AMI) data will be used to improve the accuracy of its OMS. It is expected that additional automated processes will continue to be developed and integrated to improve storm response as AMI continues to be deployed by the utilities.

Information Technology

The utilities' websites, and any event-specific information pages, are two of the primary means used by utilities to inform customers during emergencies. When reviewing the many communication channels used to provide information to customers, e.g., press releases, social media posts, Interactive Voice Response (IVR) messages, etc., the utilities routinely direct customers to go to their website to gather more information. Therefore, the ERPs contain language that the utility website is required to be available around the clock. During storms, the information on the utilities' outage maps must be updated continuously at the typical cycle (e.g., every 15 minutes) and must be updated at least hourly, so to provide customers with information that is timely and accurate. If critical repairs and/or maintenance must be done, such activities will occur after 10:00 p.m. or before 6:00 a.m. to avoid any downtime and/or disruption of availability during the remaining time periods. In addition, all non-critical maintenance is deferred until all customer outages have been restored, which is currently an industry best practice.

We cannot stress enough the need for critical systems to be operating properly at all times. Therefore, the Commission finds that it is imperative that as soon as a utility website or underlying data sources go down, it reasonable to require that the utilities notify on-call IT personnel expeditiously and without delay. IT personnel will work to resolve the issue and will provide updates on corrective action regularly and/or notify third party vendors in addition to facilitating any interactions between the vendors the utility's IT group.

When customer facing applications are experiencing a technical issue, the ERPs indicate the title of the employee responsible for directing that such notice be posted, e.g., message banner, on the main website homepage and the outage map, as appropriate, providing customers the details and necessary steps for reporting outages and obtaining other event-related information. This notice will also be used to communicate if system maintenance is underway and, if so, provide status updates on such as well as if the outage map is unavailable due to other issues, such as data quality concerns. The notices must provide outage counts and ETRs. In addition, the electric utilities also use social media to inform customers when these issues exist.

Furthermore, the ERPs designate the title of the employee responsible for stricter monitoring and maintenance of critical systems to ensure continuous access to the important information contained in those systems. These enhanced measures provide the utilities higher visibility and responsibility for making certain the critical information technology systems are properly maintained, but more importantly, diligently monitored so to identify issues quickly and be in a better position to ameliorate those issues effectively.

* * *

Footnotes:

1/ The investor-owned electric utilities are Consolidated Edison Company of New York, Inc. (Con Edison), Central Hudson Gas & Electric Corporation (Central Hudson), New York State Electric & Gas Corporation (NYSEG), Niagara Mohawk Power Corporation, d/b/a National Grid (National Grid), Rochester Gas and Electric Corporation (RG&E), and Orange & Rockland Utilities, Inc. (Orange & Rockland or O&R).

2/ Case 20-E-0586, supra, Notice of Apparent Violation Related to Tropical Storm Isaias and Direction of Prompt Remedial Action (filed August 19, 2020)(NOAV) and Case 20-E-0586, supra, CEO Rhodes Letter Re: Tropical Storm Isaias After-Action Responses (filed August 19, 2020) (Corrective Action Letters).

3/ Tropical Storm Isaias struck on August 4, 2020, significantly impacting Long Island, New York City, and several counties in the mid and lower Hudson Valley. The storm's impacts resulted in peak outages of approximately one million. On August 5, 2020, Governor Andrew M. Cuomo directed the Department of Public Service (Department or DPS) to investigate New York State's major electric utilities' (utilities) following the slow and inadequate response of certain electric utilities to Tropical Storm Isaias.

4/ The PSEG LI ERP review followed a parallel process to the IOUs under Matter 20-02608, In the Matter of the December 15, 2020 Emergency Response Plan of the Long Island Power Authority and PSEG Long Island (2021 Plan) but is not subject to this Order.

5/ The damage in National Grid's Capital Region service area was not as severe. National Grid had a peak of approximately 35,000 customer outages that were restored in less than 36 hours.

6/ The accuracy of this figure cannot be verified due to the issues PSEG LI had with its OMS.

7/ See, Matter 20-01633, In the Matter of DPS Investigation into the Utilities Preparation and Response to August 2020 Tropical Storm Isaias and Resulting Electric Power Outages, Part 105 Reports.

8/ Case 13-E-0198, In the Matter of 2013 Electric Emergency Response Plan Review, Order Approving Electric Emergency Plans (issued August 16, 2013).

9/ Case 19-M-0285, In the Matter of Utility Preparation and Response to Power Outages During the March 2018 Winter and Spring Storms, 2018 Winter and Spring Storms Investigation Report (filed April 18, 2019)(Storm Report).

* * *

View full order at http://documents.dps.ny.gov/public/Common/ViewDoc.aspx?DocRefId={C911E976-7617-4D3F-8259-012E4163E596}