On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff <mh...@ox.com> wrote: > Reminds me of something that happened about 25 years ago when an > elementary school visited our data center of the insurance company where I > worked. One of our operators strategically positioned himself between the > kids and the mainframe, leaned back and hit it's EPO button. > > Or when your building engineering team cuts themselves a new key for the 'main breaker' for the facility... and tests it at 2pm on a tuesday. Or when that same team cuts a second key (gotta have 2 keys!) and tests that key on the same 'main breaker' ... at 2pm on the following tuesday.
<quadruple face palm> not fakenews, a real story from a large building full of gov't employees and computers and all manner of 'critical infrastructure' for the agency occupying said building. Matthew Huff | Director of Technical Operations | OTA Management LLC > > Office: 914-460-4039 > mh...@ox.com | www.ox.com > > ........................................................................................................................................... > > -----Original Message----- > From: NANOG <nanog-bounces+mhuff=ox....@nanog.org> On Behalf Of Sean > Donelan > Sent: Friday, September 10, 2021 12:38 PM > To: nanog@nanog.org > Subject: Never push the Big Red Button (New York City subway failure) > > NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER > OUTAGE ISSUE ON AUGUST 29, 2021 > Key Findings > September 8, 2021 > > > > https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf > > Key Findings > [...] > > 3. Based on the electrical equipment log readings and the manufacturer’s > official assessment, it was determined that the most likely cause of RCC > shutdown was the “Emergency Power Off” button being manually activated. > > Secondary Findings > > 1. The “Emergency Power Off” button did not have a protective cover at the > time of the shutdown or the following WSP investigation. > > [...] > Mitigation Steps > > 1. Set up the electrical equipment Control and Communication systems > properly to stay active so that personnel can monitor RCC electrical > system operations. > > [...] >
