ATLANTA — A “software defect” led to the computer system outage that disrupted Norfolk Southern operations on Monday, the railroad said in a follow-up report.
In a statement today, the railroad said during routine maintenance by its data storage vendor, “a leading global technology provider,” this software defect created an error that affected both the primary and backup data storage systems, which in turn affected core operating systems.
“We are working closely with the vendor to understand the steps they are taking to strengthen their systems to prevent this incident from reoccurring and identify opportunities for Norfolk Southern to become more resilient,” the statement reads.
The Monday outage affected NS positive train control and yard operating systems, as well as the railroad’s customer portal and other systems. Trains that were unable to connect to the PTC system were not allowed to proceed, leading to delays for Amtrak and Virginia Rail Express commuter trains, as well as NS movements [see “Norfolk Southern system outage …,” Trains News Wire, Aug. 28, 2023]. The outage began Monday morning, with all systems restored by 7 p.m. EDT, but the railroad said at the time that impacts could last for a couple of weeks.
That estimate appears to be holding, based on today’s update.
“We are making progress on the temporary backlog of trains that resulted from this incident,” NS said in its statement, “and expect a full recovery of network fluidity in the coming weeks.”
The railroad reaffirmed today that it has found “no indication” that the incident was related to a cybersecurity breach.
Hmmm. Wonder who that could be.
Curious.
Outsourcing data storage to save money means you no longer control your own destiny.
What a way to run a railroad. Wonder how we managed in the old days.
A “Prime” example of consideration of an independent, secure, off-site back up system. ‘Look Ahead, Look South”!
The FRA would do better to let the trains run on signal indication instead of treating the entire system as dark territory.
Maybe the FRA should be duplicating the banking regulators and run operational stress tests against the railroads to see how they fare in an IT faux pas in the future.