In the afternoon of Monday, September 28, several states, including Arizona, California, Colorado, Delaware, Florida, Illinois, Indiana, Minnesota, Nevada, North Carolina, North Dakota, Ohio, Pennsylvania and Washington have reported 911 outages in various cities and towns. .
Several press reports suggest the outages may have been linked to an ongoing service disruption at Microsoft. But a spokesperson for the software giant told KrebsOnSecurity, “We saw no indication that the 911 multi-state outage was the result of yesterday’s Azure service disruption.
Investigations of emergency dispatch centers in several of the cities affected by the 911 outage highlighted a different source: Intrado, based in Omaha, New York – until last year known as name of West Safety Communications – a provider of 911 and emergency communications infrastructure, systems and services to telecommunications companies and public safety agencies across the country.
Intrado did not respond to multiple requests for comment. But according to officials in Henderson County, North Carolina, which experienced its own 911 outages yesterday, Intrado said the outage was the result of an issue with an unspecified service provider.
“On September 28, 2020, at 4:30 p.m. EST, our 911 service provider observed internal conditions in their network that affected the delivery of 911 calls,” a statement read. ‘Intrado provided to county officials. “The impact was mitigated and service was restored and confirmed to be operational at 5:47 pm Eastern Time. Our service provider is currently working to determine the root cause. ”
The service provider mentioned in Intrado’s statement appears to be Lumen, a communications company and 911 provider that until very recently was known as CenturyLink Inc. A glance at the company’s status page shows that several Lumen systems suffered full or partial service disruptions on Monday, including its private and internal cloud networks and network of control systems.
In a statement provided to KrebsOnSecurity, Lumen blamed the problem on Intrado.
“At approximately 4:30 p.m. Montreal time, some Lumen customers were affected by a supplier partner event that impacted 911 services in AZ, CO, NC, ND, MN, SD and UT,” states the press release. “Service was restored in less than an hour and all 911 traffic is routed correctly at this time. The partner supplier is investigating the event. ”
It may not be a coincidence that these two companies are now operating under new names, as it wouldn’t be the first time that an issue between them has disrupted access to 911 for a large number of Americans.
In 2019, Intrado / West and CenturyLink agreed to pay $ 575,000 to settle an investigation of the Federal Communications Commission (FCC) in an August 2018 outage that lasted 65 minutes. The FCC found that the incident was the result of a West Safety technician messing up a configuration change to the company’s 911 routing network.
On April 6, 2014, some 11 million people across the United States were disconnected from 911 services for eight hours due to a “fully preventable” software error related to Intrado’s systems. The incident affected 81 call dispatch centers, rendering emergency services inoperative throughout Washington state and parts of North Carolina, South Carolina, Pennsylvania, California, from Minnesota and Florida.
According to a 2014 Washington Post article on a subsequent investigation and report released by the FCC, this issue involved an issue with how Intrado’s automated system assigns a unique identification code to each incoming call before transmitting it. at the “appropriate public safety response point,” or PSAP.
“On April 9, the software responsible for assigning codes reached a predefined limit,” The Post explained. “The meter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, resulting in a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure. ”
The FCC found that the duration of the 2014 outage was made worse by the fact that the Intrado server responsible for categorizing and tracking service outages classified them as “low level” incidents that had never been reported for. manual examination by human beings.
The FCC ultimately fined Intrado and CenturyLink $ 17.4 million for the 2014 multi-state outage. An FCC spokesperson declined to comment on Monday’s outage, but said the agency was investigating the incident.
Mots clés: panne du 911, Azure, CenturyLink, FCC, Federal Communications Commission, Intrado, microsoft, The Washington Post, West Corporation