On June 15th just after 12 noon, T-Mobile users began to experience problems with making and receiving calls as well as sending text messages over T-Mobile’s VoLTE (Voice over LTE) services. Local 9-1-1 authorities posted on social media accounts that a T-Mobile VoLTE outage could have potentially affected 9-1-1 calls and urged citizens to use landlines to contact local 9-1-1 services. Frustrated users flooded the crowd-sourced website downdetector.com by posting reports of network issues. While voice and texting were down, T-Mobile stated that customers could use apps and services like FaceTime, iMessage, Google Meet, Zoom, and Skype to reach people. The outage was so widespread that rumors of a DDOS attack began spreading on social media that had to be quickly debunked.
A public statement from T-Mobile stated that the problem was triggered by a leased fiber outage from a third-party provider that exploited a routing platform configuration, subsequently causing circuit overload. That resulted in “an IP traffic storm that spread from the Southeast” creating substantial issues across T-Mobile’s core radio network. The VoLTE outage caused phones to stop registering and call completion, so many users could not make or receive calls until the outage resolved approximately 13 hours later. During the 13-hour outage, users also experienced intermittent outages with text messages.
RELATED: T-Mobile hit with widespread network issue, customers report service outages
The VoLTE outage was fully resolved around 1 AM ET on June 16, with T-Mobile posting the following statement on their website from Neville Ray, T-Mobile’s President of Technology:
Update on T-Mobile Voice and Text Performance
“Every day we see the vital role technology plays in keeping us connected, and we know T-Mobile customers rely on our network to ensure they have connections with family, loved ones and service providers. This is a responsibility my team takes very seriously and is our highest priority. Yesterday, we didn’t meet our own bar for excellence.
Many of our customers experienced a voice and text issue yesterday, specifically with VoLTE (Voice over LTE) calling. My team took immediate action — hundreds of our engineers worked tirelessly alongside vendors and partners throughout the day to resolve the issue starting the minute we were aware of it. Data connections continued to work, as did our non-VoLTE calling for many customers and services like FaceTime, iMessage, Google Meet, Google Duo, Zoom, Skype and others allowed our customers to stay in touch. Additionally, many customers were able to use circuit-switched voice connections and customers on the Sprint network were unaffected. VoLTE and text in all regions were fully recovered by 10 p.m. PDT last night. I’m happy to say the network is fully operational… and we’re working day in and day out to keep it that way.
Our engineers worked through the night to understand the root cause of yesterday’s issues, address it and prevent it from happening again. The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.“
During the VoLTE outage, there were reported issues with Sprint service; however, Sprint issued statement that they were not impacted. AT&T and Verizon’s networks were also functioning normally. While the other carriers were functioning normally, calls to and from T-Mobile from the other carriers’ customers may have resulted in error messages.
Later the same day, FCC Chairman Ajit Pai with a posted to his official FCC Twitter account stated that the FCC was “demanding answers—and so are American consumers”.
On June 23, the FCC announced that the agency’s Public Safety and Homeland Security Bureau is investigating the outage “given the large area affected and the critical importance of dependable and resilient 911 service throughout the United States”. The FCC is seeking comments from affected public safety agencies on the number of calls to PSAPs that were affected and the impact to services caused by the VoLTE outage. The FCC is also seeking comments from consumers on not only how the outage affected them but how they were notified of the outage and how affective were T-Mobile’s communications on the outage. The deadline for comments is July 8, 2020. Comments may be filed using the Commission’s Electronic Comment Filing System (ECFS).
More EmComm Brief News
Similar Stories: