T-Mobile explains why its network went down, hard, on Monday

quote:

---

If you’ve been wondering what could knock out one of the United States’ three big cellular carriers’ ability to deliver calls and text messages — and keep it that way for most of an entire day — T-Mobile now has a partial answer that pertains to its extensive nationwide outage Monday.

---

quote:

---

The short version, if we’re reading this correctly: a fiber-optic circuit failed, and its backup circuit also failed, which caused a chain reaction that strained the network to the point that many calls and texts couldn’t make it through.

---

quote:

---

The longer version:

---

quote:

---

Many of our customers experienced a voice and text issue yesterday, specifically with VoLTE (Voice over LTE) calling. My team took immediate action — hundreds of our engineers worked tirelessly alongside vendors and partners throughout the day to resolve the issue starting the minute we were aware of it.

---

quote:

---

Our engineers worked through the night to understand the root cause of yesterday’s issues, address it and prevent it from happening again. The trigger event is known to be a leased fiber circuit failure from a third party provider in the Southeast. This is something that happens on every mobile network, so we’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors. This overload resulted in an IP traffic storm that spread from the Southeast to create significant capacity issues across the IMS (IP multimedia Subsystem) core network that supports VoLTE calls.

We have worked with our IMS (IP Multimedia Subsystem) and IP vendors to add permanent additional safeguards to prevent this from happening again and we’re continuing to work on determining the cause of the initial overload failure.

---

quote:

---

It’s not clear which third-party provider’s fiber circuit failed. There was a report on Monday that Level 3, one of the world’s major internet backbone providers, was experiencing an outage, but a spokesperson told TechCrunch differently.

---