ATT Wireless Cell phone outage

[Lee Schierer]

It would appear that several of the major and minor Cell phone services are unable to provide service this morning. States affected appear to be California, Texas, New York, Pennsylvania, Massachusetts, Missouri, Illinois, North Carolina, Oklahoma, Kansas, Colorado, Georgia and probably others by now.

[Jim Becker]

While this primarily affects ATT Wireless and it's private label pre- and post-paid wireless entities, there are also issues with other major and sub carriers, such as VZ and TMo, likely because of all the necessary interconnection and infrastructure sharing, particularly in non-urban areas. Also affected is a lot of the vehicle telematics as many vehicle brands use the ATT Wireless network for private backbone services. So folks using apps for things like remote start may not be able to use those features until things are resolved.

[Michael Weber]

My wife is sitting close to me with “no service” showing on her iPhone. I had her reset the phone and the results were the same. Five feet from her my iPhone is fine. Weird. We share the same service on the same account.

[Jim Becker]

Turn on WiFi calling to get around the outage at least where your device has access to WiFi.

[Brian Elfert]

As of a few hours ago Verizon and T-Mobile both said they were having no outages. They said the issues reported by their customers was because they can't reach AT&T customers.

[Lee Schierer]

Our ATT service was restored about 1:00 pm.

[Bill Dufour]

I read if the cell service is not working you can text 911 in an emergency. Not exactly sure how to do that. Just send a text to 911 with no area code? Not really something to try on a whim.
Bill D

[Alan Lightstone]

Ours was out in Tampa too. First no bars, just great wifi. Then SOS showed up a few hours later, then service restored.

This clearly affected far more people than they were admitting to.

[George Yetka]

NJ out as well. Was a pretty bad day to be out. 3 major jobs scheduled all with issues and I was stuck in an hvac license refresher.

[Jim Becker]

As it turned out, they had a coding mistake while adding to the network that took it down; I suspect a serious routing issue. But at least it wasn't something nefarious...

[Bill Dufour]

I read it happened at exactly 3:00 AM east coast time. This is late enough the west coast is asleep and the east coast is not awake yet. This is when most national companies do their software updates. Gives them enough slow time to catch and fix any code errors. In theory.
used to be underground atom bomb test were confused with earthquakes. But how many quakes occur on the stroke of the hour?
Bill D

[Bill Dufour]

I heard folks were clogging up 911 calling in to say their cell service was down. Just like people call 911 to ask if they really felt an earthquake and the police/fire departments need to know.
Of course if Taylor Swift is selling tickets to a show in your town the phone lines are jammed in minutes of the announcement.
Bill D

[Bill Dufour]

I have no idea who this guy is.
Bill D.

[Brian Tymchak]

As it turned out, they had a coding mistake while adding to the network that took it down; I suspect a serious routing issue. But at least it wasn't something nefarious...

Not the first time something like this has happened. In 1990 a misplaced break statement in 4ESS code caused a massive cascade failure after a network upgrade. I was testing network management software at AT&T Bell Labs at the time. Although my project was not directly involved with the failure, the next few months were a little uncomfortable.

Here's the best online accounting I've read of what really happened.

https://users.csc.calpoly.edu/~jdalb...s/att_collapse

[Jason Roehl]

I have a HS classmate who said, "I am the battering ram for the crack team that fixed the problem." That was after I mentioned that I assumed he was part of the crack team dealing with the outage, but he also added that he no longer "does the real work." Unfortunately, he won't (probably can't at the risk of his job) share details about what happened. He also said he was really proud of what his team did to fix the outage.

About 10 years ago, I PMed him on Facebook during a local outage, and he looked up my number, could see what tower I was connected to, then dug a little deeper, and found out that something had happened in Indianapolis, 60 miles away. Maybe a dead server or construction work took out a trunk line, something like that (I've slept since then).