ATT Wireless Cell phone outage

[Lee Schierer]

It would appear that several of the major and minor Cell phone services are unable to provide service this morning. States affected appear to be California, Texas, New York, Pennsylvania, Massachusetts, Missouri, Illinois, North Carolina, Oklahoma, Kansas, Colorado, Georgia and probably others by now.

[Jim Becker]

While this primarily affects ATT Wireless and it's private label pre- and post-paid wireless entities, there are also issues with other major and sub carriers, such as VZ and TMo, likely because of all the necessary interconnection and infrastructure sharing, particularly in non-urban areas. Also affected is a lot of the vehicle telematics as many vehicle brands use the ATT Wireless network for private backbone services. So folks using apps for things like remote start may not be able to use those features until things are resolved.

[Michael Weber]

My wife is sitting close to me with “no service” showing on her iPhone. I had her reset the phone and the results were the same. Five feet from her my iPhone is fine. Weird. We share the same service on the same account.

[Jim Becker]

Turn on WiFi calling to get around the outage at least where your device has access to WiFi.

[Brian Elfert]

As of a few hours ago Verizon and T-Mobile both said they were having no outages. They said the issues reported by their customers was because they can't reach AT&T customers.

[Lee Schierer]

Our ATT service was restored about 1:00 pm.

[Bill Dufour]

I read if the cell service is not working you can text 911 in an emergency. Not exactly sure how to do that. Just send a text to 911 with no area code? Not really something to try on a whim.
Bill D

[Alan Lightstone]

Ours was out in Tampa too. First no bars, just great wifi. Then SOS showed up a few hours later, then service restored.

This clearly affected far more people than they were admitting to.

[George Yetka]

NJ out as well. Was a pretty bad day to be out. 3 major jobs scheduled all with issues and I was stuck in an hvac license refresher.

[Jim Becker]

As it turned out, they had a coding mistake while adding to the network that took it down; I suspect a serious routing issue. But at least it wasn't something nefarious...

[Bill Dufour]

I read it happened at exactly 3:00 AM east coast time. This is late enough the west coast is asleep and the east coast is not awake yet. This is when most national companies do their software updates. Gives them enough slow time to catch and fix any code errors. In theory.
used to be underground atom bomb test were confused with earthquakes. But how many quakes occur on the stroke of the hour?
Bill D

[Bill Dufour]

I heard folks were clogging up 911 calling in to say their cell service was down. Just like people call 911 to ask if they really felt an earthquake and the police/fire departments need to know.
Of course if Taylor Swift is selling tickets to a show in your town the phone lines are jammed in minutes of the announcement.
Bill D

[Bill Dufour]

I have no idea who this guy is.
Bill D.

[Brian Tymchak]

As it turned out, they had a coding mistake while adding to the network that took it down; I suspect a serious routing issue. But at least it wasn't something nefarious...

Not the first time something like this has happened. In 1990 a misplaced break statement in 4ESS code caused a massive cascade failure after a network upgrade. I was testing network management software at AT&T Bell Labs at the time. Although my project was not directly involved with the failure, the next few months were a little uncomfortable.

Here's the best online accounting I've read of what really happened.

https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

[Jason Roehl]

I have a HS classmate who said, "I am the battering ram for the crack team that fixed the problem." That was after I mentioned that I assumed he was part of the crack team dealing with the outage, but he also added that he no longer "does the real work." Unfortunately, he won't (probably can't at the risk of his job) share details about what happened. He also said he was really proud of what his team did to fix the outage.

About 10 years ago, I PMed him on Facebook during a local outage, and he looked up my number, could see what tower I was connected to, then dug a little deeper, and found out that something had happened in Indianapolis, 60 miles away. Maybe a dead server or construction work took out a trunk line, something like that (I've slept since then).

[mike stenson]

As it turned out, they had a coding mistake while adding to the network that took it down; I suspect a serious routing issue. But at least it wasn't something nefarious...

All it takes is the right typo in the wrong place....


and I typically find out about it several years later, in the middle of the night.

[Patty Hann]

Saw not a few articles that have questioned the wisdom of not having a LL as backup.
Some people in remote areas had no way to contact anyone.
People with medical conditions come to mind, but also other folks.
Still hangin' onto my LL and I live in a Metro area.
Not putting "all the eggs in one basket", juneau...

[Jim Becker]

Patty, unfortunately, land lines are not just disappearing because people opt to just be wireless. It's also about infrastructure. The old copper wiring plant is in beyond horrible condition in many geographies and the gear that runs it is no longer made with parts to keep existing class 5 switches relegated to EBAY in many cases. While in some areas, fiber has replaced copper or is in process of replacing it, the financial incentive to move to fiber in huge portions of the geography is just not there for investors. I used that word on purpose...the telecommunications business is totally pegged to the stock market and return on investment rules there.

There's nothing wrong with having a "landline" if it's available to you and you want to pay for it. But overall, society has moved on in a big way globally and that's where the money is. Consequently, that's also where the investment is going.

[Bill Howatt]

Our provider, serving a small rural area, offers VOIP phone service which we have and is pretty good but they will tell subscribers if they have a medical condition it should not be the only telephone mechanism they have.

[Brian Elfert]

Patty, unfortunately, land lines are not just disappearing because people opt to just be wireless. It's also about infrastructure. The old copper wiring plant is in beyond horrible condition in many geographies and the gear that runs it is no longer made with parts to keep existing class 5 switches relegated to EBAY in many cases. While in some areas, fiber has replaced copper or is in process of replacing it, the financial incentive to move to fiber in huge portions of the geography is just not there for investors. I used that word on purpose...the telecommunications business is totally pegged to the stock market and return on investment rules there.

There's nothing wrong with having a "landline" if it's available to you and you want to pay for it. But overall, society has moved on in a big way globally and that's where the money is. Consequently, that's also where the investment is going.

I have read that telephone companies in some locations are replacing the big old 5ESS and similar switches with SIP to analog gateways inside the central office buildings. You don't need many gateways when only about 15% of households have a land line these days. Businesses of any decent size have long used digital phone lines instead of analog. Further, a lot of companies with enough employees to have an actual phone system have gone to cloud based telephony.

[Jim Becker]

I have read that telephone companies in some locations are replacing the big old 5ESS and similar switches with SIP to analog gateways inside the central office buildings. You don't need many gateways when only about 15% of households have a land line these days. Businesses of any decent size have long used digital phone lines instead of analog. Further, a lot of companies with enough employees to have an actual phone system have gone to cloud based telephony.
That's been going on for decades for both Lucent (5ESS) and Nortel (DMS). But even that is not inexpensive and with the copper plant "out there" is really horrible shape in so many areas, even using the softswitches isn't going to keep land lines running forever. Here in Verizon land, at least where FiOS fiber is supported, if a copper line fails, the customer gets fiber. They can still choose "regulated" service, but they have to pay a lot more for it. If they choose the now-standard, non-regulated offer, it's still expensive but at least has more features. In areas that are not built out for fiber and are not intended to get fiber due to investment barriers relative to geography and population, wireless is the only choice other than the local cableco if they happen to offer phone service...which is still unregulated, unlike traditional copper.

As to companies, yea, there's been a big movement over the past decade and a half to cloud based subscriptions. In addition to CPE solutions, I also sold cloud setups using the same technology prior to retirement.

[Brian Elfert]

About 10 years ago, I PMed him on Facebook during a local outage, and he looked up my number, could see what tower I was connected to, then dug a little deeper, and found out that something had happened in Indianapolis, 60 miles away. Maybe a dead server or construction work took out a trunk line, something like that (I've slept since then).

My employer has two fiber lines between buildings about two miles apart. One fiber line goes pretty much direct and the other takes a diverse route. The diverse route goes at least 25 miles south of our location because we have had fiber cuts in the diverse line multiple times and the fiber cuts were at least 25 miles south. I have no idea why the line is routed that far away. We assumed a diverse route meant it is routed through the other side of downtown, not miles away. Both buildings are close to the size of a city block. The fiber comes out of different sides of the buildings.