Curt Harms
12-31-2015, 10:17 AM
in this sparky's shoes? Not completely the electrician's fault but we know who got probably got blamed.
.................................................. ...........
The existing facility sported a trio of 220KvA UPSes, but at the new bit barn it was decided to run with just two.
“The decision was made to save costs by relocating one of them to the new building rather than buying all new equipment.”
JF says he “begged the business to call a complete shutdown to remove the UPS. They asked me what the odds of something going wrong, and I made the error of trying to provide an accurate estimate of the risk by saying there was about a one in 100 chance of problems.”
JF thought a one per cent risk of power failure across 25,000 square feet packed full of server racks, live, in production, would scare off the bean counters.
He was wrong.
“The business figured that was a perfectly acceptable risk,” he recalls. So JF decided to come in on the Saturday of the move, just in case.
“I was sitting at a desk in the middle of the data center floor that weekend when the electricians began the delicate work of removing a 220KvA UPS unit from the mains,” JF wrote. “They put the system in bypass mode without a problem. They then cut the output breakers for the units to be removed. No problems. Then they wanted to isolate the inputs for the UPS units.”
Bad idea, because “they cut the input breaker for the master electrical panel, not the outputs that went to the UPS systems. That master panel also supplied the circuits to the bypass feed, which meant we had no power to anything at all.”
“25,000 square feet suddenly went silent. I ran into the electrical room expecting to find bit of dead electrician all over the place, but they were just calmly disconnecting wires.”
“I yelled, 'We're down!' and they said, 'No, we're in bypass mode.' I repeated, 'Noooo. We are down". They paused for 10 seconds and then their eyes got really wide.”
.................................................. ........
http://www.theregister.co.uk/2015/12/11/electrician_cuts_the_wrong_wire_and_brings_down_25 000_square_feet_of_data_centre/
.................................................. ...........
The existing facility sported a trio of 220KvA UPSes, but at the new bit barn it was decided to run with just two.
“The decision was made to save costs by relocating one of them to the new building rather than buying all new equipment.”
JF says he “begged the business to call a complete shutdown to remove the UPS. They asked me what the odds of something going wrong, and I made the error of trying to provide an accurate estimate of the risk by saying there was about a one in 100 chance of problems.”
JF thought a one per cent risk of power failure across 25,000 square feet packed full of server racks, live, in production, would scare off the bean counters.
He was wrong.
“The business figured that was a perfectly acceptable risk,” he recalls. So JF decided to come in on the Saturday of the move, just in case.
“I was sitting at a desk in the middle of the data center floor that weekend when the electricians began the delicate work of removing a 220KvA UPS unit from the mains,” JF wrote. “They put the system in bypass mode without a problem. They then cut the output breakers for the units to be removed. No problems. Then they wanted to isolate the inputs for the UPS units.”
Bad idea, because “they cut the input breaker for the master electrical panel, not the outputs that went to the UPS systems. That master panel also supplied the circuits to the bypass feed, which meant we had no power to anything at all.”
“25,000 square feet suddenly went silent. I ran into the electrical room expecting to find bit of dead electrician all over the place, but they were just calmly disconnecting wires.”
“I yelled, 'We're down!' and they said, 'No, we're in bypass mode.' I repeated, 'Noooo. We are down". They paused for 10 seconds and then their eyes got really wide.”
.................................................. ........
http://www.theregister.co.uk/2015/12/11/electrician_cuts_the_wrong_wire_and_brings_down_25 000_square_feet_of_data_centre/