Telecom Informer

    

by The Prophet

Hello, and greetings from the Central Office!

As I write this, "Bella," the portable toilet that has graced our parking lot for far too many months with her presence, is being loaded onto a flatbed truck.  I will not miss her.  We have water that runs, a toilet that flushes, and a working sewer line (with a snazzy new clean-out) again.  I never thought I'd be so excited to flush a toilet, but after months of trudging out to a very ripe porta-potty this summer, I'll be happy if I never use one again in my life.

In a way, however, this is trading one problem for another.

Here in the Pacific Northwest, we were plagued for weeks over the summer with some of the world's worst air quality.  Having lived in Beijing for three years, I was shocked to see the Puget Sound area socked in with the kind of acrid smog that I thought I'd escaped when I left China.  Fortunately, my experience operating there meant that I knew we needed to step up our maintenance here.

In the Central Office, we have a very large Heating, Ventilation, and Air Conditioning (HVAC) system which is designed to cool the equipment and keep it operating at a consistent temperature of 74 degrees.  Owing largely to the Network Equipment-Building System (NEBS) standards under which it was certified, the system is uncanny in how consistent it is and, despite its age, works flawlessly.  (The same cannot be said for our switch, which is well over 20 years old at this point and is increasingly temperamental.)  It can be 20 degrees or 100 degrees outside, and it'll still be 74 degrees inside the Central Office.

We also have a smaller HVAC system that cools the data center space downstairs.  This was initially installed in the early 2000s to house Competitive Local Exchange Carriers (CLEC), but there are also several racks of equipment owned by the Internet Service Provider part of our business.

Technically, they are a paying customer just like the CLECs because this is an unregulated service, so it's treated like a separate company.  The temperature requirements are different, and it's a cold, noisy 64 degrees in this facility.  Unfortunately, the company chose to invest in technology that was cutting-edge at the time, which meant that the bugs weren't worked out yet.  There are two Computer Room Air Conditioner (CRAC) units.  They're more efficient than the older chillers we're using upstairs, but one or the other of them will get confused every now and then and stop operating.  The temperature will slowly creep up until it trips an alarm, at which point a bleary-eyed Central Office technician will drive out and reboot the unit.  I'm not kidding.  We'll shut off the CRAC, wait five minutes, turn it back on, and magically the temperatures will go down again.

What does all of this have to do with the bad air here in the Pacific Northwest?  These units move a massive amount of air through our facility, and we filter that air before it comes in here.  Normally, we change the filters once a year.  However, in our Beijing facility, I learned that we needed to change the filters every two months or bad things would start to happen.  Dirty filters cause the units to be less efficient at best, and can also cause inconsistent temperatures along with that (this is bad).  Now that the autumn rains have begun and the fires are (more or less) out, I knew we'd need to replace the filters.

Of course, that involved a lot of paperwork.  The company doesn't want to spend any money on maintenance that isn't absolutely required, and I didn't have the budget for an out-of-cycle filter replacement.

This meant that I had to ask the bean counters in Denver for approval.  Fortunately, Denver got hit by the smoke and fires too, so I got an unusually sympathetic ear when I called in to find out the exception code.  As it turns out, the company has an exception code for smoke damage and this can be used not just to order new filters, but to request budget to repair any damage caused by smoke.

"Hmmm..." I thought, "This could be an opportunity."  The paint on most of our building is peeling.  However, some sort of high school gang has been painting graffiti on the walls.  The company only gives us enough budget to paint over it, but not to paint the whole building, so we have new paint mixed with old.

But what if I could explain the peeling paint as caused by fire damage?  I mean, where there's smoke there's fire, even though the fires were over 100 miles away.  I sent a couple of techs out to take the most unflattering pictures possible and practically held my breath while I submitted a budget request to Denver.

Amazingly, it came back approved!  I decided to press my luck.  The paint on our ancient GMC bucket truck is peeling too, and it would be nice to get that repainted as well, so we took pictures and I sent those in too.  This request, naturally, was almost instantly denied.  "Truck is on file as being garaged and no recent jobs shown near fire zone," said the denial.  I decided to stop pressing my luck.

The entire Central Office exterior is being painted, and it's going to be in our own "gang" colors: light and dark green!

Both of our HVAC systems are designed to be fully redundant, which allows us to perform maintenance without any outages.

This is important, because telecommunications switching equipment is all sensitive to heat with very tight tolerances.  When I say "designed," you might pick up on the difference between what the manufacturer claims and the reality of the situation.

The system serving the switch uses traditional water-cooled chillers and is redundant in the sense you'd expect.  We can take one of the chillers offline for maintenance and everything stays exactly at 74 degrees.  Part of the reason why I think this works is because the switch we currently have installed just isn't running very hot these days.  It's serving far fewer lines of service than it could, and phone lines aren't as busy as they used to be (where voice traffic is concerned; data traffic, of course, is another story).  The system serving our data center (which is very full) is a different story.

This one uses an air-cooled CRAC.  While this is allegedly more efficient, it's also more finicky.  Unfortunately, the redundant design doesn't get the job done when only a single cooling tower is in use, and it especially doesn't get the job done when the system is operating at reduced capacity (which is the case when the filters are very dirty).

The procedure to replace the filters is more or less as you'd expect, except we do it in the middle of the night so the outside temperature is as low as possible.  We shut down the first chiller, change out the filters one by one (these are very dirty and the dust is something you don't want to breathe, so the technician doing it wears heavy gloves and a respirator), bring it back online and, when everything is verified to be working, we'll repeat the process on the other chiller.

This is done first for the telephone switch and next for the data center, so we get the benefit of the lowest possible outside temperatures for our troublesome CRAC units to face alone.  Even though changing the filters themselves only takes a few minutes, shutting down and restarting each chiller or CRAC and running through our verification procedures takes about an hour overall.

It's a lot of stress on the units to take them offline and bring them up, and this is when you're most likely to have service outages.  I'm most concerned about this for the data center, because the temperature will slowly creep up with only one CRAC unit running until equipment begins to fail.  This happened once, and nearly knocked out Internet service in half the city!

Since then, I always notify the vendor 48 hours in advance of planned maintenance so they can have staff on standby.  Although we have a 24x7x365 service contract with the manufacturer, this doesn't necessarily mean that they always have a supply of spare parts available and ready to dispatch.  By notifying them in advance, I can ensure they are ready to save the day if the need arises.

And with that, it's time for me to meet with the painting contractor.

Have an amazing autumn, and I'll see you again in the winter!