Telecom Informer

    

by The Prophet

Hello, and greetings from the Central Office!

I am in sunny Southern California after several months of traveling around the world.  I have enjoyed fine hotels and luxury travel from London to Los Angeles while researching some of my favorite topics, all on the company dime and never in economy class.  The management lifestyle is definitely a cut above the union rate.  I'm never going back!

I love the smell of a 4ESS tandem in the morning, but today, I'm standing in a new style of central office.  It is what is known as a carrier hotel.  Much like the Central Office tandems that remain as part of the old Bell System, it is a hub and interconnection point.  However, this facility is for Internet carriers, many of which are also long distance phone companies.  They all pay to exchange traffic here, and there are efficiencies to be achieved.

Bell facilities were built to exacting standards and maintained nationwide according to detailed and consistent Bell System Practices.  Bell System employees were generally unionized (some still are) and received extensive and exacting training.  Most traffic between telephone companies is exchanged at specialized offices called access tandems.  These are carefully managed and run by dedicated staff, most of which have extensive telecommunications experience.

Carrier hotels are a much different type of facility than a traditional Central Office.  These are operated by private Internet companies, and while some operational aspects are consistent, there isn't anything approaching the management rigor of Bell System Practices.  The three largest companies, who operate most Internet interexchange points, are Equinix, Telx, and CoreSite.  These facilities can be very large.  For example, One Wilshire in Los Angeles, one of the densest carrier hotels, hosts more than 260 different Internet service providers exchanging traffic within the facility.

My employer recently acquired a carrier hotel and assigned me to a project to achieve additional efficiencies.  All of my usual management tricks didn't apply.  I like to streamline processes wherever possible (leaving a paper trail could be bad for my bonus in the future), but I found out that the engineers who manage peering often turn up circuits based on just a few email messages.  Sure, security procedures are theoretically in place, but the guys all know one another, so what's the harm?  In 2010, a single routing error (using the Internet equivalent of SS7, which is called BGP) routed 15 percent of the world's Internet traffic through China until it was corrected 18 minutes later.  These sorts of errors happen all the time, but I correctly recognized that there is no real harm to the business, so I have streamlined security procedures even more.  For the most part, security is an unnecessary cost.  After all, for Internet service providers with more than one connection, TCP/IP is largely a self-healing protocol.  Even if traffic is routed through China, it still gets there eventually and it doesn't really cost any more, even if you misroute traffic halfway around the world.  This is much different than a similar SS7 error, which would inevitably result in a flurry of debit memos (assuming the unwitting recipient of the erroneous traffic has configured their system to complete the call, which is usually the case).  There is an exception for smaller providers, who are often connected to only one "upstream provider."  A single routing error could put them out of business, so more testing is done before "flipping the switch" to ensure that changes are accurate.

I next looked at HR to find out whether I could save money by firing better paid older workers or breaking a union.  Unfortunately for me, I discovered the average age of the staff in the facility is 21, they are all non-union, and the average wage is about $10.50 per hour.  It's hard to wring many cost savings from this, but I still found some.  As it turns out, management brings in new hires from the local technical school.  A grizzled old hand complained bitterly, and told me they don't really know what they are doing.  Customers complain of constant issues with people unplugging the wrong things or damaging things while trying to work with them.  And the company had to pay compensation!

I had been tuning him out until I heard this, but the word "compensation" definitely got my attention.  I had found something to streamline!  As it turned out, our contracts promised a particular service level that implied our staff was competent and even provided compensation for errors and omissions on our part.  I instructed the legal department to update our contracts to promise that we would provide service "in good faith" and based on "best efforts."  They also helped me with new compensation clauses.  I opted not to entirely eliminate promises of any particular service level or response time, because customers demand these clauses in the contract and competitors would gain an advantage if they didn't exist.

Instead, I just watered down the clauses to the point where they are effectively meaningless.  I love "new" and "updated" contracts - they always mean a bigger bonus!  I laid off the grizzled old hand.  He was a great engineer, but he was twice as expensive as the kids, spent most of his time posting on the North American Network Operators' Group (NANOG) mailing list, and our new contracts meant we didn't need anyone competent anymore.  I then boosted sales by updating our sales program to promise expert 24x7x365 service while also raising our rates.  Our customers have no idea they are paying more and getting less.

Finally, I helped the company with a very successful "green initiative."  The carrier hotel was hot and uncomfortable, with temperatures reaching up to 100 degrees in the "hot rows."  However, I observed that July temperatures in Phoenix are as hot as 130° F (54° C) so there was room for our facility to be even hotter!  Obviously, if a major city in the United States can be 130 degrees in the shade outside, so could our carrier hotel; I believe this is a fully justifiable position to OSHA.  I immediately instructed that the air conditioning be turned down, raising indoor temperatures to 130 degrees.  A couple of the kids turned green and threw up, so I saw immediate green results.  We are saving a fortune in electricity costs and the environment along with it!  I will hopefully receive a larger bonus for my environmental stewardship.  Sure, it's bad for the equipment hosted in our facility, but it's not our equipment and there is nothing in the contract guaranteeing any particular temperature.

And with that, I declare our newly acquired carrier hotel streamlined!  I have improved profitability by seven percent with a few simple changes.  If you complain about warm weather this summer, think of our employees toiling away in 130 degree temperatures and consider that if I were your manager it could be you!  I will be at DEFCON and HOPE this summer, and will look forward to meeting all of you and exchanging ideas about success.  For now, I will enjoy the ice cold air conditioning in my brand new car.  Management life is the best!

References
Return to $2600 Index