There is No Magic in the Clouds

by kyber

Technology is built and maintained by fallible humans working with imperfect information on a problem they may not fully understand.  These are all realities of life and must be accepted.  Take all of your worry about what came before you and abstract it behind APIs.  Now take those APIs and build something new.  Iterate over this an arbitrary number of times and now we have innovation.  Continue this cycle for a while and suddenly you have abstracted away much of computing itself.  The concept of highly available disks and high-performance compute resources have essentially become reduced to a function call.  These abstractions have given us the cloud, a highly powerful but highly dangerous tool.

We no longer have to think about backing up our photos, documents, source code, game saves, and messages.  We pay a provider a small fee and they take care of the messy details.  As an 11-year-old aspiring technologist in the 1990s, this is the kind of reality that could only exist in my dreams.  I often feared that I'd return to my computer after school to discover that my hard drive had crashed and my MP3s, Unreal saves, 1337 Perl scripts, and Simpsons episodes were gone forever (spoiler: that happened).

Services like Google Drive and AWS were not bestowed upon us by a higher power.  They are built by imperfect people that require a seemingly unending amount of upkeep, much like the seemingly limitless amount of storage and computing power that they offer.  We rarely think about the on-call engineer at a data center who loses a night of sleep to make sure you can keep uploading pictures of Dick Butt to your friend's message board.  We go about our day as soon as we see that our upload completed successfully.

Now all is well in paradise.  The facade of the cloud fades when you peer too closely.  Artifacts of implementation remind us that imperfect humans built everything on both sides of our laptop screens.  Peer through the installation directory of the Dropbox client or go poking around at the file system on a Heroku dyno.  You will see reminders of humanity and past mistakes.  These seemingly magic resources are just servers running somebody's code.  This person is not so different from you, just a programmer trying to get their work done.

Years ago, I watched Honda's Asimo fall to the ground during a routine demo.  Immediately, a team of engineers hurried over, curtains were put up, and an awkward mix of silence and disbelief filled the room.  That thing that once appeared perfect had now just failed before our eyes.  While not catastrophic, it does remind us to not confuse routine with easy.  Cloud services are no different.

We rarely think about what happens in cloud services until they fail.  I've been working in the backstage area of the cloud for quite a few years now.  I've developed war stories and battle scars.  Seemingly promising organizations have crashed and burned due to bad luck, naive assumptions, or a little of both.  I watched a company's finances grind to a halt for days because they assumed the cloud could handle their sub-par code.  Sat helpless while a prominent e-commerce site nearly went out of business because they never planned for a developer accidentally deprovisioning a database (hello access controls).  Got a sick feeling in my stomach when I ran a script that disabled the account of an abusive customer, knowing that this would be the company's death sentence.

The impacted parties chose to blame the cloud or, more specifically, the cloud provider every time.  Initially, they saw this technology as a magical way to absolve themselves of responsibility.  Warnings were often given and often fell upon deaf ears, that is, until the worst finally came to pass.  Isn't it funny how "not on our road map this quarter" becomes "nobody goes home until this is done" all because a single disk decides to fail?  The cloud is merely an abstraction, the computers are still there.

Life is all about a series of tradeoffs.  The advantages of the cloud will often outweigh the potential disadvantages.  Not having to worry about physical hardware has advantages.  Your organization can focus on the bigger picture and developers can allocate the resources they see fit.  However, some people see this as a way to abdicate responsibility when the worst comes to pass.  Just because you don't see the server doesn't mean it can't hurt you.  The cloud is nothing more than imperfect people trying to maintain the illusion by giving their blood, sweat, and tears.

There is no magic in the clouds.