Unexpected Denial-of-Service

by J. Savidan

I work as an Enterprise Resource Planning (ERP) consultant for a big IT company in France (this precision should explain my relative lack of fluency in English), and it is not particularly fun on a daily basis.

But a few days ago, something a little more spicy happened: I had to install a patch in production, nothing really difficult or dangerous.

As part of the procedure, my customer wanted a proof that only the impacted programs were installed.  I had that requirement on the previous installation also, so a few weeks ago I wrote a utility program that scans two environments and yields the differences between them.

To check the version and the signature of one program in particular, all we have to do is to use a web form that checks this for you after being given the name of the program.

To do that for almost 9000 programs, I needed a way to automate this in one way or another.

My solution was to write a Java program that opens a HTTP connection to the address of the lookup form, providing the name of the program to look up in the HTTP request.  Then all I needed was to do that for every line of a file containing the list of all programs, and I had a super fast web bot.

I added a parameter corresponding to the time the program has to wait before issuing the next request.

The code is very simple, and can be shortened to something like this:

FileReader fr = newFileReader(newFile(programsListing)); BufferedReader br = newBufferedReader(fr); while(br.ready()) { String prg = br.readLine().trim(); URL finalURL = newURL("www.a-server.com/findClass?port=6100&p1="+prg); URLConnection conn = finalURL.openConnection(); BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream())); ... Thread.sleep(Integer.parseInt(wait)); }

The rest of the code reads the buffer to do some Document Object Model (DOM) parsing to retrieve what the web form would produce on screen.

On D-Day, I launched my scan with a 500 millisecond delay.  Strangely, an hour after this, the production became unstable.

The IT operations team suspected my patch to be the troublemaker, but none of the issues reported could be logically caused by mere software patch.

I was the only one to suspect my analysis tool, and I began to check the log files for some hint.

It was simple to figure out: the application container was trying to create a new JVM (for load balancing) using a servlet, and a HTTP exception was occurring.

That was just in the middle of my long scan, and I realized that my tool was flooding the server with HTTP requests, causing other requests to timeout, and in particular one with a URL ending with: /addJVM

That was an involuntary denial-of-service, and it was quite efficient, judging by the raising of temperature on the client's side!

What's most funny is that they have an expensive supervision tool which triggers an alarm as soon as you act on a JVM, even if the action is a necessary maintenance action - but it's not able to detect an unusual count of hits on the web port...

Of course, they still don't know the truth.

That's the price you pay when you don't know how to read a log file...