Exploring Road Runner's Internal Network

by Tim

Most ISPs require you to have a modem of some sort.

For broadband cable, this is usually a Data Over Cable Service Interface Specifications (DOCSIS) compatible device, version 1.0, 1.1, 2.0, or 3.0, depending on your ISP's needs.

This device is essential to cable Internet as it isolates and uses the various frequencies on the cable line which have been reserved for Internet service.  All of this information is determined by your ISP and is delivered to the cable modem via TFTP from some server on your ISP's non-public network.

Your cable modem has a MAC address like any other network device, and it is usually this that the ISP uses to authenticate you to the network.

The Cable Modem Termination System (CMTS) is where the transition between cable and fiber happens, for those that are interested.

At any rate, once your device is determined to be legitimate - again, the method is determined by the ISP, but is most likely the MAC address - you are leased a public IP address.

There is also an internal IP address granted to the modem, and it usually resides somewhere in the 10.x.x.x private subnet.  This address should never be accessible either from your own computer or by anyone else that isn't correctly authenticated on the network.  This is to prevent various horrible things from happening, such as the use of one of the many in-band configuration methods for routers and switches that reside on the networks.

Most devices decide who should be able to access the device remotely only by seeing which network they reside on.  If you access the 10.x.x.x side of the device, the odds are good that you'll be allowed access at least at the same level as the ISP.  Simple enough.

Now, once your device is given the correct network configuration, it then forwards those settings onto your computer.  If you are not using a router or some middle-man appliance, then your computer will inherit the TCP/IP configuration, allowing you to access the Internet at large.

The cable modem is essentially doing very simple routing for your computer.

It is simply taking everything given to it and pushing it through the other side in accordance with the ISP's settings.  This is how it was intended to be.  The cable company can terminate your connection by sending a series of commands to the device.  It can similarly throttle your connection, do troubleshooting, and so on.

They do this either by using proprietary tools such as Orion, which has some phenomenal CMTS tools, or by using in-house tools, usually PHP, ASP.NET, or Perl scripts running on some machine that manages the network.  (See the resources at the end of this article for some interesting sites on the Road Runner network).

From there, they can do all sorts of stuff, but the important thing to remember is that they are not using your public IP address to do this; they are using the private IP address given to your modem.  This is where my story begins.

I was sitting in my office, configuring my router to support the addition of a couple more subnets in the 10.0.0.0/24 range.  As I was doing this, I decided that the easiest way to test for connectivity among the various subnets was to simply allow all traffic on the 10.0.0.0/8 network to pass to any of the other subnets.  So, I set all this up and let some ICMP traffic fly across the wires.  This is where it got interesting.

I typed an IP address incorrectly.

To be specific, I typed 10.0.0.10 and pressed Enter.  Knowing that this IP address would not be found on my network I went to Ctrl+C the command.

What did I see appear on my console?:  Reply from 10.0.0.10: bytes=32 time=76ms TTL=128

My first thought was that someone had penetrated my network and established an entire subnet without me noticing.  Then I saw the latency and decided to do a traceroute.  Sure enough, the trace passed through my router, through the ISP-provided modem, and over the Road Runner network, eventually coming to a stop at some poor soul's Ambit cable modem.

Admittedly, I was very curious, so I ran some simple Nmap commands and discovered that this device was listening on port 80.

So, I loaded Firefox and hit the device with HTTP.  Sure enough, I saw the cable modem's management screen.  Being the concerned citizen that I am, I tested the login to make sure the defaults had been changed.

Much to my surprise, I could log in and get full viewing and configuration access with username and password: user

I then had admin access to someone's cable modem, complete with an internal IP address range on Road Runner's network, the public IP address, the MAC address, and everything else needed to clone their cable modem and steal their service.

From the screen which came up, you can restart the device, reset it to the factory defaults, or do pretty much anything you want.  My mind boggles at the concept.  And this is just 10.x.x.x addresses into a 16 million host subnet.

I immediately powered up nmap with OS fingerprinting and version scanning with the target network of 10.0.0.0/8.

I watched as the log file grew from 1k to 10k to 100k to 1000k.  After a couple of hours, I had a 5 MB file, full of cable modems running HTTP, SSH, Telnet, and various other services, all of them using default logins and passwords.  Most of them are running vulnerable version of SSH, and all of them will fall back to SSH1, which means that any passwords that may be in place protecting the shell access are useless.

I suddenly realized that Road Runner might notice all of the scanning that I was doing, so I called up Road Runner tech support and asked to speak to someone in the security department.  They put me on hold, and I listened to crappy music for about ten minutes before someone finally picked up.  We will call him Bill.

"Hello, thank you for calling Road Runner technical support.  My name is Bill, how can I help you?"

"Hi, Bill.  My name is Tim.  I'm just calling to report some strange behavior on your network.  It seems that I am able to see some of your internal IP addresses.  I can access your entire class A subnet as if it were public."

"Oh... hold on a minute.  I have to make a call."

I was then put on hold for about twenty minutes.  Eventually Bill returned, with an edge of concern in his voice.

"Can you give me some more information about this?  What addresses are you seeing?  What do you think is allowing you to do this?"

"Well, any IP address on the Road Runner network that starts with 10 is visible to me.  There don't seem to be any restrictive measures in place or anything, Bill.  As for how this has been happening, I'm not sure."

"Okay, do you see any other private IP addresses, anything like 192?"

"Doesn't seem like it, Bill, but I haven't really looked either."

"How are you seeing these IP addresses?  Are you using a packet sniffer or something?"

At this point, I realized that he was very concerned and that he was fishing for information.  I told the truth, as I don't want to go to jail for terrorism or some other equally absurd reason.  (Hooray for abusive and unconstitutional laws!)

"I'm just using Nmap to scan the subnet, no packet sniffers or anything.  So, yeah, I'm actually very concerned about this.  If I can see these internal IP addresses, it means that I can sniff traffic off the network as well, Bill.  I don't like that.  If I found this by mistake, someone out there will certainly find it as well.  I mean, if I were malicious, I could cause some serious damage.  These devices have default admin logins.  Oh, and the guy at 10.0.0.10 is having network issues."

"Really?" he chuckled nervously.  "Well, hold on a minute.  I have to make a call."

I waited on hold again, this time for only a couple of minutes.

"Alright, the security specialists say that this is normal for the network.  Since you're a part of the network, you should be able to see the other machines, so it's okay.  You're on a business account and, since you have a static IP, you are able to see some things that most of our customers cannot.  I'll make some notes on your account so that it's clear that you mentioned this to us and were concerned.  You might get a call from the Road Runner security department some time in the future.  Is there anything else?"

The conversation ended with the standard scripted closing, and I hung up the phone.  Normal operational behavior?  An entire internal IP address range available publicly?

I could see not just an entire subnet, but the entire 10.x.x.x network, the entire Road Runner network.

I decide to test Bill's theory about the business connection.  I SSH'd into my Linux box at home and issued a Ping to 10.0.0.10.  Sure enough, it responded.  So, everyone on the Road Runner network can simply use this private IP range to access network equipment.  I quickly loaded up Nmap and continued the scan.

At this point in time, I had found several thousand modems, nearly all of them running web servers, many of them also running SSH and Telnet.  I also found several cable modems acting as routers.  If someone were to log into one of those devices, it wouldn't be hard to set up forwards into the NAT'd network or to forward all their traffic through a tunnel to some other PC.

The possibilities then would be nearly limitless: hijacking VoIP service by cloning their hardware, stealing Internet service by cloning the MAC address, changing settings, or redirecting the location of the default DOCSIS servers, among other things.

As far as ISP-level equipment goes, Road Runner's DHCP servers, DNS servers, and network monitoring services are all available for scanning.

Worse, Nmap's version reporting option ( -sV ) shows version numbers for the services running.  Many of these are reported correctly, and several of them are vulnerable to very well-known exploits.

For instance, on one particular server the SSH daemon is set to roll-back to SSH1 if the client doesn't support SSH2.

Aside from all of that, a quick scan of the log file reveals the type of IDS they're using, the type of network monitoring software they're using, strange and unneeded third-party applications such as screencast, and other pieces of information, all freely available.

Honestly, I don't imagine that it would take a skilled hacker more than an hour or two to successfully compromise the systems.  The servers are pretty homogeneous, apparently consisting mainly of Linux servers running essentially the same applications, so the odds are good that if you can comproex mise one system, then you can take the rest as well.

Also, each system seems to be a central IDS reporting center, most likely for whatever section of the network it controls, and syslog information is forwarded to those machines.  The information that could be gleaned from the log files alone would be worth its weight in gold.

Of the 25,000 or so devices that showed up, about 100 of them seemed to be ISP servers.  I stopped scanning after about 12 hours because I felt like I had seen enough, but anyone who were to scan the entire 10.x.x.x subnet would undoubtedly discover much more than I have.

Needless to say, the potential for abuse here is tremendous, and it's shocking that this kind of network behavior was ever engineered to begin with.  Under normal circumstances, their routers and firewalls should filter public requests for private IPs, but I guess this isn't being done.

I guess it's true what they say about corporate networks: hard on the outside, gooey on the inside.

One final note:  There are interesting sites at tools.location.rr.com, where location is your geographical region, usually pretty easy to figure out.

For example, the Tampa, Florida area is tools.tampabay.rr.com.  The login and password have recently changed, but these sites contain all the information needed to hijack someone's account or to change most, if not all, of the services attached to the account.

Pretty slick stuff.