Linux Pwned - Just Not By Me

by Edster @ 2600 Dublin

If you ask Linux experts or admins, they will tell you "Linux doesn't get viruses."

It is common to hear people saying you do not need to worry about anti-virus software unless you are living in Windows land.  This is not quite as true as you think.

About a month ago, my main home system rebooted a few times without warning and made me suspicious.  I ran a virus check / malware check / rootkit check and none of them found anything at all.  As always - clean.  A week later, a letter came in the post from my ISP letting me know I had the Linux/Ebury virus - they monitor for various types of traffic and spotted it coming out from my IP.

I had never heard of the Linux/Ebury virus and spent the first few minutes trying to work out if this could be a scam letter that had not come from my ISP at all.  A bit of Googling later - I had some more facts and started the investigation.

This is a very interesting virus (at least to me).  It is spread by infected machines SSHing to non-infected machines.  The virus is then injected from outside the network.  Nothing is spread during the connection (I guess by definition, this stops it being a virus).

If my machine is infected and I then SSH to another server to do some maintenance work, the machine I am on sends out the username, the password, the IP address, and the port you attach on.  If any SSH connection goes in or out of the infected machine, this information gets sent.

How it gets sent is also interesting.  In an attempt to get the information out from behind firewalls and also maybe to hide it from scanners looking for suspect traffic, it sends the information as a DNS request to a server that knows the request is really an information packet to give them someone's login details: 1357924680acef123bcd.192.168.0.1

The first part (up to the first dot) is a hashed value that has the name or password in it.  This is then followed by the IP address in what looks like a valid DNS name.  The DNS request is then sent to their server.  Each time you SSH in or out, two or three of these packets get sent.  The username and password get sent in two different request packets.

Sometime after this connection is made, the server on the outside then connects back to your IP and logs in as you.

It attempts to gain access as root (which, if you have sudo access, will be pretty easy as it has your password) and, if you logged in as root then it is instant.

If it gets control, it downloads a ready built file and replaces a library used in SSH and SSHD (the client and server software on your machine or server).

On the machine I was testing (a Ubuntu desktop), it originally had these two files:

/lib/x86_64-linux-gnu/libkeyutils.so.1 /lib/x86_64-linux-gnu/libkeyutils.so.1.4

The top one is just a link file which points to the other file.

After infection it looked like this:

/lib/x86_64-linux-gnu/libkeyutils.so.1 /lib/x86_64-linux-gnu/libkeyutils.so.1.4 /lib/x86_64-linux-gnu/libkeyutils.so.1.4.0

The original file was still there, but the link file now pointed to the new version of the file (which was also about 30k in size instead of about 10k).

The machine rebooted at this point, and now all incoming or outgoing SSH was logged.

ClamAV - No hits.  RKhunter - no hits, etc., etc., etc.  Nothing was finding this virus.  It is well hidden and doesn't do anything obvious to the PC.  I think it is lying in wait - maybe for a massive zombie net powered by millions of Linux servers (damn scary thought).

So how do you know if you have it?  Good question.  The first step is to do:

$ ipcs -m

This shows you small packets of shared memory that have been put aside to allow two separate processes or programs to talk and swap data.  This allows the SSH and SSHD to report back and share the login details.

# ipcs -m ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 786433 bob 600 393216 2 dest 0x00000000 458754 bob 600 554432 2 dest 0x00000000 819203 bob 666 3048576 2 dest 0x00000000 131072 gbppr 600 28672 2 dest 0x00000000 131073 gbppr 600 393216 2 dest 0x00000000 131078 gbppr 600 393216 2 dest 0x00000000 131079 gbppr 600 323584 2 dest 0x00000000 131080 gbppr 600 323584 2 dest 0x00000000 32777 gbppr 600 524288 2 dest 0x00000000 10 gbppr 600 524288 2 dest 0x00000000 65548 gbppr 600 524288 2 dest 0x00000000 14 gbppr 600 524288 2 dest 0x00000000 17 gbppr 600 524288 2 dest 0x00000000 131090 gbppr 600 524288 2 dest 0x00000000 131091 gbppr 600 77824 2 dest 0x00000000 131092 gbppr 600 77824 2 dest 0x00000000 32789 gbppr 600 4194304 2 dest 0x00000000 32803 gbppr 600 4194304 2 dest 0x00000000 41 gbppr 600 524288 2 dest 0x00000000 49 gbppr 600 524288 2 dest 0x00000000 98355 gbppr 600 675840 2 dest 0x00000000 98356 gbppr 600 675840 2 dest 0x00000000 54 gbppr 600 524288 2 dest ...

The first two are probably fine.  They do not raise suspicion.

The third one has a couple of telltale signs.  Number one: its security value (permissions) is "666" - it is open for any process to be able to attach to and read/write.  This is pretty lapsed security and most programmers would hopefully not do it.  It is also approximately three megabytes in size.

This is probably a hit.

Note when you reboot, these shared files will not exist.  It only makes them after it has the first login to transmit.  The next step would be to check the libkeyutils files and the SSH and SSHD files.  If it looks modified, the last test is to capture some network traffic and look for strange DNS requests while you SSH in or out.  Think Wireshark / tcpdump, etc.

Another sample output of ipcs -m on an infected machine:

------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x000006e0 32763 user 666 3018428 0 dest 0x00000469 65538 root 666 4313584 0 dest 0x0000047a 131072 smmsp 666 3966496 0 dest

In order to check the owner, use the -p flag:

# ipcs -mp ------ Shared Memory Creator/Last-op -------- shmid owner cpid lpid 1769472 root 1975 1975 2129921 root 2931 2940 1736706 root 1965 1965 2162691 root 2931 2940 2195460 root 2931 2940 2228229 postgres 4011 6813

Now you can verify if sshd owns any of the shared memory segments - normally it shouldn't, so if it does you can try to inspect the system further.

If the process matches sshd:

# ipcs -m -p ------ Shared Memory Creator/Last-op PIDs -------- shmid owner cpid lpid 0 root 4162 4183 32769 root 4162 4183 65538 root 4162 4183 465272836 root 15029 17377 # ps aux | grep [cpid] root 11531 0.0 0.0 103284 828 pts/0 S+ 16:40 0:00 grep 15029 root 15029 0.0 0.0 66300 1204 ? Ss Jan26 0:00 /usr/sbin/sshd

An sshd process using shared memory segments larger than three (3) megabytes (3,145,728 bytes) and with broad permissions (666) is a strong indicator of compromise.

Now for the cleanup.

The first step is to lock them out of your machine.  SSH onto the machine (or, if possible, do it from the terminal).  Set the file back to the original.

On my example machine:

# cd /lib/x86_64-linux-gnu # rm libkeyutils.so.1 # ln -s libkeyutils.so.1.4 libkeyutils.so.1 # mv libkeyutils.so.1.4.0 libkeyutils.offline

Reboot and check again.  If it is all clear, then change all of your passwords (especially root and any users with Sudo access) and delete all SSH keys.

This is really only a temporary patch to give you some time to make sure your backups are ready for a reload.  You have no idea what else they have changed or embedded into your system while they were logged on, so please rebuild and restore from a good backup.

Final thoughts: Remember, Linux can get viruses.

There are a lot fewer than Windows has - but they do exist.

Be careful and keep your system as secure as possible.