I have had an Ensim 3.1.x server here for just over a year. I had no problems until last night when I suddenly noticed the server wasn't responding. I put in a trouble ticket about this and support tried to reboot it. Apparently it would not reboot so they said I had to order a restore. $75 later I have a new hard disk (minus all my customers sites of course) and it doesn't work. It keeps saying that there is no disk space left even though there should be about 30 gigs. I put in another TT about this and was told it was being checked. That was over 2 hours ago.

Furthermore, I see I am being DDoSd and I cannot get anyone from support to comment on that at all. /var/log/maillog is filling up like crazy and showing about 15 incoming emails per second to randomly generated usernames at my domain, from many different IP addresses. This is most likely the actual reason that my server went down in the first place. Sendmail just keeps spawning more and more processes to handle it and the server load goes up and up.

I'm not really sure what the point of this post is except to vent and wonder why ev1 support seemingly won't try to help.

My customers' sites have been offline for well over 12 hours now and I can't even begin to restore them because I haven't been provided with a working hard disk.

I feel your pain, but your best bet to get your server back online is to be patient, acceptive of what measures RS must take to protect themselves (and you of course) before they put your box back online.

I find with issues such as these it is ALWAYS better to phone rather than use live chat or TT as it is much better method of getting answers

Are the attackers using a group of ip addresses that are being repeated or are they all unique..?

If they are a group ( even if its a 100 of them), heres an ugly potential solution that may work

I'd temproarily disable sendmail (no one can use the box anyway) if you can get SSH access long enough to do this.

service sendmail stop

Install APF or another firewall if you havent already

download your /var/log/maillog and use excel or something to isolate the attacking ip address entries, and add each UNIQUE ip address to the firewall

clear your maillog and restart sendmail

My TIPs:

after EV1Servers installed your server for you (with new hdd/image stuff like that).

-- start by securing your server.
<- if again theres a attack on your server through sendmail ->
service sendmail stop
cp -R /var/log/maillog /root/logs/badguys
pico -w /root/logs/badguys (remove the crap like sendmail starting bla bla, only provide a plain log with attacks)
mail the log to abuse department.
<- end snedmail ->

I recommend installing a firewall (APF?) and IDS (Snort).

Then it might be an idea to tell your customers about the current situation.

This is my real problem to be honest. I could probably deal with the DDoS if I had a working server. When they put the new hard disk in it had ensim 3.1.9. I upgraded to 3.1.10 fine. Then when I tried to upgrade to 3.1.11 and then 3.1.12 it gave errors about not being able install RPMs due to being out of disk space (which it wasn't). So I now have Ensim thinking it is 3.1.12 when it doesn't have the RPMs installed right. As as result Ensim won't work properly.

I have a backup of all my customers' sites made with ensimbackup 0.8 (written by a user of this forum). I made this backup with sites based on ensim 3.1.10. However, I cannot restore the sites because Ensim is all screwed. I'm trying to get them to just re-image the hard disk again but it hasn't happened yet.

its probably your maillog filling up your hard drive like you suspected.

You can use a command to find any large files on you server (this one will find files larger that 1MB

find / -size +1048576c


do this at your own risk

download the files you want to keep for debugging purposes (such as /var/log/messages/ and mailog to your local hard drive using sftp or something

Once you have a copy, you can empty them file on your server. that should give you room to install ensim

Well, I've just had the hard disk re-imaged again. This time I have stopped sendmail, upgraded ensim 3.1.9 to 3.1.10 and I am now going to have another go at restoring the web sites.

I really hope this ensimrestore script works properly.

If anyone has any bright ideas about how to deal with this DDoS it would be much appreciated.

I have port 25 firewalled by iptables. As soon as I open it up the incoming traffic shoots to about 34Mbs, loads of sendmail processes are started and the server load goes way up. If I leave it like this the server crashes.

I have spoked to support on the phone but nobody really knows how to stop it. Its been going for nearly 48 hours now.

I tried blocking all the attacker IP addresses separately but it didn't seem to work. Maybe there are more than I thought. I also deleted /var/log/maillog and it doesn't seem to get re-created any more when I open port 25 so I can't get any more IP addresses to block.

well you can recreate the maillog file

i think that can be done if you are logged in as root and issue

touch /var/log/maillog

that will at least let you get some more ip addresses to block

Note: if you remove the maillog file, you must restart sendmail to create it again. If that does not work, run "touch /var/log/maillog; chmod 0600 /var/log/maillog" as the root user and then restart sendmail.

Unfortunately, there is no easy way to "block" a ddos attack like you are experiencing. You can prevent it from crashing your site, however.

In your /etc/sendmail.cf file, there will be a line like

#O MaxDaemonChildren=12

This option will limit the number of running sendmail processes, temporarily refusing new mail if the limit is reached. Remove the # symbol to activate this option, and change the number to 20.

#O RefuseLA=12

This option will refuse connections if the load average is above a certain number. Again, remove the # and change the number to something much higher such as 40 or 50.

Save and quit, and restart the sendmail service. Try un-firewalling port 25 ... does the server still skyrocket out of control? It should stay running now, rejecting the excessive number of incoming mail connections.

Admittedly, containing this DDOS attack will also prevent valid mail from coming through. And it is likely that the relay addresses in the maillog are spoofed, so that blocking them at the firewall level actually blocks the mail servers of legitmate users.

The only thing you can do is have EV1 technicians go over your log files and work with the admins of the attack's source network to track down and block the user.

I wish you the best of luck in overcoming this attack.

I asked them to on the phone but they said I'm pretty much on my own. The number of attacking IPs is so huge and I think the attacker can get more zombies faster than I can get them shut down.

Thanks for the tips on sendmail config though. I'm going to give them a try now and see what happens.

I can't help wonder why someone would do something like this. I guess this is how some people get their kicks.

Okay, I have set them both to 20 and opened port 25. Sure enough there are exactly 20 sendmail processes and the server load is currently only about 1.00. However, email still doesn't seem to be getting through, presumably because the 20 processes are so busy dealing with the attack. I guess I'll keep increasing the number of processes until the server loads gets up to 10 or so to give myself the maximum chance of receiving the legitimate emails. Is there any risk of Ensim regenerating the sendmail.cf file while I'm asleep and crashing my server?