QUOTE
Originally posted by madsere I've not been hacked. Chkrootkit comes up blank and anyway, I'm convinced I would know for a number of reasons I don't want to disclose publicly.
It's possible to be hacked and not be able to find any trace of it on the machine itself. I haven't ever seen a spammer get this sophisticated with taking over a machine, but I have seen other examples of it.
QUOTE
If there was an open proxy on this server we would have found
1) an unknonwn process listening on a port somewhere (it doesn't, been through netstat quite extensively)
With a sophisticated hack job, netstat and other programs would have been changed to lie to you. But this is beyond the sophistication or motivation of most spammers, so it is unlikely. It is however, possible that the spammer is opening the port at a time that he/she believes you are not on to monitor the box.
QUOTE
2) traces of outgoing spam in /var/log/maillog (nothing)
An open proxy (or a closed one set up solely for the spammer's use) will not leave log entries in /var/log/maillog. The spammer connects to the proxy and back out again directly to port 25 of the spammee without bothering with your mailserver, thus no log entries.
QUOTE
3) one of the endless open-proxy tests already run on my server would surely have flagged it down
It could be a closed proxy that gives no sign of being there to outside scanners except to previously defined IP's.
QUOTE
It's possible that a client may have installed his own "sendmail" program, though I've alrady been through all client accounts many times, gone through the directories where they could possible have stored such a binary, i.e. /tmp, /var/tmp, /home/* etc. but not been able to find anything. But then again, it could be possible for someone to login, upload his binary, use it and delete it. I've checked history files, again nothing. Again, could be edited by a knowledgable user.
Uploading at a time they believe you are not monitoring, and deleting later is quite possible. I used this myself in the commission of an April Fools joke (harmless). Are your customers allowed to have binaries of any kind on the machine? How would you know what the binary did if you found one? As for history files, yes, they can be edited, but there are other ways to do things without leaving traces.
QUOTE
Question is: If this is indeed the way it happened how can we possible find the guilty? I'm really open for suggestions here but feel a bit at a dead end.
First we'd have to establish that the spam in question was in fact coming from your box, and that the headers were not forged. I'll address that below. If the spammer were just hopping in and out when you are presumably not watching the box, you could just check in the middle of the night and see if anything was going on, for instance. If your mail server uses one port or a specific range of ports below 1024, you could use a firewall to stop all packets from source ports on your machine above 1024 from going to port 25 anywhere else. That way, nothing but your mail server can send email anywhere. In the unlikely event that you have been rooted and the hacker is sophisticated enough to use tools to completely hide himself, the only way to detect what is going on is by outside traffic analysis (such as by Rackshack) or by booting off other media and doing a forensic analysis of the hard disk. Both are not fun to do and would need to get Rackshack involved, so hope this isn't the case. Most spammers are stupid gits, but there are a few who are very bright and technically talented and could pull this off.
QUOTE
Below is an example of such a spam. Avi-systems.com is my server (the green lines). The IP address matches, but the sendmail version doesn't. There is, as always, no trace of the receiving server in my maillog.
At first appearance, the spam appears likely to have your server in as a forged header. The green lines are the *second* received header on the mail, and the first appears to have unrelated mailservers, which is usually a sign of a forged header through a proxy. However, 216.218.215.35 is not listed as a spam source anywhere, appears to be a normal mail server, and if you'll notice, the "for" destination email changes on the last (topmost) Received line. This indicates that this last hop was probably due to mail forwarding, and that mx.user.kolo.net did in fact receive this email from your machine. If you have other example spams that show your server in the topmost (last hop) received header, then you can be certain that the spam did in fact originate on your machine. Note that mx.user.kolo.net says that the transmission from you box originated at port 61588, and that you helo'd as "localhost". These are not things your *mailserver* program is likely to do, but would be something that a spamming program would (high port number, meaning they probably haven't rooted you, and use of a generic HELO). The third received line is forged, which is why the sendmail version isn't right, and why it doesn't look like your usual received headers. mail.com is likely forged here because they are known to be rabidly antispam and have pissed off a large number of spammers. The IP address listed in the third received header doesn't belong to mail.com, but to Kraft Foods. Why they are being used is anybody's guess.
QUOTE
QUOTE
Received: from mx.user.kolo.net (roo.kolo.net [216.218.215.35])
by seagull.nest.org (8.8.8/8.8.8/KOLO.NET-SINK-20030423-CHECK-19990919) with ESMTP
id WAA13476
for ; Tue, 5 Aug 2003 22:58:30 -0700 (PDT)
(envelope-from offers@bestspecials.biz)
Received: from avi-systems.com ([207.218.206.118.61588] helo=localhost)
by mx.user.kolo.net with smtp (Exim 3.34 #1)
id 19kHJf-000HbQ-00
for sales@exumas.net; Tue, 05 Aug 2003 22:58:31 -0700
Received: from mail.com ([192.123.46.212])
by localhost (8.11.9/8.11.9) with ESMTP id qwert009392220
for ; Wed, 6 Aug 2003 05:58:28 +0000 (UTC)
Message-ID: <894826225152.ws892bA5Qv163G@greatbizservices.com>
From: "FlashCraft"
To: sales@exumas.net
Subject: Flash logo animation and design
Date: Wed, 6 Aug 2003 05:58:28 +0000 (UTC)
