Help - Search - Members - Calendar
Full Version: HOWTO: Email Bounce handling with PHP, Exim & CPanel
The Planet Forums > Control Panels > cPanel/WHM > Cpanel/WHM HOWTOs
anders0x
Recently it has become even more important to properly take care of bounced emails. The big email vendors such as hotmail, yahoo and gmail all keep track of the percentage of your email that go to non-existent mailboxes, for example from users who have changed email address. If you cross a threshold, you are likely to get a block on your IP. Not fun.

I use this simple method, and thought to share it with everyone, since I found no complete tutorial on the Internet for this task:


1. Define a way to identify your users.
I use my primary key, userid, from my database, e.g. User 12345, but it can also be a textual string, just encode any special characters.


2. Encode the user identity like this:
user-12345-bounce@yoursite.com
This technique is called Variable Envelope Return Path (VERP)


3. Include the Return-Path: header when sending emails
When you send emails from PHP from now on, you should include the Return-Path.
Example: Assume the variables $to_name, $to_email, $user_id
contains the user you are planning to send an email to:
"John Smith", "johns@gmail.com", 12345 respectively.

CODE
$return_path = "user-".((int)$user_id)."-bounce@yoursite.com";
$headers = "MIME-Version: 1.0\n";
$headers .= "From: YourSite <support@yoursite.com>\n";
$headers .= "Reply-To: YourSite <support@yoursite.com>\n";
$headers .= "X-Priority: 3\n";
$headers .= "X-MSMail-Priority: Normal\n";
$to_address = $to_name . " <" . $to_email . ">";
$result=mail($to_address, $subject, $message, $headers, "-f".$return_path );


What this brings is that you get these headers on your outgoing emails to this user:
Return-Path: <user-12345-bounce@yoursite.com>
To: John Smith <johns@gmail.com>
From: YourSite <support@yoursite.com>

Note the last argument of mail(). You need to switch SafeMode Off for that argument to be accepted. That's usually ok as you probably run this as a cronjob, so you can still keep SafeMode On for your PHP web pages if you want, and SafeMode is removed in PHP6 anyway.


4. Now invalid email addresses will bounce
If it now turns out that the email address johns@gmail.com is no longer valid, the email will bounce back to:
user-12345-bounce@yoursite.com


5. Set up a script that can process that bounce.
Create a script file at /home/yoursite/misc/bounceparse.php

CODE
#!/usr/local/bin/php -q
<?php

// Reading in the email
$fd = fopen("php://stdin", "r");
while (!feof($fd)) {
  $email .= fread($fd, 1024);
}
fclose($fd);

// Parsing the email
$lines = explode("\n", $email);
$stillheaders=true;
for ($i=0; $i < count($lines); $i++) {
  if ($stillheaders) {
    // this is a header
    $headers .= $lines[$i]."\n";

    // look out for special headers
    if (preg_match("/^Subject: (.*)/", $lines[$i], $matches)) {
      $subject = $matches[1];
    }
    if (preg_match("/^From: (.*)/", $lines[$i], $matches)) {
      $from = $matches[1];
    }
    if (preg_match("/^To: (.*)/", $lines[$i], $matches)) {
      $to = $matches[1];
    }
  } else {
    // not a header, but message
    break;
    // Optionally you can read out the message also, instead of the break:
    //$message .= $lines[$i]."\n";
  }

  if (trim($lines[$i])=="") {
    // empty line, header section has ended
    $stillheaders = false;
  }
}

list($part1,$dum1) = explode("-bounce@yoursite.com", trim($to) );
list($dum2,$user) = explode("user-", $part1);
//
// $user now contains the user id "12345" in the example
//
// Here you put in your custom code
// like open up your database connection and
// mark the user as invalid email address,
// so you don's send to him again.

return true;
?>



6. Set the permissions
The script must be executable and owned by the user who is running your site (the one you log into Cpanel with)
cd /home/yoursite/misc/
chmod a+x bounceparse.php
chown youruser:youruser bounceparse.php


7. Add a Filter
Now log into Cpanel, Go to Mail, click E-Mail Filtering and Add Filter.

Select Filter: TO CONTAINS bounce@yoursite.com
Destination: |/home/yoursite/misc/bounceparse.php

(Note the Pipe | in the beginning.)


8. Enable default address
If you set the default address to :fail: the email will never reach the Filter stage, unfortunately.
So you need to have a default address.
Go into Cpanel / Mail / Default Address and set your default address. (Actually I pipe it to null.)


9. Check that it works!
Check that it works by sending an email to one valid gmail user and one non-existing and check what happens in /var/log/exim_mainlog

=> means email sent by you to someone, e.g. you would find johns@gmail.com

That if that fails, you will get one of two behaviours, which will both be processed by your bounceparse.php script:
<= means email delivered to you, e.g. a returned bounce message to user-12345-bounce@yoursite.com
** means email that couldn't be sent non-existent mailbox discovered already at the SMTP stage


Improvement possibility 1
The only thing I'm not 100% satisfied with, is that I would have liked to :fail: instead of having a catch-all default address. Because with :fail: you reject spam already at the SMTP stage before reading the email to your server. Maybe someone can figure out how to :fail: all emails to the domain, except those with a valid address and those which have the *bounce@yoursite.com structure , I'm sure it's possible to tweak /etc/exim.conf to do that, but I leave that as future improvement for someone to post.


Improvement possibility 2
With this implementation, all bounces are treated as bad email address, and are removed from the email list, better safe than sorry. If you want to be more elaborate, you can of course use the $headers and $message fields to parse the bounce and possibly refine different issues, but in practice, my experience is that a bounce like this is enough reason to take him off the list, to not risk offending the receiptent sysadmin. The big 4 have become much more stingy.


Enjoy safe emailing! And of course add SPF records.
Different parts of this tutorial is compiled from various good sources and friends. All credits to those who deserve it!

Best Regards, Anders
http://www.phoneimage.com/
adaulton
Thanks for this post. This is almost exactly the solution I came up with. However I like yours much better. I was unaware of the concept of VERP before reading your post. I was using a forwarder instead of a filter and altered the domain of the Message-ID to handle the user identification. I.e...

CODE
$headers .= "Message-ID: <"md5(uniqid(time()))."@user-".((int)$user_id)."-bounce@yoursite.com">;


This created a challenge in extracting the user id since the message-id isn't neatly included in the header of the bounce. And I don't know what the side affects of altering the domain of a Message-ID would be or if it violates RFC standards. Which brought me to find your VERP solution. Very elegant. Thanks again.

I have been trying to add the functionality of reading the error message in the bounce (as touched on in your improvement possibility 2) and proceeding accordingly. I was looking at two possible classes that sort-of accomplish this. However, they are based on actually receiving the bounce message to a mailbox and sifting through them at some later date. Not processing them in real time as we wish to do.

http://sourceforge.net/projects/bmh/

and

http://www.phpclasses.org/browse/package/2691.html

I know that with a little tweaking we should be able to determine if a bounce is a permenant or temporary error, or just an autoreply. I plan on working on this in my spare time and I will post any progress I make here. Please let me know if you have any ideas about making this work.

Thanks again for a great tutorial,
Ash
brainjuice
QUOTE
Improvement possibility 1
The only thing I'm not 100% satisfied with, is that I would have liked to :fail: instead of having a catch-all default address. Because with :fail: you reject spam already at the SMTP stage before reading the email to your server. Maybe someone can figure out how to :fail: all emails to the domain, except those with a valid address and those which have the *bounce@yoursite.com structure , I'm sure it's possible to tweak /etc/exim.conf to do that, but I leave that as future improvement for someone to post.


I don't have any complete solution to this issue but I would suggest that you set up a subdomain like bounce.yoursite.com and use the format as follows: user-12345@bounce.yoursite.com

The advantage this provides is that you can still set the catchall account on your primary domain, where most spam is prone to be delivered, to :fail: and presumably less spam would be directed to a very specific use subdomain like bounce.yoursite.com. So, as I said, not a fool proof solution but one that should address your primary concern of not having to open the floodgates on the catchall address on your primary domain.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2009 Invision Power Services, Inc.