I was trying to make a recent, plain Redhat Enterprise 3.0 server here work with SMTP Auth using Sendmail (just plain text AUTH).
A first step is to set the M4 configuration with the following lines:
TRUST_AUTH_MECH(`EXTERNAL LOGIN PLAIN')dnl
define(`confAUTH_MECHANISMS', `EXTERNAL LOGIN PLAIN')dnl
then rebuilding the file. (I use webmin, from www.webmin.com, for administrative tasks.)
The M4 configuration wasn't the hard part though. AUTH still wasn't working.
I finally discovered the actual problems. Sendmail relies on a program called saslauthd to perform the actual authentication. There are two important issues.
1) saslauthd is NOT turned on by default. It needs to be set to start at boot time (and of course manually started if it's not running already.)
2) There is a configuration file at /etc/sysconfig/saslauthd which by default sets a variable:
MECH=pam
To authenticate against the actual Unix user/password, change this to:
MECH=shadow
And that's it. AUTH then works as expected.
I hope this saves somebody the same headaches I had. I'm considering working up a web page to guide people through turning a "stock" EV1 system into a functional box.
Phil