In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play - A Full Recipe

Recipes in our field are all too often offered with little or no commentary to help the user understand the underlying principles of how a specific configuration works. To counter the trend and offer some free advice on a common configuration, here is my recipe for a sane mail setup.

Mailing lists can be fun. Most of the time the discussions on lists like openbsd-misc are useful, entertaining or both.

But when your battle with spam fighting technology ends up blocking your source of information and entertainment (like in the case of the recent thread titled "spamd greylisting: false positives" - starting with this message), frustration levels can run high, and in the process it emerged that some readers out there place way too much trust in a certain site offering barely commented recipes (named after a rare chemical compound Cl-Hg-Hg-Cl).

I did pitch in at various points in that thread, but then it turned out that the real problem was a misconfigured secondary MX, and I thought I'd offer my own recipe, in the true spirit of sharing works for me(tm) content. So without further ado, here is

Setting Up OpenBSD's spamd(8) With Secondary MXes In Play in Four Easy Steps

Yes, it really is that simple. The four steps are:

1. Make sure your MXes (both primary and secondary) are able to receive mail for your domains
2. Set set up content filtering for all MXes, since some spambots actually speak SMTP
3. Set up spamd in front of all MXes
4. Set up synchronization between your spamds

These are the basic steps. If you want to go even further, you can supplement your greylisting and publicly available blacklists with your own greytrapping, but greytrapping is by no means required.

For steps 1) and 2), please consult the documentation for your MTA of choice and the content filtering options you have available.

If you want an overview article to get you started, you could take a peek at my longish Effective spam and malware countermeasures article (originally a BSDCan paper - if you feel the subject could bear reworking into a longer form, please let me know). Once you have made sure that your mail exchangers will accept mail for your domains (checking that secondaries do receive and spool mail when you stop the SMTP service on the primary), it's time to start setting up the content filtering.

At this point you will more likely than not discover that any differences in filtering setups between the hosts that accept and deliver mail will let spam through via the weakest link. Tune accordingly, or at least until you are satisfied that you have a fairly functional configuration.

When you're done, leave top or something similar running on each of the machines doing the filtering and occasionally note the system load numbers.

Before you start on step 3), please take some time to read relevant man pages (pf.conf, spamd, spamd.conf and spamlogd come to mind), or you could take a peek at the relevant parts of the PF FAQ, or my own writings such as The Book of PF, the somewhat shorter Firewalling with PF online tutorial or the most up to date tutorial slides with slightly less text per HTML page.

The book and tutorial both contain material relevant to the FreeBSD version and other versions based on the older syntax too (really only minor tweaks needed). In the following I will refer to the running configuration at the pair of sites that serve as my main lab for these things (and provided quite a bit of the background for The Book of PF and subsequent columns here).

As you will have read by now in the various sources I cited earlier, you need to set up rules to redirect traffic to your spamd as appropriate. Now let's take a peek at what I have running at my primary site's gateway. greping for rules that reference the smtp should do the trick:

peter@primary $ doas grep smtp /etc/pf.conf

which yields

pass in log quick on egress proto tcp from <nospamd> to port smtp
pass in log quick on egress proto tcp from <spamd-white> to port smtp
pass in log on egress proto tcp to port smtp rdr-to 127.0.0.1 port spamd queue spamd
pass out log on egress proto tcp to port smtp

Hah. But these rules differ both from the example in the spamd man page and in the other sources! Why?

Well, to tell you the truth, the only thing we achieve by doing the quick dance here is to make sure that SMTP traffic from any host that's already in the nospamd or spamd-white tables is never redirected to spamd, while traffic from anywhere else will match the first non-quick rule quoted here and will be redirected.

I do not remember the exact circumstances, but this particular construct was probably the result of a late night debugging session where the root cause of the odd behavior was something else entirely. But anyway, this recipe is offered in a true it works for me spirit, and I can attest that this configuration works.

The queue spamd part shows that this gateway also has a queue based traffic shaping regime in place. The final pass out rule is there to make sure spamlogd records outgoing mail traffic and maintains whitelist entries.

---

Update 2017-05-25: At some point after this was originally written, I revised that rule set. They now read, with no quick dance:

pass in on egress inet proto tcp from any to any port smtp \
    divert-to 127.0.0.1 port spamd set queue spamd set prio 0
pass in on egress inet6 proto tcp from any to any port smtp \
    divert-to ::1 port spamd set queue spamd set prio 0

pass in log(all) on egress proto tcp from <nospamd> to port smtp 
pass in log(all) on egress proto tcp from <spamd-white> to port smtp 

pass out on egress proto tcp from { self $mailservers } to any port smtp

---

And of course for those rules to load, you need to define the tables before you use them by putting these two lines

table <spamd-white> persist
table <nospamd> persist file "/etc/mail/nospamd"

somewhere early in your /etc/pf.conf file.

Now let's see what the rules on the site with secondary MX looks like. We type:

$ doas grep smtp /etc/pf.conf

and get

pass in log on egress proto tcp to port smtp rdr-to 127.0.0.1 port spamd
pass log proto tcp from <moby> to port smtp
pass log proto tcp from <spamd-white> to port smtp
pass log proto tcp from $lan to port smtp

which is almost to the letter (barring only an obscure literature reference for one of the table names) the same as the published sample configurations.

Pro tip: Stick as close as possible to the recommended configuration from the spamd(8) man page. The first version here produced some truly odd results on occasion.

Once again the final rule is there to make sure spamlogd records outgoing mail traffic and maintains whitelist entries. The tables, again earlier on in the /etc/pf.conf file, are:

table <spamd-white> persist counters
table <moby> file "/etc/mail/nospamd"

At this point, you have seen how to set up two spamds, each running in front of a mail exchanger. You can choose to run with the default spamd.conf, or you can edit in your own customizations.

The next works for me item is bsdly.net's very own spamd.conf file, which automatically makes you a user of my greytrapping based blacklist.

Once you have edited the /etc/rc.conf.local files on both machines so the spamd_flags= no longer contains NO (change to spamd_flags="" for now), you can start spamd (by running /usr/libexec/spamd and /usr/libexec/spamdlogd and run /usr/libexec/spamd-setup manually).

Note (update 2021-03-19): On modern OpenBSD versions, the easiest way to enable and start spamd is (assuming you have configured doas to allow your user to run rcctl:)

$ doas rcctl enable spamd

$ doas rcctl start spamd

Or if you want, reboot the system and look for the spamlogd and spamd startup lines in the /etc/rc output.

The fourth and final required step for a spamd setup with backup mail exchangers it to set up synchronization between the spamds. The synchronization keeps your greylists in sync and transfers information on any greytrapped entries to the partner spamds. As the spamd man page explains, the synchronization options -y and -Y are command line options to spamd.

So let's see what the /etc/rc.conf.local on the primary has in its spamd_flags options line:

peter@primary-gw $ doas grep spamd /etc/rc.conf.local
spamd_flags="-v -G 2:8:864 -w 1 -y bge0 -Y secondary.com -Y secondary-gw.secondary.com "

Here we see that I've turned up verbose logging (-v), for some reason I've fiddled with the greylisting parameters (-G). But more significantly, I've also set up this spamd to listen for synchronization messages on the bge0 interface (-y) and to send its own synchronization messages to the hosts designated by the -Y options.

On the secondary, the configuration is almost identical. The only difference is the interface name and that the synchronization partner is the primary gateway.

$ doas grep spamd /etc/rc.conf.local spamd_flags="-v -G 2:8:864 -w 1 -y xl0 -Y primary-gw.primary.com -Y primary.com"

With these settings in place, you have more or less completed step four of our recipe. But if you want to make sure you get all spamd log messages in a separate log file, add these lines to your /etc/syslog.conf:

# spamd
!!spamd
daemon.err;daemon.warn;daemon.info;daemon.debug /var/log/spamd

After noting the system load on your content filtering machines, restart your spamds. Then watch the system load values on the content filterers and take a note of them from time to time, say every 30 minutes or so.

Step 4) is the last required step for building a multi-MX configuration. You may want to just leave the system running for a while and watch any messages that turn up in the spamd logs or the mail exchanger's logs.

The final embellishment is to set up local greytrapping. The principle is simple: If you have one or more addresses in your domain that you know will never be valid, you add them to your list of trapping addresses with a command such as

$ doas spamdb -T -a noreply@mydomain.nx

and any host that tries to deliver mail to noreply@mydomain.nx will be added to the local blacklist spamd-greytrap to be stuttered at for as long as it takes.

Greytrapping can be fun, you can search for posts here tagged with the obvious keywords. To get you started, I offer up my published list of trap addresses, built mainly from logs of unsuccessful delivery attempts here, at The BSDly.net traplist page, while the raw list of trap email addresses is available here. If you want to use that list in a similar manner for your site, please do, only remember to replace the domain names with one or more that you will be receiving mail for.

This is the list that is used to trap the addresses I publish here with a faster mirror here. The list is already in the spamd.conf file I offered you earlier.

If you want more background on the BSDly.net list, please see the How I Run This List, Or The Ethics of Running a Greytrapping Based Blacklist page or search this blog on the obvious keywords.

By the way, what happened to the load on those content filtering machines?

---

Update 2012-05-30: Titles updated to clarify that the main feature here is the spamd(8) spam deferral daemon from the OpenBSD base system, not the identically-named program from the SpamAssassin content filtering suite.

Update 2013-04-16: Added the Pro tip: Stick as close as possible to the recommendend configuration from the spamd(8) man page. The first version here produced some truly odd results on occasion.

Update 2015-01-23: Changed the Book of PF links to point to the most recent edition.

Update 2015-08-01: Several correspondents have asked me for a useful nospamd file. Here's the one I use at bsdly.net, (http://www.bsdly.net/~peter/nospamd) collected over the years from various incidents and some SPF records extracting via dig -ttxt domain.tld.

Update 2017-05-25: In an act of procrastination while preparing slides for the upcoming BSDCan PF and networking tutorial, I added man.openbsd.org links for man page references, and edited in some minor fixes such as up to date rules.

Update 2021-03-19: Added a note about using rcctl to enable and start spamd on recent OpenBSD versions. A tangentially related incident had me review this article, and I found that it would be more useful to readers to be pointed at the recommended way to run a system. Returning readers may also be interested in the care and feeding activites for the traplist and other data we offer as described in the more recent articles Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting and Badness, Enumerated by Robots (both 2018).

And of course, if you're still wondering why OpenBSD is good for you, the slides from my OpenBSD and you presentation might help.