Measuring the effect of what you do is important. Equally important is knowing what is the measure of your actions.
A question turned up on IRC that had me thinking.
Do you have a percentage of the spam traffic you catch on your MXes? The reason I ask is I lust learned that fastmail.com claim they catch 71% of all incoming spam. Also a rate of false positives would be nice to have, but that's likely harder to measure.
My first impulse was that I would consider a seventy-one percent hit rate on the low side of what we are seeing here at bsdly.net and associated domains.
But getting actually useful data would require some thinking. That said, comparing a major mail operator that sells deliverability and promises a 71 percent catch rate for incoming spam and bsdly.net would be like comparing apples and oranges at best.
While bsdly.net (which is also known under a few other domain names) is my main mail service for my personal use and for a very select number of other people, to the rest of the world it is primarily a honeypot that generates security relevant data that other sites use, and that contributes to IP reputation rankings.
The site has been in operation in those roles for a little more than 15 years, since shortly before the original announcement in the article Hey, spammer! Here's a list for you!. When we started using the greylisting and greytrapping based setup, we saw a sharp drop in undesirable messages actually reaching inboxes, and I observed a marked decrease in load on the mail servers that did the content filtering.
Not long after I had set up our early greylisting setup, a message turned up on the openbsd-misc mailing list that pretty much matched our experience — a 95% reduction in spam in line to be treated to content filtering — so setting up precise measuring became a thing to do when we could get around to it.
Now enough with the background. It is relatively easy to extract at least some data that would give us a rough picture of the relative effectiveness of the greylisting and greytrapping versus the content filtering on receipt. The setup is very similar to the one described in the practically-oriented parts of the Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools and is part of a syncronizing multi-domain setup rougly as described in the earlier article In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play - A Full Recipe.
Using only tools found in the OpenBSD base system, I went on to collect data.
Whenever spamd(8) closes a connection it logs a message to that effect, so
$ zgrep "Nov 1" /var/log/spamd.6.gz | grep -c disconnected
Supplies the total number of connections closed by spamd(8) during November 1st, fetched from the archived log file.
$ zgrep "Nov 1" /var/log/spamd.6.gz | grep -c BLACK
provides the number of connections during the same 24 hour period initiated by hosts that were already in one of the blocklists used.
The command to get the number of connections that had cleared the first hurdle and entered greylisted status would be
$ zgrep "Nov 1" /var/log/spamd.6.gz | grep -c GREY
And the number of hosts that had been well behaved enough to enter the whitelist and be allowed to talk to the real SMTP service comes out of
$ zgrep "Nov 1" /var/log/spamd.6.gz | grep -c whitelisting
For hosts that have reached this far and did not fail the content filtering we do during receipt, we get the number with
$ doas zgrep 2022-11-02 /var/spool/exim/logs/main.log.6.gz | grep -c Completed
It is however worth noting that our MTA exim reports Completed for apparently message deliveries in both directions, so the number of received messages, or messages that did inbox is likely about thirty percent lower.
The number of messages rejected for one reason or the other, by being addressed to an undeliverable address or by failing content filtering we find with
$ doas zgrep 2022-11-02 /var/spool/exim/logs/main.log.6.gz | grep -c rejected
And finally, a side effect of a frequently run log reading script that adds hosts with certain kinds of characteristics such as not having a correct reverse DNS entry to a blocklist and kills all their connections will at times produce an unexpected disconnection while reading SMTP command message. We find those with
$ doas zgrep 2022-11-02 /var/spool/exim/logs/main.log.6.gz | grep -c unexpected
Those are hosts that somehow got past spamd(8) by behaving enough like a real SMTP server to clear greylisting. However spamd(8) does not have the ability to check for valid reverse, so that part is left in our case to check for by reading the log files at intervals.
The following table has the data for November 2022 —
The table is also available as a comma separated (CSV) file.
As I mentioned earlier, the number of connections to the outer layer spamd(8) is likely higher than what would be expected on sites that are not considered a honeypot and home to in excess of three hundred thousand imaginary friends (see The Things Spammers Believe - A Tale of 300,000 Imaginary Friends or the trackerless version.
That said, I think the data shows that catching the unwanted traffic early, and discarding as much as possible of that traffic before it reaches the resource hungry content filtering is definitely beneficial.
Even sites that do not actively bait the baddies out there would likely see noticeable energy bill savings by having their mail servers run quiter and cooler, as they definitely will after getting a greylisting, and optionally greytrapping setup in front of them. Those services have a truly low energy consumption profile.
If you found this article interesting, useful or just simply irritating, I would like to hear from you. Please use the comment field, or if you prefer, send email to nix at nxdomain dot no with a subject that at least tries to sound sensible and relevant.
As always, if you are interested in research on items mentioned in this article, I will be able to provide data for study. I will honor reasonable requests.