IETF failed to account for greylisting

The potential for conflict between greylisting and sites with large pools of outgoing SMTP senders is well known and in need of resolution. Why does the SMTP RFC moving along the standards track fail to address this?

Standardization efforts rarely grab headlines. Except in rather exceptional circumstances (think Microsoft's recent ISO buyout), standards are formulated in response to specific technical needs, or as frequently happens, standards documents are written to codify existing and well known best practices.

Note: This piece is also available without trackers but classic formatting only here.
As regular readers of this column will be aware, I tend to argue that in the email domain, greylisting is one such 'best practice' that would deserve to be included in a standards or best practice document. In a way it already is, but it was never actually referred to in any standard or standards draft, and its main claim to standards compliance relied on extrapolating some crucial points in RFC2821.

The technical alibi for claiming that greylisting is a valid, RFC-compliant technique comes from reading section 4.5.4.1, Sending Strategy. It's really quite straightforward: An SMTP sender that receives a temporary error (451) when it tries to deliver is required to try again later, after a reasonable time. The RFC gives a few more details such as recommended retry intervals and time to give up trying, but does not explicitly say where those repeated attempts should come from. At the time RFC2821 was written in early 2001, it was almost certainly implicit in the formulation used that the retries would come from the same hosts, and specifying that as an explicit requirement most likely seemed more than a little redundant.

The MUST requirement in RFC2821 is what cleared the way for greylisting (see greylisting.org and the relevant parts of my PF tutorial (or the book)), and the earliest implementations started appearing from 2003 onwards.

Fast-forward a few years, and you have a situation where several large operators have set up their networks with large pools of outgoing SMTP servers and no guarantee which machine in the pool will be the next to handle a message queued for a delivery retry. Some greylisting implementations use a modified algorithm that stores or at least acts upon the subnet a greylistes message came from instead of the specific IP address, while others such as OpenBSD's spamd operate strictly on individual IP addresses.

It does not exactly take a rocket scientist to figure out that here is a potential problem, at least when it comes to the stricter implementations. Sites that do not retry from the same IP address can claim to be RFC2821 compliant, since the RFC does not contain a specific requirement that the delivery retries have to be from the same IP address. Again quoting my tutorial, the solution so far has been to whitelist those sites, extracting their SPF info for whitelisting purposes if necessary.

The large number of outgoing SMTP hosts per site problem has been widely discussed, and anybody even marginally interested in mail and spam avoidance should be aware if this. Then here's a surprise for you: Apparently the writers of RFCs are actually unaware of this, or chose not to care. On October 1, 2008, I found in my IETF mailbox RFC5321, which obsoletes RFC2821. The new RFC contains a number of things, but section 4.5.4.1, "Sending Strategy" (yes, they even kept the numbering) is unchanged.

That means that the working group were either unaware of the problem or chose not to resolve the conflict at this time. It would have been a very sensible thing to explicitly state that retries MUST come from the same IP address. The only halfway sane reason not to resolve the conflict that way would be that this would have made greylisting powerful enough to possibly go a long way towards eliminating the need for SPF and other more convoluted schemes.

I can not bring myself to believe that the working group does not have at least one member who is aware of what greylisting is and knows about that sole remaining problem that needs to be addressed. RFC5321 has almost everything else covered, so I hope the working group will listen to reason and move resolve this problem before the standard is finalized.

Update 2021-04-25: It was recently pointed out to me that per RFC6647, issued in 2012, the IETF does acknowledge the usefulness of greylisting. The documents lists a number of useful implementation recommendations.

---

In other news, this year's EuroBSDCon in Strasbourg went smoothly with a lot of good content if a somewhat smaller number of participants than the previous one. The next chance to catch my PF tutorial live will be in London on November 26th, 2008. Contact the UKUUG for details and booking.

---