Here is a story involving a web-based contact form that indicates that this may be the case.
A wise man once noted that only two things in life are inevitable: death and taxes.
Just how taxes and interactions with the tax authorities are handled vary widely from jurisdiction to jurisdiction, but in Norway, where I live, the default mode of contact with the tax authorities for most of us is via web forms protected by sometimes cumbersome authentication procedures and the occasional alert via SMS text message to your phone.
Note: This piece is also available without trackers but classic formatting only here.
And we're used to that the things just work with only occasional and very minor technical embarrassments for the people who operate the thing.
Then in August 2016, I tried to report a bug via the contact form at Altinn.no, the main tax authorities web site.
The report in itself was fairly trivial: The SMS alert I had just received about an invoice for taxes due contained one date, which turned out to be my birth date rather than the invoice due date. Not a major issue, but potentially confusing to the recipient until you actually log in and download the invoice as PDF and read the actual due date and other specifics.
The next time I checked my mail at bsdly.net, I found this bounce.
The meat of the message is
support@altinn.no
SMTP error from remote mail server after RCPT TO:
host mx.isp.as2116.net [193.75.104.7]: 550 5.7.23 SPF validation failed
which means that somebody, somewhere tried to send a message to support@altinn.no, but the message could not be delivered because the sending machine did not match the published SPF data for the sender domain.
SPF expands to "Sender Policy Framework", which is one of the early and still valid ways for a domain to publish which hosts are supposed to be sending mail on the domain's behalf.
There is no requirement to refuse delivery of mail from a host that is not in a domain's SPF record, and emphatically so when no such record exists. On the other hand, when a domain does publish SPF data, rejecting mail from hosts that are not in the set published is a valid and accepted action.
What happened is actually quite clear even from the part quoted above: the host mx.isp.as2116.net [193.75.104.7] tried to deliver mail on my behalf (I received the bounce, remember), and since I have no agreement for mail delivery with the owners and operators of that host, it is not in bsdly.net's SPF record either, and the delivery fails.
The bounce message contained one Received: header which tells part of the story, and after decoding the MIME-encoded part it becomes clear that it contains my bug report with some slightly odd markup.
So it's clear that what produced the bounce was that the contact form had tried to deliver mail using the address I had given the contact form.
I can hear your groans all the way from there to here.
My original bug report had not been delivered, but I thought perhaps I could help them with the other problem, so I sent off a new message to the addresses that were given as contacts in the altinn.no domain's whois info, taking care to send it from a machine that was properly authorized to send mail for my domain.
The full text of the message is available here, in Norwegian. The message includes the bounce with a short introduction that said essentially "this is the result of trying to submit a bug report via your web contact form. This is clearly an SPF problem, please look into that".
Then that message bounced, too.
The exact reason is likely buried in the configuration files for altinn.no's MX, but it could be that it has some reservations against accepting mail from what it sees as a subdomain that doesn't have an MX record of its own.
Anyway, by then I had spent far too much time on this rather trivial matter while I was supposed to be doing other things, so I quickly wrote up a summary and sent it off to the same contact addresses, this time from my gmail.com account, with the various accumulated data as attachments.
Then, as often happens when dealing with the authorities, nothing happened. For quite a while.
It was only on October 18, 2016 that my gmail.com account received a reply from the support address, which quoted my last message in full, and added their resolution text saying:
"Løsning:
Det kan se ut som du har Sender Policy Framework (SPF) aktivert på din mailserver, Det er en kjent svakhet ved vårt kontaktskjema at mailservere med SPF ikke støttes."
Translation:
"Solution:
It looks like you have Sender Policy Framework (SPF) enabled on your mailserver, It is a known weakness of our contact form that mailervers with SPF are not supported."
Once again, I can hear and fully sympathize with your groans.
This is as perfectly wrong as can be, in fact the exact opposite of right.
The obvious answer should be, as you will agree if you're still reading: The form's developer should place the user's email address in the Reply-To: field, and send the message as its own, valid local user. That would solve the problem.
So I put it to you: Is SPF, the Sender Policy Framework, simply too hard for application developers to understand?
Yes, I'm well aware that SPF also breaks traditional forwarding of the type generally used by mailing lists and a few other use cases. Just how afraid should we be when those same developers come to do battle with the followup specifications such as DKIM and (shudder) the full DMARC specification?
I anticipate your feedback in comments, or if you have things you really want me to only paraphrase in public, email.
Update 2016-12-02: While looking for something else entirely, I stumbled across the DMARC FAQ, where the next to final entry describes the exact problem described in this column. Developers should perhaps learn to read FAQs.
To wit,
Why are messages I send on behalf of visitors to my website being blocked?
This depends on how you are sending these messages. If you are simply taking the website visitor’s email address and inserting it into the “From:” header of the message, and sending that message from your own servers, then you are impersonating the domain in their email address – in a way that is indistinguishable from spammers. These practices may have worked previously – in many cases for decades – because before spam became a literally overwhelming problem, nobody checked. The most successful initial mechanisms to combat such spam were IP address-based blocklists, and so your site may have been allowed to continue because it did not appear on such a list.
For the past decade, however email authentication has been introduced as a filtering mechanism, and is increasingly being used to detect and block such messages.As a best practice, you should instead be using a domain you control in the address of the From: header, and use mechanisms like SPF, DKIM, and DMARC to show that this message is authorized to use your domain. In the example below, the site visitor’s name is shown in the descriptive part of the From: header, and the Reply-To: header is set to the website visitor’s address, but the actual address used in the From: header clearly indicates that your website is the origin of the message.
Taken from the DMARC.ORG FAQ: 4.23 Why are messages I send on behalf of visitors to my website being blocked? as of December 2. 2016.From: "John Doe via the Example Website" <service@website.example.com> Reply-To: "John Doe" <john@firstmailboxprovider.com> To: "Bob Smith" <bob@secondmailboxprovider.com> Subject: "An article I thought you would find interesting"
Posts
If SPF is too hard, whatever messaging tech replaces e-mail will surely be much easier to use correctly from a web-app -- NOT.
ReplyDeleteIn other words: giving up and moving on to some other technology than e-mail is not a solution.
The only solution is that tool-kits and frame-works contain enough pointers to correct documentation to nudge developers towards correct usage. Won't happen over night, but there is no other option.
P.S: Thinking about this, I sort of wish you had replied something like: "This is not caused by broken SPF support, it is caused by you forging sender addres on an e-mail, using MY address as sender."
ReplyDeleteMy initial reply, which sounds significantly muted compared to my inital reaction, was (quoting):
Delete"Det er verd å gjøre oppmerksom på at det er *deres* mailserver (mx.isp.as2116.net) som avviser meldingen.
Det enkleste er helt åpenbart at de som har ansvar for den serveren slår av SPF-sjekken. Da vil post sendt fra deres kontaktskjema komme gjennom, også for avsendere i domener som publiserer SPF.
At dette forholdet var kjent for dere fra før, gjør ikke saken noe bedre."
Translation:
"It's worth pointing out that it's *your* mailserver (mx.isp.as2116.net) that rejects the message
The obviously simplest approach is to have those responsible for the server disable the SPF check. That way, mail sent from your contact for will make it through, including senders in domains that publish SPF.
The fact that you knew about this already, does not improve matters."
Not a correct approach in any shape way or form, but at least it would have had messages from senders with correctly configured SPF records limp through.
The *correct* approach is in the article, initially formulated a few moments after I'd hit send on that reply.
Vel, nå er ikke mx.isp.as2116.net Altinn sin. Det er Broadnet (om du gidder å sjekke, så er AS2116 Broadnet i RIPE). Du finner den også som smtp.bluecom.no av historiske årsaker.
DeleteSå kontaktskjemaet til Altinn går altså via Broadnet SMTP
Very interesting read.
ReplyDeleteNo, SPF is not too hard. If I'm understanding correctly, SPF is not the issue in this scenario. More of a side-effect. SPF is just doing exactly what it is supposed to do! The real question is "Are contact forms too hard for application developers". In this case, yes. :P
It is not SPF that is too hard, but mail that is too complex. Too often sending and receiving domains deploying SPF and DKIM don't consider the variety of paths email can take. These include mailing lists, recipient forwarding, and "mail this article to a friend" use cases. They apply harsh policies that cause some legitimate mail to fail.
ReplyDeleteUnfortunately, it does not seem like that was the case here, as the domain administrators knew of the problem but did not consider it important enough to fix.
Similar problems have arisen with DMARC, where domains have published "reject" policies that have interfered with their users participating on mailing lists. This occurred with Yahoo a couple of years ago: http://www.pcworld.com/article/2141120/yahoo-email-antispoofing-policy-breaks-mailing-lists.html
My opinion is that we need alternatives to email for some classes of messages, particularly transactional messages (from banks, etc.) that are widely phished. If we can agree on more suitable media for those messages, maybe we can start to unwind the pressure on the email system. We still need email, but we are asking too much of it.
I too have experienced mail bounces due to SPF and abuses of my own email address in the from-address field. The fire department in Gjøvik, Norway is no exception. I never tried to make the fire department aware of this bug, as I figured my notice would never be forwarded to a competent person in the first place. As a tax payer I would demand a refund.
ReplyDelete