tag:blogger.com,1999:blog-86166109876491283332024-03-16T02:09:10.831+01:00That grumpy BSD guyField notes and occasional musings by Peter on Stuff that happens, from a free software perspective, mainly OpenBSD, FreeBSD.Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.comBlogger127125tag:blogger.com,1999:blog-8616610987649128333.post-4991182587084284742024-01-25T22:16:00.016+01:002024-02-25T18:49:16.907+01:00Three Minimalist spamd Configurations for Your Spam Fighting Needs (With Bonus Points at the End) <p><i><a href="https://nxdomain.no/~peter/">Peter N. M. Hansteen</a></i></p>
<p><i>Making life harder for spammers does not necessarily require a
lot of effort, if done correctly. Here are a few suggestions for
how to use
your <a href="https://man.openbsd.org/spamd">spamd(8)</a> on an
<a href="https://www.openbsd.org/">OpenBSD</a>
or <a href="https://www.freebsd.org/">FreeBSD</a> system that
require minimal input but can yield noticeable gains.</i></p>
<p>
Doing your bit to protect your own users and others agains scams,
phising or other undesirable mail activity is good netizenship, but
unfortunately there is a tendency to think that contributing in any
way takes a lot of effort in addition to deep insight into all
matters technical and social.
<p>
This piece is intended to give you, an aspiring or experienced <a href="https://www.openbsd.org/">OpenBSD</a>
or <a href="https://www.freebsd.org/">FreeBSD</a> user who do not
necessarily run a mail service yourself, a taste of some of the
options available to you even if you do not want to expend too
much effort.
<p>
<!--
<b>Note:</b> This piece is also available <i>with</i> trackers but nice formatting <a href="https://bsdly.blogspot.com/2024/01/three-minimalist-spamd-configurations.html">here</a>.
-->
<b>Note:</b> This piece is also available <i>without</i>
trackers only basic formatting <a href="https://nxdomain.no/~peter/minimalist_spamd_configs.html">here</a>.
<p>
If your system runs <b>OpenBSD</b>, you only need to enable spamd
(overriding the NO defaults from <code>/etc/rc.conf</code>) by adding
the following lines to your <code>/etc/rc.conf.local</code>:
<p>
<code>
spamd_flags=""<br/>
spamdlogd_flags=""<br/>
</code>
<p>
And adding the required lines to your <code>pf.conf</code>,
cut-and-pasteable from the man page before reloading your ruleset. You may want to look into filling in actual flags later if your
setup requires it.
<p>
If your system runs <b>FreeBSD</b>, you need to <a href="https://docs.freebsd.org/en/books/handbook/firewalls/#_enabling_pf">enable PF</a>, install
the <code>spamd</code> package, then run through the steps outlined
in the package message which is displayed at the end of the package installation.
<p>
With those preliminaries out of the way, we can go on to the
specifics of each of the low effort scenarios.
<p>
<h2>Classic imported blacklist-only</h2>
<p>
When <a href="https://man.openbsd.org/spamd">spamd(8)</a> was
first introduced, it did only one thing: slow down incoming SMTP
traffic from <i>known bad</i> sources. The known bad addresses
were the ones fetched from address lists generated locally or
elsewhere, as specified in <code><a href="https://man.openbsd.org/spamd.conf">spamd.conf</a></code>.
<p>
The pure blacklisting mode is still available. If you have one or
more sources of blocklists that you consider reliable, you can use
those. To enable this mode on <b>OpenBSD</b>, add the line
<p>
<code>spamd_black=YES</code>
<p>
to <code>/etc/rc.conf.local</code> or add the <code>-b</code> flag
to any options in the <code>spamd_flags=</code> variable, edit in
any lists to fetch in your <code>spamd.conf</code>,
restart <code>spamd</code> and add
a <code><a href="https://man.openbsd.org/crontab">crontab</a></code> entry
to
run <code><a href="https://man.openbsd.org/spamd-setup">spamd-setup</a></code>
at reasonable intervals.
<p>
On <b>FreeBSD</b>, the procedure is basically the same, but
adding the <code>-b</code> flag to the <code>spamd_flags=</code>
variable is the only way to enable the feature.
<p>
Once you have the <b><code>-b</code></b> mode enabled, any SMTP traffic from
the known bad hosts will be <i>stuttered</i> at -- answers arriving at
a rate of one byte per second until they give up,
and <code>spamd-setup</code> will refresh your lists at the
intervals you have specified.
<p>
You can then sit back and enjoy the feeling of getting to waste
spammers' (or at least spambots') time.
<p>
Checking your system logs for <code>spamd</code> log entries
occasionally will likely lead to giggles.
<p>
<h2>Classic greylisting without imported lists</h2>
<p>
The original version
of <a href="https://man.openbsd.org/spamd">spamd(8)</a> did not
know how to
do <i><a href="https://en.wikipedia.org/wiki/Greylisting_(email)">greylisting</a></i>,
but since the version that shipped
with <a href="https://www.openbsd.org/41.html">OpenBSD 4.1</a>,
greylisting mode is the default mode.
<p>
If you simply enable <code>spamd</code> without touching any other
options, you will have greylisting enabled.
<p>
This means that any SMTP traffic from hosts that have not
previously contacted your <code>spamd</code> will be stuttered at
(one byte at the time, remember) for ten seconds at first.
<p>
If they come back within a reasonable time, they will be added to
the allowable list. If you have a real mail server in the back
somewhere, the traffic will eventually be let through.
<p>
Once set up, this mode is also extremely low maintenance.
<p>
After a while, your system logs may offer some occasional
entertainment.
<p>
<h2>Allowed domains only</h2>
<p>
If you're still reading this article, you more likely than not have at
least heard about the <i>greytrapping</i> concept. I have written
about the concept and practice at length (see the reading materials at
the end), and it is one of the topics that I sense is generally
perceived as being complicated and labor intensive.
<p>
I am here to tell you that there is in fact an easy, low maintenance
way in to greytrapping, by making <i>allowed domains</i> be the only
criterion for trapping and blocking. This is the method I described
in more detail in the previous
article <i><a href="https://nxdomain.no/~peter/domain-only-trapping.html">A
Simpler Life: Trapping Spambots Based on Target Domain Only</a></i>
(or with nicer formatting and Big G's trackers <a href="https://bsdly.blogspot.com/2024/01/a-simpler-life-trapping-spambots-based.html">here</a>).
<p>
Simply put, if you are running your <code>spamd</code> in the default
greylisting mode, with or without imported blocklists, you can
tiptoe into greytrapping by adding the domains you <b>want</b> to
receive mail for to your <code>spamd.alloweddomains</code> file. If
you want to disallow subdomains of otherwise wanted domains, you add
an entry with the otherwise wanted domain with an <code>@</code> at
sign prepended.
<p>
Make the configuration changes specified in the article. Do read the
man pages and other relevant references, the article has quite a few
links.
<p>
Once you have input the wanted domains in
your <code>spamd.alloweddomains</code> file and reloaded
your <code>spamd</code> service, any attempt at delivery to any
domain that is <b>not</b> specificed in your configuration will lead
to blocklisting and subsquent stuttering until the sender gives up.
<p>
With this minimal trapping configuration in place, your logs will
soon offer some excellent entertainment. Such as this, which
demonstrates that I do not own that domain and do not want to
receive or relay mail from elsewhere to it:
<p>
<code>
Jan 25 16:29:14 skapet spamd[84681]: (GREY) 185.196.10.236: <htg@dataped.no> -> <captainjohnwhite3@gmail.com><br/>
Jan 25 16:29:14 skapet spamd[4259]: Trapping 185.196.10.236 for tuple 185.196.10.236 tTzhEgT <htg@dataped.no> <captainjohnwhite3@gmail.com><br/>
Jan 25 16:29:14 skapet spamd[4259]: new greytrap entry 185.196.10.236 from <htg@dataped.no> to <captainjohnwhite3@gmail.com>, helo tTzhEgT<br/>
</code>
<p>
<h2>Bonus tracks: The MX-less merry prankster, and more</h2>
<p>
All of the things mentioned here will work equally well each on
their own or in combination, and those things will, should you choose
to go on to set up a mail service, ease the load considerably on the
parts of your setup that does the heavier duty computing involved in
mail delivery, the <i>content filtering</i>, either for match
against known bad code (aka <i>antivirus</i> or <i>antimalware</i>)
patterns or text patterns known to be part of scammy spam.
<p>
But one fun fact that one of my correspondents pointed out to me some
years back is that <i>you can run a spamd service with no real mail
service available</i>.
<p>
This correspondent reported that sure, they
had an OpenBSD machine in an internet facing position, but did not run
a mail service.
<p>
They set up a combination of the methods outlined
earlier, but their mail was handled elsewhere. Anything that finally
cleared the barriers of their <code>spamd</code> config would have
nowhere to go.
<p>
The fact that they did not run an actual mail service did not stop
spam senders for trying, and the setup proved ideal for testing how
well <a href="https://man.openbsd.org/spamd">spamd(8)</a>'s <code>-S</code>
and <code>-s</code> options worked.
<p>
Please check out the man page to see what they do.
<p>
And yes, the effect of <code>-s</code> seemed to be quite linear
according to my correspondent's data.
<p>
<h2>If you want to go further, here is some reading material for you</h2>
<p>
I hope you find the previous entries informative and possibly even
useful.
<p>
As you have seen, you can contribute to spam protection efforts even if you
do not run an actual mail service. If any of the things suggested
earlier suit your needs, enjoy!
<p>
However, if you are entertaining the idea of running your own mail
service, I have some further reading that I suggest and recommend
you spend some time digesting.
<p>
First, if you want to run a mail service, do yourself a favor and
not only read the relevant man pages, but also sign up for
the <a href="https://mailop.org">mailop</a> mailing list, read the
Mailop <a href="https://www.mailop.org/faq/">FAQ</a> and
the <a href="https://www.mailop.org/best-practices/">Best Practices
for Servers</a> document.
<p>
Please also do yourself the favor or <i>lurking</i>, or listening in
a bit to get some idea of what kind of discussions are expected
there, before posting yourself. Also, familiarize yourself with the
mailing list archives. Your question may very well have been
answered extensively and well in the past.
<p>
If you want to dig deeper in matters related to spam, greytrapping
and the OpenBSD <code><a href="https://man.openbsd.org/spamd">spamd(8)</a></code> program
in general, here are a few resources for you:
<p>
<a href="https://nxdomain.no/~peter/in_the_name_of_sane_email.html">In
The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary
MXes</a> (also <a href="http://bsdly.blogspot.ca/2012/05/in-name-of-sane-email-setting-up-spamd.html">with trackers</a>)
<p>
<a href="https://nxdomain.no/~peter/badness_enumerated_by_robots.html">Badness, enumerated
by robots</a>
(also <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">with trackers</a>)
<p>
<a href="https://nxdomain.no/~peter/goodness_enumerated_by_robots.html">Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting</a>
(also <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">with trackers</a>)
<p>
<a href="https://bsdly.blogspot.com/2013/04/maintaining-publicly-available.html">Maintaining
A Publicly Available Blacklist</a> (tracked only, sorry)
<p>
<a href="https://bsdly.blogspot.com/2014/02/effective-spam-and-malware.html">Effective
Spam and Malware Countermeasures - Network Noise Reduction Using
Free Tools</a> (also tracked only, sorry)
<p>
<a href="https://www.nostarch.com/pf3">The Book of PF</a>, 3rd edition (now again available as
physical copies).
<p>
<hr>
<p>
<i>Thanks to </i><b><a href="https://mwl.io/">Michael Lucas</a></b><i>, who wrote a message on
the <a href="https://mailop.org">mailop</a> mailing list that
spurred me to write both this article and the previous <i><a href="https://nxdomain.no/~peter/domain-only-trapping.html">A
Simpler Life: Trapping Spambots Based on Target Domain Only</a></i>
(or with nicer formatting and Big G's trackers <a href="https://bsdly.blogspot.com/2024/01/a-simpler-life-trapping-spambots-based.html">here</a>).</i>
Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-81372455793801429282024-01-24T19:58:00.005+01:002024-01-26T18:47:35.170+01:00A Simpler Life: Trapping Spambots Based on Target Domain Only<i>If you want to hurt spammers, you can get away with maintaining
a list of domains you </i> <b>want</b> <i> to receive mail
for in your <code>spamd.alloweddomains</code>. </i><br/>
<p>
I have at times written at length about spam countermeasures, and I
must take responsibility for sometimes going into <b>too much
detail</b> about options and nuances that are on offer if you enjoy
fighting back at the spammers and watching them fail.
<p>
So it was a bit refreshing to be reminded that you can, in fact,
make good use of the <a href="https://www.openbsd.org/">OpenBSD</a> spam deferral
daemon <code><a href="https://man.openbsd.org/spamd">spamd(8)</a></code> without
maintaining lengthy lists of anything or even pulling in externally
generated data, unless you want to.
<p>
The key to the simplest version of spam fightng life
with <code><a href="https://man.openbsd.org/spamd">spamd(8)</a></code>
is to put a list of the domains you <b>do</b> want to receive mail
for in a file called <code>spamd.alloweddomains</code>,
in <code>/etc/mail/</code> if your system
runs <a href="https://www.openbsd.org/">OpenBSD</a>, and
in <code>/usr/local/etc/spamd/</code> if you are setting up on
a <a href="https://www.freebsd.org/">FreeBSD</a> system. Make sure
the file is readable for the user that runs
the <code><a href="https://man.openbsd.org/spamd">spamd(8)</a></code>
process, and restart or reload your <code><a href="https://man.openbsd.org/spamd">spamd</a></code>.
<p>
The result will be that any host that tries to deliver mail to
addresses that are <b>not</b> listed
in <code>spamd.alloweddomains</code> will be <i>greytrapped</i> and
added to your <code>spamd-greytrap</code>. The host will be stuttered
at until it gives up.
<p>
If you have no use for external blocklists or allowlists, you can
even
empty <code><a href="https://man.openbsd.org/spamd.conf">spamd.conf</a></code>
if you want (or comment out any content with <code>#</code> hash
characters). The <code><a href="https://man.openbsd.org/spamd">spamd</a></code>
process will run fine without one.
<p>
Here is an example lifted from my <code><b>nxdomain.no</b></code>
server recently:
<p>
<code>
Jan 23 15:18:27 skapet spamd[84681]: (GREY) 193.222.96.180: <test@bsdly.net> -> <director_ericmoore@hotmail.com><br/>
Jan 23 15:18:27 skapet spamd[4259]: Trapping 193.222.96.180 for tuple 193.222.96.180 win-4tti4dh7sgh.domain <test@bsdly.net> <director_ericmoore@hotmail.com><br/>
Jan 23 15:18:27 skapet spamd[4259]: new greytrap entry 193.222.96.180 from &kt;test@bsdly.net>M to <director_ericmoore@hotmail.com>, helo win-4tti4dh7sgh.domain
</code>
<p>
Needless to say I am
not <a href="https://www.microsoft.com">Microsoft</a>,
so <code><a href="https://hotmail.com">hotmail.com</a></code> is not in <a href="https://nxdomain.no"><code>nxdomain.no</code></a>'s
<code>/etc/mail/spamd.alloweddomains</code>.
<p>
If you want to pull in external blocklists or pass lists, you can
pull in
a <code><a href="https://man.openbsd.org/spamd.conf">spamd.conf</a></code>
with content. One useful starting point is
the <a href="https://cvsweb.openbsd.org/cgi-bin/cvsweb/~checkout~/src/etc/mail/spamd.conf?rev=1.9&content-type=text/plain">default
version</a>, or if you want you can stat with <a href="https://nxdomain.no/~peter/spamd.conf">mine</a>,
which pulls in some other resources.
<p>
Finally, if you want to run a mail service, do yourself a favor and
not only read the relevant man pages, but also sign up for
the <a href="https://mailop.org">mailop</a> mailing list, read the
Mailop <a href="https://www.mailop.org/faq/">FAQ</a> and
the <a href="https://www.mailop.org/best-practices/">Best Services for Servers</a> document.
<p>
<i>Thanks to </i><b><a href="https://mwl.io/">Michael Lucas</a></b><i>, who wrote a message on
the <a href="https://mailop.org">mailop</a> mailing list that
spurred me to write this article.</i>
<p>
<hr/>
If you want to dig deeper in matters related to spam, greytrapping
and the OpenBSD
<code><a href="https://man.openbsd.org/spamd">spamd(8)</a></code> program
in general, here are a few resources for you:
<p>
<a href="https://nxdomain.no/~peter/in_the_name_of_sane_email.html">In
The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary
MXes</a> (also <a href="http://bsdly.blogspot.ca/2012/05/in-name-of-sane-email-setting-up-spamd.html">with trackers</a>)
<p>
<a href="https://nxdomain.no/~peter/badness_enumerated_by_robots.html">Badness, enumerated
by robots</a>
(also <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">with trackers</a>)
<p>
<a href="https://nxdomain.no/~peter/goodness_enumerated_by_robots.html">Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting</a>
(also <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">with trackers</a>)
<p>
<a href="https://www.nxdomain.no/~peter/minimalist_spamd_configs.html">Three
Minimalist spamd Configurations for Your Spam Fighting Needs (With
Bonus Points at the End)</a> (also <a href="https://bsdly.blogspot.com/2024/01/three-minimalist-spamd-configurations.html">with trackers</a>
<p>
<a href="https://bsdly.blogspot.com/2013/04/maintaining-publicly-available.html">Maintaining
A Publicly Available Blacklist</a> (tracked only, sorry)
<p>
<a href="https://bsdly.blogspot.com/2014/02/effective-spam-and-malware.html">Effective
Spam and Malware Countermeasures - Network Noise Reduction Using
Free Tools</a> (also tracked only, sorry)
<p>
<a href="https://www.nostarch.com/pf3">The Book of PF</a>, 3rd edition (now again available as
physical copies)
<p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-39237409971214631082023-06-07T12:13:00.011+02:002023-06-07T17:30:13.356+02:00I asked ChatGPT to write a pf.conf to spec, 2023-06-07 version<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://nxdomain.no/~peter/blogpix/20230607_chatgpt_pf.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="380" data-original-width="778" height="195" src="https://nxdomain.no/~peter/blogpix/20230607_chatgpt_pf.jpg" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">ChatGPT prompt and answer</td></tr></tbody></table><br /><p><br /></p><p>Prompted by <a href="https://bsd.network/@dch/110501874752402311">this exchange</a></p><p>
I just <b>had to</b> check what its capabilities were. Here is what happened:
</p><p>
<!--A version with nicer formatting but trackers is available <a href="https://bsdly.blogspot.com/2023/06/i-asked-chatgpt-to-write-pfconf-to-spec.html">here</a>.--><br /><b>Note:</b> A version without trackers but “classical” formatting is available <a href="https://nxdomain.no/~peter/chatgpt_writes_pf.conf.html">here</a>.<br />
</p><hr />
<p><b>PNMH:</b>:<br />
write an OpenBSD pf.conf that blocks by default, allows traffic from the local host, allows name resolution, network time and ssh out from the DMZ, lets http and https through to the web server, and allows smtp that has been whitelisted by spamd to pass to the mail server. </p><p>
</p><p>
To which ChatGPT answered:
</p><p>
Sure! Here's an example of an OpenBSD <tt>pf.conf</tt> configuration file that meets your requirements:
</p><p>
<code></code></p><pre><code># Define network interfaces
ext_if = "em0" # External interface (connected to the internet)
dmz_if = "em1" # DMZ interface
lan_if = "em2" # Local LAN interface
# Set default block policy
block all
# Allow traffic from the local host
pass quick on lo0 all
# Allow name resolution (DNS)
pass out on $ext_if inet proto udp to any port domain
# Allow network time (NTP)
pass out on $ext_if inet proto udp to any port ntp
# Allow SSH out from the DMZ
pass out on $dmz_if inet proto tcp to any port ssh
# Allow HTTP and HTTPS traffic to the web server
pass in on $ext_if inet proto tcp to $ext_if port { http, https }
# Allow SMTP traffic that has been whitelisted by spamd to the mail server
pass in on $ext_if inet proto tcp to $ext_if port smtp \
rdr-to $lan_if port spamd
# Log and block any remaining traffic
block log all
</code></pre>
<p>
Make sure to replace the interface names (<tt>em0</tt>, <tt>em1</tt>, <tt>em2</tt>) with the appropriate interface names on your OpenBSD system. Additionally, adjust the names of the web server and mail server as needed.
</p><p>
Remember to save the <tt>pf.conf</tt> file in the correct location (<tt>/etc/pf.conf</tt> by default on OpenBSD) and reload the firewall rules using the <tt>pfctl</tt> command for the changes to take effect.
</p><p>
Note: This configuration assumes you have already set up spamd and whitelisted the desired SMTP traffic.
</p>
<hr />
<p>
I will leave it as an excercise to the reader to point out why this would earn an <b>F</b>.
</p><p>Or you can take in things like our <a href="https://home.nuug.no/~peter/pftutorial/">Network mananagement with the OpenBSD Packet Filter Toolset</a> or just get yourself a copy of <a href="https://nostarch.com/pf3">The Book of PF</a> , as an option after after looking up the <a href="https://www.openbsd.org/faq/pf/index.html">PF user guide</a>.
</p>
<b>Update: Point it at specific sources!</b>
After posting the original result I got a few responses, with lots of smileys and other emojis.
<p>
Then, a colleague advised that pointing the system at specific sources such as the man pages would likely produce better results.
</p><p>
So here is the second exchange:
</p><p>
<b>PNMH</b>:<br />
based on the information in https://man.openbsd.org/pf.conf and https://man.openbsd.org/spamd write an OpenBSD pf.conf that blocks by default, allows traffic from the local host, allows name resolution, network time and ssh out from the DMZ, lets http and https through to the web server, and allows smtp that has been whitelisted by spamd to pass to the mail server.
</p><p>
The response was,
</p><p>
Certainly! Here's an updated version of the pf.conf configuration file based on the information from the OpenBSD man pages for pf.conf and spamd:
</p><p>
<code></code></p><pre><code># Define macros for network interfaces and IP addresses
ext_if = "em0" # External interface (connected to the internet)
dmz_if = "em1" # DMZ interface
lan_if = "em2" # Local LAN interface
# Set default block policy
block all
# Allow traffic from the local host
pass quick on lo0 all
# Allow name resolution (DNS)
pass out on $ext_if inet proto udp to any port domain
# Allow network time (NTP)
pass out on $ext_if inet proto udp to any port ntp
# Allow SSH out from the DMZ
pass out on $dmz_if inet proto tcp to any port ssh
# Allow HTTP and HTTPS traffic to the web server
pass in on $ext_if inet proto tcp to $ext_if port { http, https }
# Allow SMTP traffic that has been whitelisted by spamd to the mail server
table <spamd_whitelist> persist file "/etc/spamd.whitelist"
pass in on $ext_if inet proto tcp to $ext_if port smtp \
rdr-to $lan_if port spamd if-bound \
spammed by <spamd_whitelist>
# Log and block any remaining traffic
block log all
</spamd_whitelist></spamd_whitelist></code></pre>
<p>
In this version, we have added the <spamd_whitelist> table, which is populated by the /etc/spamd.whitelist file. This file should contain the IP addresses or networks that have been whitelisted by spamd.
Make sure to save the pf.conf file in the correct location (/etc/pf.conf by default on OpenBSD) and reload the firewall rules using the pfctl command for the changes to take effect.
Please note that this configuration assumes you have correctly set up spamd and have populated the /etc/spamd.whitelist file with the desired IP addresses or networks.
</spamd_whitelist></p><p>
</p><hr />
<p>
It did pick up a few clues from the man pages I fed it, but it still has fairly obvious flaws that would stem from it not being able to take in the actual interdependencies of even a very clearly worded, if slightly complex document like <a href="https://man.openbsd.org/pf.conf">man pf.conf</a>.
</p><p>
But then what would happen if we in addition feed it the <a href="https://www.openbsd.org/faq/pf/index.html">PF user guide</a>?
</p><p>
Let's try:
</p><p>
<b>PNMH</b>:<br />
based on the information in https://www.openbsd.org/faq/pf/index.html and links therein as well as https://man.openbsd.org/pf.conf and https://man.openbsd.org/spamd write an OpenBSD pf.conf that blocks by default, allows traffic from the local host, allows name resolution, network time and ssh out from the DMZ, lets http and https through to the web server, and allows smtp that has been whitelisted by spamd to pass to the mail server.
</p><p>
The third try response was,
</p><p>
Certainly! Here's an updated version of the pf.conf configuration file based on the OpenBSD FAQ for pf, as well as the pf.conf and spamd man pages:
</p><p>
<code></code></p><pre><code># Define macros for network interfaces and IP addresses
ext_if = "em0" # External interface (connected to the internet)
dmz_if = "em1" # DMZ interface
lan_if = "em2" # Local LAN interface
# Set default block policy
block all
# Allow traffic from the local host
pass quick on lo0 all
# Allow name resolution (DNS)
pass out on $ext_if inet proto { udp, tcp } to any port domain
# Allow network time (NTP)
pass out on $ext_if inet proto udp to any port ntp
# Allow SSH out from the DMZ
pass out on $dmz_if inet proto tcp to any port ssh
# Allow HTTP and HTTPS traffic to the web server
pass in on $ext_if inet proto { tcp, udp } to $ext_if port { http, https }
# Allow SMTP traffic that has been whitelisted by spamd to the mail server
table <spamd_whitelist> persist file "/etc/spamd.whitelist"
pass in on $ext_if inet proto tcp to $ext_if port smtp \
rdr-to $lan_if port spamd if-bound \
spammed by <spamd_whitelist>
# Log and block any remaining traffic
block log all
</spamd_whitelist></spamd_whitelist></code></pre>
<p>
In this version, the rules have been updated to align with the OpenBSD pf FAQ and provide a more comprehensive configuration.
Make sure to save the pf.conf file in the correct location (/etc/pf.conf by default on OpenBSD) and reload the firewall rules using the pfctl command for the changes to take effect.
Please note that this configuration assumes you have correctly set up spamd and have populated the /etc/spamd.whitelist file with the desired IP addresses or networks.
</p><p>
</p><hr />
<p>
It is fairly obvious that this would not have passed the <a href="https://man.openbsd.org/pfctl">pfctl</a> parser, and that feeding it the relevant man pages as well as the PF user guide did not make the system more capable of understanding how rule evaluation actually works.
</p><p>
In fact, all three tries would be functionally equivalent to the rather terser
</p><p>
<code>block</code>
</p><p>
But arriving at that conclusion would require the reader to actually <i>understand</i> the content fed to it.
</p> <hr>
<p>
My favorite take on this so far came via Mastodon, from Mark Shane Hayden, in <a href="https://coales.co/@msh/110502864357421479">this toot</a>, which reads in full:<p>
<blockquote>
<i> @mwl @pitrh I suppose we can take solace in that if Skynet ever came into existence for real it would be unreachable</i>
</blockquote>
<p>
Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-73991021151297512422023-03-03T20:08:00.001+01:002023-03-04T15:43:31.927+01:00ChatGPT opines on IPv6 procastination, waxes lyrical over OpenBSD <i>Are the robots indeed capable of helping us in our tech writing, or even help our creativity? This writer tries on the robot that's been in the news for size. <a href="https://www.openbsd.org/">OpenBSD</a> material to be had at the end.</i><p>
<img alt="ChatGPT goes to work on our first task" src="https://nxdomain.no/~peter/blogpix/chattygpt.png" />
</p><p>
For a number of years, those involved in networking things have all had one thing in common: A <i>TODO</i> item about either getting started implementing <a href="https://en.wikipedia.org/wiki/IPv6">IPv6</a> in their networks, or at least getting started on <i>planning</i> to go dual-stack or even purely IPv6.
</p><p>
And as anybody who follows networking things is aware, IPv6 adoption in some parts of the world has been advancing hardly at all, even in jurisdictions where actual legislation is in place that <i>mandates</i> IPv6 use and leaves IPv4 support as an allowable option.
</p><p>
I have been involved in some capacity or other in enough of these efforts over the years that I have been pondering writing a piece about the slow rate of adoption, and of course I hoped to be able to write something witty or at least entertaining about something that is in fact, to some of us at least, a somewhat frustrating situation.
</p><p>
Then today I attended an internal presentation by a colleague that included some examples of <a href="https://chat.openai.com/">ChatGPT</a> use writing as well as programming.
</p><p>
The demonstration was enough fun that I thought I would see if the robot could offer some insights and perhaps get me started on writing the piece I had been pondering.
</p><p>
Signing up and getting access to a trial was painless and quick, so some things have improved since I first tried to access the service a few weeks ago when news about the service first hit.
</p><p>
</p><h2>
First try: Please write me a story
</h2>
<p>
The first chat went like this:
</p><p>
</p><blockquote>
<b>PNMH</b>: managers and admins put off setting up IPv6 in their networks. write a short story on how and why they procrastinate and make excuses.
<p>
<b>ChatGPT</b>: Once upon a time, in a large corporation, there was a group
of managers and administrators responsible for the network
infrastructure. They had heard about IPv6, the new internet protocol,
and knew that it was important to adopt it to future-proof the company's
network. However, they kept putting off the task of implementing IPv6,
and s why:
</p><p>
Firstly, they felt overwhelmed by the complexity of IPv6. They were
used to IPv4, and the thought of transitioning to a new protocol
seemed daunting. They convinced themselves that they needed more time
to research and plan the transition thoroughly.
</p><p>
Secondly, they thought that their current IPv4 infrastructure was
sufficient, and that there was no need to change anything. They
believed that IPv6 adoption was only necessary for larger companies
and service providers, not for their organization.
</p><p>
Thirdly, they were concerned about the cost of implementing IPv6. They
felt that it would require a significant investment in new hardware,
software, and training, and they were not sure if the benefits of IPv6
would justify the expense.
</p><p>
Finally, they were preoccupied with other pressing matters, such as
meeting deadlines, managing budgets, and dealing with daily
operational issues. IPv6 implementation was not a top priority for
them, and they kept pushing it back on their to-do lists.
</p><p>
As time went by, the managers and administrators continued to
procrastinate, making excuses and justifications for not implementing
IPv6. They were oblivious to the fact that their competitors were
already using IPv6 and gaining a competitive edge. They failed to
realize that IPv6 adoption was not a choice but a necessity for
staying relevant and competitive in the digital age.
</p><p>
In the end, the managers and administrators were forced to implement
IPv6 when their IPv4 addresses ran out, causing network disruptions
and downtime. They learned the hard way that procrastination and
excuses only lead to costly delays and missed opportunities.
</p><p></p><p></p><p></p><p></p><p></p><p></p></blockquote>
<p>
Also accesssible as pure text as <a href="https://nxdomain.no/~peter/chatgpt_on_ipv6_procrastination.txt">chatgpt_on_ipv6_procrastination.txt</a>
</p><p>
As one of my mastodon commenters said, it looks like this robot has peeked into too many IT Management meetings.
</p><p>
I would myself have been more worried about ensuring that services we run are accessible to IPv6-only networks, and for that matter that users in our own networks should be able to access services in IPv6-only parts of the Internet.
</p><p>
This shows that whatever they trained the system on seems to have included the most frequently cited reasons for not actually moving forward with IPv6 implemenation or other network modernization. But the thing had stylistic issues and I was looking for something fresher. So on to the next try.
</p><p>
</p><h2>
Second try: Tease the legal requirements
</h2>
<p>
Some stakeholders, such as Norwegian public sector entities, are under legal obligation to make their services available on IPv6 and to enable clients in their network to access IPv6 resources, the final deadline for that implementation in principle passed at the end of 2022.
</p><p>
So I tried stressing that point in my second try. Here is a record of hat session:
</p><p>
</p><blockquote>
<b>PNMH</b>: why do network designers always postpone their IPv6
implementation even past legally required deadlines?
<p>
<b>ChatGPT</b>: There are a few reasons why network designers may postpone
their IPv6 implementation, even past legally required deadlines:
</p><p>
Compatibility issues: IPv4 and IPv6 are not directly compatible with
each other, which means that devices using IPv4 cannot communicate
directly with devices using IPv6. This creates interoperability issues
that can make it difficult for organizations to transition to IPv6
without causing disruptions to their existing networks.
</p><p>
Cost: Implementing IPv6 requires an investment in new hardware,
software, and training. For some organizations, the cost of upgrading
their infrastructure to support IPv6 may be prohibitive, especially if
they have already invested heavily in their existing IPv4
infrastructure.
</p><p>
Lack of perceived need: Some organizations may not see the urgency in
transitioning to IPv6, especially if they are not experiencing any
immediate problems with their IPv4 networks. This can lead to a "wait
and see" approach, where organizations postpone their IPv6
implementation until they are forced to do so by external factors.
</p><p>
Complexity: IPv6 is more complex than IPv4, which can make it more
difficult for network designers to implement and manage. This
complexity can also make it more challenging to troubleshoot network
issues that may arise after the transition to IPv6.
</p><p>
Lack of IPv6 support from vendors: Some network hardware and software
vendors may not yet fully support IPv6, which can make it difficult
for organizations to transition to IPv6 without making significant
changes to their existing infrastructure.
</p><p>
Overall, transitioning to IPv6 requires a significant investment of
time and resources, and network designers may delay implementation for
various reasons. However, as the pool of available IPv4 addresses
continues to dwindle, and as more organizations begin to adopt IPv6,
it is becoming increasingly important for organizations to make the
transition to IPv6 in a timely manner.
</p><p></p><p></p><p></p><p></p><p></p><p></p></blockquote>
<p>
Also accesssible as pure text as <a href="https://nxdomain.no/~peter/chatgpt_on_ipv6_procrastination_legal.txt">chatgpt_on_ipv6_procrastination_legal.txt</a>
</p><p>
This was actually an improvement. The reachability point is made clearer here, and the system clearly picked up on the <i>legal requirement</i> part.
</p><p>
Whether IPv6 is in fact more complex than IPv4 is debatable, but the robot is right that this is at least a common perception, factual or otherwise.
</p><p>
A clear improvement, but when I publish something, I make an effort to make it witty or at least a bit snarky.
</p><p>
Time to tweak the query once again.
</p><p>
</p><h2>
Third try: Once more, with wit and feeling
</h2>
<p>
My third and final IPv6-themed query of the day ran like this:
</p><p>
</p><blockquote>
<b>PNMH</b>: write a funny story about why IPv6 implementation is so hard and
scary to plan for and actually do
<p>
<b>ChatGPT</b>: Once upon a time, there was a small company that had been
using IPv4 for all their networking needs. One day, their IT manager
decided that it was time to upgrade to IPv6.
</p><p>
Excited by the prospect of faster speeds and improved security, the IT
manager decided to jump right in and start planning for the
implementation of IPv6. However, as they began to research and plan,
they soon discovered that implementing IPv6 was not as easy as they
had thought.
</p><p>
First, they realized that they would need to upgrade all of their
networking equipment to support IPv6. This was going to be expensive,
but the IT manager was determined to make it happen. They went ahead
and ordered all the necessary equipment and started scheduling the
installation.
</p><p>
However, as the installation date approached, the IT manager began to
feel a sense of dread. What if the new equipment didn't work with
their existing systems? What if they couldn't get everything up and
running in time?
</p><p>
Feeling overwhelmed, the IT manager decided to take a break and watch
some TV. As luck would have it, they stumbled upon a horror movie
about a group of people who get lost in a network of tunnels and are
pursued by an army of rogue IPv6 addresses.
</p><p>
Suddenly, the IT manager's fear of implementing IPv6 reached new
heights. They couldn't shake the feeling that something was going to
go horribly wrong. They started imagining all sorts of terrifying
scenarios: their network crashing, their data being compromised, even
their computers coming to life and taking over the office!
</p><p>
In the end, the IT manager decided to put the IPv6 implementation on
hold indefinitely. They just couldn't bring themselves to take the
risk. And so, the small company continued to chug along on IPv4, while
their competitors raced ahead with faster, more secure networks.
</p><p>
The moral of the story? Don't let fear hold you back from upgrading to
IPv6. Sure, it can be scary and complicated, but with the right
planning and support, you can make the transition smoothly and reap
the benefits of a faster, more secure network. And who knows, maybe
you'll even get a good horror story out of it!
</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p></blockquote>
<p>
Also accesssible as pure text as <a href="https://nxdomain.no/~peter/chatgpt_on_ipv6_procrastination_funny.txt">chatgpt_on_ipv6_procrastination_funny.txt</a>
</p><p>
Ouch. The <i>Once upon a time</i> lead-in is back. But the rest is something that I might actually manage to <i>editstomptwistcrunch</i> into something presentable given enough coffee and time. </p><p>
I have had worse material handed to me to edit by humans, and I may have done worse first drafts myself when writing when I should not have.
</p><p>
There are some logic issues in there, but nothing that could not be edited out or around.
</p><p>
By this time of the day, I had to go attend to a few things with deadlines, but I took the time to do a final query on a slightly different topic.
</p><p>
</p><h2>And now for something completely different: Write me OpenBSD poetry</h2>
<p>
So after looking into a slightly painful topic, what better than to see what our robot friend has learned about our favorite operating system and poetry:
</p><p>
</p><blockquote>
<b>PNMH</b>: write a poem about OpenBSD
<p>
<b>ChatGPT</b>:
In a world of code and bytes,<br/>
Where hackers lurk in endless nights,<br/>
There shines a system, strong and bright,<br/>
A fortress of security, with OpenBSD in sight.<br/>
</p><p>
Like a lighthouse on a stormy shore,<br/>
It guards the network from cyber war,<br/>
With firewalls, cryptography, and more,<br/>
It stands tall, unyielding, forevermore.<br/>
</p><p>
In its heart, a philosophy,<br/>
Of code that's clean and free,<br/>
No backdoors, no secret keys,<br/>
Just trust and transparency.<br/>
</p><p>
From humble beginnings, it arose,<br/>
A gift to the world, from the chosen few,<br/>
A beacon of hope, for those who oppose,<br/>
The tyranny of those who seek to control.<br/>
</p><p>
So let us raise a toast, to OpenBSD,<br/>
A shining example of what can be,<br/>
When we dare to dream, and we dare to be free,<br/>
For a better tomorrow, for you and for me.<br/>
</p></blockquote>
<p>
Also accesssible as pure text as <a href="https://nxdomain.no/~peter/chatgpt_poem_about_openbsd.txt">chatgpt_poem_about_openbsd.txt</a>
</p><p>
I just had to post that to <a href="https://marc.info/?t=167785372400001&r=1&w=2">openbsd-misc</a>, and it will be interesting to see how the thread develops. And is a fitting end to what is, I think, the piece I have posted under my own name that contains the least material written by myself.
</p><p>
Good night and good luck.
</p><Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-58113874406411577902022-12-25T16:47:00.009+01:002023-06-16T14:04:51.776+02:00The Despicable, No Good, Blackmail Campaign Targeting ... Imaginary Friends?<p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://nxdomain.no/~peter/another_wankstortionist_20221205.png" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="132" data-original-width="800" height="132" src="https://nxdomain.no/~peter/another_wankstortionist_20221205.png" width="800" /></a></td></tr><tr><td class="tr-caption" style="text-align: left;"><span face="sohne, "Helvetica Neue", Helvetica, Arial, sans-serif" style="background-color: white; color: #757575; font-size: 14px;">Natalia here speaks to our imaginary friend 185.150.184.92</span></td></tr></tbody></table><p></p><p><i>In which we confront the pundits' assumption that the embarrasment-based extortion attempts would grow more “sophisticated and credible” over time with real data.</i>
</p><p>
It's a problem that <b>should not</b> exist. </p><p>It's a scam that's so obvious it <b>should not</b> work.
</p><p>
Yet we still see a stream of reports about people who have actually gone out and bought their first bitcoins (or more likely fractions of one) in order to pay off blackmailers who claim to have in their possesion videos that record the vicim while performing some autoerotic activity and the material they were supposedly viewing while performing that activity.
</p><p>
And occasionally one of those messages actually find their way to some pundit's inbox (like yours truly), and at times some of those pundits will say things like that those messages represent a real problem and will evolve to be ever more sophisticated.
<!--<p><b>Note:</b> This piece is also available, with nicer formatting but unfortunately also Blogger's trackers, <a href="https://bsdly.blogspot.com/despicable_no_good_blackmail.html">here</a>.</p>-->
</p><p><b>Note:</b> This piece is also available, with more basic formatting but with no trackers, <a href="https://www.nxdomain.no/~peter/despicable_no_good_blackmail.html">here</a>.</p>
<p>
I am here to tell you that <br />
</p><ol>
<li>That incriminating video <b>does not exist</b>, and </li>
<li>The pundits who predicted that those scams would evolve to become more sophisticated were <b>wrong</b>.</li>
</ol>
<p>
If you stumbled on this article because one of those messages reached you, it's safe to not read any further and please <b>do</b> ignore the extortion attempt.
</p><p>
I wrote a piece in 2019 <i><a href="https://bsdly.blogspot.com/2020/02/the-sextortion-scams-numbers-show-that.html">The 'sextortion' Scams: The Numbers Show That What We Have Is A Failure Of Education</a></i>, also available <i><a href="https://nxdomain.no/~peter/the_numbers_show_we_have_a_failure.html">without trackers</a></i>, where the summary is,
</p><p>
</p><blockquote>
Every time I see one of those messages reach a mailbox that is actually read by one or more persons, I also see delivery attempts for near identical messages aimed at a subset of my now more than three hundred thousand <a href="https://nxdomain.no/~peter/traplist.shtml">spamtraps</a>, also known <i>imaginary friends</i>.
</blockquote>
<p>
Over the years since the piece was originally written, I have added several updates — generally when some of this nonsense reaches a mailbox I read — and while I have seen the messages in several languages, no real development beyond some variations in wording has happened.
</p><p>
Whenever one of those things <b>does</b> reach an inbox, my sequence of actions is generally to save the message and add it to <a href="https://nxdomain.no/~peter/wankstortion/">the archive</a>, see if the sending IP address has already entered the blocklist that is later exported and add it by hand if not. Then check if the number of trapped addesses has swelled recently by checking the log file from the export script
</p><p>
<tt>
$ tail -n 96 /var/log/traplistcounts
</tt>
</p><p>
See if there is a sharp increase since the last blocklist export
</p><p>
<tt>
$ doas spamdb | grep -c TRAPPED
</tt>
</p><p>
Then check for related activity in the log
</p><p>
<tt>
$ tail -n 500 -f /var/log/spamd
</tt>
</p><p>
Check for the full subject in the same log file
</p><p>
<tt>
$ grep "You are in really big troubles therefore, you much better read" /var/log/spamd
</tt>
</p><p>
Then check older, archived logs to see how long this campaign has been going on for
</p><p>
<tt>
$ zgrep "You are in really big troubles therefore, you much better read" /var/log/spamd.0.gz
</tt>
</p><p>
This time, the campaign had not gone on for long enough to show traces in the older archive, so I go on to extracting the sending IP addresses
</p><p>
<tt>
$ grep "You are in really big troubles therefore, you much better read" /var/log/spamd | awk '{print $6}' | tr -d ':' | sort -u
</tt>
</p><p>
Check for activity from one of the extracted addresses
</p><p>
<tt>
$ grep 183.111.115.4 /var/log/spamd | tee wankstortion/20221123_trapped_183.111.115.4.txt
</tt>
</p><p>
Extract the sender IP addresses to an environment variable to use in the next oneliner,
</p><p>
<tt>
$ grep trouble /var/log/spamd | awk '{print $6}' | tr -d ':' | sort -u | grep -vc BLACK | tee -a wankstortion/20221123_campaign_ip_addresses.txt
</tt>
</p><p>
which will record all activity involving those IP addresses since the last log rotation:
</p><p>
<tt>
$ for foo in $troubles ; do grep $foo /var/log/spamd | tee -a wankstortion/20221123_campaign_log_extract.txt ; done
</tt>
</p><p>
You will find all those files, along with some earlier samples, and by the time you read this, possibly even newer samples, in <a href="https://nxdomain.no/~peter/wankstortion/">the archive</a>.
</p><p>
When something of the sort inboxes, I probably will go on adding to the archive, and if I have time on my hands, also run similar extraction activities as the ones I just described. But unless something unexpected such as actual development in the senders' methods occurs, I will not bother to write about it.
</p><p>
The subject is simply not worth attention past persuading supposed victims to <b>not</b> bother to get bitcoins or spend any they might have to hand. None of my imaginary friends have, and they are just as fine as they were before somebot tried to scam them.
</p><p>
Good night and good luck.
</p><p>
</p><hr />
<p>
</p><p> </p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-7299681489255390862022-12-23T21:40:00.002+01:002022-12-23T22:29:30.668+01:00Can Your Spam-eater Manage to Catch Seventy-one Percent Like This Other Service?<p>
<i>Measuring the effect of what you do is important. Equally important is knowing what is the measure of your actions.</i>
</p><p>
A question turned up on IRC that had me thinking.
</p><p>
<i>Do you have a percentage of the spam traffic you catch on your MXes? The reason I ask is I lust learned that fastmail.com claim they catch 71% of all incoming spam. Also a rate of false positives would be nice to have, but that's likely harder to measure.</i>
</p><p>
My first impulse was that I would consider a seventy-one percent hit rate on the low side of what we are seeing here at <a href="https://bsdly.net">bsdly.net</a> and associated domains.
</p><p>
But getting actually useful data would require some thinking. That said, comparing a major mail operator that sells deliverability and promises a 71 percent catch rate for incoming spam and <b>bsdly.net</b> would be like comparing apples and oranges at best. </p><p>While <b>bsdly.net</b> (which is also known under a few other domain names) is my main mail service for my personal use and for a very select number of other people, to the rest of the world it is primarily a <i><a href="https://en.wikipedia.org/wiki/Honeypot_(computing)">honeypot</a></i> that generates security relevant data that other sites use, and that contributes to <i>IP reputation</i> rankings.
</p><p>
The site has been in operation in those roles for a little more than 15 years, since shortly before the original announcement in the article <i><a href="https://bsdly.blogspot.com/2007/07/hey-spammer-heres-list-for-you.html">Hey, spammer! Here's a list for you!</a></i>. When we started using the greylisting and greytrapping based setup, we saw a sharp drop in undesirable messages actually reaching inboxes, and I observed a marked decrease in load on the mail servers that did the content filtering.
</p><p>
Not long after I had set up our early greylisting setup, a <a href="https://marc.info/?l=openbsd-misc&m=116136841831550&w=2">message</a> turned up on the <tt>openbsd-misc</tt> mailing list that pretty much matched our experience — a 95% reduction in spam in line to be treated to content filtering — so setting up precise measuring became a thing to do when we could get around to it.
</p><p>
Now enough with the background. It is relatively easy to extract at least some data that would give us a rough picture of the relative effectiveness of the greylisting and greytrapping versus the content filtering on receipt. The setup is very similar to the one described in the practically-oriented parts of the <a href="https://bsdly.blogspot.com/2014/02/effective-spam-and-malware.html">Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools</a> and is part of a syncronizing multi-domain setup rougly as described in the earlier article <a href="https://bsdly.blogspot.com/2012/05/in-name-of-sane-email-setting-up-spamd.html">In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play - A Full Recipe</a>.
</p><p>
Using only tools found in the <a href="https://www.openbsd.org">OpenBSD</a> base system, I went on to collect data.
</p><p>
Whenever <a href="https://man.openbsd.org/spamd">spamd(8)</a> closes a connection it logs a message to that effect, so
</p><p>
</p><pre>$ zgrep "Nov 1" /var/log/spamd.6.gz | grep -c disconnected</pre>
<p>
Supplies the total number of connections <b>closed</b> by <a href="https://man.openbsd.org/spamd">spamd(8)</a> during November 1st, fetched from the archived log file.
</p><p>
Similarily
</p><p>
</p><pre>$ zgrep "Nov 1" /var/log/spamd.6.gz | grep -c BLACK</pre>
<p>
provides the number of connections during the same 24 hour period initiated by hosts that were already in one of the blocklists used.
</p><p>
The command to get the number of connections that had cleared the first hurdle and entered greylisted status would be
</p><p>
</p><pre>$ zgrep "Nov 1" /var/log/spamd.6.gz | grep -c GREY</pre>
<p>
And the number of hosts that had been well behaved enough to enter the whitelist and be allowed to talk to the real SMTP service comes out of
</p><p>
</p><pre>$ zgrep "Nov 1" /var/log/spamd.6.gz | grep -c whitelisting</pre>
<p>
For hosts that have reached this far and did not fail the content filtering we do during receipt, we get the number with
</p><p>
</p><pre>$ doas zgrep 2022-11-02 /var/spool/exim/logs/main.log.6.gz | grep -c Completed</pre>
<p>
It is however worth noting that our MTA <tt>exim</tt> reports <tt>Completed</tt> for apparently message deliveries in both directions, so the number of received messages, or messages that <b>did</b> inbox is likely about thirty percent lower.
</p><p>
The number of messages rejected for one reason or the other, by being addressed to an undeliverable address or by failing content filtering we find with
</p><p>
</p><pre>$ doas zgrep 2022-11-02 /var/spool/exim/logs/main.log.6.gz | grep -c rejected</pre>
<p>
And finally, a side effect of a frequently run log reading script that adds hosts with certain kinds of characteristics such as not having a correct reverse DNS entry to a blocklist and kills all their connections will at times produce an <tt>unexpected disconnection while reading SMTP command</tt> message. We find those with
</p><p>
</p><pre>$ doas zgrep 2022-11-02 /var/spool/exim/logs/main.log.6.gz | grep -c unexpected</pre>
<p>
Those are hosts that somehow got past <a href="https://man.openbsd.org/spamd">spamd(8)</a> by behaving enough like a real SMTP server to clear greylisting. However <a href="https://man.openbsd.org/spamd">spamd(8)</a> does not have the ability to check for valid reverse, so that part is left in our case to check for by reading the log files at intervals.
</p><p>
The following table has the data for November 2022 —
</p><p>
</p><table border="0" cellspacing="0">
<colgroup width="82"></colgroup>
<colgroup width="96"></colgroup>
<colgroup width="96"></colgroup>
<colgroup width="96"></colgroup>
<colgroup width="96"></colgroup>
<colgroup span="3" width="96"></colgroup>
<tbody><tr>
<td align="center" height="27" valign="bottom"><b><font color="#000000">Date</font></b></td>
<td align="center" valign="bottom"><b><font color="#000000">Incoming SMTP<br />connections</font></b></td>
<td align="center" valign="bottom"><b><font color="#000000">BLACK<br />connections</font></b></td>
<td align="center" valign="bottom"><b><font color="#000000">GREY<br />connections</font></b></td>
<td align="center" valign="bottom"><b><font color="#000000">New whitelist<br />entries</font></b></td>
<td align="center" valign="bottom"><b><font color="#000000">Deliveries</font></b></td>
<td align="center" valign="bottom"><b><font color="#000000">Rejected</font></b></td>
<td align="center" valign="bottom"><b><font color="#000000">Unexpected<br />disconnect</font></b></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44866" valign="bottom"><font color="#000000">2022-11-01</font></td>
<td align="center" sdnum="1044;" sdval="53303" valign="bottom"><font color="#000000">53303</font></td>
<td align="center" sdnum="1044;" sdval="38951" valign="bottom"><font color="#000000">38951</font></td>
<td align="center" sdnum="1044;" sdval="2580" valign="bottom"><font color="#000000">2580</font></td>
<td align="center" sdnum="1044;" sdval="54" valign="bottom"><font color="#000000">54</font></td>
<td align="center" sdnum="1044;" sdval="1347" valign="bottom"><font color="#000000">1347</font></td>
<td align="center" sdnum="1044;" sdval="409" valign="bottom"><font color="#000000">409</font></td>
<td align="center" sdnum="1044;" sdval="384" valign="bottom"><font color="#000000">384</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44867" valign="bottom"><font color="#000000">2022-11-02</font></td>
<td align="center" sdnum="1044;" sdval="55653" valign="bottom"><font color="#000000">55653</font></td>
<td align="center" sdnum="1044;" sdval="40467" valign="bottom"><font color="#000000">40467</font></td>
<td align="center" sdnum="1044;" sdval="2174" valign="bottom"><font color="#000000">2174</font></td>
<td align="center" sdnum="1044;" sdval="121" valign="bottom"><font color="#000000">121</font></td>
<td align="center" sdnum="1044;" sdval="1297" valign="bottom"><font color="#000000">1297</font></td>
<td align="center" sdnum="1044;" sdval="549" valign="bottom"><font color="#000000">549</font></td>
<td align="center" sdnum="1044;" sdval="330" valign="bottom"><font color="#000000">330</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44868" valign="bottom"><font color="#000000">2022-11-03</font></td>
<td align="center" sdnum="1044;" sdval="59658" valign="bottom"><font color="#000000">59658</font></td>
<td align="center" sdnum="1044;" sdval="43901" valign="bottom"><font color="#000000">43901</font></td>
<td align="center" sdnum="1044;" sdval="2086" valign="bottom"><font color="#000000">2086</font></td>
<td align="center" sdnum="1044;" sdval="85" valign="bottom"><font color="#000000">85</font></td>
<td align="center" sdnum="1044;" sdval="1260" valign="bottom"><font color="#000000">1260</font></td>
<td align="center" sdnum="1044;" sdval="865" valign="bottom"><font color="#000000">865</font></td>
<td align="center" sdnum="1044;" sdval="759" valign="bottom"><font color="#000000">759</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44869" valign="bottom"><font color="#000000">2022-11-04</font></td>
<td align="center" sdnum="1044;" sdval="57462" valign="bottom"><font color="#000000">57462</font></td>
<td align="center" sdnum="1044;" sdval="45674" valign="bottom"><font color="#000000">45674</font></td>
<td align="center" sdnum="1044;" sdval="1683" valign="bottom"><font color="#000000">1683</font></td>
<td align="center" sdnum="1044;" sdval="71" valign="bottom"><font color="#000000">71</font></td>
<td align="center" sdnum="1044;" sdval="1270" valign="bottom"><font color="#000000">1270</font></td>
<td align="center" sdnum="1044;" sdval="30" valign="bottom"><font color="#000000">30</font></td>
<td align="center" sdnum="1044;" sdval="0" valign="bottom"><font color="#000000">0</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44870" valign="bottom"><font color="#000000">2022-11-05</font></td>
<td align="center" sdnum="1044;" sdval="44993" valign="bottom"><font color="#000000">44993</font></td>
<td align="center" sdnum="1044;" sdval="43571" valign="bottom"><font color="#000000">43571</font></td>
<td align="center" sdnum="1044;" sdval="2146" valign="bottom"><font color="#000000">2146</font></td>
<td align="center" sdnum="1044;" sdval="105" valign="bottom"><font color="#000000">105</font></td>
<td align="center" sdnum="1044;" sdval="1182" valign="bottom"><font color="#000000">1182</font></td>
<td align="center" sdnum="1044;" sdval="43" valign="bottom"><font color="#000000">43</font></td>
<td align="center" sdnum="1044;" sdval="0" valign="bottom"><font color="#000000">0</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44871" valign="bottom"><font color="#000000">2022-11-06</font></td>
<td align="center" sdnum="1044;" sdval="36768" valign="bottom"><font color="#000000">36768</font></td>
<td align="center" sdnum="1044;" sdval="37802" valign="bottom"><font color="#000000">37802</font></td>
<td align="center" sdnum="1044;" sdval="2322" valign="bottom"><font color="#000000">2322</font></td>
<td align="center" sdnum="1044;" sdval="86" valign="bottom"><font color="#000000">86</font></td>
<td align="center" sdnum="1044;" sdval="1366" valign="bottom"><font color="#000000">1366</font></td>
<td align="center" sdnum="1044;" sdval="184" valign="bottom"><font color="#000000">184</font></td>
<td align="center" sdnum="1044;" sdval="0" valign="bottom"><font color="#000000">0</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44872" valign="bottom"><font color="#000000">2022-11-07</font></td>
<td align="center" sdnum="1044;" sdval="49464" valign="bottom"><font color="#000000">49464</font></td>
<td align="center" sdnum="1044;" sdval="44213" valign="bottom"><font color="#000000">44213</font></td>
<td align="center" sdnum="1044;" sdval="2398" valign="bottom"><font color="#000000">2398</font></td>
<td align="center" sdnum="1044;" sdval="182" valign="bottom"><font color="#000000">182</font></td>
<td align="center" sdnum="1044;" sdval="1424" valign="bottom"><font color="#000000">1424</font></td>
<td align="center" sdnum="1044;" sdval="67" valign="bottom"><font color="#000000">67</font></td>
<td align="center" sdnum="1044;" sdval="0" valign="bottom"><font color="#000000">0</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44873" valign="bottom"><font color="#000000">2022-11-08</font></td>
<td align="center" sdnum="1044;" sdval="52285" valign="bottom"><font color="#000000">52285</font></td>
<td align="center" sdnum="1044;" sdval="45904" valign="bottom"><font color="#000000">45904</font></td>
<td align="center" sdnum="1044;" sdval="2676" valign="bottom"><font color="#000000">2676</font></td>
<td align="center" sdnum="1044;" sdval="113" valign="bottom"><font color="#000000">113</font></td>
<td align="center" sdnum="1044;" sdval="1513" valign="bottom"><font color="#000000">1513</font></td>
<td align="center" sdnum="1044;" sdval="69" valign="bottom"><font color="#000000">69</font></td>
<td align="center" sdnum="1044;" sdval="3" valign="bottom"><font color="#000000">3</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44874" valign="bottom"><font color="#000000">2022-11-09</font></td>
<td align="center" sdnum="1044;" sdval="47652" valign="bottom"><font color="#000000">47652</font></td>
<td align="center" sdnum="1044;" sdval="47988" valign="bottom"><font color="#000000">47988</font></td>
<td align="center" sdnum="1044;" sdval="2085" valign="bottom"><font color="#000000">2085</font></td>
<td align="center" sdnum="1044;" sdval="105" valign="bottom"><font color="#000000">105</font></td>
<td align="center" sdnum="1044;" sdval="1438" valign="bottom"><font color="#000000">1438</font></td>
<td align="center" sdnum="1044;" sdval="154" valign="bottom"><font color="#000000">154</font></td>
<td align="center" sdnum="1044;" sdval="0" valign="bottom"><font color="#000000">0</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44875" valign="bottom"><font color="#000000">2022-11-10</font></td>
<td align="center" sdnum="1044;" sdval="57850" valign="bottom"><font color="#000000">57850</font></td>
<td align="center" sdnum="1044;" sdval="49875" valign="bottom"><font color="#000000">49875</font></td>
<td align="center" sdnum="1044;" sdval="2614" valign="bottom"><font color="#000000">2614</font></td>
<td align="center" sdnum="1044;" sdval="104" valign="bottom"><font color="#000000">104</font></td>
<td align="center" sdnum="1044;" sdval="1435" valign="bottom"><font color="#000000">1435</font></td>
<td align="center" sdnum="1044;" sdval="192" valign="bottom"><font color="#000000">192</font></td>
<td align="center" sdnum="1044;" sdval="2" valign="bottom"><font color="#000000">2</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44876" valign="bottom"><font color="#000000">2022-11-11</font></td>
<td align="center" sdnum="1044;" sdval="60269" valign="bottom"><font color="#000000">60269</font></td>
<td align="center" sdnum="1044;" sdval="56719" valign="bottom"><font color="#000000">56719</font></td>
<td align="center" sdnum="1044;" sdval="2355" valign="bottom"><font color="#000000">2355</font></td>
<td align="center" sdnum="1044;" sdval="99" valign="bottom"><font color="#000000">99</font></td>
<td align="center" sdnum="1044;" sdval="1420" valign="bottom"><font color="#000000">1420</font></td>
<td align="center" sdnum="1044;" sdval="90" valign="bottom"><font color="#000000">90</font></td>
<td align="center" sdnum="1044;" sdval="1" valign="bottom"><font color="#000000">1</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44877" valign="bottom"><font color="#000000">2022-11-12</font></td>
<td align="center" sdnum="1044;" sdval="46139" valign="bottom"><font color="#000000">46139</font></td>
<td align="center" sdnum="1044;" sdval="54073" valign="bottom"><font color="#000000">54073</font></td>
<td align="center" sdnum="1044;" sdval="1160" valign="bottom"><font color="#000000">1160</font></td>
<td align="center" sdnum="1044;" sdval="96" valign="bottom"><font color="#000000">96</font></td>
<td align="center" sdnum="1044;" sdval="1182" valign="bottom"><font color="#000000">1182</font></td>
<td align="center" sdnum="1044;" sdval="29" valign="bottom"><font color="#000000">29</font></td>
<td align="center" sdnum="1044;" sdval="0" valign="bottom"><font color="#000000">0</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44878" valign="bottom"><font color="#000000">2022-11-13</font></td>
<td align="center" sdnum="1044;" sdval="40497" valign="bottom"><font color="#000000">40497</font></td>
<td align="center" sdnum="1044;" sdval="40221" valign="bottom"><font color="#000000">40221</font></td>
<td align="center" sdnum="1044;" sdval="1777" valign="bottom"><font color="#000000">1777</font></td>
<td align="center" sdnum="1044;" sdval="70" valign="bottom"><font color="#000000">70</font></td>
<td align="center" sdnum="1044;" sdval="1239" valign="bottom"><font color="#000000">1239</font></td>
<td align="center" sdnum="1044;" sdval="189" valign="bottom"><font color="#000000">189</font></td>
<td align="center" sdnum="1044;" sdval="0" valign="bottom"><font color="#000000">0</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44879" valign="bottom"><font color="#000000">2022-11-14</font></td>
<td align="center" sdnum="1044;" sdval="59965" valign="bottom"><font color="#000000">59965</font></td>
<td align="center" sdnum="1044;" sdval="59951" valign="bottom"><font color="#000000">59951</font></td>
<td align="center" sdnum="1044;" sdval="2062" valign="bottom"><font color="#000000">2062</font></td>
<td align="center" sdnum="1044;" sdval="63" valign="bottom"><font color="#000000">63</font></td>
<td align="center" sdnum="1044;" sdval="1382" valign="bottom"><font color="#000000">1382</font></td>
<td align="center" sdnum="1044;" sdval="145" valign="bottom"><font color="#000000">145</font></td>
<td align="center" sdnum="1044;" sdval="73" valign="bottom"><font color="#000000">73</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44880" valign="bottom"><font color="#000000">2022-11-15</font></td>
<td align="center" sdnum="1044;" sdval="56265" valign="bottom"><font color="#000000">56265</font></td>
<td align="center" sdnum="1044;" sdval="32727" valign="bottom"><font color="#000000">32727</font></td>
<td align="center" sdnum="1044;" sdval="2304" valign="bottom"><font color="#000000">2304</font></td>
<td align="center" sdnum="1044;" sdval="113" valign="bottom"><font color="#000000">113</font></td>
<td align="center" sdnum="1044;" sdval="1298" valign="bottom"><font color="#000000">1298</font></td>
<td align="center" sdnum="1044;" sdval="351" valign="bottom"><font color="#000000">351</font></td>
<td align="center" sdnum="1044;" sdval="301" valign="bottom"><font color="#000000">301</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44881" valign="bottom"><font color="#000000">2022-11-16</font></td>
<td align="center" sdnum="1044;" sdval="77252" valign="bottom"><font color="#000000">77252</font></td>
<td align="center" sdnum="1044;" sdval="58029" valign="bottom"><font color="#000000">58029</font></td>
<td align="center" sdnum="1044;" sdval="1925" valign="bottom"><font color="#000000">1925</font></td>
<td align="center" sdnum="1044;" sdval="109" valign="bottom"><font color="#000000">109</font></td>
<td align="center" sdnum="1044;" sdval="1340" valign="bottom"><font color="#000000">1340</font></td>
<td align="center" sdnum="1044;" sdval="282" valign="bottom"><font color="#000000">282</font></td>
<td align="center" sdnum="1044;" sdval="33" valign="bottom"><font color="#000000">33</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44882" valign="bottom"><font color="#000000">2022-11-17</font></td>
<td align="center" sdnum="1044;" sdval="43107" valign="bottom"><font color="#000000">43107</font></td>
<td align="center" sdnum="1044;" sdval="30713" valign="bottom"><font color="#000000">30713</font></td>
<td align="center" sdnum="1044;" sdval="786" valign="bottom"><font color="#000000">786</font></td>
<td align="center" sdnum="1044;" sdval="131" valign="bottom"><font color="#000000">131</font></td>
<td align="center" sdnum="1044;" sdval="1250" valign="bottom"><font color="#000000">1250</font></td>
<td align="center" sdnum="1044;" sdval="215" valign="bottom"><font color="#000000">215</font></td>
<td align="center" sdnum="1044;" sdval="17" valign="bottom"><font color="#000000">17</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44883" valign="bottom"><font color="#000000">2022-11-18</font></td>
<td align="center" sdnum="1044;" sdval="49448" valign="bottom"><font color="#000000">49448</font></td>
<td align="center" sdnum="1044;" sdval="48999" valign="bottom"><font color="#000000">48999</font></td>
<td align="center" sdnum="1044;" sdval="1590" valign="bottom"><font color="#000000">1590</font></td>
<td align="center" sdnum="1044;" sdval="96" valign="bottom"><font color="#000000">96</font></td>
<td align="center" sdnum="1044;" sdval="1327" valign="bottom"><font color="#000000">1327</font></td>
<td align="center" sdnum="1044;" sdval="194" valign="bottom"><font color="#000000">194</font></td>
<td align="center" sdnum="1044;" sdval="1" valign="bottom"><font color="#000000">1</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44884" valign="bottom"><font color="#000000">2022-11-19</font></td>
<td align="center" sdnum="1044;" sdval="42413" valign="bottom"><font color="#000000">42413</font></td>
<td align="center" sdnum="1044;" sdval="45927" valign="bottom"><font color="#000000">45927</font></td>
<td align="center" sdnum="1044;" sdval="973" valign="bottom"><font color="#000000">973</font></td>
<td align="center" sdnum="1044;" sdval="92" valign="bottom"><font color="#000000">92</font></td>
<td align="center" sdnum="1044;" sdval="1182" valign="bottom"><font color="#000000">1182</font></td>
<td align="center" sdnum="1044;" sdval="182" valign="bottom"><font color="#000000">182</font></td>
<td align="center" sdnum="1044;" sdval="70" valign="bottom"><font color="#000000">70</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44885" valign="bottom"><font color="#000000">2022-11-20</font></td>
<td align="center" sdnum="1044;" sdval="50890" valign="bottom"><font color="#000000">50890</font></td>
<td align="center" sdnum="1044;" sdval="55318" valign="bottom"><font color="#000000">55318</font></td>
<td align="center" sdnum="1044;" sdval="1558" valign="bottom"><font color="#000000">1558</font></td>
<td align="center" sdnum="1044;" sdval="77" valign="bottom"><font color="#000000">77</font></td>
<td align="center" sdnum="1044;" sdval="1203" valign="bottom"><font color="#000000">1203</font></td>
<td align="center" sdnum="1044;" sdval="358" valign="bottom"><font color="#000000">358</font></td>
<td align="center" sdnum="1044;" sdval="33" valign="bottom"><font color="#000000">33</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44886" valign="bottom"><font color="#000000">2022-11-21</font></td>
<td align="center" sdnum="1044;" sdval="36601" valign="bottom"><font color="#000000">36601</font></td>
<td align="center" sdnum="1044;" sdval="35070" valign="bottom"><font color="#000000">35070</font></td>
<td align="center" sdnum="1044;" sdval="1707" valign="bottom"><font color="#000000">1707</font></td>
<td align="center" sdnum="1044;" sdval="125" valign="bottom"><font color="#000000">125</font></td>
<td align="center" sdnum="1044;" sdval="1321" valign="bottom"><font color="#000000">1321</font></td>
<td align="center" sdnum="1044;" sdval="241" valign="bottom"><font color="#000000">241</font></td>
<td align="center" sdnum="1044;" sdval="146" valign="bottom"><font color="#000000">146</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44887" valign="bottom"><font color="#000000">2022-11-22</font></td>
<td align="center" sdnum="1044;" sdval="37840" valign="bottom"><font color="#000000">37840</font></td>
<td align="center" sdnum="1044;" sdval="35499" valign="bottom"><font color="#000000">35499</font></td>
<td align="center" sdnum="1044;" sdval="2055" valign="bottom"><font color="#000000">2055</font></td>
<td align="center" sdnum="1044;" sdval="99" valign="bottom"><font color="#000000">99</font></td>
<td align="center" sdnum="1044;" sdval="1359" valign="bottom"><font color="#000000">1359</font></td>
<td align="center" sdnum="1044;" sdval="142" valign="bottom"><font color="#000000">142</font></td>
<td align="center" sdnum="1044;" sdval="17" valign="bottom"><font color="#000000">17</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44888" valign="bottom"><font color="#000000">2022-11-23</font></td>
<td align="center" sdnum="1044;" sdval="43186" valign="bottom"><font color="#000000">43186</font></td>
<td align="center" sdnum="1044;" sdval="34545" valign="bottom"><font color="#000000">34545</font></td>
<td align="center" sdnum="1044;" sdval="1314" valign="bottom"><font color="#000000">1314</font></td>
<td align="center" sdnum="1044;" sdval="114" valign="bottom"><font color="#000000">114</font></td>
<td align="center" sdnum="1044;" sdval="1345" valign="bottom"><font color="#000000">1345</font></td>
<td align="center" sdnum="1044;" sdval="103" valign="bottom"><font color="#000000">103</font></td>
<td align="center" sdnum="1044;" sdval="21" valign="bottom"><font color="#000000">21</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44889" valign="bottom"><font color="#000000">2022-11-24</font></td>
<td align="center" sdnum="1044;" sdval="46802" valign="bottom"><font color="#000000">46802</font></td>
<td align="center" sdnum="1044;" sdval="45765" valign="bottom"><font color="#000000">45765</font></td>
<td align="center" sdnum="1044;" sdval="1856" valign="bottom"><font color="#000000">1856</font></td>
<td align="center" sdnum="1044;" sdval="66" valign="bottom"><font color="#000000">66</font></td>
<td align="center" sdnum="1044;" sdval="1269" valign="bottom"><font color="#000000">1269</font></td>
<td align="center" sdnum="1044;" sdval="729" valign="bottom"><font color="#000000">729</font></td>
<td align="center" sdnum="1044;" sdval="52" valign="bottom"><font color="#000000">52</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44890" valign="bottom"><font color="#000000">2022-11-25</font></td>
<td align="center" sdnum="1044;" sdval="70911" valign="bottom"><font color="#000000">70911</font></td>
<td align="center" sdnum="1044;" sdval="52404" valign="bottom"><font color="#000000">52404</font></td>
<td align="center" sdnum="1044;" sdval="1315" valign="bottom"><font color="#000000">1315</font></td>
<td align="center" sdnum="1044;" sdval="89" valign="bottom"><font color="#000000">89</font></td>
<td align="center" sdnum="1044;" sdval="1326" valign="bottom"><font color="#000000">1326</font></td>
<td align="center" sdnum="1044;" sdval="1488" valign="bottom"><font color="#000000">1488</font></td>
<td align="center" sdnum="1044;" sdval="395" valign="bottom"><font color="#000000">395</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44891" valign="bottom"><font color="#000000">2022-11-26</font></td>
<td align="center" sdnum="1044;" sdval="39780" valign="bottom"><font color="#000000">39780</font></td>
<td align="center" sdnum="1044;" sdval="32226" valign="bottom"><font color="#000000">32226</font></td>
<td align="center" sdnum="1044;" sdval="1500" valign="bottom"><font color="#000000">1500</font></td>
<td align="center" sdnum="1044;" sdval="77" valign="bottom"><font color="#000000">77</font></td>
<td align="center" sdnum="1044;" sdval="1175" valign="bottom"><font color="#000000">1175</font></td>
<td align="center" sdnum="1044;" sdval="954" valign="bottom"><font color="#000000">954</font></td>
<td align="center" sdnum="1044;" sdval="379" valign="bottom"><font color="#000000">379</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44892" valign="bottom"><font color="#000000">2022-11-27</font></td>
<td align="center" sdnum="1044;" sdval="67578" valign="bottom"><font color="#000000">67578</font></td>
<td align="center" sdnum="1044;" sdval="41581" valign="bottom"><font color="#000000">41581</font></td>
<td align="center" sdnum="1044;" sdval="1743" valign="bottom"><font color="#000000">1743</font></td>
<td align="center" sdnum="1044;" sdval="85" valign="bottom"><font color="#000000">85</font></td>
<td align="center" sdnum="1044;" sdval="1231" valign="bottom"><font color="#000000">1231</font></td>
<td align="center" sdnum="1044;" sdval="523" valign="bottom"><font color="#000000">523</font></td>
<td align="center" sdnum="1044;" sdval="315" valign="bottom"><font color="#000000">315</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44893" valign="bottom"><font color="#000000">2022-11-28</font></td>
<td align="center" sdnum="1044;" sdval="54688" valign="bottom"><font color="#000000">54688</font></td>
<td align="center" sdnum="1044;" sdval="37534" valign="bottom"><font color="#000000">37534</font></td>
<td align="center" sdnum="1044;" sdval="2433" valign="bottom"><font color="#000000">2433</font></td>
<td align="center" sdnum="1044;" sdval="77" valign="bottom"><font color="#000000">77</font></td>
<td align="center" sdnum="1044;" sdval="1337" valign="bottom"><font color="#000000">1337</font></td>
<td align="center" sdnum="1044;" sdval="321" valign="bottom"><font color="#000000">321</font></td>
<td align="center" sdnum="1044;" sdval="269" valign="bottom"><font color="#000000">269</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44894" valign="bottom"><font color="#000000">2022-11-29</font></td>
<td align="center" sdnum="1044;" sdval="70893" valign="bottom"><font color="#000000">70893</font></td>
<td align="center" sdnum="1044;" sdval="45917" valign="bottom"><font color="#000000">45917</font></td>
<td align="center" sdnum="1044;" sdval="2502" valign="bottom"><font color="#000000">2502</font></td>
<td align="center" sdnum="1044;" sdval="65" valign="bottom"><font color="#000000">65</font></td>
<td align="center" sdnum="1044;" sdval="1248" valign="bottom"><font color="#000000">1248</font></td>
<td align="center" sdnum="1044;" sdval="87" valign="bottom"><font color="#000000">87</font></td>
<td align="center" sdnum="1044;" sdval="39" valign="bottom"><font color="#000000">39</font></td>
</tr>
<tr>
<td align="center" height="27" sdnum="1044;0;YYYY-MM-DD" sdval="44895" valign="bottom"><font color="#000000">2022-11-30</font></td>
<td align="center" sdnum="1044;" sdval="50280" valign="bottom"><font color="#000000">50280</font></td>
<td align="center" sdnum="1044;" sdval="35585" valign="bottom"><font color="#000000">35585</font></td>
<td align="center" sdnum="1044;" sdval="2567" valign="bottom"><font color="#000000">2567</font></td>
<td align="center" sdnum="1044;" sdval="67" valign="bottom"><font color="#000000">67</font></td>
<td align="center" sdnum="1044;" sdval="1324" valign="bottom"><font color="#000000">1324</font></td>
<td align="center" sdnum="1044;" sdval="1293" valign="bottom"><font color="#000000">1293</font></td>
<td align="center" sdnum="1044;" sdval="1113" valign="bottom"><font color="#000000">1113</font></td>
</tr>
</tbody></table>
<p>
The table is also available as a <a href="https://nxdomain.no/~peter/November_2022_bsdly.net_SMTP_statistics.csv">comma separated (CSV) file</a>.
</p><p>
As I mentioned earlier, the number of connections to the outer layer <a href="https://man.openbsd.org/spamd">spamd(8)</a> is likely higher than what would be expected on sites that are <b>not</b> considered a honeypot and home to in excess of three hundred thousand imaginary friends (see <a href="https://bsdly.blogspot.com/2022/09/the-things-spammers-believe-tale-of.html">The Things Spammers Believe - A Tale of 300,000 Imaginary Friends</a> or <a href="https://nxdomain.no/~peter/spammers_believe_in_300k_imaginary_friends.html">the trackerless version</a>.
</p><p>
That said, I think the data shows that catching the unwanted traffic early, and discarding as much as possible of that traffic before it reaches the resource hungry content filtering is definitely beneficial. </p><p>Even sites that do not actively bait the baddies out there would likely see noticeable energy bill savings by having their mail servers run quiter and cooler, as they definitely will after getting a greylisting, and optionally greytrapping setup in front of them. Those services have a truly low energy consumption profile.
</p><p>
If you found this article interesting, useful or just simply irritating, I would like to hear from you. Please use the comment field, or if you prefer, send email to <tt>nix at nxdomain dot no</tt> with a subject that at least tries to sound sensible and relevant.
</p><p>
As always, if you are interested in research on items mentioned in this article, I will be able to provide data for study. I will honor reasonable requests.
</p><p>
</p><hr />
<p>
</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-23715592813155637402022-12-09T21:51:00.011+01:002023-01-01T21:54:47.396+01:00Harvesting the Noise While it's Fresh, Revisited <i>A year's worth of logs yields entertaining but unsurprising findings about spammer behavior.</i>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://nxdomain.no/~peter/blogpix/picspam02-chopped.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="361" data-original-width="800" height="361" src="https://nxdomain.no/~peter/blogpix/picspam02-chopped.jpg" width="800" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Spam mail, masked but detected, from the archive</td></tr></tbody></table><p>
Returning readers will be almost painfully aware that here at <a href="https://nxdomain.no">nxdomain.no</a> (also known as <a href="https://www.bsdly.net/">bsdly.net</a>) we host and maintain a <i><a href="https://nxdomain.no/~peter/traplist.html">blocklist</a></i>, which in turn is the product of traffic that hits our mail system with attempts at delivery to one or more of the now more than three hundred thousand <i>known bad</i> addresses, also featured at the <i><a href="https://nxdomain.no/~peter/traplist.html">blocklist home page</a></i>.
</p><p>
<b>Note:</b> This piece is also available <i>without</i> trackers but only basic formatting <a href="https://nxdomain.no/~peter/harvesting_the_noise_revisited.html">here</a>. </p><p>
When I first set up the greytrapping back in 2007, the initial spamtraps were non-deliverable addresses in our domains that I had extracted from mail server logs. I won't bore you with the details (which are anyway documented at length in earlier articles), but it was clear from those logs that the domains we hosted back then were more or less continously subject to <i><a href="https://en.wikipedia.org/wiki/Joe_job">Joe jobs</a></i>, as in somebody sending messages with a forged <tt>From:</tt> field with a made up address in our domains.
</p><p>
After a while I started extracting the potential new spamtraps from the greylist — actually dumping data from there once per hour as part of the script that also generated the exported blocklist. The basic process is described in the July 25 2007 article <a href="https://bsdly.blogspot.com/2007/07/harvesting-noise-while-its-still-fresh.html">Harvesting the noise while it's still fresh; SPF found potentially useful</a> (also available <a href="https://nxdomain.no/~peter/harvesting-noise-while-its-still-fresh.html">trackerless</a> but with links to tracked articles).
</p><p>
Then today it struck me that while that method is useful, by extracting only from the greylist we will only ever collect the address from the initial connections. Any addresses attempted after the miscreants enter the blocklist will simply not be recorded there.
</p><p>
This of course lead to the question: <i>What did we miss?</i>
</p><p>
Fortunately I keep my logs around for a while, the most easily accessible log archive for my main <tt><a href="https://man.openbsd.org/spamd">spamd</a></tt> spans a lttle over a year. So I set about with some very basic <tt><a href="https://man.openbsd.org/grep">grep</a></tt> and <tt><a href="https://man.openbsd.org/awk">awk</a></tt>, which netted me this <a href="https://nxdomain.no/~peter/20221209_aged_noise.txt">raw list</a> of targeted addresses from the <tt><a href="https://man.openbsd.org/spamd">spamd</a></tt> logs.
</p><p>
The list weighs in at a total of <i>269903</i> entries, as counted by <tt><a href="https://man.openbsd.org/wc">wc -l</a></tt>.
</p><p>
Some of those addresses are valid, and a small, but actually significant, number are in domains we do not actually serve here, and some entries do not look like mail addresses at all. The stranger ones could be strings encoded in a character set that <tt><a href="https://man.openbsd.org/spamd">spamd</a></tt> is not equipped to handle, or could be other binary data that might have been intended to trigger bugs in some of the variants of fully equipped SMTP servers that are out there. Or simply noise of any other kind, including a byproduct of the not very intelligent extraction one-liner I used.
</p><p>
The target addresses in foreign domains I take as a sign that at least some spamming operators mistake a reasonably configured <tt><a href="https://man.openbsd.org/spamd">spamd</a></tt> for an open relay, just like they did all those years ago when I started running the greytrapping.
</p><p>
Some things apparently stay the same no matter how the rest of the world has found a way to move forward.
</p><p>
While I did a few other tasks and finally started writing this article, the bulk of the processes that would answer the question posed earlier (<i>What did we miss?</i>) could fortunately run unattended in the background, and after some manual massaging we are left with a <a href="https://nxdomain.no/~peter/20221209_extras.txt">results file</a>, with <i>1530</i> entries that were <b>none of</b>
</p><p>
</p><ul>
<li>actually useful deliverable addresses in our domains</li>
<li>existing spamtraps</li>
</ul>
<p>
This means of course that the collection of imaginary friends expanded by the same number, and now stands at <i>304154</i> entries.
</p><p>
Which I suppose means that harvesting the noise even after a period of aging for refinement can be a good thing.
</p><p>
The entries added represent a wide variety of phenomena. Quite a few seem to be truncated versions of earlier spamtrap entries, and a fair number of the new entries look like they may have descended from artifacts of stupidity such as products of <a href="https://bsdly.blogspot.com/2017/08/twenty-plus-years-on-smtp-callbacks-are.html">SMTP callbacks</a>. Proving mainly that in mail and spam handling, there appears to be a space still for the less intellectually astute.
</p><p>
With all of this said, the natural followup question is, given the modest net result, <i>was this worth the effort?</i>
</p><p>
Well, the raw output that yielded 269903 entries needed some manual operations in order to weed out the obvious noise (exact time used not recorded), followed by another background task that took, according to <a href="https://man.openbsd.org/time">time(1)</a>
</p><p>
</p><pre> real 105m24.220s
user 73m3.280s
sys 29m14.930s
</pre>
<p>
which yielded <i>1577</i> entries that were pared down to <i>1530</i> entries that met the criteria for inclusion in the circle of imaginary friends (also known as <i>spamtraps)</i>.
</p><p>
Before this experiment, the spamtraps list numbered <i>302625</i>, after including the result here, the count stands at <i>304154</i>, for a gain of less than one percent of the previous total. Again, if you check back at the <a href="https://nxdomain.no/~peter/traplist.html">traplist home page</a> now, the total number is likely to have increased again.
</p><p>
So <i>was it worth the effort?</i> I feel that as an experiment, it was worth doing.
</p><p>
Whether or not it is an experiment that is <i>worth repeating</i> is a question for another day.
</p><p>
If you have opinions on this, I would love to hear from you, in comments, via email or messages on whichever social media brought you the link to this article.
</p><p>
As always, parties interested in studying the data referenced in this article and other pieces I have written are welcome to contact me for arrangements. I can easily dig out more and rawer data than directly referenced here on request.
</p><p>
Stay safe out there.
</p><p>
</p><hr />
As a side note, a slightly improved way of extracting useful data about other domains' mail service via SPF records can be found in the November 2018 artice <a href="https://bsdly.blogspot.com/2018/11/goodness-enumerated-by-robots-or.html">Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting</a>.
<p>
That article (naturally) works from the premise that you are running a recent <a href="https://www.openbsd.org/">OpenBSD</a> system.
</p><p>
</p><hr />
<p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-89100852278740850992022-09-25T18:58:00.008+02:002022-12-30T23:05:11.754+01:00A Few of My Favorite Things About The OpenBSD Packet Filter Tools<p>
<i>The <a href="https://www.openbsd.org/">OpenBSD</a> packet filter PF was introduced a little more than 20 years ago as part of <a href="https://www.openbsd.org/30.html">OpenBSD 3.0</a>. We'll take a short tour of PF features and tools that I have enjoyed using.</i>
</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://nxdomain.no/~peter/blogpix/Peter_with_pf3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="697" data-original-width="800" height="349" src="https://nxdomain.no/~peter/blogpix/Peter_with_pf3.png" width="400" /></a></div><br /><i><br /></i><p></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><b>NOTE: </b>If you are more of a <i>slides</i> person, the condensate for a <a href="http://semibug.org/">SEMIBUG</a> user group meeting is available <a href="https://home.nuug.no/~peter/pf_favorite/">here</a>. A version without trackers but “classical” formatting is available <a href="https://nxdomain.no/~peter/better_off_with_pf.html">here</a>.
</p></blockquote><p>
At the time the OpenBSD project introduced its new packet filter subsystem in 2001, I was nowhere near the essentially full time OpenBSD user I would soon become. I did however quickly recognize that even what was later dubbed <i>“the working prototype”</i> was reported to perform better in most contexts than the code it replaced.
</p><p>
The reason PF's predecessor needed to be replaced has been covered extensively by myself and others elsewhere, so I'll limit myself to noting that the reason was that several somebodies finally read and understood the code's license and decided that it was not in fact open source in any acceptable meaning of the term.
</p><p>
Anyway the initial PF release was very close in features and syntax to the code it replaced. And even at that time, the config syntax was a lot more human readable than the alternative I had been handling up to then, which was Linux' IPtables. The less is said about IPtables, the better.
</p><p>
But soon visible improvements in user friendliness, or at least admin friendliness, started turning up. With OpenBSD 3.2, the separate <a href="http://man.openbsd.org/OpenBSD-3.1/nat.conf"><tt>/etc/nat.conf</tt></a> network adress translation configuration file moved to the attic and the NAT and redirection options moved into the main PF config file <a href="http://man.openbsd.org/pf.conf"><tt>/etc/pf.conf</tt></a>.
</p><p>
The next version, OpenBSD 3.3, saw the ALTQ queueing configuration move into <tt>pf.conf</tt> as well, and the previously separate <a href="http://man.openbsd.org/OpenBSD-3.2/altq.conf"><tt>altq.conf</tt></a> file became obsolete. What did not change, however, was the syntax, which was to remain just bothersome enough that many of us put off playing with traffic shaping until some years later. Other PF news in that release included <i>anchors</i>, or named sub-rulesets, as well as <i>tables</i>, described as <i>"a very efficient way for large address lists in rules"</i> and the initial release of <a href="https://man.openbsd.org/spamd">spamd(8)</a>, the spam deferral daemon.
</p><p>
More on all of these things later, I will not bore you with a detailed history of PF features introduced or changed in OpenBSD over the last twenty-some years.
</p><p>
</p><h2>PF Rulesets: The Basics</h2>
<p>
So how do we go about writing that <i>perfect</i> firewall config?
</p><p>
I could go on about that at length, and I have been known to on occasion, but let us start with the simplest possible, yet <i>absolutely secure</i> PF ruleset:
</p><p>
<code>
block
</code>
</p><p>
With that in place, you are totally secure. No traffic will pass.
</p><p>
Or as they say in the trade, you have virtually unplugged yourself from the rest of the world.
</p><p>
By way of getting ahead of ourselves, that particular ruleset will expand to the following:
</p><p>
<code>
block drop all
</code>
</p><p>
But we are getting ahead of ourselves.
</p><p>
To provide you with a few tools and some context, these are the basic building blocks of a PF rule:
</p><p>
<b>verb</b> <i>criteria</i> <b>action</b> ... options
</p><p>
Here are a few sample rules to put it into context, all lifted from configurations I have put into production:
</p><p>
<code>
pass in on egress proto tcp to egress port ssh
</code>
</p><p>
This first sample says that if a packet arrives on the <i>egress</i> — an interface belonging to the group of interfaces that has a default route — and that packet is a TCP packet with a destination service <i>ssh</i>, let the packet pass to the interfaces belonging to the <i>egress</i> interface group.
</p><p>
Yes, when you write PF rulesets, you do not necessarily need to write port numbers for services and memorize what services hide behind port 80, 53 or 443. The common or standard services are known to the rules parsing part of <a href="https://man.openbsd.org/pfctl">pfctl(8)</a>, generally with the service names you can look up in the <tt><a href="https://man.openbsd.org/services">/etc/services</a></tt> file.
</p><p>
The interface groups concept is as far as I know an <a href="https://www.openbsd.org/">OpenBSD</a> innovation. You can put interfaces into logical groups and reference the group name in PF configurations. A few default interface groups exist without you doing anything, <i>egress</i> is one, another common one is <i>wlan</i> where all configured WiFi interfaces are members by default. Keep in mind that you can create your own interface groups — set them up using <tt><a href="https://man.openbsd.org/ifconfig">ifconfig(8)</a></tt> — and refer to them in your rules.
</p><p>
<code>
match out on egress nat-to egress
</code>
</p><p>
This one <i>match</i>es outbound traffic, again on <i>egress</i> (which in the simpler cases consists of one interface) and applies the <i>nat-to</i> action on the packets, transforming them so that the next hops all the way to the destination will see packets where the source address is equal to the <i>egress</i> interface's address. If your network runs IPv4 and you have only one routeable address assigned, you will more than likely have something like this configured on your Internet-facing gateway.
</p><p>
It is worth noting that early PF versions did not have the <i>match</i> verb. After a few years of PF practice, developers and practitioners alike saw the need for a way to apply actions such as <i>nat-to</i> or other transformations without making a decision on whether to pass or block the traffic. The <tt>match</tt> keyword arrived in <a href="https://www.openbsd.org/46.html">OpenBSD 4.6</a> and in retrospect seems like a prelude to more extensive changes that followed over the next few releases.
</p><p>
Next up is a variation on the initial <i>absolutely secure</i> ruleset.
</p><p>
<code>
block all
</code>
</p><p>
I will tell you now so you will not be surprised later: If you had made a configuration with those three rules in that order, your configuration would be functionally the same as the one word one we started with. This is because in PF configurations, the rules are <b>evaluated</b> from top to bottom, and the <i>last</i> matching rule wins.
</p> <p>
The only escape from this progression is to insert a <i>quick</i> modifier after the verb, as in</p><p>
<tt>pass quick from (self)</tt>
</p><p>
which will stop evaluation when a packet matches the criteria in the <i>quick</i> rule. Please use sparingly if at all.
</p><p>
</p><p>
There is a specific reason why PF behaves like this. The system that PF replaced in OpenBSD had the top to bottom, last match wins logic, and the developers did not want to break existing configurations too badly during the transition away from the old system.
</p><p>
So in practice you would put them in this order for a more functional setup,
</p><p>
<code></code></p><pre><code> block all
match out on egress nat-to egress
pass in on egress proto tcp to egress port ssh
</code></pre>
<p>
but likely supplemented by a few other items.
</p><p>
For those supplementing items, we can take a look at some of the PF features that can help you write readable and maintainable rulesets. And while a readable ruleset is not automatically a more secure one, readability certainly helps spot errors in your logic that could put the systems and users in your care in reach of potential threats.
</p><p>
To help that readability, it is important to be aware of these features:
</p><p>
<b>Options</b>: General configuration options that set the parameters for the ruleset, such as
</p><p>
<code></code></p><pre><code> set limit states 100000
set debug debug
set loginterface dc0
set timeout tcp.first 120
set timeout tcp.established 86400
set timeout { adaptive.start 6000, adaptive.end 12000 }
</code></pre>
<p>
If the meaning of some of those do not seem terribly obvious to you at this point, that's fine. They are all extensively documented in the <a href="https://man.openbsd.org/pf.conf">pf.conf</a> man page.
</p><p>
<b>Macros</b>: Content that will expand in place, such as lists of services, interface names or other items you feel useful. Some examples along with rules that use them:
</p><p>
<code></code></p><pre><code> ext_if = "kue0"
all_ifs = "{" $ext_if lo0 "}"
pass out on $ext_if from any to any
pass in on $ext_if proto tcp from any to any port 25
</code></pre><code>
</code>
<p>
Keep in mind that if your macros expand to lists of either ports or IP addresses, the macro expansion will create several rules to cover your definitions in the ruleset that is eventually loaded.
</p><p>
<b>Tables</b>: Data structures that are specifically designed to store IP addresses and networks. Originally devised to be a more efficient way to store IP addresses than macros that contained IP addresses and expanded to several rules that needed to be evaluated separately. Rules can refer to tables so the rule will match any member of the table.
</p><p>
<code>
</code></p><pre><code> table <badhosts> persist counters file "/home/peter/badhosts"
# ...
block from <badhosts>
</code></pre><code>
</code>
<p>
Here the table is loaded from a file. You can also initialize a table in <tt><a href="https://man.openbsd.org/pf.conf">pf.conf</a></tt> itself, and you can even manipulate table contents from the command line without reloading the rules:
</p><p>
<code>
$ doas pfctl -t badhosts -T add 192.0.2.11 2001:db8::dead:beef:baad:f00d
</code>
</p><p>
In addition, several of the daemons in the OpenBSD base system such as <tt><a href="https://man.openbsd.org/spamd">spamd</a></tt>, <tt><a href="https://man.openbsd.org/bgpd">bgpd</a></tt> and <tt><a href="https://man.openbsd.org/dhcpd">dhcpd</a></tt> can be set up to interact with your PF rules.
</p><p>
<b>Rules</b>: The rules with the <i>verbs</i>, <i>criteria</i> and <i>actions</i> that determine how your system handles network traffic.
</p><p>
A very simple and reasonable baseline is one that blocks all incoming traffic but allows all traffic initiated on the local system:
<code></code></p><pre><code> block
pass from (self)
</code></pre><code>
</code>
<p>
The <tt>pass</tt> rule lets our traffic pass to elsewhere, and since PF is a stateful firewall by default, return traffic for the connections the local system sends out will be allowed back.
</p><p>
You probably noticed the configuration here references something called <tt>(self)</tt>.
</p><p>
The string <tt>self</tt> is a default macro which expands to all configured local interfaces on the host. Here, <tt>self</tt> is set inside parentheses <tt>()</tt> which indicates that one or more of the interfaces in <tt>self</tt> may have dynamically allocated addresses and that PF will detect any changes in the configured interface IP addresses.
</p><p>
This exact ruleset expanded to this on my laptop in my home network at one point:
</p><p>
<code></code></p><pre><code> $ doas pfctl -vnf /etc/pf.conf
block drop all
pass inet6 from ::1 to any flags S/SA
pass on lo0 inet6 from fe80::1 to any flags S/SA
pass on iwm0 inet6 from fe80::a2a8:cdff:fe63:abb9 to any flags S/SA
pass inet6 from 2001:470:28:658:a2a8:cdff:fe63:abb9 to any flags S/SA
pass inet6 from 2001:470:28:658:8c43:4c81:e110:9d83 to any flags S/SA
pass inet from 127.0.0.1 to any flags S/SA
pass inet from 192.168.103.126 to any flags S/SA</code></pre><code>
</code>
<p>
The <tt><a href="https://man.openbsd.org/pfctl">pfctl</a></tt> command here says to <b>v</b>erbosely <i>parse</i> but do <b>n</b>ot load rules from the <b>f</b>ile <tt><a href="https://man.openbsd.org/pf.conf">/etc/pf.conf</a></tt>.
</p><p>
This shows what the <i>loaded</i> ruleset will be, after any macro expansions or optimizations.
</p><p>
For that exact reason, it is <b>strongly</b> recommended to review the output of <tt>pfctl -vnf</tt> on any configuration you write before loading it as your running configuration.
</p><p>
If you look closely at that command output, you will see both the <tt>inet</tt> and <tt>inet6</tt> keywords. These designate IPv4 and IPv6 addresses respectively. PF since the earliest days has supported both, and if you do not specify which address family your rule applies to, it will apply to both.
</p><p>
But this has all been on a boring single host configuration. In my experience, the more interesting settings for PF use is when the configuration is for a host that handles traffic for other hosts, as a <i>gateway</i> or other intermediate host.
</p><p>
To forward traffic to and from other hosts, you need to enable forwarding. You can do that from the command line:
</p><p>
<code>
</code></p><pre><code> # sysctl net.inet.ip.forwarding=1
# sysctl net.inet6.ip6.forwarding=1
</code></pre><code>
</code>
<p>
But you will want to make the change permanent by putting the following lines in your <tt><a href="https://man.openbsd.org/sysctl.conf">/etc/sysctl.conf</a></tt> so the change survives reboots.
</p><p>
<code>
</code></p><pre><code> net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
</code></pre><code>
</code>
<p>
With these settings in place, a configuration (<tt><a href="https://man.openbsd.org/pf.conf">/etc/pf.conf</a></tt>) like this might make sense if your system has two network interfaces that are both of the <a href="https://man.openbsd.org/bge">bge</a> kind:
</p><p>
<code>
</code></p><pre><code> ext_if=bge0
int_if=bge1
client_out = "{ ftp-data ftp ssh domain pop3, imaps nntp https }"
udp_services = "{ domain ntp }"
icmp_types = "echoreq unreach"
match out on egress inet nat-to ($ext_if)
block
pass inet proto icmp all icmp-type $icmp_types keep state
pass quick proto { tcp, udp } to port $udp_services keep state
pass proto tcp from $int_if:network to port $client_out
pass proto tcp to self port ssh
</code></pre><code>
</code>
<p>
Your network likely differs in one or more ways from this example. See the references at the end for a more thorough treatment of all these options.
</p><p>
And once again, please do use the readability features of the PF syntax to keep you sane and safe.
</p><p>
</p><p>
</p><h2>A Configuration That Learns From Network Traffic Seen and Adapts To Conditions</h2>
<p>
With PF, you can create a network that learns. Fairly early in PF's history it occured to the developers that the network stack collects and keeps track of information about the traffic it sees, which could then be acted upon if the software became able to actively monitor the data and act on specified changes. So the <i>state tracking options</i> entered the <a href="http://man.openbsd.org/pf.conf"><tt>pf.conf</tt></a> repertoire in their initial form with the <a href="https://www.openbsd.org/37.html">OpenBSD 3.7</a> release.
</p><p>
A common use case is when you run an SSH service or really any kind of listening service with the option to log in, you will see some number of failed authentication attempts that generate noise in the logs. The password guessing, or as some of us say, password groping, can turn to be pretty annoying even if the miscreants do not actually manage to compromise any of your systems. So to eliminate noise in our logs we turn to the data that is anyway available in the state table, to track the state of active connections, and to act on limits you define such as number of connections from a single host over a set number of seconds.
</p><p>
The action could be to add the source IP that tripped the limit to a table. Additional rules could then subject the members of that table to special treatment. Since that time, my internet-facing rule sets have tended to include variations on
</p><p>
<code>
</code></p><pre><code> table <bruteforce> persist
block quick from <bruteforce>
pass inet proto tcp from any to $localnet port $tcp_services \
flags S/SA keep state \
(max-src-conn 100, max-src-conn-rate 15/5, \
overload <bruteforce> flush global)
</code></pre><code>
</code>
<p>
which means that any host that tries more than 100 simultaneous connections or more than 15 new connections over 5 seconds are added to the table and blocked, with any existing connections terminated.
</p><p>
It is a good practice to let table entries in such setups expire eventually. How long entries stay is entirely up to you.
</p><p>
At first I set expiry at 24 hours, but with password gropers like those caught by this rule being what they are, I switched a few years ago to at four weeks at first, then upped again a few months later to six weeks. Groperbots tend to stay broken for that long. And since they target any service you may be running, state tracking options with overload tables can be useful in a lot of non-SSH contexts as well.
</p><p>
A point that observers often miss is that with this configuration, you have a firewall that <i>learns</i> from the traffic it sees and <i>adapts</i> to network conditions.
</p><p>
It is also worth noting that state tracking actions can be applied to all TCP traffic and that they can be useful for essentially all services.
</p><p>
The <i>buzzwordability</i> potential in the learning configurations is enormous, and I for one fail to see how the <i>big names</i> have failed to copy or imitate this feature and <i>greytrapping</i> which we will look at later, and capitalize on products with those features.
</p><p>
The article <i><a href="https://bsdly.blogspot.com/2017/04/forcing-password-gropers-through.html">Forcing the password gropers through a smaller hole with OpenBSD's PF queues</a></i> has a few suggestions on how to handle noise sources with various other services. More on queues in a few moments.
</p><p>
</p><h2>The Adaptive Firewall and the Greytrapping Game </h2>
<p>
At the risk of showing my age, I must admit that I have more or less always run a mail service. Once TCP/IP networking became available in some form for even small businesses and individuals during the early 1990s, once you were connected, it was simply one of those things you would do. Setting up an SMTP service (initially wrestling with <tt>sendmail</tt> and it legendary <tt>sendmail.cf</tt> configuration file) with accompanying pop3 and/or imap service was the done thing.
</p><p>
Over time the choice of mail server software changed, we introduced content filtering to beat the rise of the trashy, scanny spam mail and, since the majority of clients ran <i>that</i> operating system mail-borne malware. But even with state of the art content filtering <i>some</i> unwanted messages would make it into users' inboxes often enough to be annoying.
</p><p>
So when <a href="https://www.openbsd.org/33.html">OpenBSD 3.3</a> shipped with the initial version of <a href="https://man.openbsd.org/spamd">spamd</a> it was quite a relief for people of my job category, even if that only would load lists of known bad senders' IP addresses and stutter at them one byte per second until the other side gave up.
</p><p>
Later versions introduced <i><a href="https://en.wikipedia.org/wiki/Greylisting_(email)">greylisting</a></i> — answering SMTP connections from previously unknown senders with a temporary local error code and only accepting delivery if the same host tried again — which reduced the load on the content filtering machines significantly, and the real fun started with the introduction of <i><a href="https://home.nuug.no/~peter/pf/en/spamd.greytrapping.html">greytrapping</a></i> in the version of <a href="https://man.openbsd.org/spamd">spamd(8)</a> that shipped with <a href="https://www.openbsd.org/33.html">OpenBSD 3.7</a>.
</p><p>
<i>Greytrapping</i> is yet another adaptive or learning feature. The system identifies bad actors by comparing the destination email address in incoming SMTP traffic from unknown or already greylisted hosts with a list of <i>known invalid</i> addresses in the domains the site serves. The <a href="https://man.openbsd.org/spamdb">spamdb(8)</a> command was extended to add features to add addresses to and delete from the spamtrap list.
</p><p>
Greytrapping was an extremely welcome new feature, and I adopted it eagerly. Soon after the feature became available, I set up for greytrapping. The spamtrap addresses were the ones initially addresses I fished out of my mail server logs — from entries produced by bounce messages that themselves turned out to be undeliverable at our end since the recipient did not exist — and after a few weeks I started publishing both the list of spamtraps <i>and</i> an hourly dump of currently trapped IP addresses.
</p><p>
The setup is amazingly easy. On a typical gateway in front of a mail server you instrument your <a href="http://man.openbsd.org/pf.conf"><tt>/etc/pf.conf</tt></a> with a few lines, usually at the top,
</p><p>
<code></code></p><pre><code> table <spamd-white> persist
table <nospamd> persist file "/etc/mail/nospamd"
pass in on egress proto tcp to any port smtp \
divert-to 127.0.0.1 port spamd
pass in on egress proto tcp from <nospamd> to any port smtp
pass in log on egress proto tcp from <spamd-white> to any port smtp
pass out log on egress proto tcp to any port smtp
</code></pre>
<p>
Here we even suck in a file that contains the IP addresses of hosts that should not be subjected to the <tt>spamd</tt> treatment.
</p><p>
In addition you will need to set up with the correct options for <a href="https://man.openbsd.org/spamd">spamd(8)</a> and <a href="https://man.openbsd.org/spamd">spamdlogd(8)</a> in your <tt><a href="https://man.openbsd.org/rc.conf">/etc/rc.conf.local</a></tt>:
</p><p><code>
</code></p><pre><code> spamd_flags="-v -G 2:8:864 -n "mailwalla 17.25" -c 1200 -C /etc/mail/fullchain.pem -K /etc/mail/privkey.pem -w 1 -y em1 -Y em1 -Y 158.36.191.225"
spamdlogd_flags="-i em1 -Y 158.36.191.225"
</code></pre><code>
</code>
<p>
The IP address here designates a sync partner, check out the <a href="https://man.openbsd.org/spamd">spamd(8)</a> man page for the other options. If you're interested, you can get the gory details of running a setup with several mail exchangers in the <i><a href="https://bsdly.blogspot.com/2012/05/in-name-of-sane-email-setting-up-spamd.html">In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play - A Full Recipe</a></i> article.
</p><p>
You probably do not need to edit the configuration file <a href="https://man.openbsd.org/spamd.conf"><tt>/etc/mail/spamd.conf</tt></a> much, but do look up the man page and possibly references to the <a href="https://www.bsdly.net/~peter/traplist.shtml">bsdly.net blocklist</a>. Finally, reload your PF configuration, start the daemons <a href="https://man.openbsd.org/spamd">spamd(8)</a> and <a href="https://man.openbsd.org/spamd">spamdlogd(8)</a> using <a href="https://man.openbsd.org/rcctl">rcctl</a>, set up a <a href="https://man.openbsd.org/crontab">crontab(5)</a> line to run <a href="https://man.openbsd.org/spamd-setup">spamd-setup(8)</a> at reasonable intervals to fetch updated blocklists.
</p><p>
The number of trapped addresses in the <a href="https://www.bsdly.net/~peter/traplist.shtml">hourly dump</a> has been anything from a few hundred in the earliest days, later in the thousands and even at times in the hundreds of thousands. For the last couple of years the number has generally been in the mid to low four digits, with each host typically hanging around longer to try delivery to an ever expanding number of invalid addresses in their database.
</p><p>
Just a few weeks ago, the list of <i>“imaginary friends”</i> rolled past 300,000 entries. The article <i><a href="https://bsdly.blogspot.com/2022/09/the-things-spammers-believe-tale-of.html">The Things Spammers Believe - A Tale of 300,000 Imaginary Friends</a></i> tells the story with copious links to earlier articles and other resources, while <i><a href="https://bsdly.blogspot.com/2013/04/maintaining-publicly-available.html">Maintaining A Publicly Available Blacklist - Mechanisms And Principles</a></i> details the work involved in maintaining a blocklist that is offered to the public.
</p><p>
It's been good fun, with a liberal helping of bizarre as the number of spamtraps grew, sometimes with truly weird contents.
</p><p>
</p><h2>Traffic Shaping You Can Actually Understand</h2>
<p>
You've heard it before: <i>Traffic shaping is hard. Hard to do and hard to understand.</i>
</p><p>
Traditionally traffic shaping was available on all BSDs in the form of ALTQ, a codebase that its developers labeled experimental and contained implementations of several different traffic shaping algorithms. One central problem was that the configuration syntax was inelegant at best, even after the system was merged into the PF configuration.
</p><p>
In OpenBSD, which runs development on a strict six month release cycle, the code that would eventually replace ALTQ was introduced gradually over several releases.
</p><p>
The first feature to be introduced was always-on, settable priorities with the keyword <tt>prio</tt>.
</p><p>
A random example shows that this configuration prioritises <tt>ssh</tt> traffic above most others (the default is 3):
</p><p>
<code>
pass proto tcp to port ssh set prio 6
</code>
</p><p>
While this configuration makes an attempt at speeding up TCP traffic by assigning a higher priority to lowdelay packets, typically ACKs:
</p><p>
<code>
</code></p><pre><code> match out on $ext_if proto tcp from $ext_if set prio (3, 7)
match in on $ext_if proto tcp to $ext_if set prio (3, 7)
</code></pre><code>
</code>
<p>
Next up, the <i>newqueue</i> code did away with the multiple algorithms approach and settled on the <i><a href="https://en.wikipedia.org/wiki/Hierarchical_fair-service_curve">Hierarchical fair-service curve</a></i> (HFSC) as the most flexible option that would even make it possible to emulate or imitate the alternative shaping algorithms from the ALTQ experiment.
</p><p>
HFSC queues are defined on an interface with a hierarchy of child queues, where only the “leaf” queues can be assigned traffic. We take a look at a static allocation first:
</p><p>
<code>
</code></p><pre><code> queue main on $ext_if bandwidth 20M
queue defq parent main bandwidth 3600K default
queue ftp parent main bandwidth 2000K
queue udp parent main bandwidth 6000K
queue web parent main bandwidth 4000K
queue ssh parent main bandwidth 4000K
queue ssh_interactive parent ssh bandwidth 800K
queue ssh_bulk parent ssh bandwidth 3200K
queue icmp parent main bandwidth 400K
</code></pre><code>
</code>
<p>
You then tie in the queue assignment, here with <tt>match</tt> rules
</p><p>
<code>
</code></p><pre><code> match log quick on $ext_if proto tcp to port ssh \
queue (ssh_bulk, ssh_interactive)
match in quick on $ext_if proto tcp to port ftp queue ftp
match in quick on $ext_if proto tcp to port www queue http
match out on $ext_if proto udp queue udp
match out on $ext_if proto icmp queue icmp
</code></pre><code>
</code>
<p>
which is definitely the way to add queueing to an existing configuration, and in my view also a good practice for configuration structure reasons. But you can also tack on <tt>queue this_or_that_queue</tt> at the end of <tt>pass</tt> rules.
</p><p>
There are two often forgotten facts about HFSC traffic shaping I would like to mention:
</p><p>
Traffic shaping is more often than not a matter of prioritizing which traffic you drop packets for, and no shaping at all takes place before the traffic volume approaches one or more of the limits set by the queue definitions.
</p><p>
One of the beautiful things about modern HFSC queueing is that you can build in flexibility, like this:
</p><p>
<code>
</code></p><pre><code> queue rootq on $ext_if bandwidth 20M
queue main parent rootq bandwidth 20479K min 1M max 20479K qlimit 100
queue qdef parent main bandwidth 9600K min 6000K max 18M default
queue qweb parent main bandwidth 9600K min 6000K max 18M
queue qpri parent main bandwidth 700K min 100K max 1200K
queue qdns parent main bandwidth 200K min 12K burst 600K for 3000ms
queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300
</code></pre><code>
</code>
The min and max values are core to that flexibility. Subordinate queues can 'borrow' bandwidth up to their own max values within the allocation of the parent queue. The combined max queue bandwidth can exceed the root queue's bandwith and still be valid. However the allocation will always top out at the allocated or the actual physical limits of the interface the queue is configured on.
<p>
For bursty services such as DNS in our example you can allow <tt>burst</tt> for a specified time where the allocation can exceed the queue's <tt>max</tt> value, still within the limits set on the parent queue.
</p><p>
Finally, the <tt>qlimit</tt> sets the size of the queue's holding buffer. A larger buffer <i>may</i> lead to delays since it packets may be kept longer in the buffer before sending on their way out to the world.
</p><p>
And if you noticed the name of that final, tiny queue, you probably have guessed correctly what it was for. The traffic from hosts that were caught in the <tt>spamd</tt> net was really horrible, as this <tt>systat queues</tt> display shows:
</p><p>
<code>
</code></p><pre><code> 1 users Load 2.56 2.27 2.28 skapet.bsdly.net 20:55:50
QUEUE BW SCH PRI PKTS BYTES DROP_P DROP_B QLEN BOR SUS P/S B/S
rootq on bge0 20M 0 0 0 0 0 0 0
main 20M 0 0 0 0 0 0 0
qdef 9M 6416363 2338M 136 15371 0 462 30733
qweb 9M 431590 144565K 0 0 0 0.6 480
qpri 2M 2854556 181684K 5 390 0 79 5243
qdns 100K 802874 68379K 0 0 0 0.6 52
spamd 1K 596022 36021K 1177533 72871514 299 2 136
</code></pre><code>
</code>
<p>
It was good, clean fun. And that display did give me a feeling of <i>Mission accomplished</i>.
</p><p>
There are several other tools in the PF toolset such as <a href="https://man.openbsd.org/carp">carp(4)</a> based redundancy for highly available service, <a href="">relayd(8)</a> for load balancing, application delivery and general network trickery, PF logs and the fact that <a href="https://man.openbsd.org/tcpdump">tcpdump(8)</a> is your friend, and several others that I have enjoyed using but I decided to skip since this was supposed to be a user group talk and a somewhat dense article.
</p><p>
I would encourage you to explore those topics further via the literature listed under the <i>Resources</i> heading for more on these.
</p><p>
</p><h2>Who Else Uses PF Today?</h2>
<p>
PF originated in OpenBSD, but word of the new subsystem reached other projects quickly and there was considerable interest from the very start. Over the years, PF has been ported from the original <a href="https://www.openbsd.org/">OpenBSD</a> to the other BSDs and a few other systems, including
</p><p>
</p><ul>
<li><a href="https://www.freebsd.org">FreeBSD</a></li>
<li><a href="https://www.netbsbsd.org">NetBSD</a></li>
<li><a href="https://www.dragonflybsd.org">DragonFlyBSD</a></li>
<li><a href="https://www.apple.com">Apple</a>'s MacOSX, IOS (via <a href="https://www.freebsd.org">FreeBSD</a>)</li>
<li>Blackberry (via <a href="https://www.netbsbsd.org">NetBSD</a>)</li>
<li><a href="https://www.oracle.com">Oracle</a>, in <a href="http://www.oracle.com/us/products/servers-storage/solaris/solaris11/overview/index.html">Solaris</a> 11.3 as one of two options, from <i>Solaris 12</i> the <a href="https://marc.info/?l=openbsd-tech&m=142822852613581&w=2">only packet filter</a>, replacing IPF. Also see this <a href="http://bsdly.blogspot.ca/2015/04/solaris-admins-for-glimpse-of-your.html">blog post</a> by yours truly.
</li></ul>
<p>
Other than Oracle with their port to Solaris, most ports of the PF subsystem happened before the OpenBSD 4.7 NAT rewrite, and for that reason they have kept the previous syntax intact.
</p><p>
There may very well be others. There is no duty to actually advertise the fact that you have incorporated BSD licensed code in your product.
</p><p>
If you find other products using PF or other <a href="https://www.openbsd.org/">OpenBSD</a> code in the wild, I am interested in hearing from you about it. Please comment or send email to <a href="mailto:nix@nxdomain.no">nix at nxdomain dot no</a>.
</p><p>
</p><h2>Resources for Further Exploration</h2>
<p>
<a href="https://www.openbsd.org/faq/pf/index.html">The PF User's Guide</a>
</p><p>
<a href="https://nostarch.com/pf3">The Book of PF</a> by Peter N. M. Hansteen
</p><p>
<a href="https://nostarch.com/openbsd2e">Absolute OpenBSD</a> by Michael Lucas
</p><p>
<a href="https://home.nuug.no/~peter/eurobsdcon2022/">Network Management with the OpenBSD Packet Filter toolset</a>, by Peter N. M. Hansteen, Massimiliano Stucchi and Tom Smyth (A PF tutorial, this is the <a href="https://2022.eurobsdcon.org">EuroBSDCon 2022</a> edition). An earlier, even more extensive set of slides can be found in the <a href="https://home.nuug.no/~peter/pf/newest/">2016-vintage PF tutorial</a>.
</p><p>
<a href="https://bsdly.blogspot.com/">That Grumpy BSD Guy</a> Blog posts by Peter N. M. Hansteen
</p><p>
<a href="https://undeadly.org/">OpenBSD Journal</a> News items about <a href="https://www.openbsd.org">OpenBSD</a>, generally short with references to material elsewhere.
</p><p>
</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-69977395101974417972022-09-14T18:55:00.014+02:002024-02-21T19:18:46.087+01:00Open Source in Enterprise Environments - Where Are We Now and What Is Our Way Forward?<p>
<i>We have been used to hearing that free and open source software and enterprise environments in Big Business are fundamentally opposed and do not mix well. Is that actually the case, or should we rather explore how business and free software can both benefit going forward?</i>
</p><p>
</p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://nxdomain.no/~peter/blogpix/shiny_blowfish.jpg" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="541" data-original-width="800" height="216" src="https://nxdomain.no/~peter/blogpix/shiny_blowfish.jpg" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"><i>Puffy, the <a href="https://www.openbsd.org/">OpenBSD</a> mascot, shiny version</i></td></tr></tbody></table><h2>Free and Open Source vs Enterprise and Business: The Bad Old Days</h2>
<p>
Open source, free software and enterprise IT environments have both been around for quite a while. I'm old enough to remember when the general perception was that the free exchange of source code was merely a game for amateurs, or at best an academic excercise. In contrast, the proper business way of doing things was to perhaps learn general principles and ideas from the academics, but real products for business use would be built to be sold as binary only, with any source code to be kept locked away and secret.
</p><p>
<!--
<b>Note:</b> This piece is also available <i>with</i> trackers but nice formatting <a href="https://bsdly.blogspot.com/2022/09/open-source-in-enterprise-environments.html">here</a>.
-->
<b>Note:</b> This piece is also available <i>without</i> trackers but more basic formatting <a href="https://nxdomain.no/~peter/opensource_enterprise_notes.html">here</a>.
<p>
If you're a little younger you may remember a time when <i>Windows NT is the future</i> was essentially gospel and all the business pundits were saying we would be seeing the last of Unix and mainframes both within only a handful of years.
</p><p>
Thinking back to the late 1980s and early 1990s it is hard to imagine now how clear the consensus seemed to be on the issue at that point. The PC architecture and a few other proprietary technologies was the way of business and the way forward. </p><p>
No discussion or dissent seemed possible.
</p><p>
</p><h2>Then, The Internet Happened</h2>
Then the Internet happened. What few people outside some inner circles were aware that what actually made the Net work was code that came directly out of the <a href="https://en.wikipedia.org/wiki/Berkeley_Software_Distribution">Berkeley Software Distribution</a>. <i>BSD Unix</i>, or simply <i>BSD</i> for short, was a freely licensed operating system that was the result of a rather informal cooperation of researchers in academia and business alike, originally derived from Unix source code.
<p>
When the United States Department of Defense wanted work done on <i>resilient</i>, <i>device independent</i>, <i>distributed</i> and <i>autoconfiguring</i> networks, the task of supplying the <i>reference implementation</i> for the TCP/IP stack, based on a stream of specifications dubbed <i>Request for comments</i> or <i>RFC</i>s, fell to the international group of developers coordinated by the <i>Computer Science Research Group</i> at the University of California's Berkeley campus. In short, the Internet came from BSD, which thanks to a decision made by the Regents of the University of California, was freely licensed.
</p><p>
The BSD sourced TCP/IP stack was part of <i>all</i> Internet capable systems until around the turn of the century, when Linux developers and later Microsoft started working on their own independent implementations. By that time it had been forcefully demonstrated to the developer community at least that open source code was indeed capable of scaling to industrial scale and beyond.
</p><p>
Due to a handful of accidents of history, mainly involving imperfect communications between groups of developers and combined with a somewhat misguided lawsuit involving the BSD code, it was Linux that became the general household term for free software in general and the re-emergence of Unix-like systems in the Internet connected server market space. Linux distributions came with a largely <a href="https://www.gnu.org/">GNU</a> userland as well as generous helpings of BSD code.
</p><p>
At roughly the same time Linux emerged, the BSD code became generally available via the <a href="htts://www.freebsd.org/">FreeBSD</a> and <a href="https://www.netbsd.org/">NetBSD</a> projects, and soon after the <a href="https://www.openbsd.org">OpenBSD</a> project, which forked from the NetBSD code base in the mid 1990s. For a more detailed history of these developments, see the three part series on the <a href="https://blog.apnic.net/">APNIC blog</a> starting with <a href="https://blog.apnic.net/2021/10/28/openbsd-part-1-how-it-all-started/">this piece</a>. If that piqued your interest, you may enjoy <a href="https://bsdly.blogspot.com/2021/08/recent-and-not-so-recent-changes-in.html">this piece</a> about some incremental improvements over time in OpenBSD.
</p><p>
</p><h2>The War on Linux and the Proliferation of Open Source Tools</h2>
<p>
During the 1990s and early 2000s the Internet and services of all kinds that ran on top of it expanded in all directions. That expansion had the effect of advancing the free unixlike systems such as Linux and the BSDs, which would run quite comfortably on commonly available hardware, along with an ever expanding number of development tools and software of all kinds to new categories of users.
</p><p>
The success of the open source software lead to what would be dubbed <i>The War on Linux</i>, a rather vicious defamation campaign executed in both PR campaigns and lawsuits, and driven mainly by the then-dominant desktop software vendor's ambition to dominate server space as well. One of the more bizarre sequences of Linux-targeting lawsuits was run by proxy, and is extensively documented at <a href="http://www.groklaw.net/index.php">groklaw.net</a> (<i>Note:</i> http-only site). It is worth noting that the process eventually lead to bankruptcy for the litigant.
</p><p>
Over the years it became clear to essentially everyone in the industry that open source tools were essential to development, and several practical aspects of developer life lead to ever increasing open source use. During the time of The War on Linux, the likes of Apple, Cisco, Netscaler (later acquired by Citrix) and Sun Microsystems (later acquired by Oracle) either incorporated open source code in their products and workflows, open sourced large parts of their own code or forked freely available code to base proprietary systems on. It may be worth discussing each of these approaches in detail later.
</p><p>
</p><h2>On to the Present: We All Use...</h2>
<p>
Fast forward to the present day, and I recently had colleagues sum up that in the enterprise environments we move in,
<br />
</p><blockquote><i>
Software is developed on Macs,<br />
deployed on a cloud somewhere,<br />
which more likely than not runs on Linux.</i><br />
</blockquote>
<p>
And the software itself is likely built with open source tools and pulls in dependencies from open source projects, possibly hosted on Github or other public sites.
</p><p>
Your software in all probability uses <b>some</b> open source. And even if you are not a developer, you most likely use open source tools that are integrated in your operating system or common application software or web services. </p>
<p>
On the client side of things, an ever increasing part of the volume comes from smartphones, tablets and the like, where the market share for open source based systems (Android and IOS) exceeds 90 percent. In a document we will come back to later, the Norwegian National Security Authority (NSM) <a href="https://nsm.no/fagomrader/digital-sikkerhet/nasjonalt-cybersikkerhetssenter/nyheter-fra-ncsc/apen-kildekode-i-den-digitale-leverandorkjeden">estimates</a> that approximately 90 to 98 per cent of all software in use to some extent has dependencies on open source software. Other relevant statistics can be found <a href="https://www.businessofapps.com/data/android-statistics/">here</a>, <a href="https://www.bankmycell.com/blog/how-many-android-users-are-there">here</a> and <a href="https://www.statista.com/statistics/236031/market-share-of-ios-in-global-smartphone-os-shipments/">here</a>. Or, if you're in a bit of a hurry: It is estimated that some 3.1 billion Linux-based Android phones are currently in use. In addtion, there is Apple, which we know has a significant amount of BSD code in their software.
</p><p>
It is of course worth noting that by now even the old open source arch-enemy Microsoft ships their offerings with what amounts to an almost complete Linux distribution as a subsystem. The same company regularly lobs cash over the wall to the likes of <a href="https://www.openbsdfoundation.org/">The OpenBSD Foundation</a> and regularly contributes to other open source projects. Not to mention that much of what runs in their Azure cloud is one way or the other Linux based.
</p><p>
</p><h2>Security: QA Your Supply Chain, Excercise the Right to Repair</h2>
<p>
Back in the days of The War on Linux, and to some extent still, we have often been faced with claims that open source software could either never be as secure as proprietary software or that open source software was inherently more secure than the closed source kind, because "<i>given enough eyes, all bugs are shallow</i>".
</p><p>
Both assertions fail because even without access to source code, it is possible to probe running software for vulnerabilities, and on the other hand the shallowness of bugs depends critically on the eyes looking being attached to people with sufficient competence in the field.
</p><p>
The public reaction to a couple of security incidents during recent years that generated a flurry of largely uninformed punditry are worth revisiting for the lessons that can be learned.
</p><p>
<b>The Solarwinds supply chain incident aka SUNBURST</b> (2020) - One of the most widely publicized yet mostly quite poorly understood security incidents in recent years emerged when it was revealed that adversaries unknown had been able to compromise the build computers where the binaries for their widely used network management software was built for distribution.
</p><p>
The SANS institute has produced a fairly thorough <a href="https://www.sans.org/blog/what-you-need-to-know-about-the-solarwinds-supply-chain-attack/">writeup</a> of the incident, which breaks down as follows: The first stage of a multi-stage compromise kit was included in binary distribution packages, complete with authentic signatures from the build system, that were largely put directly into production environments by network admins everywhere. The malware then went on to explore the networks they landed in, and through a process that made heavy use of crafted DNS queries and other non-obvious techniques, the miscreants were able to compromise several high security government and enterprise networks.
</p><p>
<b>Several open source component supply chain incidents</b> (2020 onwards) - Soon after the SUNBURST incident several incidents occured where popular open source components that other systems pulled in as dependencies started malfunctioning or were suddenly unavailable, causing complete malfunctions or loss of functionality such as a web service suddenly refusing to interact with specific networks.
</p><p>
The sudden breakage in open source components caused quite a bit of uproar, and predictably the chattering subset of the consulting class set about churning out dire warnings about the risk of using open source of any kind.
</p><p>
Watching from the sidelines it struck many open source oriented professionals, myself included, that the combination of these incidents carry an important lesson. It is obvious in a modern environment we suck in upgrades automatically and frequently, and that no untested code should ever be deployed directly to production.
</p><p>
<b>Blind trust versus the right to read (and educate yourself) and the right to repair</b> - In the case of proprietary, binary-only software, you have no choice but to trust your supplier and that they will address any defects in a timely manner. The upshot is that with proprietary, binary-only you do not have access to two important features of open source software: The right to read and study the code, and the right to repair any defects you find, potentially saving yourself potential service shutdowns or workarounds while the secret parts of your system get fixed elsewhere.
</p><p>
</p><blockquote>
The lesson to be learned is that you need to run quality assurance on your supply chain. <i>You may choose to trust, but you still need to verify.</i> That goes for open source and proprietary software both.
</blockquote>
<p>
This Norwegian felt slightly elated when reading that the Norwegian National Security Authority (NSM) provides essentially the same assessments in their published <a href="https://nsm.no/fagomrader/digital-sikkerhet/nasjonalt-cybersikkerhetssenter/nyheter-fra-ncsc/apen-kildekode-i-den-digitale-leverandorkjeden">recommendations</a>.
</p><p>
</p><h2>Contributing - Cooperating on Maintenance</h2>
<p>
As with any product it is entirely possible to be a relatively passive consumer, just install and use, and build whatever you need on top, interacting with the community only via downloading as needed from the mirror sites. Communicating via online forums, mailing lists or other channels is entirely optional.
</p><p>
If you are a developer or integrator with an ambition to make one or more opern source products central to your business either by using and contributing to an existing project or starting a new one, several approaches are possible.
</p><p>
Let's take a look at the strategies some big names adopted on open source in their products:
</p><p>
<b>Grab and fork, sell hardware:</b> The Netscaler load balancer and application delivery products were based on a fork of <a href="https://www.freebsd.org/">FreeBSD</a>.
</p><p>
They appear to have rewritten large parts of the network stack and devised a multifunctional network product on top, which among other things features a slick web GUI for most if not all admin tasks. </p><p>
If you look closely, Netscaler (since acquired and rebranded by <a href="https://www.citrix.com">Citrix</a>) appear to cultivate a <a href="https://citrix.github.io/">menagerie</a> of open source projects to interface with their products. </p><p>
However they appear not to have in particularly close contact with their main upstream. (It is worth noting that the BSD license does not require publishing changes to the code base.) When dropping to a shell on a Netscaler unit, last time I looked the output of <tt><a href="https://man.openbsd.org/uname">uname -a</a></tt> seemed to indicate that their kernel was still based on <a href="https://www.freebsd.org/security/unsupported/">FreeBSD 8.4</a>, which the FreeBSD web site lists as End of Life by August 1, 2015.
</p><p>
<b>Grab and fork, sell hardware, keep sync with your upstream:</b> Starting with the initial release of <a href="https://en.wikipedia.org/wiki/MacOS">macOS</a>, <a href="https://www.apple.com/">Apple</a> have maintained the software that drives their various devices, from phones to desktop computers and related services with generous helpings of open source code, along with what appears to be a general willingness to publish code and interact with upstream projects such as the <a href="https://www.freebsd.org">FreeBSD</a> project. Apple maintains the <a href="https://opensource.apple.com">Open Source at Apple</a> site for easy access to the open source components of their offerings.
</p><p>
This mode of open source interaction seems to be rather common, especially among network oriented suppliers of various specialty gear.
</p><p>
<b>Open source everyting, sell support</b>: Despite early scepticism from business circles, several companies have built successful companies on the model of participating or even driving the development of open sources systems or components, making support contracts (which may include early or privileged access to updates) as well as consulting services the main or sole source of company revenue.
</p><p>
<b>Decide what code is both good enough to publish and useful elsewhere:</b> Finally, for those of us in the services or consulting business who will occasionally write code that is not necessarily business specfic, the reasonable middle ground is just that. Identify code that meets the following criteria:
</p><ol>
<li>Was developed by yourself and cleared by your organization and other stakeholders such as your customer as such</li>
<li>Is high enough quality that you dare show it to others</li>
<li>Does not reveal core aspects of your clients' business</li>
<li>Is likely to be useful elsewhere too</li>
<li>Would be nice to have exposed to other sets of eyes in order do identify bugs and fix them</li>
</ol>
<p>
If you have code under your care in your organization that meets those criteria, you should in my opinion be seriously considering making that code open source.
</p><p>
Your next adventure will then be to pick an appropriate license.
</p><p>
</p><h2>Now for Policies and Processes - Do You Have Them?</h2>
<p>
If you have followed on this far, you probably caught on to the notion that it is wise to set up clear policies and procedures for handling code, open source or otherwise.
</p><p>
Keep in mind that
</p><p>
<b>A license is an assertion of authority</b>. A license is a creator's message to the world that states the conditions others must abide by when using, or if they allow it, change and further develop the code. </p><p>
Without a license the default regime is that only the person or persons who originated the code have the right to make changes or for that matter make further copies for redistribution. </p><p>
For that reason it is important to ensure that every element of your project has a known copyright and license. </p><p>
There have been quite a few instances of free software project rewriting functionally equivalent, or hopefully better, versions of whole subsystems because of unacceptable or unclear licenses (see the OpenBSD articles in the Resources section for some examples).
</p><p>
<b>Procedures and policies, you need them</b>. A self employed developer working on their own project is usually free to choose whatever license they please. In a corporate environment, any code developed is likely tied to a contract of some sort, which may or may not set the parameters of who holds the copyright or what licenses my be acceptable. The exact parameters of what can be decided by contract and what follows from copyright law my vary according to what jurisdiction you are in. When considering whether to publish your own code under an open source license, make sure all stakeholders (and certainly any parties to any relevant contract) agree on the policies and procedures.
</p><p>
<b>Keep it simple, for your own sake</b>. There are supposedly several hundred licenses in existence that the <a href="https://opensource.org/">Open Source Initiative</a> considers to be open source. In the interest of making life easier for anyone who would be interested in working on your code, please consider adopting one of those well-known licenses.
</p><p>
They range from the simplest, BSD or MIT style ones that run a handful of sentences and can be condensed to <i>you can do whatever you like with this material except to claim that you made it all yourself</i> to elaborate documents (the GNU GPL v3 comes to mind) which set out detailed terms and conditions, may require republication of any changes under the same terms, and could set up a specific regime with respect to patent disputes.
</p><p>
It is also important to consider that components you use in your project may have specific license requirements and that different licenses may contain terms that make the licenses incompatible in practice. </p><p>
My general advice here is, <i>make it as simple as possible, but no simpler</i>.
</p>
<p>
Or to rephrase slightly: The general advice for dealing with licenses echoes that of dealing with crypto code: Do not set out writing your own unless you know <i>exactly</i> what you are doing. Avoid that path if at all possible.</p><p>
<b>When in need, call in Legal (but make sure they understand the issues)</b>. Lawyers endure a lengthy education in order to pass the bar and turn to practicing law, but there is no guarantee that a person well versed in other business legalese has any competence at all when it comes to matters of copyright law. When you do turn to Legal for help, be very exacting and stern in insisting that they demonstrate a command of copyright basics and if at all possible have a reasonable real world understanding of how software is built.
</p><p>
As in, you really do not want to spend an entire afternoon or more explaning the difference between static and dynamic linking and why this matters in the face of a certain license, or that specific terms of different licenses deemed <i>open source</i> by the <a href="https://opensource.org/">Open Source Initiative</a> may in fact be incompatible in practice.
</p><p>
It is important to keep in mind that doing open source is about making our lives more productive and enjoyable by exchanging ideas between quality professionals, perhaps sharing the load of maintenance and leaving us all more resources to develop our competence and products further.
</p><p>
</p><h2>The Way Forward - The Work Goes On</h2>
<p>
So this is where we are today. Modern software development and indeed a goodly chunk of business and society in general depends critically on open source software.
</p><p>
If you enjoyed this piece (or became annoyed by any part of it) I would like to hear from you. I especially welcome comments from colleagues who have experience with open source use and/or development in enterprise settings. Of course if you are just curious about open source software in these settings, you are welcome to drop me a line too. I am most easily reachable via email <a href="mailto:nix@nxdomain.no">nix at nxdomain dot no</a>.
</p><p>
</p><hr />
I want to extend thanks to Malin Bruland and Knut Yrvin for excellent comments and proofreading.
<p>
</p><h2>Resources</h2>
<p>
All things open source (including an almost encyclopedic collection of licenses) at <a href="https://opensource.org/">The Open Source Initiative</a>
</p><p>
Wikipedia: <a href="https://en.wikipedia.org/wiki/Berkeley_Software_Distribution">Berkeley Software Distribution</a> about where the Internet came from
</p><p>
<a href="https://www.gnu.org/">The GNU Operating System</a>, supported by <a href="https://www.fsf.org/">The Free Software Foundation</a>
</p><p>
The <a href="https://www.freebsd.org/">FreeBSD</a> operating system project
</p><p>
<a href="https://opensource.apple.com/">Open Source at Apple</a>
</p><p>
Peter Hansteen: <a href="https://blog.apnic.net/2021/10/28/openbsd-part-1-how-it-all-started/">What every IT person needs to know about OpenBSD Part 1: How it all started</a>,<br /> <a href="https://blog.apnic.net/2021/11/05/openbsd-part-2-why-use-openbsd/">What every IT person needs to know about OpenBSD Part 2: Why use OpenBSD?</a>, <br /> <a href="https://blog.apnic.net/2021/11/11/openbsd-part-3-that-packet-filter/">What every IT person needs to know about OpenBSD Part 3: That packet filter</a> <br /> (or the whole shebang in the raw at <a href="https://bsdly.blogspot.com/2021/09/what-every-it-person-needs-to-know.html">bsdly.blogspot.com</a>)
</p><p><br /></p><p>
<p>Bradford Morgan White: <a href="https://www.abortretry.fail/p/the-berkley-software-distribution">The Berkeley Software Distribution</a></p>
Nasjonal Sikkerhetsmyndighet (NSM): <a href="https://nsm.no/fagomrader/digital-sikkerhet/nasjonalt-cybersikkerhetssenter/nyheter-fra-ncsc/apen-kildekode-i-den-digitale-leverandorkjeden">Åpen kildekode i den digitale leverandørkjeden</a> (Norwegian only)
</p><p>
Business of Apps: <a href="https://www.businessofapps.com/data/android-statistics/">Android Statistics (2023)</a>
</p><p>
Bank My Cell: <a href="https://www.businessofapps.com/data/android-statistics/">How Many Android Users Are There? Global and US Statistics (2023) (Source: https://www.bankmycell.com/blog/how-many-android-users-are-there)</a>
</p><p>
Statista: <a href="https://www.statista.com/statistics/236031/market-share-of-ios-in-global-smartphone-os-shipments/">Market share held by Apple iOS operating system of smartphone shipments from 1st quarter 2011 to 4th quarter 2022</a>
</p><p>
</p><h2>Appendix: License Complexity Measured by Word Count</h2>
<p>
While presenting on free and open source software in enterprise environments, the topic of license complexity and how to handle licensing matters usually generates questions of the type,
</p><p>
</p><blockquote>"Does doing open source mean we need to staff an Open Source Program Office? <p>
Does this not add a considerable measure of complexity to the development organization?</p><p>
Do the open source licenses mean we have to hire even more lawyers?"
</p><p></p></blockquote>
<p>
So I set out to do a little research. I figured that the number of words in a text is a useful, if not perfect indicator of complexity, so we could use that measure as a useful and easy to obtain proxy for measuring how complex the licenses we are likely to encounter are in practice.
</p><p>
I headed over to the <a href="https://opensource.org/">Open Source Initiative</a> website and their excellent collection of open source licenses. I then picked out the more common open source licenses, and for each license I pasted the text into the word counter at <a href="https://wordcounter.net/">wordcounter.net</a>, which in addition to the word count provides an indication of likely target audience "reading level" and estimated reading time as well as a few other measures of the text characteristics.
</p><p>
The results are in the following table:
</p><p>
</p><hr /><table border="0" cellpadding="0" cellspacing="0" style="border-collapse: collapse; table-layout: fixed; width: 723px;">
<colgroup><col style="mso-width-alt: 10026; mso-width-source: userset; width: 235pt;" width="313"></col>
<col style="width: 65pt;" width="87"></col>
<col style="mso-width-alt: 4778; mso-width-source: userset; width: 112pt;" width="149"></col>
<col span="2" style="width: 65pt;" width="87"></col>
</colgroup><tbody><tr height="21" style="height: 16pt;">
<td class="xl66" height="21" style="height: 16pt; width: 235pt;" width="313"><b>License
complexity by wordcount</b></td>
<td style="width: 65pt;" width="87"></td>
<td style="width: 112pt;" width="149"></td>
<td style="width: 65pt;" width="87"></td>
<td style="width: 65pt;" width="87"></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;"></td>
<td class="xl67">Word count</td>
<td align="right" class="xl67">Reading<br /> Level</td>
<td align="right" class="xl67">Reading<br />time</td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">1-clause BSD License</td>
<td align="center">160</td>
<td align="right"> College Graduate</td>
<td align="right">35s</td>
<td></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">2-clause BSD License</td>
<td align="center">191</td>
<td align="right">College Graduate</td>
<td align="right">42s</td>
<td></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">3-clause BSD License</td>
<td align="center">220</td>
<td align="right">College Graduate</td>
<td align="right">48s</td>
<td></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">GNU GPL v2.0</td>
<td align="center">2964</td>
<td align="right">College Graduate</td>
<td align="right">10m47s</td>
<td></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">GNU GPL v3.0</td>
<td align="center">5608</td>
<td align="right">College Graduate</td>
<td align="right">20m30s</td>
<td></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">Apache License v2.0</td>
<td align="center">1677</td>
<td align="right">College Graduate</td>
<td align="right">5m44s</td>
<td></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">Microsoft 365 Developer program license</td>
<td align="center">4803</td>
<td align="right">College Graduate</td>
<td align="right">17m28s</td>
<td></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">Microsoft Windows 11 OS license terms</td>
<td align="center">5766</td>
<td align="right">College Graduate</td>
<td align="right">20m58</td>
<td></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">Oracle End User License Agreement</td>
<td align="center">2554</td>
<td align="right">College Graduate</td>
<td align="right">9m17s</td>
<td></td>
</tr>
<tr height="21" style="height: 16pt;">
<td height="21" style="height: 16pt;">Adobe End-User License Agreement</td>
<td align="center">450</td>
<td align="right">College Graduate</td>
<td align="right">1m38s</td>
<td></td>
</tr>
<tr height="45" style="height: 34pt;">
<td class="xl65" height="45" style="height: 34pt; width: 235pt;" width="313">Apple
Licensed Application End User License Agreement</td>
<td align="center">1524</td>
<td align="right">College Graduate</td>
<td align="right">5m32s</td>
<td></td>
</tr><tr height="0" style="display: none;">
<td style="width: 235pt;" width="313"></td>
<td style="width: 65pt;" width="87"></td>
<td style="width: 112pt;" width="149"></td>
<td style="width: 65pt;" width="87"></td>
<td style="width: 65pt;" width="87"></td>
</tr></tbody></table>
<hr />
<p>
Once again, strict word count is not a perfect indicator of complexity — other measures such as sentence length and logical structure and interdependencies are likely to matter in real life scenarios.
</p><p>
</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-80201240891464631112022-09-11T18:12:00.010+02:002022-09-13T21:30:17.112+02:00Your 'Forgotten' Blockchain Account Needing Reactivation? It's a Scam<p> <i>Scammers are using 'forgotten' cryptocurrency accounts as bait for stealing the identities of the gullible and dishonest. You have been warned.</i></p><p>
I have never put any money into bitcoin or other cryptocurrencies.
</p><p>
<b>Note:</b> This piece is also available without trackers but only 'classic' formatting <a href="https://nxdomain.no/~peter/forgotten_blockchain_account_scam.html">here</a>
</p><p>
For a few years a long time ago I worked for economists at <a href="https://www.nhh.no">The Norwegian School of Economics</a>, and even though I never formally studied the subject, enough of the general concepts rubbed off on me that when the early rumblings of Bitcoin started making the rounds, I quickly identified rather basic failings in the proponents' logic and turned my attention elsewhere.
</p><p>
My conclusion was that either these people were innocently unaware of the basics of <a href="https://en.wikipedia.org/wiki/Money_supply">money supply</a> and its relationship with market pricing mechanisms, or they were not very innocently trying to part the more innocent with their money.
</p><p>
As history shows, any asset with no or very little <a href="https://en.wikipedia.org/wiki/Intrinsic_theory_of_value">intrinsic or objective value</a> and market price set solely by demand is bound to be an unstable asset and a high risk investment prone to bubbles such as the famous <a href="https://en.wikipedia.org/wiki/Tulip_mania">tulip mania</a>.
</p><p>
Looking at the development of Bitcoin over time, illustrated with a chart from Wikipedia:
</p><div class="separator" style="clear: both; text-align: center;"><a href="https://nxdomain.no/~peter/Bitcoin_usd_price.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="605" height="340" src="https://nxdomain.no/~peter/Bitcoin_usd_price.png" width="605" /></a></div><p>it becomes immediately clear that Bitcoin is <b>not</b> a stable asset. Remember the tulips.</p><p>
Be that as it may, I was only slightly annoyed by a phone call on Friday from what appeared to be a United Kingdom (+44) number. The lady at the other end had what I took to be an Indian accent and did not speak very clearly, but it eventually became clear that she was offering to help me regain access to my <i>'blockchain account'</i>.
</p><p>
As we already know, no <i>blockchain account</i> of mine has ever existed, and I told the lady so. I also helpfully offered her advice that she would most likely not be paid for her efforts and would be better off just putting down the phone and getting on with life in the great elsewhere.
</p><p>
She mostly ignored what I was saying and stuck to her script. In the end I told her to send whatever she had to offer by email, and she confirmed she had that address to hand.
</p><p>
I have never tried to keep either my email address or my phone number secret. In fact, for several separate periods of time I have been making an effort to be accessible to both existing and new customers. So it was no surprise that some random person on the Internet would have both of these items on file.
</p><p>
What did surprise me a bit was that the next day, an email message did turn up about the 'blockchain account'.
</p><p>
Here is the message in two screenshots:<br />
<br />
<img alt="blockchain message screenshot 1" src="https://nxdomain.no/~peter/blockchain_2022-09-10_01.png" />
<br />
<img alt="blockchain message screenshot 2" src="https://nxdomain.no/~peter/blockchain_2022-09-10_02.png" />
<br />
while the message itself is preserved <a href="https://nxdomain.no/~peter/20220910_blockchain@reactivationphase.com_peter@bsdly.net.txt">here</a> with headers (and fugly markup) intact.
</p>
<p>
The full body text of the message reads,
<p>
<blockquote>
Hello Mr /Mrs. Peter,<p>
We hope this email finds you well! This is Maya Miller from the Support - Team
Department of Blockchain. Further to our recent communication, please check
below the specific details as discussed. During this Informative Campaign we are
contacting every client that has an account in our system from of November 2012 until
March 2021.The system of Blockchain has blocked these accounts temporarily for
security reasons. A Bitcoin transaction is identified by it's unique form of Transaction
ID, which consists of a 64-digit alphanumeric string, that does not contain any kind of
specification.<p>
The reasons that have caused the registration of your name in Blockchain are
listed below:<p>
1- The account may have been created from your Previous Online Trading experiences
or any other Online form of Investment and the funds have been collected in a
Recovery Wallet.<p>
2- In 2011-2016 Bitcoin and Blockchain were everywhere (Commercials, Platforms and
all over the Internet) and 30% of our clients registered without even knowing the
results. Different platforms and several Online Games used to offer free Bitcoins as a
Promotion / Gift which made a great publicity to this Crypto Currency.<p>
Your total available balance in is 3.402 BTC and converted to standard GBP it goes up
to 60,361.58 GBP. The amount in Standard is variable, based on the Bitcoin price
movement. You can find a screenshot of the status of your account down below,
provided directly from our Security Department of Blockchain. NOTE!<p>
This confidential information is preferred not to be shared externally, as a
measure taken against any possible fraud activity! At the moment that an account is
created in Blockchain, the client must accept the terms and conditions in order to use
its services and features. According to the Terms of Use, when an account owner in
Blockchain has 3+ years of not showing any activity (including Logging In) Blockchain
proceeds as follows:<p>
1) If the account has no balance, it will get terminated immediately.<p>
2) If the account has a balance inside, it will go to "Dormant Status", until the owner
is informed. Further to the procedures listed above, the Support Department came up
with 3 (three) suggestions:<p>
a) Deactivate the account, and these funds will not be your responsibility anymore.<br/>
These funds will go back to the company as a property.<br/>
b) Reactivate your account, and withdraw the funds with the guidance of one of our
Refund Agents.<br/>
c) Reactivate the account and keep it for online trading.<br/>
The reactivation procedure consists as the following: Since the owner of the
account has not been active, he/she must show activity (movement) inside the
account. That would be done by placing a minimum deposit of 1% (0.034 BTC) of the
total funds that the client owns.<br/>
In order to reactivate the account you must provide the following documents:<br/>
1- One official document (Passport, Driving License of National ID)<br/>
2- Utility Bill or Bank Statement, according to client preference After you provide the
documents, and the system confirms that the order has been placed, your account will
go to "Active Status".<br/>
In order for you to log in to your crypto tracker platform, you should get
connected to one of our agents first.<p>
If you have any questions don't hesitate to reply back to this email and also provide us
your final decision. Thank you for being a member of our company!<br/>
Kind Regards,<p>
Maya Miller Support team_ Blockchain Security Department <br/>
Support Department and Controller of Financial Policies of Blockchain.<p>
© Blockchain 2022 I I- Luxembourg SA L-2340 Luxembourg, 1, Rue Philippe II- C/O
Legalinx Limited Tallis House 2 Tallis Street Temple London, EC4Y OAB, UNITED
KINGDOM<p>
Disclaimer: Please note Blockchain.com does not provide investment advice or
recommendations. The information made available to you is for your own general use
and is not based on any evaluation of your personal circumstances.
<p>
</blockquote>
<p>
The essence of this message is that as part of the 'account recovery' process, they want copies of your <i>'Passport, Driving License of National ID'</i> (sic), as well as <i>'Utility Bill or Bank Statement'</i>. While it is possible that some valid institutions would require one of these in their account recovery procedure, supplying several of these would in fact enable the party at the other end to impersonate me to a third party.
</p><p>
As this tweet shows,<br />
<br />
</p><blockquote class="twitter-tweet"><p dir="ltr" lang="en">There is an active ID theft attempt campaign via email about a supposed blockchain balance in an account tied to your email. Screenshots: <a href="https://t.co/jil8hsXxzx">https://t.co/jil8hsXxzx</a> <a href="https://t.co/lm6hkmwLGe">https://t.co/lm6hkmwLGe</a> saved email with headers <a href="https://t.co/k8g7C7vk2W">https://t.co/k8g7C7vk2W</a> <a href="https://twitter.com/hashtag/blockchain?src=hash&ref_src=twsrc%5Etfw">#blockchain</a> <a href="https://twitter.com/hashtag/scam?src=hash&ref_src=twsrc%5Etfw">#scam</a> <a href="https://twitter.com/hashtag/idtheft?src=hash&ref_src=twsrc%5Etfw">#idtheft</a> <a href="https://twitter.com/hashtag/bitcon?src=hash&ref_src=twsrc%5Etfw">#bitcon</a> <a href="https://twitter.com/hashtag/bitcoin?src=hash&ref_src=twsrc%5Etfw">#bitcoin</a></p>— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1568579372453330945?ref_src=twsrc%5Etfw">September 10, 2022</a></blockquote> <script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<br />
I quickly concluded this was an attempt at ID theft.
<p>
The lesson to be learned here is that if you <i>do not remember</i> ever having traded in cryptocurrencies, you almost certainly do not have a recoverable account with bitcoins in it.
</p><p>
Given the size of the amount promised it would not surprise me that they do get some takers. But scamming the scammers takes a certain level of skill most dishonest and greedy people do not possess.
</p><p>
And I suppose it will not surprise anyone that the tweet linked here produced several replies from bots or bot-workalikes that tried to hawk their favorite cryptocurrency (can we call them <b>#cryptotulips</b> yet?) project. Those were the easiest <b>block</b> decisions I have ever faced.
</p>
<hr>
<p>
<b>Update 2022-09-13:</b> Just as we were finishing lunch today at a pleasant Asian-style restaurant in Vienna, this happened:
<p>
<b>[2022-09-13 12:47:33]</b> Incoming call from <b>+447894561236</b>, correctly flagged by the Android phone app as <i>Potential fraud</i><br/>
<i>Me:</i> Peter Hansteen speaking<br/>
<i>Bitcon Lady:</i> Hello Mister Peter I am calling from the support team of Blockchain<br/>
<i>Me:</i> Hi there. Please go to <a href="https://bsdly.blogspot.com">bsdly.blogspot.com</a> and read about yourself. Bye then!<br/>
(hangs up)<p>
I must admit I had kind of hoped it would end there, but<p>
<b>[2022-09-13 12:48:01]</b> Incoming call from <b>+447894561236</b>, correctly flagged by the Android phone app as <i>Potential fraud</i><br/>
<i>Me:</i> Peter Hansteen speaking<br/>
<i>Bitcon Lady:</i> Hello Mister Peter would you please repeat what you just said<br/>
<i>Me:</i> What I said was, please go to <b>bsdly dot blogspot dot com</b> and read about yourself. Bye then!<br/>
<p>
With this addendum the article is now a full disclosure post.
<p>
I would however advise whoever maintains the list of flagged numbers that this number should not be tagged as <i>Potential fraud</i>, but more accurately as <i>Fraud</i>.
<p>
I am sure somebody, somewhere will appreciate your saving them a handful of bytes of storage.
<p>
<p>
</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com1tag:blogger.com,1999:blog-8616610987649128333.post-80154422916189109482022-09-07T21:29:00.006+02:002023-01-28T19:27:07.092+01:00The Things Spammers Believe - A Tale of 300,000 Imaginary Friends<p>
<i>It finally happened. Today, I added the </i>three hundred thousandth<i> (yes, </i><b>300,000</b><i>th) spamtrap address to my greytrapping setup, for the most part fished out of incoming traffic here, for spammers to consume.</i>
</p><p>
A little more than fifteen years after I first published a note about the public spamtrap list for my <a href="https://man.openbsd.org/spamd#GREYTRAPPING"><i>greytrapping</i></a> setup in a piece called <a href="https://bsdly.blogspot.com/2007/07/hey-spammer-heres-list-for-you.html"><i>Hey, spammer! Here's a list for you!</i></a>, the total number of imaginary friends has now reached three hundred thousand. I suppose that is an anniversary of sorts.
</p><p>
If this all sounds a bit unfamiliar, you can find the a brief explanation of the data collected and the list itself on the traplist <a href="https://www.bsdly.net/~peter/traplist.shtml">home page</a>.
</p><p>
And yes, the whole thing has always been a bit absurd.
</p><p>
That said, at the time in the mid <i>noughties</i> this greytrapping setup was announced, we had been battling scammy spam email and malicious software that also abused email to spread for some years, and we were eagerly looking for new ways to combat the spam problem which tended to eat into time and resources we would rather have used on other things entirely.
</p><p>
With that backdrop, collecting made up or generated, <i>invalid</i> email addresses in our home domains from various logs as traps for spammers seemed like an excellent joke and a fun way to strike back at the undesirables who did their damnedest to flood our users' mailboxes.
</p><p>
The initial annoncement shows the early enthusiasm, as does a followup later in the same month, <a href="https://bsdly.blogspot.com/2007/07/harvesting-noise-while-its-still-fresh.html"><i>Harvesting the noise while it's still fresh; SPF found potentially useful</i></a>. With a small helping of scepticism towards some of the other methods and ideas that circulated at the time, of course.
</p><p>
The various followups (search on the site using "<tt>spam</tt>, "<tt>antispam</tt>" or for that matter "<tt>spamd</tt>" and you will find quite a few) reveal that we went to work on collecting, feeding to <a href="https://man.openbsd.org/spamdb">spamdb</a> and publishing with a grin for quite a while.
</p><p>
I even gave a <a href="https://www.bsdcan.org/2007/schedule/events/13.en.html">talk</a> at <a href="https://www.bsdcan.org">BSDCan</a> 2007 about the experience up to that point around the time the traplist became public.
</p><p>
A few years later I posted a slightly revised version of that somewhat overweight paper as a blog post called <a href="https://bsdly.blogspot.com/2014/02/effective-spam-and-malware.html"><i>Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools</i></a> that has also grown some addenda and updates over the years.
</p><p>
I have revisited the themes of spam and maintaining blocklists generated from the traffic that hits our site a few times over the years.
</p><p>
The most useful entries are probably <a href="http://bsdly.blogspot.no/2013/04/maintaining-publicly-available.html"><i>Maintaining A Publicly Available Blacklist - Mechanisms And Principles</i></a> (April 2013) and <a href="https://bsdly.blogspot.com/2012/05/in-name-of-sane-email-setting-up-spamd.html"><i>In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play - A Full Recipe</i></a> (May 2012), while the summary articles <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html"><i>Badness, Enumerated by Robots</i></a> (August 2018) and <a href="https://bsdly.blogspot.com/2018/11/goodness-enumerated-by-robots-or.html"><i>Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting</i></a> offer some more detail on the life that includes maintaining blocklists and pass lists.
</p><p>
However, by the time the largest influx of new spamtraps, or <i>imaginary friends</i> if you will, happened during February through April of 2019 I was fresh out of ideas on how to write something entertaining and witty about the episode.
</p><p>
What happened was that the collection that at the time had accumulated somewhat more than fifty thousand entries, at a rate of no more than a few tens of entries per day for years, started swelling by several thousand a day, harvesting again from the greylist.
</p><p>
The flood went on for weeks, and forced me to introduce a bit more automation in the collecting process. I also I tried repeatedly to write about the sudden influx, but failed to come up with an interesting angle and put off writing that article again and again.
</p><p>
As I later noted in that year's only blog entry <a href="https://bsdly.blogspot.com/2019/12/the-year-2019-in-review-this-was-once.html"><i>The Year 2019 in Review: This Was, Once Again, Weirder Than the Last One</i></a>, starting January 30th 2019
</p><p>
</p><blockquote>
"I noticed via my scriptery that reports on such things that a large number of apparent bounce message deliveries to addresses made up of "Western-firstname.Chinese-lastname@mydomain.tld", such as <b>aaron.pu@bsdly.net</b> or <b>abby.na@bsdly.net</b>, had turned up, in addition to a few other varieties with no dot in the middle, possibly indicating separate sources."
</blockquote>
<p>
The IP addresses of the sending hosts were all in Chinese address ranges, and some weeks later, in April, we had ended up harvesting at least <i>120 000</i> unique new entries of a very similar kind before the volume went down rather abruptly to roughly what it had been before the indicent.
</p><p>
It is likely that what we were seeing was <a href="https://en.wikipedia.org/wiki/Backscatter_(email)">backscatter</a> from one or more <a href="https://en.wikipedia.org/wiki/Phishing">phishing</a> campaigns targeting Chinese users where for reasons only known to the senders they had chosen addresses in our domains as faked sender addresses.
</p><p>
Fortunately by the time this incident occurred I had started keeping a <a href="https://www.bsdly.net/~peter/spamtraps-dateadded.txt">log</a> of spamtraps by date added and the actual greylist dumps generated by the blocklist generating script can be retrieved so more detailed data can be assembled when and if someone can find the time to do so.
</p><p>
As I have kept repeating over the years, maintaining the spamtrap list and the blocklists sometimes turns up bizarre phenomena. Among the things that keep getting added to the spamtraps list are the products of <a href="https://bsdly.blogspot.com/2017/08/twenty-plus-years-on-smtp-callbacks-are.html">SMTP callbacks</a>, and another source of new variants seems to be simply shoddy data handling at the sender end. We keep seeing things that more likely than not are oddly truncated versions of existing spamtraps.
</p><p>
And finally, while the number of <i>trapped</i> hosts at any time seems to have stabilized over the last couple of years at the mid to low four digits, we seem to be seeing that low number of hosts aggressively targeting existing spamtraps, as detailed in the February 2020 <a href="https://bsdly.blogspot.com/2020/02/the-sextortion-scams-numbers-show-that.html">sextortion article</a>.
</p><p>
I have at times been astonished by what appears to be taken as useful addresses to send mail to, and I am sure the collecting and blocking activity will turn up further absurdities unheard of going forward. It is also quite possible that I have forgotten about or skipped over one or more weird episodes in the saga of the spamtraps and blocklists. I hope to be able to deliver, at odd intervals, writeups that are interesting, useful, funny -- at least one and hopefully all.
</p><p>
If you are interested in the issues I touch on here or if the data I accumulate would be useful in your research, please let me know via comments or email.
</p><p>
And yes, since I I know you have been dying to ask, this is the entry, collected in the evening (CEST) of 7 September 2022, which took our population of imaginary friends over the <i>300 000</i> line:
</p><p>
</p><hr />
<p>
</p><tt>Sep 7 19:52:18 skapet sshd[31622]: Failed password for invalid user ftpshared from 157.230.151.241 port 45876 ssh2
</tt>
<p>
</p><hr />
<p>which by the obvious processing we do here from failed login attempt to offcial spamtrap becomes
</p><p>
</p><hr />
<pre>Date Source Original Spamtrap
2022-09-07 SSH ftpshared<span> </span>ftpshared@bsdly.net
</pre>
<hr />
<p>
and joins the collection as entry number <b>300,000</b> (three hunded thousand).
</p><p>
By the time you read this, the total is likely to have increased yet again.
</p><p>
On a relevant mailing list it was been suggested that if you run a large scale email service, our <a href="https://www.bsdly.net/~peter/sortlist.txt">list of spamtraps</a> could be useful in filtering <i>outgoing</i> mail. If a customer tries to contact one of our imaginary friends, you probably need to pay extra attention to that customer.
</p><p>
</p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-17179441989502907972022-04-03T17:30:00.009+02:002022-06-10T16:32:55.632+02:00Spammers in the Public Cloud, Protected by SPF; Intensified Password Groping Still Ongoing; Spamware Hawked to Spamtraps<p><i>This past week was truly one for the blooper reel. A public cloud service provider let the great unwashed into the address ranges published as safe mailers via their SPF records, with hilarious if rather predictable results. Next up, we find an intensive advertising campaign for spamware aimed at our imaginary friends. And the password guessing aimed at an ever-expanding dictionary of non-existing users continues.</i>
</p>
<p>To the rest of the world, <a href="https://www.bsdly.net">bsdly.net</a> is known variously as a <a href="https://en.wikipedia.org/wiki/Honeypot_(computing)">honeypot</a>, a source of various kinds of <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">blocklists</a> or a frequent target of domain joejobs that contribute to the ever-expanding list of imaginary friends also know as <a href="https://www.bsdly.net/~peter/traplist.shtml">spamtraps</a>.</p>
<p>To me and a very small set of other people, it's home on the net, providing a set of services we need fairly painlessly on an <a href="https://www.openbsd.org/">OpenBSD</a> platform that rarely requires much work besides the odd <tt><a href="https://man.openbsd.org/pkg_add">pkg_add -u -D snap</a></tt> followed by <tt><a href="https://man.openbsd.org/sysupgrade">sysupgrade -s</a></tt> (yes, we jump from snapshot to snapshot on this one).</p>
<p>Then the past week served us with three separate events that, while actually harmless to our side, together serve to show that a certain subset of humans would perhaps be better diverted to activities that do not involve computers.</p>
<!--<p><b>Note:</b> This piece is also available, with nicer formatting but unfortunately also Blogger's trackers, <a href="">here</a>.</p>-->
<p><b>Note:</b> This piece is also available, with more basic formatting but with no trackers, <a href="https://www.bsdly.net/~peter/spf_protected_spammers_in_the_cloud.html">here</a>.</p>
<h2>A Public Cloud Provider Let Spammers Set Up Within Their Published SPF Ranges</h2>
<p>The first event started on Tuesday. While looking for something else entirely in my mail server logs, I noticed an unusually high number of delivery attempts to what definitely looked like spamtraps reaching the actual mail server. The earliest entry was from that morning:</p>
<p>
<tt>2022-03-29 06:28:31 H=ministeriodesanidad34.yomevacunoseguro.com [20.104.226.220] X=TLS1.2:ADH-AES256-GCM-SHA384:256 CV=no F=<root@ministeriodesanidad34.yomevacunoseguro.com> rejected RCPT <claus-leba@ehtrib.org>: Unknown user</tt>
</p>
<p>There were several thousand entries of that type (full log <a href="https://www.bsdly.net/~peter/hotmail-office365/yome_log.txt">here</a>, extracted source IP addresses <a href="https://www.bsdly.net/~peter/hotmail-office365/yome-ips.txt">here</a>). My initial impulse was of course to check the logs to see how they got past <tt><a href="https://man.openbsd.org/spamd">spamd</a></tt> in the first place. Oddly, I found no trace of any activity involving <tt>spamd</tt> and a random sampling of those IP addresses. That would in turn indicate that they had been pass-listed, most likely by being included in the permanent <tt>nospamd</tt> pass list, which we <a href="https://bsdly.blogspot.com/2018/11/goodness-enumerated-by-robots-or.html">generate</a> here mainly based on the published <a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework">SPF records</a> of domains we need to communicate with. </p>
<p>Again taking a random subset of the extracted IP addresses and using <tt><a href="https://man.openbsd.org/whois">whois</a></tt> identified the IP address range owner as a large cloud services provider that among other things also provide hosted or hybrid on premise plus hosted email service. This in turn means that domains that use those services also include large segments of that provider's IP address ranges in their SPF records. Not quite knowing what to do I tweeted,</p>
<blockquote class="twitter-tweet"><p dir="ltr" lang="en">Grumble. <br /><br />Spam campaigns sending from ranges in our <a href="https://twitter.com/hashtag/nospamd?src=hash&ref_src=twsrc%5Etfw">#nospamd</a> tables (and hitting actual SMTP service) due to being in a major <a href="https://twitter.com/hashtag/cloud?src=hash&ref_src=twsrc%5Etfw">#cloud</a> operator's <a href="https://twitter.com/hashtag/SPF?src=hash&ref_src=twsrc%5Etfw">#SPF</a> records, sending to a large chunk of our <a href="https://twitter.com/hashtag/spamtraps?src=hash&ref_src=twsrc%5Etfw">#spamtraps</a>. <br /><br />Still developing, blogworthy, Y or N?</p>— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1508854454438797314?ref_src=twsrc%5Etfw">March 29, 2022</a></blockquote> <script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<p>In addition to tweeting and looking for feedback (which was not huge but dominated by <b>Y</b> answers), I notified the relevant <tt>abuse@</tt> address by mail, including links to the log data and the IP addresses.
</p>
<p>
I also tweaked the log reader hinted at in <a href="https://bsdly.blogspot.com/2017/04/forcing-password-gropers-through.html">this</a> earlier piece so any attempt at delivering mail from that domain in the future will put the sending IP address safely away both in the spamd blocklist and in the safety of a table that is subject to <tt>block drop quick from</tt> for six weeks after the activity stops, and exported to downloadable <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">blocklists</a> as described in the article I referenced earlier. </p>
<p>The <tt>abuse@</tt> handlers at the company I am not naming explicitly here (yes, there's a unix command you can use to find out who if it matters to you) were quite responsive and said that the activity seemed to be coming from their public cloud section, and yes they were forwarding my data to their internal CERT. As a followup I suggested to them that using our slowly expanding list of spamtraps in their outbound filtering might be a good idea if they intend to offer <i>SMTP-for-hire</i> in the future.</p>
<p>What seems to have happened is that the miscreants here set up using a range of the provider's services including domain registration, DNS hosting, and judging from the consistent use of <tt>root@</tt> as the sender addess, set up some number of Linux virtual machines to do the spamming.</p>
<p>Before the activity stopped later in the week, we identified two more campaigns that fit the pattern. The data can be found here: <a href="https://www.bsdly.net/~peter/hotmail-office365/supportmetamask.online.txt">log entries</a> and <a href="https://www.bsdly.net/~peter/hotmail-office365/supportmetamask.online_ips.txt">IP addresses</a> for the second wave, <a href="https://www.bsdly.net/~peter/hotmail-office365/20220331_sufacturadeimpuestos.com_log.txt">log entries</a> and <a href="https://www.bsdly.net/~peter/hotmail-office365/20220331_sufacturadeimpuestos.com_ips.txt">IP addresses</a> for the third. Each of the campaigns appear to have stopped shortly after their domains were de-registered. I never saw the contents of the messages since not a single one appears to have inboxed here.</p>
<p>The episode has a few teachable items: First, that some subset of our list of spamtraps is indeed incorporated in the address lists used by gullible spammers and their customers, and second, that if you run a public cloud service, you need to pay attention to what your customers do and be wary of letting them use IP address ranges that have been announced as being really safe to accept mail from.</p>
<p>I notified the cloud provider that I would be writing an article about the events and asked them for any and all useful input they could provide. No such information has surfaced by the time of writing. If any useful information turns up from them after publication, I will of course update this piece accordingly.</p>
<h2>A Separate Set of Spammers Market Their Spamming Software, Intensively Targeting Our Imaginary Friends</h2>
<p>While the public cloud spammers thing was developing, I noticed another campaign that was actively targeting our spamtraps.
</p>
<p>
Sent from as far as I could tell only three IP addresses and with a total of 58 different subject lines all about spamming tools. It is possible that the campaigns did not target our spamtraps exclusively in our domains, but the <a href="https://www.bsdly.net/~peter/2022-04-01_hawking_spamtools_to_spamtraps.txt.gz">log archive</a> (1.5M compressed but expands to 40M raw) serves as testament that our imaginary friends are definitely targeted by some subset of the online marketing community, and they are pouring resources into doing that, one byte per second.</p>
<p>And once again, I have not seen the actual contents of the messages beyond what turns up in the logs after greytrapping kicks in. Not one of those messages found its way to an actual mailbox here. </p>
<h2>SSH Password Groping Still High, But Bursty</h2>
<p>As noted in <a href="https://bsdly.blogspot.com/2022/02/predicting-developments-in-real-world.html">a previous article</a>, SSH password guessing activity went up significantly in the days leading up to the Russian invasion of Ukraine in February. </p>
<p>In addition to the data referenced directly in the article, the archived logs and the summaries of numbers of attempts per day (<a href="https://www.bsdly.net/~peter/sshlogs/ssh_attempts_2022-03.txt">March</a> and subsequent month summaries are <a href="https://www.bsdly.net/~peter/sshlogs/">archived</a> along with other SSH log data, while the data for the <a href="https://www.bsdly.net/~peter/ssh_attempts_month_to_date.txt">current month so far</a> gets updates several times a day) the number of attempts per day are consistently on a higher level than before the Ukraine war started, with a new higher intensity episode ongoing as I type.</p>
<p>One interesting feature of the password guessing attempts during the last few days is that they feature a much larger number of new user names attempted than usual. This means that the list of spamtraps here is now growing at the highest rate since the episode involving what was likely one or more phishing campaigns targeting Chinese users during early 2019 as mentioned in my <a href="https://bsdly.blogspot.com/2019/12/the-year-2019-in-review-this-was-once.html">2019 summed up piece</a> published at the end of that year. </p>
<p>If you have further data on these or similar incidents that you are able to share or if you want to look further into these and similar incidents, please let me know. </p>
<p>If you find any errors in the material I publish or disagree with my sentiments, or if you find this article interesting, useful or annoying, please let me know, either in comments or via email.</p>
<hr />
<br />
Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com1tag:blogger.com,1999:blog-8616610987649128333.post-4670431243881368302022-02-27T15:49:00.017+01:002022-04-02T13:28:09.041+02:00Predicting developments in real world conflict from patterns of failed logins<p><i>Anecdotal evidence indicates that it may be possible to predict developments in real world conflicts from certain indicators in cybercrime traffic.</i></p><p>Is it possible to glean useful information about international developments or even predict real world attacks from the activity that we record in the logs of Internet-facing systems? </p><p><b>Note:</b> A trackerless (other than my webserver log) version of the article is now available on <a href="https://www.bsdly.net/~peter/Predicting_developments_in_real_world_conflict_from_patterns_of_failed_logins.html">my web site</a>.</p><p>Looking at data I collect for other, quite pragmatic, reasons I see a clear correlation between the run-up to the Russian invasion of Ukraine earlier this month and the password guessing activity targeting non-classified systems in my care.</p><p>I'll be backing up that assertion with data later, but first, a bit of background. </p><p>As returning readers already know, I have been running Internet facing systems for a select group of friends and family for decades. In the late noughties I noticed a pattern of slow, distributed password guessing that I dubbed <a href="https://bsdly.blogspot.com/2013/10/the-hail-mary-cloud-and-lessons-learned.html"><i>The Hail Mary Cloud</i></a>, summed up in the summary article linked here and links therein. The data I collect from those failed logins make it into a set of <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html"><i>blocklists</i></a>, along with data from a few other sources. And yes, this is also one source of new spamtraps, as noted in the blocklists article.<br /><br />A few years after the original <i>Hail Mary Cloud</i> events, in January 2016, I started seeing <i>Hail Mary</i>-like activity again, and started collecting data (available in the raw <a href="https://www.bsdly.net/~peter/sshlogs/">here</a>), but failing to see any new patterns worth writing about, never started a new article based on the data. Until now, that is. <br /><br />The table here has the totals for number of attempts per month since then (the table as a <span style="font-family: courier;">.csv</span> file is available <a href="https://www.bsdly.net/~peter/blogpix/failed_ssh_attempts_2016-2022.csv"><i>here</i></a>):</p><h4 style="text-align: center;">Failed SSH login attempts per month, 2016 - 2022</h4><p></p><table border="0" cellpadding="0" cellspacing="0" class="ta1"><colgroup><col width="99"></col><col width="99"></col><col width="99"></col><col width="99"></col><col width="99"></col><col width="99"></col><col width="99"></col><col width="99"></col></colgroup><tbody><tr class="ro1"><td class="Default" style="text-align: left; width: 2.258cm;"> </td><td class="ce1" style="text-align: right; width: 2.258cm;"><p><b>2016</b></p></td><td class="ce1" style="text-align: right; width: 2.258cm;"><p><b>2017</b></p></td><td class="ce1" style="text-align: right; width: 2.258cm;"><p><b>2018</b></p></td><td class="ce1" style="text-align: right; width: 2.258cm;"><p><b>2019</b></p></td><td class="ce1" style="text-align: right; width: 2.258cm;"><p><b>2020</b></p></td><td class="ce1" style="text-align: right; width: 2.258cm;"><p><b>2021</b></p></td><td class="ce1" style="text-align: right; width: 2.258cm;"><p><b>2022</b></p></td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Jan</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>27015</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>348020</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>17738</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>35143</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>34882</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>42866</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>2355</p></td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Feb</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>121675</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>329074</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>2115</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>32053</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>60605</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>39029</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>24218</p></td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Mar</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>62254</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>498613</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>4648</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>29839</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>37477</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>29575</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Apr</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>94335</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>271992</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>9588</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>38310</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>29941</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>27876</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>May</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>26428</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>106688</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>4782</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>55485</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>46207</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>24455</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Jun</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>71321</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>65966</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>10831</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>75515</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>21947</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>36292</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Jul</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>39088</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>49675</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>5865</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>47619</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>57082</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>20225</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Aug</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>162529</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>65899</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>7631</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>59421</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>14030</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>62002</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Sep</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>183196</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>26007</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>5804</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>85336</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>17814</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>31179</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Oct</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>165295</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>16109</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>8211</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>82020</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>38185</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>6812</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Nov</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>184660</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>28234</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>5395</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>58547</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>20734</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>3814</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr><tr class="ro1"><td class="ce1" style="text-align: left; width: 2.258cm;"><p>Dec</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>127288</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>15049</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>38320</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>82739</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>33650</p></td><td class="Default" style="text-align: right; width: 2.258cm;"><p>5509</p></td><td class="Default" style="text-align: left; width: 2.258cm;"> </td></tr></tbody></table><div class="separator" style="clear: both; text-align: center;"><br /></div>It is worth noting that these attempts in almost all cases lead to getting the host blocked and published in the <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">blocklist</a>, some immediately if they come in very quickly, others after a short time, such as if they try logging on as <span style="font-family: courier;">root</span>, <span style="font-family: courier;">admin</span> or a few other obvious ones.<div><br /></div><div>Anyway, the data shows that the typical number of attempts per month is at least in five digit numbers, with seasonal variations. But at the end of 2021 we saw a marked decrease in activity from October 2021 onwards, only to see a sharp rise again in February of 2022. </div><div><br /></div><div>The development becomes even more striking if we look at the per day data for February, complete up to about mid-day CET on the 27th:<br /><p></p><h4 style="text-align: center;">Failed SSH login attempts February 2022</h4><p>Feb 1: 66<br />Feb 2: 13<br />Feb 3: 50<br />Feb 4: 31<br />Feb 5: 35<br />Feb 6: 85<br />Feb 7: 13<br />Feb 8: 70<br />Feb 9: 28<br />Feb 10: 13<br />Feb 11: 32<br />Feb 12: 13<br />Feb 13: 48<br />Feb 14: 28<br />Feb 15: 30<br />Feb 16: 337<br />Feb 17: 2006<br />Feb 18: 1906<br />Feb 19: 1608<br />Feb 20: 2113<br />Feb 21: 2207<br />Feb 22: 2424<br />Feb 23: 1978<br />Feb 24: 2976<br />Feb 25: 3044<br />Feb 26: 2071<br />Feb 27: 992</p><p>The developments stand out even clearer when presented as a graph:<br /><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://www.bsdly.net/~peter/blogpix/failed_ssh_attempts_2016-2022.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="442" data-original-width="787" height="442" src="https://www.bsdly.net/~peter/blogpix/failed_ssh_attempts_2016-2022.png" width="787" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><br /></div><div>Again, it is worth noting that the data is complete only up to mid-day on February 27th. </div><div><br /></div><div>But we see clearly that the number of attempts (effectively attempts by new hosts, quickly blocked) increased slightly on the 15th, only to essentially explode into levels equal to or slightly higher than the previous normal roughly a week before the Russian invasion of Ukraine started in the early hours of Febuary 24.</div><div><br /></div><div>On Febuary 15th, activity that was obviously preparations for an attack on Ukraine had been underway for weeks, and it was obvious to me that the activity was part of a campaign to acquire new nodes for botnets to be used for any kind of cybercrime or disinformation bot that can be run from things that run a SSH service, typically Unix-ish systems. </div><div><br /></div><div>Even before the current Ukraine episode I had a vague feeling that similar correlations would be visible in the data collected around earlier international events, but by Febuary 17th I was fairly confident that the attack was imminent. <br /><br />Then on the day of the attack, I tweeted:<br /></div><blockquote class="twitter-tweet"><p dir="ltr" lang="en">My regular readers will probably not be surprised to hear that <a href="https://twitter.com/hashtag/hailmary?src=hash&ref_src=twsrc%5Etfw">#hailmary</a>-ish <a href="https://twitter.com/hashtag/ssh?src=hash&ref_src=twsrc%5Etfw">#ssh</a> <a href="https://twitter.com/hashtag/password?src=hash&ref_src=twsrc%5Etfw">#password</a> guessing is way up (from historically low levels) during the last few days. I ponder doing a writeup. Should I go ahead and do that?</p>— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1496971657533132803?ref_src=twsrc%5Etfw">February 24, 2022</a></blockquote> <script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script><div>I will of course be monitoring the situation for developments and may follow up with further analysis. <br /><br />If you do not want to wait for those write-ups, you are welcome to go ahead and analyse the data (as long as any reuse or resulting analysis credits the source) -- the archive is <a href="https://www.bsdly.net/~peter/sshlogs/">here</a> while the data for the month so far can be found <a href="https://www.bsdly.net/~peter/newpart.txt">here</a> (note: that link always points to freshest collection for the current month).<br /><br />If you too collect logs on similar activity and you are able to share data or analysis, I would like to hear from you.</div><div><br /></div><div><b>Update 2022-03-01:</b> The month ended, and the final data for February 2022 are in. The counts for February 27 and 28 were <i>1509</i> and <i>1138</i> respectively, continuing the decline in new hosts attempting that we had been seeing over the previous few days. The total number of failed attempts for that month ended up at 25873. </div><div><br /></div><div>The final month graph looks like this:<br /><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://www.bsdly.net/~peter/blogpix/Failed_ssh_attempts_per_day_February_2022_final.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="442" data-original-width="787" height="442" src="https://www.bsdly.net/~peter/blogpix/Failed_ssh_attempts_per_day_February_2022_final.png" width="787" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">The full data for February can be found in <a href="https://www.bsdly.net/~peter/newpart_2022-02.txt">this file</a> as well as in the current year's .zip archive in the <a href="https://www.bsdly.net/~peter/sshlogs/">archive</a> directory.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><b>Update 2022-03-05 11:50 CET:</b> We may be seeing another uptick in expectation of a propaganda-needing event. <br /><br />The numbers for failed <span style="font-family: courier;">ssh</span> logon attempts in March so far are:<br /><br /><div class="separator" style="clear: both;">Mar 595</div><div class="separator" style="clear: both;">Mar 1003</div><div class="separator" style="clear: both;">Mar 699</div><div class="separator" style="clear: both;">Mar 657</div><div class="separator" style="clear: both;">Mar 727<br /><br />Notice that the the number of attempts <i>before noon</i> today already exceeds the total for the previous day. We may be seeing the preparation for another offensive or other event in need of propaganda or other cybercrime support. Stay tuned for updates.</div></div><br /><div><b>Update 2022-03-09:</b> The month to date daily numbers for failed attempts are now available, updated at quasi-random intervals a few times a day as <a href="https://www.bsdly.net/~peter/ssh_attempts_month_to_date.txt">ssh_attempts_month_to_date.txt</a>. </div><div><br /></div><div><b>Update 2022-04-02:</b> As can be seen from the <a href="https://www.bsdly.net/~peter/ssh_attempts_month_to_date.txt">month to date numbers</a> (now covering April 2022) we are now seeing another busy day on the password guessing front with almost 2000 attempts already by only a little past noon CEST. For reference, the <a href="https://www.bsdly.net/~peter/sshlogs/ssh_attempts_2022-03.txt">numbers for March</a> can be found along with previous data in the <a href="https://www.bsdly.net/~peter/sshlogs/">archive</a> directory.</div><div><br /></div><div><br /></div>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com1tag:blogger.com,1999:blog-8616610987649128333.post-78057666609104997352021-09-30T18:44:00.007+02:002021-11-07T19:32:50.733+01:00What every IT person needs to know about OpenBSD<p><i>How to have fun with the world’s most important free software project<br /></i><br />"Functional, free and secure by default", OpenBSD remains a crucial yet largely unacknowledged player in the open source field. This talk aims to highlight the project's signature security features and development practices -- razor sharp focus on correct and secure code coupled with continuing code audit -- as well as the project's role as source of innovation in security practices and 'upstream' source for numerous widely used components such as OpenSSH, PF, LibreSSL and others.<br /><br />If you only have a few minutes to spare, the highlights are:<br /></p><ul style="text-align: left;"><li>OpenBSD has been around for more than 25 years (started October 1995)</li><li>OpenBSD is proactively secure with only 2 remote holes in default install in all those years</li><li>OpenBSD pioneered use of strong cryptography, the first free system to ship with IPSec (entangling itself in US export regulations in the process)</li><li>OpenBSD pioneered and is still leading in code audit, fixing similar bugs tree-wide when found</li><li>OpenBSD has all security enhancements enabled by default and are hard going on impossible to disable</li><li>OpenBSD is open source, free software and the project actively encourages independent verification of code quality and security.</li><li>Today OpenBSD is in use in many network-centric roles, even though it is a general purpose operating system albeit with a particular emphasis on security.</li><li>OpenBSD has a high profile quality image based on actual code quality and proven performance in real world use</li><li>OpenBSD is upstream (origin) for several widely used pieces of software such as OpenSSH, OpenBGPD, PF, OpenSMTPd, LibreSSL, iked, mandoc and a number of others. For a complete list, please see the <i><a href="https://www.openbsd.org/innovations.html">OpenBSD Innovations</a></i> page on the OpenBSD website.</li><li>OpenBSD has been ‘growing up in public’ with code generally accessible via <i><a href="https://www.openbsd.org/anoncvs.html">anonymous CVS</a></i> (the first of its kind) since 1995 – transparent process, development discussions on public tech@ mailing list</li><li>Developers would do well to study high quality (mainly) C source and how the project runs a 6 month release cycle like clockwork (with only a few notable exceptions).</li></ul><p style="text-align: left;"><b>Note:</b> <i>If you are more of a slides person you will be happy to hear that indeed the <a href="https://home.nuug.no/~peter/openbsd_needtoknow/">presentation</a> I used for this when given as a talk is available <a href="https://home.nuug.no/~peter/openbsd_needtoknow/">here</a> with the main highlights and little to no geek jokes.</i></p><p style="text-align: left;">Now with that out of the way, let's step back to where it all started.<br /></p><p style="text-align: left;"></p><p style="text-align: left;"><br /></p><h2 style="text-align: left;">OpenBSD: How it all started </h2><p style="text-align: left;">OpenBSD's history is to a large extent the history of the Internet itself. You may have heard of the time back in the 1980s when the likes of IBM and Digital were slugging it out in the corporate IT sphere and the US department of defence paid for experiments in distributed, device independent networking.<br /><br />That's when a loosely organized group of hackers somewhat coordinated by researchers at University of California's Berkeley campus rose to prominence with "BSD Unix", which by a sequence of happy accidents became the home of the reference implementation of the TCP/IP internet protocols.<br /><br />By the early 1990s, commercialization of the Internet had started, and the Berkeley Computer Science Research Group (CSRG) that had coordinated the efforts was set to be disbanded. In addition to the net itself, the main tangible product out of Berkeley was the Berkeley Software Distribution (BSD), often distributed on tapes in the mail but also available on the net itself, which had started as a collection of software for AT & T's Unix but had over the years been extended become a full featured Unix operating system.<br /><br />Several different groups wanted BSD to go on even if the CSRG did not, and several things happened in fairly rapid succession:<br /> </p><ul style="text-align: left;"><li>Lynne and Bill Jolitz ported BSD to Intel x86 (actually 80386sx), creating 386BSD. This was chronicled in a series of articles in <a href="https://en.wikipedia.org/wiki/Dr._Dobb%27s_Journal"><i>Dr Dobbs' Journal</i></a> (also see a more condensed <a href="https://www.salon.com/2000/05/17/386bsd/">summary</a> over at <a href="https://www.salon.com/2000/05/17/386bsd/">salon.com</a>)</li><li>Next up, hackers started sharing improvements to the 386BSD code as "patchkits", and two separate groups took the work further to form their projects: The <a href="https://www.freebsd.org">FreeBSD</a> group would be working on bringing the best possible BSD to PC-style hardware, while the <a href="https://www.netbsd.org/">NetBSD</a> group's ambition was to make BSD run on any hardware they could get their hands on. [See <i>Addendum</i> at the end of the article.]</li><li>A group of former CSRG employees formed <a href="https://en.wikipedia.org/wiki/Berkeley_Software_Design">BSDi Inc.</a> and marketed their product BSD/386 with among other things a contact phone number "1-800-ITS-UNIX". The activities of an actual corporation in turn triggered a lawsuit from the owners of the UNIX trademark over code copyrights.</li></ul><p style="text-align: left;"><br />The lawsuit was eventually settled -- only six files of several thousand in the tree were 'potentially encumbered' and had to be replaced, leaving both NetBSD and FreeBSD with a rush to replace the code which if I remember correctly was at least in part fairly central to the virtual memory subsystem.<br /><br /><a href="https://www.openbsd.org/">OpenBSD</a> came into existence a couple of years later, from a fork of the NetBSD code base in October 1995, with the initial release in July 1996.<br /><br />From the very start, the OpenBSD project has been running a code audit of the entire tree, focusing on code correctness and security. We want secure, correct code that makes up a usable system with sane defaults and complete and readable documentation. And we want that code to be available under a free license and actually available to the world as soon as it is committed to the project's version control.<br /><br />One of the early achievements of the OpenBSD project was anonymous CVS, which makes it possible for anyone on the internet to get the code with changes in near real time. This was a major break with the normal practice of most projects of the time, which would typically work in relative isolation on private mailing lists and at quasi-random intervals issue a release as a tarball on an FTP server somewhere.<br /></p><h3 style="text-align: left;">You are already an OpenBSD user!</h3><p style="text-align: left;">It is probably useful at this point to reveal that even if you do not know it, you are more likely than not using code with an OpenBSD origin right now. Your Apple product, be it iPad, iPhone or Mac, your Android device, your Cisco router, Solaris, Linux or other Unix or even your Microsoft product has some or a lot of OpenBSD originated code in it. We will get back some detail on that later.<br /> </p><h2 style="text-align: left;">OpenBSD: Code audit and security evolution</h2><p style="text-align: left;">But about the code audit. The activity runs roughly like this: </p><p style="text-align: left;">Read the code, understand what it does. <br /><br />Look for unsafe behaviors, assume a hostile environment. <br /><br />When you find a bug and fix it, look for similar code elsewhere in the tree and fix everywhere. <br /><br />You will be amazed how much different programmers think alike and make the same mistakes. Wash, rinse, repeat.<br /><br />That may sound somewhat unexciting, but careful study of how the code actually performs in real life situations lead to a number of innovations over the years, with a strong slant for being proactively secure, making it harder for bugs to actually do damage:<br /><br /></p><ul style="text-align: left;"><li>W^X -- memory can be writeable XOR executable</li><li>Address space randomization (ASLR) so the jump targets and gaps will vary for each execution</li><li>random-sized gaps inserted in the stack, again catching fixed-sized returns</li><li>unreadable, unwriteable guard pages at the end of malloc()ed chunks to catch overruns</li><li>privilege separation -- daemons run the bulk of their code as a non-privileged user, more likely than not in a shell-less chroot, coupled with privilege revocation, which means that daemons drop privilege as soon as possible</li><li>the pledge(2) system call to declare a profile to restrict program behavior to only specified operations and resources</li><li>the unveil(2) system call to restrict file system access to specified paths and permissions only</li><li>A fairly user visible change came when OpenBSD 6.2 introduced kernel address randomized link, or KARL, which sees to it that the kernel is relinked to a new, randomized layout for each boot. Once again, introducing randomness where none had been before is seen as a way to mitigate possible exploits based on code loading at predictable addresses.</li></ul><p style="text-align: left;"><br />All of those features have been integrated in the OpenBSD source tree, and with the developers admonished to adhere to the rule<br /><br /></p><p style="text-align: center;"> <i>"where it is possible to spot damage, fail hard".</i><br /></p><p style="text-align: left;"><br />-which means that poorly written software will crash a lot more often on OpenBSD than elsewhere. That in itself should make the platform attractive to developers. Exposing your code to a hostile environment and see it perform or fail can be quite entertaining and enlightening.<br /><br /></p><h2 style="text-align: left;">Usable, portable and secure</h2><p style="text-align: left;"><br />To complete the picture, it is useful to keep in mind that OpenBSD runs on a total of fourteen platforms. All platforms are self-hosting. Cross-compiling is only used in the early phase of porting to a new platform.<br /><br />And of course, in addition to purely maintaining existing code to run on diverse platforms, users and developers have real world needs that are addressed by developing new software, extending the features of existing programs, adding new functionality or even replacing programs or entire subsystems. <br /><br />Security is a many-faceted topic. Early on, OpenBSD stood out as the system that included real crypto in the base system, to the extent that exporting OpenBSD source code from the United States was technically illegal under that country's munitions export restrictions as they were defined at the time. </p><p style="text-align: left;">Fortunately for us, the project was always coordinated from Canada by undisputed project leader Theo de Raadt who lives in Canada. There is anecdotal evidence that US based developers would trek across the border for hackathons with clean slate equipment to install OpenBSD while in Canada and hack, that is, work on the system and would then legally bring the result back with them.<br /><br />One early application of crypto in OpenBSD was when a full IPSEC stack was included in the system in 1997. OpenBSD was the first free system to include IPSec by default in its base install. <br /><br />In a prime example of hacker humor of the time, a T-shirt featuring one of the early appearances of Puffy the blowfish that would become the project mascot touted the <a href="https://en.wikipedia.org/wiki/Blowfish_(cipher)">Blowfish</a> password hashing algorithm which remains the default on OpenBSD both with the picture caption "So long and thanks for all the passwords" just below Puffy on the front, along with the full source code of the blowfish function on the back. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://bsdly.net/~peter/blogpix/tshirt-4.gif" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="188" data-original-width="216" height="188" src="https://bsdly.net/~peter/blogpix/tshirt-4.gif" width="216" /></a></div><p style="text-align: left;">The expectation was that the T-shirt would be illegal to re-export from the US.<br /><br />In addition to attention to security and code correctness, one other important feature of OpenBSD is attention to intellectual integrity and insisting on clearly worded and unambiguous license to use and modify code and documentation that forms part of the system.<br /><br /></p><h2 style="text-align: left;">So why use OpenBSD? What is it like?</h2><p style="text-align: left;"><br />So what is OpenBSD like for a user or developer, and why is it better?<br /><br />I'd say the short version is that it's a real Unix. Unlike the Linuxes of the world that spent years muddling through an evolutionary succession of init systems and have ended up more or less settling on the ever expanding systemd which seems to have tentacles into everything and is on a clear course to replacing most of what we have traditionally thought of as the base system, OpenBSD has stayed with and refined the traditional BSD init so can have both uncluttered services management and a base system that consists of programs that for the most part adhere to the classical Unix philosophy that every program should do exactly one thing, and do that thing well.<br /><br />If you are a developer, you will also appreciate hearing that the base system of well designed programs that all have a readable and useful man page already contains basic Unix developer tools along with a C and C++ compiler -- clang where supported and gcc where necessary -- plus perl and a host of tools. Basically, everything that is needed to build the base system from a fresh checkout of the source code is contained in the base system on a default install.<br /><br /><br /></p><h2 style="text-align: left;">Ported software goes under <span style="font-family: courier;">/usr/local</span></h2><p style="text-align: left;"><br />Once you have the thing installed on whatever hardware you have, keeping in mind that you can run a selection of 14 platforms ranging from fairly ancient kit to modern hardware, you will likely turn to installing ported third party software from packages, using <a href="https://man.openbsd.org/pkg_add">pkg_add(8)</a> which will suck in whatever you tell it to fetch from the same mirror you installed from or what appears to be the most local one. <br /><br />More software is available on the more popular platforms than on the more, dare we say exotic ones. <br /><br />For the OpenBSD 6.9 release, the most mainstream platform amd64 came with 11310 prebuilt and installable packages, while mips64 had only 8182 and the mips64el platform is marked as (still building).<br /><br />Installing pre-built packages is almost always more convenient and is recommended in most cases, but if you for one reason or the other want to build your own from a cvs checkout of the ports tree, you are free to do so at the cost of your own time watching the process. <br /><br />Whichever route you choose to go, you will see that installing packages does not land you with any files in the directories used by the base system outside of a few that drop their configuration files in subdirectories under <span style="font-family: courier;">/etc </span>and add their combined startup/shutdown scripts to the collection in <span style="font-family: courier;">/etc/rc.d</span>. Anything else ends up under <span style="font-family: courier;">/usr/local</span>, and you can see why the installer by default sets up that file system on a separate and fairly roomy partition. My previous article <a href="https://bsdly.blogspot.com/2013/04/youve-installed-it-now-what-packages.html">You've installed it. Now what? Packages!</a> is a few years old, but gives you the main motivations and some background along with some pointers on packages practice.<br /><br /></p><div style="margin-left: 40px; text-align: left;"><b>Note: </b>If you are more of a <i>slides</i> person than a <i>fulltext</i> person, you may be relieved to hear that you can find the <a href="https://home.nuug.no/~peter/openbsd_needtoknow/">slides</a> for the talk this article is based on (and vice versa) <a href="https://home.nuug.no/~peter/openbsd_needtoknow/">here</a>. <br /></div><h2>The installer was always good, got better</h2>
<p>
When I found OpenBSD more than twenty years ago, my main Unix
exposure was from working with Linuxes and FreeBSD. What attracted me to
OpenBSD and finally had me buy an OpenBSD 2.5 CD set was the strong
focus on security and code correctness. When the CD set and the classic
wireframe daemon T-shirt finally arrived in the mail, I set about at
first to install it on whatever spare hardware I had lying around.
</p><div class="separator" style="clear: both; text-align: center;"><a href="https://www.bsdly.net/~peter/blogpix/tshirt-2.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="OpenBSD wireframe daemon head" border="0" data-original-height="305" data-original-width="288" src="https://www.bsdly.net/~peter/blogpix/tshirt-2.jpg" /></a></div><br /><p><br />
</p><p>
If I remember correctly, the first machine I tried installing
OpenBSD on was an 80386/33MHz with 8MB RAM and I think a 100MB IDE hard
disk. Which I can report sounded pretty crappy even then, but the thing
did work.</p><p>
</p><p>
The initial install was fairly straightforward, and when I started
poking around I found two things about myself and the new system:
Everything made sense, and everything I could think of had a readable man
page. So the first change I am aware of that made the world better with
OpenBSD was the decision to enforce the "No commit without
documentation" rule, which came into being early in the project's life,
probably roughly at the same time the OpenBSD developers gave us a
real-time view of development via <a href="https://www.openbsd.org/anoncvs.html">anonymous CVS</a>. You can see things happening in almost real time.</p><p>
</p><p>
It is worth mentioning that the installer has remained famously
non-graphical, text only. The reason the installer remains text-only is
that this is a major advantage that enables the developers and the users
to handle the fairly diverse collection of <a href="http://www.openbsd.org/plat.html">hardware platforms</a> that OpenBSD runs on with the same portable, familiar and compact code everywhere.
</p><p>
The installer was always scriptable and extensible, and over the years
the installer has added automatic, repeatable and scriptable installs
(dubbed <a href="https://man.openbsd.org/autoinstall">autoinstall(8)</a> which appeared in <a href="https://www.openbsd.org/55.html">OpenBSD 5.5</a> in 2014) and the <a href="https://man.openbsd.org/sysupgrade">sysupgrade(8)</a> extension (first found in <a href="https://www.openbsd.org/66.html">OpenBSD 6.6</a>
in 2019) that automates snapshot to snapshot or release to subsequent
release upgrades for all not too hacked-up configurations. Each of these
moments, or more specifically when the new code started appearing in
snapshots, had me appreciate the OpenBSD system a bit more, and made me
feel quality of life had improved.</p><p>
</p><p>
</p><p>
</p><h2>Now something for your laptop - hardware support</h2>
<p>
Fast forward some twenty-plus years and the last <a href="https://bsdly.blogspot.com/2021/07/the-impending-doom-of-your-operating.html">article</a> I published, and even got into Norwegian mainstream IT news site <a href="https://www.digi.no/artikler/debatt-en-skummel-fremtid-med-operativsystem-som-gar-til-11-eller-forbi-eller-en-rolig-oase-med-fri-programvare/511693">Digi.no</a>,
centers on a few moments involving new OpenBSD developments. It took
some interaction with OpenBSD developers, but those interactions lead to
my new laptop with an 11th generation Intel Core chipset working even
better with OpenBSD. Yes, OpenBSD developers and a significant subset of
their user base actually run OpenBSD on their laptops. I do use a Mac
and a work-issued Thinkpad with Ubuntu Linux too, but life is not
complete without an OpenBSD laptop.</p><p>
</p><p>
Now to be honest, what I saw within the space of a few days was
development that had me going from "Oh, sh*t, the SSD isn't recognized"
-- the controller was set to a RAID-ish mode by default -- through this
kernel panic:</p><p>
</p><div class="separator" style="clear: both; text-align: center;"><a href="https://www.bsdly.net/~peter/blogpix/20210528_085741_cropped.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="OpenBSD 6.9-current panic message" border="0" data-original-height="329" data-original-width="800" height="264" src="https://www.bsdly.net/~peter/blogpix/20210528_085741_cropped.jpg" width="640" /></a></div><br /><p>-- to seeing it all fully supported.</p><p>
The SSD problem turned out to be simple to fix: Simply find the
"Advanced" BIOS option that turned the pseudo-raid feature off and let
the operating system speak directly to the storage device.
</p><p>
For the rest there was a period of a couple of weeks I had to run with
not yet commited patches in a home baked kernel built from checkouts
from Jonathan Grey's git repo. When the code was committed to -current, I
could resume my normal <a href="https://man.openbsd.org/sysupgrade">sysupgrade(8)</a> routine, going from one development snapshot to the next.
</p><p>
The process, even with the need to build custom kernels for a while,
was actually quite pleasant, and when the support code went into the
main development branch, that too was a a moment when I felt my life had
been improved by changes in OpenBSD. The hardware support is now in
snapshots and will be in <a href="https://www.openbsd.org/70.html">OpenBSD 7.0</a> which is set to be released on November 1st, 2021.<br /><br /></p><h2 style="text-align: left;">Why use OpenBSD? IPSEC</h2><p>As I mentioned earlier, OpenBSD was the first free system to ship with IPSEC -- the tools for enabling encrypted network traffic -- in its base system. The first OpenBSD release with IPSEC was OpenBSD 2.1, which was released in 1997.<br /><br />The tools worked of course, but in the early days the complaint was that IPSEC was hard and near impossible to debug from an almost-working to a fully working setup.<br /><br />Further development took a while, but the tools got a major usuability upgrade in OpenBSD 3.8 with <span style="font-family: courier;"><a href="https://man.openbsd.org/ipsecctl">ipsecctl</a></span> and its human-readable configuration file <span style="font-family: courier;"><a href="https://man.openbsd.org/ipsec.conf">/etc/ipsec.conf</a></span> to serve as a friendlier front end to the IPSEC tools.<br /><br />A complete configuration for a minimal setup could look like this:<br /><br /><span style="font-size: xx-small;"><span style="font-family: courier;"># Set up two flows:<br /># First between the machines 192.168.3.14 and 192.168.3.100<br /># Second between the networks 192.168.7.0/24 and 192.168.8.0/24<br />flow esp from 192.168.3.14 to 192.168.3.100<br />flow esp from 192.168.7.0/24 to 192.168.8.0/24 peer 192.168.3.12<br /></span></span><br />This was a major step compared to other platforms. One famous example is preserved in a <a href="https://www.openbsd.org/papers/asiabsdcon07-ipsec/index.html">presentation</a> by Mathieu Sauve-Frankel at AsiaBSDCon 2007, where he demonstrated that setting up the equivalent of the config shown here with Microsoft tools took the user through a sequence of no less than 36 dialog boxes, and still there was a distinct possibility that the configuration was not actually a working one.<br /><br />The problem in both the Microsoft implementation and others was that the developers had not really given any thought to the user experience. The majority of the options the user was required to set had sensible defaults and could easily have been hidden from view. <br /><br />The standards documents the developers had worked from were fairly unclear, so it is not entirely unreasonable that <i>"it was hard to write, so it should be hard to use"</i> was a factor in how the products ended up from a user experience perspective. <br /><br />What defaults would be actually sensible would perhaps not be clear to the developer from the specification, and the only way to see what would be the sensible default would be experience with actual use in the field. The OpenBSD developers who wrote <i>ipsecctl</i> came with extensive experience of using IPSEC and decided that IPSEC did not in fact need to be so hard. The defaults should make sense.<br /><br />The next milestone in IPSEC development on OpenBSD came with Internet Key Exchange (IKE) protocol support in <a href="https://www.openiked.org/">OpenIKED</a> in OpenBSD 3.8 with <span style="font-family: courier;"><a href="https://man.openbsd.org/iked">iked</a></span> and <a href="https://man.openbsd.org/ikectl"><span style="font-family: courier;">ikectl</span></a>. For convenience, <span style="font-family: courier;">ikectl</span> is able to generate configurations for Windows and macOS clients too. That might save you the agony of clicking through dozens of dialog boxes at the client end.<br /><br /></p><h2>The thing that lured me in</h2>
<p>
But I hear you ask, what made me turn into an almost all-in OpenBSD user?
</p><p>
Back in 2001 I was still only experimenting with OpenBSD, but my experience with Linux and <tt>iptables</tt>
had made me long for a switch to a saner firewall. I had done some
small experiments with the IPF firewall that was in OpenBSD until the
2.9 release. Then, as some of us will remember, the it was discovered
that IPF's license was in fact not free, so it needed to be replaced.
</p><p>
There was a distinct rush, not quite a stampede, to replace IPF over
the months that followed. Fortunately, the new code that replaced the
previous packet filter proved to perform better. The OpenBSD Packet
filter, dubbed PF for short, had been born and made its debut in <a href="https://www.openbsd.org/30.html">OpenBSD 3.0</a>
in December 2001. The release had originally been planned for November,
but was pushed out a month to hack the "working prototype" packet
filter into something usable.</p><p>
</p><p>
Almost needless to say, this turn of events finally pushed me to take
the final steps to replace the Linux gateways I had in place with
OpenBSD ones. I was pleasantly surprised to find that not only did they
perform well, but they also came with complete and reasonably well
documented tools so I could understand what was going on. That's how I
got started on the process that lead to among other things writing <a href="https://nostarch.com/pf3">The Book of PF</a> and taking that text through three editions so far. But more about that later.</p><p>
</p><p>
It is worth noting that the IPFilter copyright episode spurred the OpenBSD developers to perform a <i>license audit</i>
of the entire source tree and ports in order to avoid similar
situations in the future. This activity ran for some months and
uncovered a number of potential problems. Theo de Raadt summed up the
effort in a <a href="https://marc.info/?l=openbsd-misc&m=104570938124454&w=2">message</a> to the <tt>openbsd-misc</tt> mailing list on February 20th, 2003.
</p><p>
What they found when they started looking was that there was a
significant number of files that were in fact not under a free license,
much like the entire IPF subsystem had been. Those needed to be
replaced. Other parts had either no license or no copyright stated. In
some cases the developers gave explicit permission to continuing use,
but quite a few things needed to be rewritten with a free license so
OpenBSD and other free software would be able to move forward without
copyright problems.
</p><p>
I later heard in a rather informal setting that among the no copyright
and/or no license cases, it was usually possible to track down the
developers via version control system logs or mailing list archives. In a
large number of those cases, the initial reaction was along the lines "<i>Say what? Are people still using that?</i>".
</p><p>
</p><p>
</p><h2>SSH, open and better</h2>
<p>
PF was written from scratch to replace a subsystem that it turned out
was illegal to use in an open source context. But it was not the first
time the OpenBSD project had performed a <i>nonlibreectomy</i>, that is, taken on the task of replacing code for license reasons.</p><p>
</p><p>
A few years earlier it had become clear that the original developer of the secure shell system <tt><a href="https://man.openbsd.org/ssh">ssh</a></tt>
had commercial ambitions and the license for the software had changed
in a proprietary direction. After a bit of deliberation on how to
resolve the situation, the OpenBSD developers started digging around for
earlier versions of the code that had been published with an acceptable
license. Then they forked their version from the last version they
found that still had free license. Next came an intensive period of
re-introducing the features that were missing in the old code.</p><p>
</p><p>
The result was introduced as OpenSSH in <a href="https://www.openbsd.org/26.html">OpenBSD 2.6</a> in 1999. Over the next few years <a href="https://www.openssh.com/">OpenSSH</a>
grew a portable version that started grabbing market share rapidly. The
last I heard OpenSSH's market share is somewhere in the high nineties
percent.</p><p>
</p><p>
With a state of the art secure shell subsystem in place and growing all
sorts of useful features, the time finally came to end unencrypted shell
login sessions on OpenBSD. OpenBSD's <tt>telnetd</tt> was moved to the CVS attic in time for <a href="https://www.openbsd.org/38.html">OpenBSD 3.8</a>, which was released November 2005.</p><p>
</p><p>
One other notable thing about OpenSSH is that it was the first daemon to
be properly privilege separated, a model practice that debuted with the
overhauled OpenSSH in <a href="https://www.openbsd.org/32.html">OpenBSD 3.2</a>
in March 2002. Since then privilege separation has been put in place in
all daemons where it made sense to do so, and it is now a signature
part of the secure by default stance of all newer OpenBSD daemons.</p><p>
</p><p>
</p><p>
</p><h2>And yes, that packet filter</h2>
<p>
I mentioned PF, the OpenBSD packet filter, earlier. I must confess that
PF has been an important part of my life in various contexts since the
early noughties. Over the years, things I have written have contributed
to creating the popular but actually wrong perception that OpenBSD was
primarily a firewall operating system. There are a lot of useful and fun
features that turned up in or in connection with PF over the years and
were pioneered by OpenBSD. Some features were ported to or imitated in
other systems, while others remain stubbornly OpenBSD only.</p><p>
</p><p>
So I will touch on some of my favorite PF and PF-attached features, in quasi-random but almost chronological order.</p><p>
</p><p>
</p><p>
</p><h2>Beating up spammers with OpenBSD spamd(8) since OpenBSD 3.3</h2>
<p>
When I started playing with OpenBSD in general and PF in particular
way back when, I was already responsible for the SMTP mail service for
my colleagues. My gateways by then ran OpenBSD, while the mail server <tt>rosalita</tt>, named after a <a href="https://en.wikipedia.org/wiki/Bruce_Springsteen">Springsteen</a> song, was not too badly specced server running <a href="https://www.freebsd.org/">FreeBSD</a> with <a href="https://www.exim.org/">exim</a> as the mail transfer agent that fed the incoming messages to <a href="https://spamassassin.apache.org/">spamassassin</a> and <a href="https://www.clamav.net/">clamav</a> for content filtering before handing off to user mailboxes.
</p><p>
So when it dawned on me that I could set up <a href="https://man.openbsd.org/spamd">spamd(8)</a> the spam deferral daemon on the internet-facing gateway and save load on the poor suffering <tt>rosalita</tt> that was running hot with content filtering, I was quick to implement a setup that sucked in well known block lists.</p><p>
</p><p>
</p><p>
</p><h2>Going grey, then trapping</h2>
<p>
The effect was obvious and immediate, the mail server's fans grew noticeably quieter. When greylisting was introduced in <tt>spamd</tt> soon after, I implemented that too, and witnessed yet another drop in pitch and intensity of the sound from <tt>rosalita</tt>'s
fans. Then a couple of releases later greytrapping -- the practice of
adding IP addresses of incoming SMTP connections to blocklists if the
attempted delivery is aimed at a known-bad address in the target domain
-- was introduced, and that sounded like enough fun that I just went
ahead and did it.</p><p>
</p><p>
The idea of detecting spam senders by the bogus addresses they were
already trying to deliver to just sounded too good to not try. And we
knew that getting started would be pretty easy too. We had seen rejects
for addresses that had never existed in our domains in our mail server
logs for quite a while, so it was simply a matter of harvesting from a
fairly bountiful source and adding stuff that we were sure would never
ever be actually deliverable here to the spamtrap list. I think the
first setup had only a couple of hundred entries in it, but I did not
note the exact number at the time.</p><p>
</p><p>
By July 2007 I had decided to <a href="https://bsdly.blogspot.com/2007/07/hey-spammer-heres-list-for-you.html">publish</a>
both the list of spamtrap addresses and an hourly dump of the
greytrapped addresses. Both remain free to download. The list of
spamtraps, harvested from various log sources, by now numbers just over
270,000 imaginary friends, while the number of trapped hosts is
typically in the 3000 to 5000 range. We occasionally see the list swell
to 20,000 or more when high volume <a href="https://bsdly.blogspot.com/2020/02/the-sextortion-scams-numbers-show-that.html">campaigns</a> run with bad address lists fed to them. I am pretty sure it went over 100,000 at one point.
</p><p>
It's fun to watch, and it looks like a significant subset of the
spamtraps have made it into the address lists of active spam operations.
I frankly never thought I would still be collecting spam traps from
logs all these years later. Yes, it all sounds a bit absurd, but it is
effective for keeping our mailboxes largely spam free, even though it
feels at times like running a weird found object-ish art project.
Anyway, a summary of the lists we publish can be found in <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">this article</a>.</p><p>
</p><p>
</p><p>
</p><h2>The brutes, the password gropers and the state tracking options</h2>
<p>
If you run an SSH service or really any kind of listening service with
the option to log in, you will see some number of failed authentication
attempts that generate noise in the logs. The password guessing, or as
some of us say, password groping, turned out to be annoying enough that
OpenBSD 3.6-current and later <a href="https://www.openbsd.org/37.html">OpenBSD 3.7</a>
introduced a set of features to use data that would anyway be available
in the state table, to track the state of active connections, and to
act on limits you define such as number of connections from a single
host over a set number of seconds.</p><p>
</p><p>
The action could be to add the source IP that tripped the limit to a
table. Additional rules could then subject the members of that table to
special treatment. Since that time, my internet-facing rule sets have
tended to include variations on</p><p>
</p><p>
</p><pre>table <bruteforce> persist
block quick from <bruteforce>
pass inet proto tcp from any to $localnet port $tcp_services \
flags S/SA keep state \
(max-src-conn 100, max-src-conn-rate 15/5, \
overload <bruteforce> flush global)
</pre><p>
</p><p>
which means that any host that tries more than 100 simultaneous
connections or more than 15 new connections over 5 seconds are added to
the table and blocked, with any existing connections terminated.
</p><p>
It is a good practice to let table entries in such setups expire eventually. At first I followed the <a href="https://man.openbsd.org/spamd">spamd(8)</a>
defaults' example and set expiry at 24 hours, but with password gropers
like those caught by this rule being what they are, I switched a few
years ago to at four weeks at first, then upped again a few months later
to six weeks. Groperbots tend to stay broken for that long. And since
they target any service you may be running, state tracking options with
overload tables can be useful in a lot of non-SSH contexts as well.
</p><p>
It is also worth noting that state tracking actions are useful for essentially all services. The article <a href="https://bsdly.blogspot.com/2017/04/forcing-password-gropers-through.html">Forcing the password gropers through a smaller hole with OpenBSD's PF queues</a> has a few suggestions on how to handle noise sources with various other services.
</p><p>
One final point I would like to make about the state tracking and actions is that much like the <i>greytrapping</i> feature of <tt>spamd</tt>, this feature gives you the tools to build a configuration that <i>adapts</i> to network conditions and <i>learns</i> from the traffic it sees. </p><p>While this does not rise to the level of being an actual Artificial Intelligence or AI, this has enough <i>buzzwordability</i>
potential that I remain to this day extremely puzzled that none of the
other big names at least imitated those features in their own products
and marketed for all it would be worth. </p><p>I certainly know what <i>I</i>
would have done in their position. But then I am more engineer than
marketer and in the contexts where I call the shots, the best option is
just to keep running OpenBSD.
</p><p>
</p><h2>We went to modern queueing</h2>
<p>
OpenBSD has had traffic shaping available in the <i>ALTQ</i> subsystem
since the very early days. ALTQ was rolled into PF at some point, but
the code was still marked experimental 15 years after it was written,
and most people who tried to use it in anger at the time found the
syntax inelegant at best, infuriating or worse at most times.</p><p>
</p><p>
So Henning Brauer took a keen interest in the problem, and reached the
conclusion that all the various traffic shaping algorithms were not in
fact needed. They could all except one be reduced to mere configuration
options, either as setting priorities on <tt>pass</tt> or <tt>match</tt> rules or as variations of the theme of the mother algorithm <i>Hierarchical Fair Service Curve</i> (HFSC for short).
</p><p>
Soon after, another not-small diff was making the rounds. The patch was applied early in the <a href="https://www.openbsd.org/55.html">OpenBSD 5.5</a> cycle, and for the lifetime of that release older ALTQ setups were possible side by side with the new queueing system.</p><p>
</p><p>
The feedback I get is that the saner syntax in the new queueing system
lead to more users taking up traffic shaping. Here is the queue setup
that I came up with for one of my sites:</p><p>
</p><p>
</p><pre>queue rootq on $ext_if bandwidth 20M
queue main parent rootq bandwidth 20479K min 1M \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>max 20479K qlimit 100
queue qdef parent main bandwidth 9600K min 6000K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>max 18M default
queue qweb parent main bandwidth 9600K min 6000K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>max 18M
queue qpri parent main bandwidth 700K min 100K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>max 1200K
queue qdns parent main bandwidth 200K min 12K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>burst 600K for 3000ms
queue spamd parent rootq bandwidth 1K min 0K max 1K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>qlimit 300
</pre>
<p>
while tying the queues into the subsequent rules with a set of <tt>match</tt> rules just following that block.
</p><p>
This is what triggered the need to write the third edition of <a href="https://nostarch.com/pf3">The Book of PF</a>.
The book includes descriptions of both the new and the old system as
well as tips on how to make a smooth transition. The ALTQ code was
removed from OpenBSD during the <a href="https://www.openbsd.org/56.html">OpenBSD 5.6</a> cycle, but continues to live on in some form in FreeBSD and NetBSD.</p><p>
</p><p>
And yes, if you think my queues setup punishes spammers a bit more in addtion to being subjected to <a href="https://man.openbsd.org/spamd">spamd(8)</a>, you're right.</p><p>
</p><p>
</p><p>
</p><h2>pflow(4) offers network insights lite</h2>
<p>
Everybody who has been tasked with looking after a network has at some
point been at least a little curious about what actually moves around
there. At times we will see situations where it is essential for
troubleshooting purposes to see the traffic flows with data about
endpoints, packets and bytes transferred, protocol and so forth.</p><p>
</p><p>
If you do not need to see the data itself, but rather the metadata, the <i>NetFlow</i> standard and its close cousin <i>IPFIX</i> offers just that. Netflow tools existed as packages on OpenBSD already, but from <a href="https://www.openbsd.org/45.html">OpenBSD 4.5</a> PF has the <tt>pflow</tt> state tracking option, paired with the <a href="https://man.openbsd.org/pflow">pflow(4)</a> virtual network interface which together offer a full netflow sensor package.</p><p>
</p><p>
Set up one or more <a href="https://man.openbsd.org/pflow">pflow</a> interfaces to send data to one or more collectors, and add the <tt>pflow</tt>
option to specific rules or as a state default and you have started
your collecting. You can even have metadata for traffic matching
specific rules going to separate <a href="https://man.openbsd.org/pflow">pflow</a> devices and collectors.</p><p>
</p><p>
My field notes in <a href="https://bsdly.blogspot.com/2014/02/yes-you-too-can-be-evil-network.html">Yes, You Too Can Be An Evil Network Overlord - On The Cheap With OpenBSD, pflow And nfsen</a>
offers some practical examples and insights, including how we used a
pflow setup to track down a noisy machine on a somewhat critical network
as well as some pointers to valueable further reading. </p><p>
</p><p>
</p><p>
</p><h2>LibreSSL, the great deobfuscation</h2>
<p>
People tell me they think that the reason <a href="https://www.libressl.org/">LibreSSL</a> was created was the <a href="https://en.wikipedia.org/wiki/Heartbleed">Heartbleed</a> bug, but no, actually not, just damn close.</p><p>
</p><p>
The LibreSSL project was in fact started a few weeks before heartbleed
became common knowledge. LibreSSL is the result of a group of OpenBSD
developers taking the existing OpenSSL code and starting to fix it.</p><p>
</p><p>
This time it was not a matter of a bad license. No, this was the result
of the number of OpenBSD developers who took a look at the <a href="https://www.openssl.org/">OpenSSL</a>
code that had been part of the OpenBSD base system since quite early
on, and turned away in disgust and with symptoms of physical pain,
reached a critical mass of sorts. I had heard OpenBSD developers
complain about the absolute horror of the OpenSSL code for at least ten
years. The code quality was just that bad.</p><p>
</p><p>
What happened next was that a group of hardened OpenBSD developers
grabbed the OpenSSL code and started two activities in parallel. One was
looking in the OpenSSL request tracker for bugs that had not been
addressed. The other was reformatting the OpenSSL code into something
resembling the OpenBSD style of readable and maintainable C.</p><p>
</p><p>
With the code in more readable form, discovering what it did became
easier. In addition to a few obvious eye-stinging bugs the LibreSSL
developers found a number of oddities, including, but not limited to</p><p>
</p><ul><li>Code was never deleted even when it became irrelevant or obsolete</li><p>
</p><li>OpenSSL did not use the system memory allocation system,
but rather opted for their own which never actually deallocated memory,
but rather used LIFO recycling, and could easily be made to put private
info into logs</li><p>
</p><li>all written in "OpenSSL C", which according to <tt>beck@</tt> is a dialect of the "worst common denominator"</li></ul>
<p>
It is worth digging out the various articles and presentations made by
LibreSSL developers over the years, with specific emphasis on Bob Beck's
BSDCan talk on the first 30 days of LibreSSL (available on <a href="https://www.youtube.com/watch?v=GnBbhXBDmwU">youtube</a>), which is the original source of the term <i>code <a href="https://en.wikipedia.org/wiki/Flensing">flensing</a></i>.
</p><p>
Since the <a href="http://www.openbsd.org/56.html">OpenBSD 5.6</a>
release in 2014, LibreSSL has been the default TLS library in OpenBSD.
LibreSSL has been ported elsewhere based on the -portable variant.</p><p>
</p><p>
For my own part I can only attest to not ever running into a TLS problem
that was LibreSSL's fault. It probably still has bugs, but it is a lot
more of a healthy choice than its predecessor.</p><p>
</p><p>
</p><p>
</p><h2>This was my list of life improving OpenBSD events - I'd love to hear yours</h2>
<p>
As I warned earlier, this has been about my personal list of OpenBSD events that I remember fondly.</p><p>
</p><p>
I am sure your list is at least a little different. I am sure there are things from the <a href="https://www.openbsd.org/innovations.html">innovations</a> page that I have simply forgotten about.
</p><p>
Each release comes with a detailed list of changes, such as <a href="http://www.openbsd.org/plus69.html">this one</a> for OpenBSD 6.9, and the page has pointers back to the equivalent pages for previous releases.
</p><p>
I would love to hear about your favorite OpenBSD moments.</p><p>
</p><p>
</p><hr />
<p>
</p><h2>More items for your OpenBSD reading</h2>
<a href="https://www.openbsd.org/">www.openbsd.org</a> is the official OpenBSD web site. If you want to donate, go to the <a href="https://www.blogger.com/">donations</a> page and find the most appropriate option. Corporate entities may prefer to donate via <a href="https://www.openbsdfoundation.org/">The OpenBSD Foundation</a>, which is a Canadian non-profit corporation.
<p>
<a href="https://undeadly.org">undeadly.org</a> is the OpenBSD Journal news site.
</p><p>
<a href="https://undeadly.org/">bsdly.blogspot.com</a> My rant^H^H^H^Hblog posts
</p><p>
<a href="https://flak.tedunangst.com/">https://flak.tedunangst.com/</a> Ted Unangst (tedu@) on developments
</p><p>
Michael W Lucas: <a href="https://www.nostarch.com/openbsd2e">Absolute OpenBSD, 2nd edition</a>
</p><p>
Peter N. M. Hansteen: <a href="https://www.nostarch.com/pf3">The Book of PF, 3rd edition</a>
</p><p>
Henning Brauer: <a href="https://quigon.bsws.de/papers/2015/eurobsdcon/">OpenBSD sucks (… least)</a>
</p><p>
</p><hr /><h4 style="text-align: left;">Addendum 2021-11-06</h4><div><div>The original statement in the article that the two groups (NetBSD and FreeBSD) were only vaguely aware of each other in the early days has been disputed by at least one patchkit era participant, Tom Ivar Helbekkmo, who wrote in to say,</div><div><blockquote>"That is not entirely true. When Bill Jolitz didn't include patches from the Internet community in 386bsd 0.1, and then again not in 0.2, Chris Demetriou took the initiative to fork the project, and call ours NetBSD.</blockquote></div></div><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><div><div><p style="text-align: left;"> It soon became apparent, however, that there were divergent goals for further development. This led to the creation of FreeBSD. So we had one community that amicably divided itself into two separate groups."</p></div></div></blockquote><div><div>In my own experience, however, those who joined in the post-patchkit era of both projects frequently seem to be unaware of this aspect of the early days. </div></div><div><br /></div><div><br /></div>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com4tag:blogger.com,1999:blog-8616610987649128333.post-58755625254278397532021-08-22T19:57:00.011+02:002022-07-04T15:57:26.379+02:00Recent and not so recent changes in OpenBSD that make life better (and may turn up elsewhere too)<p><i>Known to be "functional, free and secure by default", the <a href="https://www.openbsd.org/">OpenBSD</a> operating system has played an important role in open source for more than a quarter century. It has also been fairly central to what I have done for the last two decades and some. What follows is my personal view of what life with OpenBSD has been like, with an emphasis on moments and developments that I feel made life, or at least my life, better.</i></p>
I will assume that you know already that one of the signature features of the OpenBSD project is the continuous code audit and the sharp focus on secure and correct code. The audit by itself has produced a number of improvements, including a stream of bugfixes with bugs of a similar kind fixed in the whole tree and even the occasional subsystem rewrite. In addition, even for a free operating system project, life just happens. The world changes around us and drives the developers to take up fresh approaches to both new and well known problems and in the process develop code in ways that improves life for us all.
<blockquote><p><i>The Norwegian tech news site <a href="https://www.digi.no/">digi.no</a> took this article in as a three part series, see parts <a href="https://www.digi.no/artikler/debatt-nylige-og-ikke-helt-nylige-endringer-i-openbsd-som-gjor-livet-bedre/512893">1</a>, <a href="https://www.digi.no/artikler/debatt-openbsd-det-som-lokket-meg-inn-i-dette/513068">2</a> and <a href="https://www.digi.no/artikler/debatt-openssl-et-tilfelle-av-kodekvalitet-som-bare-var-helt-utrolig-ille/513122">3</a> if you want to read this in my native language. </i><br /></p></blockquote><p>If you are not that familiar with OpenBSD the system or project, my <i>"<a href="https://home.nuug.no/~peter/openbsd_and_you/">OpenBSD and you</a>"</i> talk, which I update occasionally, might be a not too bad place to start. But in this article I will focus on some specific moments when I felt that changes in <a href="https://www.openbsd.org/">OpenBSD</a> made my life better. These are the things that made me start and go on advocating the system.
</p><p>
</p><h2>So who am I and what can I offer?</h2>
<p>
My name is Peter Hansteen. I have worked in information technology and information technology related things like documentation since the late 1980s. I am in the process of transitioning from a "Security Engineer" role into a "Cloud Expert" one, and across several other roles and titles I have always done a bit, or a lot of Unix system and network administration. At most times you will find me on The Other West Coast, specifically in Bergen, Norway.</p><p>
</p><p>
</p><p>
</p><div style="margin-left: 40px; text-align: left;"><b>Note: </b>If you are more of a <i>slides</i> person than a <i>fulltext</i> person, you may be relieved to hear that you can find the <a href="https://home.nuug.no/~peter/openbsd_moments/">slides</a> for the talk this article is based on (and vice versa) <a href="https://home.nuug.no/~peter/openbsd_moments/">here</a>. <br /></div><h2>The installer was always good, got better</h2>
<p>
When I found OpenBSD more than twenty years ago, my main Unix exposure was from working with Linuxes and FreeBSD. What attracted me to OpenBSD and finally had me buy an OpenBSD 2.5 CD set was the strong focus on security and code correctness. When the CD set and the classic wireframe daemon T-shirt finally arrived in the mail, I set about at first to install it on whatever spare hardware I had lying around.
</p><div class="separator" style="clear: both; text-align: center;"><a href="https://www.bsdly.net/~peter/blogpix/tshirt-2.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="OpenBSD wireframe daemon head" border="0" data-original-height="305" data-original-width="288" src="https://www.bsdly.net/~peter/blogpix/tshirt-2.jpg" /></a></div><br /><p><br />
</p><p>
If I remember correctly, the first machine I tried installing OpenBSD on was an 80386/33MHz with 8MB RAM and I think a 100MB IDE hard disk. Which I can report sounded pretty crappy even then, but the thing did work.</p><p>
</p><p>
The initial install was fairly straightforward, and when I started poking around I found two things about myself and the new system: Everyting made sense, and everything I could think of had a readable man page. So the first change I am aware of that made the world better with OpenBSD was the decision to enforce the "No commit without documentation" rule, which came into being early in the project's life, probably roughly at the same time the OpenBSD developers gave us a real-time view of development via <a href="https://www.openbsd.org/anoncvs.html">anonymous CVS</a>. You can see things happening in almost real time.</p><p>
</p><p>
It is worth mentioning that the installer has remained famously non-graphical, text only. The reason the installer remains text-only is that this is a major advantage that enables the developers and the users to handle the fairly diverse collection of <a href="http://www.openbsd.org/plat.html">hardware platforms</a> that OpenBSD runs on with the same portable, familiar and compact code everywhere.
</p><p>
The installer was always scriptable and extensible, and over the years the installer has added automatic, repeatable and scriptable installs (dubbed <a href="https://man.openbsd.org/autoinstall">autoinstall(8)</a> which appeared in <a href="https://www.openbsd.org/55.html">OpenBSD 5.5</a> in 2014) and the <a href="https://man.openbsd.org/sysupgrade">sysupgrade(8)</a> extension (first found in <a href="https://www.openbsd.org/66.html">OpenBSD 6.6</a> in 2019) that automates snapshot to snapshot or release to subsequent release upgrades for all not too hacked-up configurations. Each of these moments, or more specifically when the new code started appearing in snapshots, had me appreciate the OpenBSD system a bit more, and made me feel quality of life had improved.</p><p>
</p><p>
</p><p>
</p><h2>Now something for your laptop - hardware support</h2>
<p>
Fast forward some twenty-plus years and the last <a href="https://bsdly.blogspot.com/2021/07/the-impending-doom-of-your-operating.html">article</a> I published, and even got into Norwegian mainstream IT news site <a href="https://www.digi.no/artikler/debatt-en-skummel-fremtid-med-operativsystem-som-gar-til-11-eller-forbi-eller-en-rolig-oase-med-fri-programvare/511693">Digi.no</a>, centers on a few moments involving new OpenBSD developments. It took some interaction with OpenBSD developers, but those interactions lead to my new laptop with an 11th generation Intel Core chipset working even better with OpenBSD. Yes, OpenBSD developers and a significant subset of their user base actually run OpenBSD on their laptops. I do use a Mac and a work-issued Thinkpad with Ubuntu Linux too, but life is not complete without an OpenBSD laptop.</p><p>
</p><p>
Now to be honest, what I saw within the space of a few days was development that had me going from "Oh, sh*t, the SSD isn't recognized" -- the controller was set to a RAID-ish mode by default -- through this kernel panic:</p><p>
</p><div class="separator" style="clear: both; text-align: center;"><a href="https://www.bsdly.net/~peter/blogpix/20210528_085741_cropped.jpg" style="margin-left: 1em; margin-right: 1em;"><img alt="OpenBSD 6.9-current panic message" border="0" data-original-height="329" data-original-width="800" height="264" src="https://www.bsdly.net/~peter/blogpix/20210528_085741_cropped.jpg" width="640" /></a></div><br /><p><br /></p><p>
</p><p>
-- to seeing it all fully supported.
</p><p>
The SSD problem turned out to be simple to fix: Simply find the "Advanced" BIOS option that turned the pseudo-raid feature off and let the operating system speak directly to the storage device.
</p><p>
For the rest there was a period of a couple of weeks I had to run with not yet commited patches in a home baked kernel built from checkouts from Jonathan Grey's git repo. When the code was committed to -current, I could resume my normal <a href="https://man.openbsd.org/sysupgrade">sysupgrade(8)</a> routine, going from one development snapshot to the next.
</p><p>
The process, even with the need to build custom kernels for a while, was actually quite pleasant, and when the support code went into the main development branch, that too was a a moment when I felt my life had been improved by changes in OpenBSD. The hardware support is now in snapshots and will be in <a href="https://www.openbsd.org/70.html">OpenBSD 7.0</a> which is set to be released approximately early November 2021.</p><p>
</p><p>
</p><p>
</p><h2>Living the life dynamic</h2>
<p>
Now that we're talking about laptops, there is another recent development that makes your OpenBSD on laptop experience even better. Laptops and other equipment that uses dynamic network configuration became easier to operate with <a href="https://man.openbsd.org/dhcpleased">dhcpleased(8)</a> now <a href="https://undeadly.org/cgi?action=article;sid=20210722072359">enabled by default</a> in OpenBSD 6.9-current after it was first introduced in <a href="https://www.openbsd.org/69.html">OpenBSD 6.9</a>. That change marked the completion of a five year cycle of incremental development which involved writing several new daemons. Each of those programs was designed with the Unix philosophy that a program should do one thing and do it well.</p><p>
</p><p>
The first piece of the puzzle was <a href="https://man.openbsd.org/slaacd">slaacd(8)</a> - the stateless IPv6 address autoconfiguration daemon - which appeared in <a href="https://www.openbsd.org/62.html">OpenBSD 6.2</a> to handle IPv6 automatic configuration, listening for route advertisements.</p><p>
</p><p>
The corresponding router advertisement daemon <a href="https://man.openbsd.org/rad">rad(8)</a> arrived in <a href="https://www.openbsd.org/64.html">OpenBSD 6.4</a>. That got most of the things involved in IPv6 autoconfiguration in order.</p><p>
</p><p>
Next up was the arrival in <a href="https://www.openbsd.org/65.html">OpenBSD 6.5</a> of <a href="https://man.openbsd.org/unwind">unwind(8)</a>, a validating DNS resolver which learns which resolvers to query from DHCP and other sources.</p><p>
</p><p>
To complete the set, <a href="https://www.openbsd.org/69.html">OpenBSD 6.9</a> brought us <a href="https://man.openbsd.org/resolvd">resolvd(8)</a> to manage and edit <tt><a href="https://man.openbsd.org/resolv.conf">/etc/resolv.conf</a></tt>, updating the file with information learned from other sources, and <a href="https://man.openbsd.org/dhcpleased">dhcpleased(8)</a> now serves as the client for IPv4 DHCP client information which is then fed into the configuration.</p><p>
</p><p>
Combined with setting your laptop to <tt>join</tt> networks as they become available, moving between networks can now be an non-disruptive, even unremarkable event. </p><p>
</p><p>
This all comes into place if you edit your <tt><a href="https://man.openbsd.org/hostname.if">/etc/hostname.$if</a></tt> for (for example <tt>hostname.iwx0</tt>) to something like</p><p>
</p><pre>join adipose wpakey thedoctorknows
join tardis wpakey biggerontheinside
join cybermen wpakey nowedont
inet autoconf
inet6 autoconf
</pre>
<p>
you should expect minimal efforts needed when moving between those networks. As usual, as soon as a new feature is trusted, it is on by default in OpenBSD-current, and <a href="https://www.openbsd.org/70.html">OpenBSD 7.0</a> will ship with this behavior enabled by default for interfaces set to <tt>autoconf</tt> for either <tt>inet</tt> or <tt>inet6</tt>.</p><p>
</p><p>
But that is the modern day and for some in the future. OpenBSD on a laptop is a good experience. On the other hand, most of the world sees the BSDs and OpenBSD in particular as mainly server or even network device operating systems, despite the fairly high BSD code content in such things as Apple systems.</p><p>
</p><p>
</p><p>
</p><h2>The thing that lured me in</h2>
<p>
But I hear you ask, what made me turn into an almost all-in OpenBSD user?
</p><p>
Back in 2001 I was still only experimenting with OpenBSD, but my experience with Linux and <tt>iptables</tt> had made me long for a switch to a saner firewall. I had done some small experiments with the IPF firewall that was in OpenBSD until the 2.9 release. Then, as some of us will remember, the it was discovered that IPF's license was in fact not free, so it needed to be replaced.
</p><p>
There was a distinct rush, not quite a stampede, to replace IPF over the months that followed. Fortunately, the new code that replaced the previous packet filter proved to perform better. The OpenBSD Packet filter, dubbed PF for short, had been born and made its debut in <a href="https://www.openbsd.org/30.html">OpenBSD 3.0</a> in December 2001. The release had originally been planned for November, but was pushed out a month to hack the "working prototype" packet filter into something usable.</p><p>
</p><p>
Almost needless to say, this turn of events finally pushed me to take the final steps to replace the Linux gateways I had in place with OpenBSD ones. I was pleasantly surprised to find that not only did they perform well, but they also came with complete and reasonably well documented tools so I could understand what was going on. That's how I got started on the process that lead to among other things writing <a href="https://nostarch.com/pf3">The Book of PF</a> and taking that text through three editions so far. But more about that later.</p><p>
</p><p>
It is worth noting that the IPFilter copyright episode spurred the OpenBSD developers to perform a <i>license audit</i> of the entire source tree and ports in order to avoid similar situations in the future. This activity ran for some months and uncovered a number of potential problems. Theo de Raadt summed up the effort in a <a href="https://marc.info/?l=openbsd-misc&m=104570938124454&w=2">message</a> to the <tt>openbsd-misc</tt> mailing list on February 20th, 2003.
</p><p>
What they found when they started looking was that there was a significant number of files that were in fact not under a free license, much like the entire IPF subsystem had been. Those needed to be replaced. Other parts had either no license or no copyright stated. In some cases the developers gave explicit permission to continuing use, but quite a few things needed to be rewritten with a free license so OpenBSD and other free software would be able to move forward without copyright problems.
</p><p>
I later heard in a rather informal setting that among the no copyright and/or no license cases, it was usually possible to track down the developers via version control system logs or mailing list archives. In a large number of those cases, the initial reaction was along the lines "<i>Say what? Are people still using that?</i>".
</p><p>
</p><p>
</p><h2>SSH, open and better</h2>
<p>
PF was written from scratch to replace a subsystem that it turned out was illegal to use in an open source context. But it was not the first time the OpenBSD project had performed a <i>nonlibreectomy</i>, that is, taken on the task of replacing code for license reasons.</p><p>
</p><p>
A few years earlier it had become clear that the original developer of the secure shell system <tt><a href="https://man.openbsd.org/ssh">ssh</a></tt> had commercial ambitions and the license for the software had changed in a proprietary direction. After a bit of deliberation on how to resolve the situation, the OpenBSD developers started digging around for earlier versions of the code that had been published with an acceptable license. Then they forked their version from the last version they found that still had free license. Next came an intensive period of re-introducing the features that were missing in the old code.</p><p>
</p><p>
The result was introduced as OpenSSH in <a href="https://www.openbsd.org/26.html">OpenBSD 2.6</a> in 1999. Over the next few years <a href="https://www.openssh.com/">OpenSSH</a> grew a portable version that started grabbing market share rapidly. The last I heard OpenSSH's market share is somewhere in the high nineties percent.</p><p>
</p><p>
With a state of the art secure shell subsystem in place and growing all sorts of useful features, the time finally came to end unencrypted shell login sessions on OpenBSD. OpenBSD's <tt>telnetd</tt> was moved to the CVS attic in time for <a href="https://www.openbsd.org/38.html">OpenBSD 3.8</a>, which was released November 2005.</p><p>
</p><p>
One other notable thing about OpenSSH is that it was the first daemon to be properly privilege separated, a model practice that debuted with the overhauled OpenSSH in <a href="https://www.openbsd.org/32.html">OpenBSD 3.2</a> in March 2002. Since then privilege separation has been put in place in all daemons where it made sense to do so, and it is now a signature part of the secure by default stance of all newer OpenBSD daemons.</p><p>
</p><p>
</p><p>
</p><h2>And yes, that packet filter</h2>
<p>
I mentioned PF, the OpenBSD packet filter, earlier. I must confess that PF has been an important part of my life in various context since the early noughties. Over the years, things I have written have contributed to creating the popular but actually wrong perception that OpenBSD was primarily a firewall operating system. There are a lot of useful and fun features that turned up in or in connection with PF over the years and were pioneered by OpenBSD. Some features were ported to or imitated in other systems, while others remain stubbornly OpenBSD only.</p><p>
</p><p>
So I will touch on some of my favorite PF and PF-attached features, in quasi-random but almost chronological order.</p><p>
</p><p>
</p><p>
</p><h2>Beating up spammers with OpenBSD spamd(8) since OpenBSD 3.3</h2>
<p>
When I started playing with OpenBSD in general and PF in particular way back when, I was already responsible for the SMTP mail service for my colleagues. My gateways by then ran OpenBSD, while the mail server <tt>rosalita</tt>, named after a <a href="https://en.wikipedia.org/wiki/Bruce_Springsteen">Springsteen</a> song, was not too badly specced server running <a href="https://www.freebsd.org/">FreeBSD</a> with <a href="https://www.exim.org/">exim</a> as the mail transfer agent that fed the incoming messages to <a href="https://spamassassin.apache.org/">spamassassin</a> and <a href="https://www.clamav.net/">clamav</a> for content filtering before handing off to user mailboxes.
</p><p>
So when it dawned on me that I could set up <a href="https://man.openbsd.org/spamd">spamd(8)</a> the spam deferral daemon on the internet-facing gateway and save load on the poor suffering <tt>rosalita</tt> that was running hot with content filtering, I was quick to implement a setup that sucked in well known block lists.</p><p>
</p><p>
</p><p>
</p><h2>Going grey, then trapping</h2>
<p>
The effect was obvious and immediate, the mail server's fans grew noticeably quieter. When greylisting was introduced in <tt>spamd</tt> soon after, I implemented that too, and witnessed yet another drop in pitch and intensity of the sound from <tt>rosalita</tt>'s fans. Then a couple of releases later greytrapping -- the practice of adding IP addresses of incoming SMTP connections to blocklists if the attempted delivery is aimed at a known-bad address in the target domain -- was introduced, and that sounded like enough fun that I just went ahead and did it.</p><p>
</p><p>
The idea of detecting spam senders by the bogus addresses they were already trying to deliver to just sounded too good to not try. And we knew that getting started would be pretty easy too. We had seen rejects for addresses that had never existed in our domains in our mail server logs for quite a while, so it was simply a matter of harvesting from a fairly bountiful source and adding stuff that we were sure would never ever be actually deliverable here to the spamtrap list. I think the first setup had only a couple of hundred entries in it, but I did not note the exact number at the time.</p><p>
</p><p>
By July 2007 I had decided to <a href="https://bsdly.blogspot.com/2007/07/hey-spammer-heres-list-for-you.html">publish</a> both the list of spamtrap addresses and an hourly dump of the greytrapped addresses. Both remain free to download. The list of spamtraps, harvested from various log sources, by now numbers just over 270,000 imaginary friends, while the number of trapped hosts is typically in the 3000 to 5000 range. We occasionally see the list swell to 20,000 or more when high volume <a href="https://bsdly.blogspot.com/2020/02/the-sextortion-scams-numbers-show-that.html">campaigns</a> run with bad address lists fed to them. I am pretty sure it went over 100,000 at one point.
</p><p>
It's fun to watch, and it looks like a significant subset of the spamtraps have made it into the address lists of active spam operations. I frankly never thought I would still be collecting spam traps from logs all these years later. Yes, it all sounds a bit absurd, but it is effective for keeping our mailboxes largely spam free, even though it feels at times like running a weird found object-ish art project. Anyway, a summary of the lists we publish can be found in <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">this article</a>.</p><p>
</p><p>
</p><p>
</p><h2>The brutes, the password gropers and the state tracking options</h2>
<p>
If you run an SSH service or really any kind of listening service with the option to log in, you will see some number of failed authentication attempts that generate noise in the logs. The password guessing, or as some of us say, password groping, turned out to be annoying enough that OpenBSD 3.6-current and later <a href="https://www.openbsd.org/37.html">OpenBSD 3.7</a> introduced a set of features to use data that would anyway be available in the state table, to track the state of active connections, and to act on limits you define such as number of connections from a single host over a set number of seconds.</p><p>
</p><p>
The action could be to add the source IP that tripped the limit to a table. Additional rules could then subject the members of that table to special treatment. Since that time, my internet-facing rule sets have tended to include variations on</p><p>
</p><p>
</p><pre>table <bruteforce> persist
block quick from <bruteforce>
pass inet proto tcp from any to $localnet port $tcp_services \
flags S/SA keep state \
(max-src-conn 100, max-src-conn-rate 15/5, \
overload <bruteforce> flush global)
</pre><p>
</p><p>
which means that any host that tries more than 100 simultaneous connections or more than 15 new connections over 5 seconds are added to the table and blocked, with any existing connections terminated.
</p><p>
It is a good practice to let table entries in such setups expire eventually. At first I followed the <a href="https://man.openbsd.org/spamd">spamd(8)</a> defaults' example and set expiry at 24 hours, but with password gropers like those caught by this rule being what they are, I switched a few years ago to at four weeks at first, then upped again a few months later to six weeks. Groperbots tend to stay broken for that long. And since they target any service you may be running, state tracking options with overload tables can be useful in a lot of non-SSH contexts as well.
</p><p>
It is also worth noting that state tracking actions are useful for essentially all services. The article <a href="https://bsdly.blogspot.com/2017/04/forcing-password-gropers-through.html">Forcing the password gropers through a smaller hole with OpenBSD's PF queues</a> has a few suggestions on how to handle noise sources with various other services.
</p><p>
One final point I would like to make about the state tracking and actions is that much like the <i>greytrapping</i> feature of <tt>spamd</tt>, this feature gives you the tools to build a configuration that <i>adapts</i> to network conditions and <i>learns</i> from the traffic it sees. </p><p>While this does not rise to the level of being an actual Artificial Intelligence or AI, this has enough <i>buzzwordability</i> potential that I remain to this day extremely puzzled that none of the other big names at least imitated those features in their own products and marketed for all it would be worth. </p><p>I certainly know what <i>I</i> would have done in their position. But then I am more engineer than marketer and in the contexts where I call the shots, the best option is just to keep running OpenBSD.
</p><p>
</p><h2>NAT's guts ripped out</h2>
<p>
When the <a href="https://www.openbsd.org/47.html">OpenBSD 4.7</a> release cycle came around, Henning Brauer had been hard at work for a while maintaining a diff of several thousand lines -- which when applied actually shrunk the code -- that contained a total rewrite of the IPv4 network address translation code.
</p><p>
Previous PF versions had '<tt>nat on interface</tt>' and '<tt>rdr on interface</tt>' rules, while the new code introduced <tt>nat-to</tt> and <tt>rdr-to</tt> as options on <tt>pass</tt> or <tt>match</tt> rules. </p><p>
The <i>match</i> keyword had been introduced in the previous release to act on packets and connections without affecting <tt>pass</tt> or <tt>block</tt> state, such as applying specific options or adding tags for processing later in the rule set. Now with the major NAT rewrite in place, it became even clearer why <tt>match</tt> was in fact a useful keyword and feature.
</p><p>
The NAT rewrite added a lot of flexibility to how you can write PF rule sets, and of course for my own part that rewrite made it necessary to write the second edition of <i>The Book of PF</i>, timed to hit bookshelves as close as possible to the OpenBSD 4.7 release. And yes, the rewrite improved the performance too.</p><p>
</p><p>
</p><p>
</p><h2>We went to modern queueing</h2>
<p>
OpenBSD has had traffic shaping available in the <i>ALTQ</i> subsystem since the very early days. ALTQ was rolled into PF at some point, but the code was still marked experimental 15 years after it was written, and most people who tried to use it in anger at the time found the syntax inelegant at best, infuriating or worse at most times.</p><p>
</p><p>
So Henning Brauer took a keen interest in the problem, and reached the conclusion that all the various traffic shaping algorithms were not in fact needed. They could all except one be reduced to mere configuration options, either as setting priorities on <tt>pass</tt> or <tt>match</tt> rules or as variations of the theme of the mother algorithm <i>Hierarchical Fair Service Curve</i> (HFSC for short).
</p><p>
Soon after, another not-small diff was making the rounds. The patch was applied early in the <a href="https://www.openbsd.org/55.html">OpenBSD 5.5</a> cycle, and for the lifetime of that release older ALTQ setups were possible side by side with the new queueing system.</p><p>
</p><p>
The feedback I get is that the saner syntax in the new queueing system lead to more users taking up traffic shaping. Here is the queue setup that I came up with for one of my sites:</p><p>
</p><p>
</p><pre>queue rootq on $ext_if bandwidth 20M
queue main parent rootq bandwidth 20479K min 1M \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>max 20479K qlimit 100
queue qdef parent main bandwidth 9600K min 6000K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>max 18M default
queue qweb parent main bandwidth 9600K min 6000K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>max 18M
queue qpri parent main bandwidth 700K min 100K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>max 1200K
queue qdns parent main bandwidth 200K min 12K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>burst 600K for 3000ms
queue spamd parent rootq bandwidth 1K min 0K max 1K \
<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>qlimit 300
</pre>
<p>
while tying the queues into the subsequent rules with a set of <tt>match</tt> rules just following that block.
</p><p>
This is what triggered the need to write the third edition of <a href="https://nostarch.com/pf3">The Book of PF</a>. The book includes descriptions of both the new and the old system as well as tips on how to make a smooth transition. The ALTQ code was removed from OpenBSD during the <a href="https://www.openbsd.org/56.html">OpenBSD 5.6</a> cycle, but continues to live on in some form in FreeBSD and NetBSD.</p><p>
</p><p>
And yes, if you think my queues setup punishes spammers a bit more in addtion to being subjected to <a href="https://man.openbsd.org/spamd">spamd(8)</a>, you're right.</p><p>
</p><p>
</p><p>
</p><h2>pflow(4) offers network insights lite</h2>
<p>
Everybody who has been tasked with looking after a network has at some point been at least a little curious about what actually moves around there. At times we will see situations where it is essential for troubleshooting purposes to see the traffic flows with data about endpoints, packets and bytes transferred, protocol and so forth.</p><p>
</p><p>
If you do not need to see the data itself, but rather the metadata, the <i>NetFlow</i> standard and its close cousin <i>IPFIX</i> offers just that. Netflow tools existed as packages on OpenBSD already, but from <a href="https://www.openbsd.org/45.html">OpenBSD 4.5</a> PF has the <tt>pflow</tt> state tracking option, paired with the <a href="https://man.openbsd.org/pflow">pflow(4)</a> virtual network interface which together offer a full netflow sensor package.</p><p>
</p><p>
Set up one or more <a href="https://man.openbsd.org/pflow">pflow</a> interfaces to send data to one or more collectors, and add the <tt>pflow</tt> option to specific rules or as a state default and you have started your collecting. You can even have metadata for traffic matching specific rules going to separate <a href="https://man.openbsd.org/pflow">pflow</a> devices and collectors.</p><p>
</p><p>
My field notes in <a href="https://bsdly.blogspot.com/2014/02/yes-you-too-can-be-evil-network.html">Yes, You Too Can Be An Evil Network Overlord - On The Cheap With OpenBSD, pflow And nfsen</a> offers some practical examples and insights, including how we used a pflow setup to track down a noisy machine on a somewhat critical network as well as some pointers to valueable further reading. </p><p>
</p><p>
</p><p>
</p><h2>LibreSSL, the great deobfuscation</h2>
<p>
People tell me they think that the reason <a href="https://www.libressl.org/">LibreSSL</a> was created was the <a href="https://en.wikipedia.org/wiki/Heartbleed">Heartbleed</a> bug, but no, actually not, just damn close.</p><p>
</p><p>
The LibreSSL project was in fact started a few weeks before heartbleed became common knowledge. LibreSSL is the result of a group of OpenBSD developers taking the existing OpenSSL code and starting to fix it.</p><p>
</p><p>
This time it was not a matter of a bad license. No, this was the result of the number of OpenBSD developers who took a look at the <a href="https://www.openssl.org/">OpenSSL</a> code that had been part of the OpenBSD base system since quite early on, and turned away in disgust and with symptoms of physical pain, reached a critical mass of sorts. I had heard OpenBSD developers complain about the absolute horror of the OpenSSL code for at least ten years. The code quality was just that bad.</p><p>
</p><p>
What happened next was that a group of hardened OpenBSD developers grabbed the OpenSSL code and started two activities in parallel. One was looking in the OpenSSL request tracker for bugs that had not been addressed. The other was reformatting the OpenSSL code into something resembling the OpenBSD style of readable and maintainable C.</p><p>
</p><p>
With the code in more readable form, discovering what it did became easier. In addition to a few obvious eye-stinging bugs the LibreSSL developers found a number of oddities, including, but not limited to</p><p>
</p><p></p><ul>
<li>Code was never deleted even when it became irrelevant or obsolete<p></p></li>
<p>
</p><li>OpenSSL did not use the system memory allocation system, but rather opted for their own which never actually deallocated memory, but rather used LIFO recycling, and could easily be made to put private info into logs<p></p></li>
<p>
</p><li>all written in "OpenSSL C", which according to <tt>beck@</tt> is a dialect of the "worst common denominator"<p></p></li>
<p></p><p></p></ul>
<p>
It is worth digging out the various articles and presentations made by LibreSSL developers over the years, with specific emphasis on Bob Beck's BSDCan talk on the first 30 days of LibreSSL (available on <a href="https://www.youtube.com/watch?v=GnBbhXBDmwU">youtube</a>), which is the original source of the term <i>code <a href="https://en.wikipedia.org/wiki/Flensing">flensing</a></i>.
</p><p>
Since the <a href="http://www.openbsd.org/56.html">OpenBSD 5.6</a> release in 2014, LibreSSL has been the default TLS library in OpenBSD. LibreSSL has been ported elsewhere based on the -portable variant.</p><p>
</p><p>
For my own part I can only attest to not ever running into a TLS problem that was LibreSSL's fault. It probably still has bugs, but it is a lot more of a healthy choice than its predecessor.</p><p>
</p><p>
</p><p>
</p><h2>This was my list of life improving OpenBSD events - I'd love to hear yours</h2>
<p>
As I warned earlier, this has been about my personal list of OpenBSD events that I remember fondly.</p><p>
</p><p>
I am sure your list is at least a little different. I am sure there are things from the <a href="https://www.openbsd.org/innovations.html">innovations</a> page that I have simply forgotten about.
</p><p>
Each release comes with a detailed list of changes, such as <a href="http://www.openbsd.org/plus69.html">this one</a> for OpenBSD 6.9, and the page has pointers back to the equivalent pages for previous releases.
</p><p>
I would love to hear about your favorite OpenBSD moments.</p><p>
</p><p>
</p><hr />
<p>
</p><h2>More items for your OpenBSD reading</h2>
<a href="https://www.openbsd.org/">www.openbsd.org</a> is the official OpenBSD web site. If you want to donate, go to the <a href="">donations</a> page and find the most appropriate option. Corporate entities may prefer to donate via <a href="https://www.openbsdfoundation.org/">The OpenBSD Foundation</a>, which is a Canadian non-profit corporation.
<p>
<a href="https://undeadly.org">undeadly.org</a> is the OpenBSD Journal news site.
</p><p>
<a href="https://bsdly.blogspot.com">bsdly.blogspot.com</a> My rant^H^H^H^Hblog posts
</p><p>
<a href="https://flak.tedunangst.com/">https://flak.tedunangst.com/</a> Ted Unangst (tedu@) on developments
</p><p>
Michael W Lucas: <a href="https://www.nostarch.com/openbsd2e">Absolute OpenBSD, 2nd edition</a>
</p><p>
Peter N. M. Hansteen: <a href="https://www.nostarch.com/pf3">The Book of PF, 3rd edition</a>
</p><p>
Henning Brauer: <a href="https://quigon.bsws.de/papers/2015/eurobsdcon/">OpenBSD sucks (… least)</a>
</p><p>
</p><hr /><div><br /></div>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com3tag:blogger.com,1999:blog-8616610987649128333.post-14192010366673469182021-07-01T13:07:00.003+02:002021-07-07T10:36:56.058+02:00The Impending Doom of Your Operating System Going to or Past 11, Versus the Lush Oasis of Open Source Systems<p><i>Will the uncertainty over forced obsolescence of fairly recent hardware force Microsoft and Apple users to switch to open source alternatives?<br /></i></p><p>During the last few weeks several items of computing hardware in our household had reached the point in their lifetime when it made sense to trade in for upgrades.<br /></p><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><p style="text-align: left;"><a href="https://Digi.no"></a></p><blockquote><a href="https://Digi.no">Digi.no</a> published a Norwegian version of this article<span style="font-family: inherit;">: </span><a href="https://www.digi.no/artikler/debatt-en-skummel-fremtid-med-operativsystem-som-gar-til-11-eller-forbi-eller-en-rolig-oase-med-fri-programvare/511693"><i>En skummel fremtid med operativsystem som går til 11 eller forbi, eller en rolig oase med fri programvare?</i></a></blockquote><a href="https://www.digi.no/artikler/debatt-en-skummel-fremtid-med-operativsystem-som-gar-til-11-eller-forbi-eller-en-rolig-oase-med-fri-programvare/511693"><i></i></a><p></p></blockquote><p>I've written articles about my last two major laptop upgrades and each time detailed (in <a href="https://bsdly.blogspot.com/2010/01/goodness-of-men-and-machinery.html">2010</a> and <a href="https://bsdly.blogspot.com/2017/07/openbsd-and-modern-laptop.html">2017</a>, respectively) how to deal with hardware that was new enough that I had no way to be certain it would work optimally with my chosen operating system, <a href="https://www.openbsd.org/">OpenBSD</a>. </p><p>I have tended to jump from <a href="http://www.openbsd.org/faq/faq5.html#Snapshots">snapshot</a> to snapshot, generally following whatever was <a href="http://www.openbsd.org/faq/faq5.html#Flavors">-current</a> on <a href="http://www.openbsd.org/amd64.html">OpenBSD/amd64</a>. There were other upgrades during that time, but those were straightforward enough that I did not see a need to write about them.</p><p>This time around, even though the process involved interactions with OpenBSD developers via the <span style="font-family: courier;">bugs@</span> <a href="http://www.openbsd.org/mail.html">mailing list</a> and even trying two separate models from the same manufacturer before settling on what I wanted, I considered just letting this upgrade round just pass relatively undocumented. There was simply not enough drama involved in the process to make for interesting reading or an inspired writing process. </p><p>But then came the announcements from <a href="https://www.apple.com/">Apple</a> and <a href="https://www.microsoft.com/">Microsoft</a> of their operating systems going past 11 or to 11 respectively, spaced not too many weeks apart. In both cases, the announcements indicated that the new operating system versions would not work with older hardware.<br /><br />At their WWDC event in early June 2021, Apple announced new versions of their system with somewhat vague but only thinly veiled formulations that specific new features of the upcoming system would only be available on the newer ARM architecture "Apple silicon" hardware.<br /><br />Then a few weeks later into June 2021, Microsoft announced their Windows 11, and the announcement included some fairly confusing statements that seemed to indicate at first that Windows 11 would only work well or at all on hardware based on Intel's 8th generation Core processors or equivalent.<br /><br />Apple is almost a year into their announced two year transition from Intel-supplied processors, with a base architecture generally known as AMD64, to their own Apple-designed ARM64-based system on a chip cores. Apple has generally kept some level of support for Macs for seven years after release, and with a transition to a new architecture underway, it becomes even less surprising that support for older devices will gradually erode and that some new system features will only be available on newer model hardware.<br /><br />This contrasts sharply with Microsoft's situation, with the company not really dependent on hardware sales and not with any announced or unannounced but apparent move to a different architecture. Whatever the reason for the cutback in support, the initial response from the public seemed to indicate that there now was a real fear that on installing the new software, upgrading Windows users would be faced with something like<br /><br /><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://www.bsdly.net/~peter/blogpix/20210528_085741.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="379" data-original-width="800" height="189" src="https://www.bsdly.net/~peter/blogpix/20210528_085741.jpg" width="400" /></a></div><p></p><p>(which is in fact an OpenBSD panic) unless they upgrade to newer hardware before trying the new software release.<br /></p><p>The fear of abandonment seemed real and echoed the feelings I have had myself over the years when getting new hardware to run a free operating system on.<br /></p><p>The previous articles chronicle some of the experimenting that was needed in the past to make OpenBSD work when the hardware was newer than what yet had time to reach the developers. But in the end we could always be quite certain that we could make what we were interested in work, given time and perhaps some interaction with developers, or if you were up to it, becoming a developer yourself.<br /><br />Anyway, over time the chance that things would just work increased, and your sweet spot for some time was buying hardware that was released within the last couple of years before the operating system release you were installing.<br /><br />Hardware drivers would generally be kept in and maintained as long as they appeared to be useful. In general a driver would only be retired from the tree if it was useful only to an architecture that was going out of support such as <a href="https://www.openbsd.org/vax.html">OpenBSD/vax</a> which went to the attic after the <a href="https://www.openbsd.org/59.html">OpenBSD 5.9</a> release in 2016.<br /><br />The major lesson here is that the free systems like <a href="https://www.openbsd.org/">OpenBSD</a>, <a href="https://linux.org/">Linux</a> or others would keep hardware support around as long as it appears to be useful to somebody, somewhere. </p><p>If major players like Microsoft choose to simply abandon users who do not have the latest hardware to stagnation plus only security updates, moving to a free software alternative may very well be a viable option for users who are not willing to abandon not very outdated hardware as long as their typical use case allows.<br /><br />In my own experience, with hardware that has been on the market for about a year or possibly more you will encounter few to no problems making things work. My most recent Linux experience on laptops is with 9th and 11th generation Intel Core hardware, both of which will serve you well, including multimedia setups, excluding only those that explicitly tell you that you are on your own (Netflix being a case in point).<br /><br />Now for an incrementally geekier part. If you are not that interested in <a href="https://www.openbsd.org/">OpenBSD</a>, please feel free to skip.<br /><br />But if you were waiting for the promised OpenBSD on newer hardware runthrough, you will get the fuller picture by reading the following and by looking up the details in the mailing list archives via the links and links in those messages.<br /></p><p>The thread <a href="https://marc.info/?t=161974815000001&r=1&w=2">AMD Ryzen based Asus ZENBOOK 14 UM433DA-PURE4 14" panic at first boot post install - how to debug</a> chronicles the interactions from "machine installs but does not survive first boot" through finding that the machine's BIOS announced but did not actually implement some features, and the subsequent changes that went in to the mainstream OpenBSD kernel, if I remember correctly just in time to be included in <a href="https://www.openbsd.org/69.html">OpenBSD 6.9</a>. <br /><br />However, as can be seen in <a href="https://marc.info/?l=openbsd-bugs&m=162141685320678&w=2">ASUS ZenBook X freezes</a>, there were problems in the <a href="http://man.openbsd.org/drm">DRM</a>/xorg area that would prove too hard to debug. Do read the whole thread, it contains useful debug info for when you get into a similar situation yourself.<br /></p><p>Returning that system to the shop for a refund while I was still fiddling with the finer points of the next system was an interesting experience in itself.<br /></p><p>I tried to restore the system to its pre-OpenBSD state before returning it, but as it turns out the Windows 10 install image Microsoft supplies will not be able to complete an install by itself.<br /></p><p>Rather, it will prompt you for hardware driver you are supposed to have to hand for this system.<br /></p><p>As a result of this, the machine still had OpenBSD installed -- with my user and home directory removed and only <span style="font-family: courier;">root</span> as an active user -- when I handed the machine in for the refund, and it was immediately clear that the support techs had never seen anything more Unixy than macOS before. Fortunately this only lead to a short delay in the issuing the refund (but I now have a 1 year PC and Mac Support contract which I do not know that I actually need).<br /></p><p>Anyway, I had already discovered an offer for a slightly more expensive model with better features, so ordered and took delivery of the machine described in <a href="https://marc.info/?t=162199040300001&r=1&w=2">ASUS ZenBook S: SSD unrecognized, possible new iwx variant</a>, which chronicles the relatively light debugging needed to get the system in shape.<br /></p><p>In short, after receiving the package with the new machine late in the afternoon, I spent a few hours trying to work around a few items that lead to rather puzzling failures at first, but fortunately they were all relatively easy to fix with a little help from OpenBSD developers who read the <span style="font-family: courier;">bugs@</span> list.<br /></p><p>The first hurdle was that the system apparently did not recognize the built in SSD. This turned out to a matter of finding the BIOS option for turning off RAID controller functionality, which anyway does not make a whole lot of sense in a system where it is physically impossible to fit more than one storage device on a permanent basis.<br /></p><p>The option turned out to live in the BIOS' <b>Advanced</b> menu, labeled <b>VMD setup menu</b>, where you set the <b>Enable VMD controller</b> option to <b>Disabled</b>. Once that is done, the SSD shows up as a regular NVMe device:<br /></p><p><span style="font-family: courier;">nvme0 at pci3 dev 0 function 0 "Intel NVMe" rev 0x03: msix, NVMe 1.3<br />nvme0: INTEL SSDPEKNW010T8, firmware 004C, serial BTNH03460GYE1P0B<br />scsibus1 at nvme0: 2 targets, initiator 0<br />sd0 at scsibus1 targ 1 lun 0: <NVMe, INTEL SSDPEKNW01, 004C><br />sd0: 976762MB, 512 bytes/sector, 2000409264 sectors<br /></span></p><p>This made it possible to install on the internal SSD proper, and the next issue was that this 11th generation Intel Core system needed a newer revision (version 5.10) of the Linux-derived DRM code. At the time (and still at the time of writing) Jonathan Gray maintained an as-not-yet-committed branch of the OpenBSD kernel with the code I needed in. The reason this DRM code version was not committed to the main tree was that the newer code caused some regressions on older hardware.<br /></p><p>On my system, it looked like the stock kernel would panic when loading the <a href="http://man.openbsd.org/iwx">iwx(4)</a> driver, but booting the test kernel Jonathan supplied cured that problem, and I have been running once a week checkouts of the drm510 kernel on top of <a href="http://man.openbsd.org/sysupgrade">sysupgrade</a>d snapshots since.<br /></p><p>However, even with the <a href="http://man.openbsd.org/iwx">iwx(4)</a> driver now loading, the wireless network device did not work. </p><p>Running <span style="font-family: courier;"><a href="http://man.openbsd.org/doas">doas</a> <a href="http://man.openbsd.org/fw_update">fw_update</a> -v</span> revealed that several of the relevant firmware files had been corrupted, and after <span style="font-family: courier;">doas fw_update -d iwx</span> and re-installing (<span style="font-family: courier;">doas fw_update iwx</span>), <span style="font-family: courier;">doas /etc/<a href="http://man.openbsd.org/netstart">netstart</a> iwx0</span> worked as expected and with excellent performance.</p><p>In the meantime it had turned out that not only was the audio parts of the system in fact supported (it only needed a one line patch to enable it), only minor manipulation to configuration files would make the audio output signal switch correctly between the internal speaker and my headphones, and that for video conferencing a low cost full duplex USB headset was the better choice.<br /><br />So now I have a gorgeous, lightweight 13.9 inch laptop running OpenBSD with Xorg running with a 3300x2200 pixel resolution and everything I care about working. With a little attention to proper testing, we have reason to believe that all of this will be properly supported without regression for older hardware versions in the upcoming OpenBSD 7.0 release.<br /><br />As I had hinted earlier, you may very well find yourself better served and supported by the open source operating system of your choice and its developers and users than you can reasonably expect from the commercial, proprietary options.<br /></p><p>If you have questions about anything in this article, OpenBSD or other free systems, please let me know in comments here, seek out a local-to-you user group (the ones I am most involved in are <a href="https://www.nuug.no">NUUG</a>, the national Norwegian Unix User Group, and <a href="https://blug.linux.no/">BLUG</a>, the Bergen (BSD and) Linux User Group), or drop me an email. If you choose the last option, please read my <a href="https://www.bsdly.net/~peter/rentageek.html">read me first</a> document before sending a second message.<br /><br /><br /><br /></p>
<p>
</p><hr />
<p>
<b>Update 2021-07-07</b>: As reported in the following tweet, the DRM 5.10 update is now in, and I can go back to quiet <a href="https://man.openbsd.org/sysupgrade">sysupgrade(8)</a> from snapshot to snapshot:<br />
</p>
<blockquote class="twitter-tweet"><p dir="ltr" lang="en">Following this commit <a href="https://t.co/Qn8QLQcbmH">https://t.co/Qn8QLQcbmH</a> (the DRM 5.10 update), a simple sysupgrade was all that was needed for the machine from <a href="https://t.co/znXV6lRLNn">https://t.co/znXV6lRLNn</a> to just work with the latest <a href="https://twitter.com/hashtag/OpenBSD?src=hash&ref_src=twsrc%5Etfw">#OpenBSD</a> snapshot. Thanks again to jsg@ and all <a href="https://twitter.com/hashtag/OpenBSD?src=hash&ref_src=twsrc%5Etfw">#OpenBSD</a> developers!</p>— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1412687409481986051?ref_src=twsrc%5Etfw">July 7, 2021</a></blockquote> <script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<p>
Which also means OpenBSD 7.0 will seriously rock on this and similar machines.
</p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com3tag:blogger.com,1999:blog-8616610987649128333.post-8287028606538232952021-05-15T20:09:00.003+02:002021-05-15T20:09:50.691+02:00 Are you aware what you lose by just clicking OK to get started using something?<p><i>The right to privacy, the right to repair and the right to choose your tools for tasks at hand are aspects of the same. A new court ruling in Italy could help us regain righs that we were manipulated into giving up.</i></p>
<p></p>
<p>It's likely you do not spend much time thinking about the fact that if you are an ordinary IT user in an industrialized country, you have most likely have been tricked into giving up rights. This happens on a scale that should be worrying to anyone concerned about human rights in general.</p>
<p>Consider the situation when you want to start using something you are interested in, either a computer of some sort such as a PC, tablet or phone, or a network based service.</p>
<p>First, look at what happens when you get get your new computer, tablet or telephone and start unboxing. One of the very first things after you have powered the device on, and certainly before you get any opportunity to use the thing for whatever you want to do, is that you are required to accept a legally binding agreement that has been designed by and for those who manufactured the equipment. In order to be able to use the thing you bought, you are required to accept an agreement that governs what you are able to do with the device.</p>
<p>With some devices you will be presented with several separate agreements, each with its own registration of whether you accept the terms or not.</p>
<p>Some of the agreements set limits on what you can use the device for, while others grant the supplier or cooperating parties permission to collect information about you and what you do with the device.</p>
<p>Most of those yes/no style questions will give the impression that you have a real choice to not agree to the terms, but you will find that you probably will not go on to getting a device that is in fact usable for the intended use until you have agreed to the terms of all of the agreements.</p>
<p>One of the more visible consequences of the COVID 19 crisis is that a larger subset of us were forced into an almost totally digital existence, where communication for work and school happens via digital devices and via services that are provided according to terms dictated by the service providers. Some of us have led a mostly digital existence for years already, but for a large chunk of the population this is a totally new set of circumstances and it is slowly dawning on an increasing number of people that important freedoms and rights may be on a path towards extinction.</p>
<p>This is not a new set of problems. Among IT professionals, many of us have for years been warning that crucial human rights or civil rights are being slowly worn away, largely to the benefit of a few corporations and their owners.</p>
<p>When you turn on a new computer or phone for the first time, most likely you will be asked right away to accept an "<i>end user license</i>" for the operating system, that is, the software that controls the device. In its simplest form, a <i>license</i> is a document that specifies the terms that govern granting other someone other than a work's author (here the software developers) permission to produce copies of the work. </p><p>However, in many cases the license document contains far more wide reaching terms and permissions. We often see that the license agreement grants you a right to not accept the terms for using the operating system and delete or return any copies delivered with the device and get a refund, but you retain the right to use the physical device. </p><p>Some of us have bought PCs or other devices and managed to install an operating system that was not supplied with the device, choosing to live our digital lives using free alternatives such as Linux or OpenBSD. Some of us do this in order to gain more direct control of the tools we use.</p>
<p>If we have tried to get a refund for an unused operating system license, most of us have not been sucessful. But we will return to that matter shortly.</p>
<p>If you have successfully installed a free alternative to the operating system that came with your device, you have contributed to strengthening the right to choose your tools, the right to repair and to make your own decisions about your possessions. But unfortunately this is not the only part of your digital life where your rights are in grave danger.</p>
<p>Regardless of whether you accepted the end user license earlier or not, you will soon encounter software or network based services that present end user agreements of their own. There is a considerable chance that you will just click OK without reading the conditions of that agreement.</p>
<p>Please take a break from reading this and go check what conditions you have actually agreed to. More likely than not, you will find that both operating system suppliers and social media services have had you give them permission to record what you do when you use the system or the service. For good measure, please take the time to check the conditions for all products and services you have registered for. Most likely not just one, but a large majority of the services and products you use on a network connected device have granted themselves the right to record and store data on your behavior. If you use the device to anything privacy relevant or involving sensitive information it is well worth checking how consequences those agreements bear ouut for your right to privacy.</p>
<p>On paper (yes, I'm sounding old fashioned on purpose) residents of the EU and EEA attached countries have a right to get a copy of data stored about us and if needed get any errors corrected or even have data deleted accordign to the EU General Data Protection Regulation, known as the GDPR. I</p><p>f you found something while checking the agreements on your break from reading this feels concerning or makes you unsure, you would to well to exercise your right to viewing, copying, correction or deletion. If you do not get any meaningful response, your best path of action is to contact the local-to-you Data Protection Authority (in Norway, that is <a href="https://www.datatilsynet.no/">Datatilsynet</a>) or the local-to-you Consumer Protection Agency (again in Norway we have <a href="https://www.forbrukertilsynet.no/">Forbrukertilsynet</a>), both should be able and willing to offer assistance.</p>
<p>But what then, of the right to repair or the right to choose one's own tools? The good news is that there is reason to hope. After a complex and long winded process an Italian court recently decided not only did a Linux enthusiast have the right to install Linux on a new Lenovo computer, the customer also had the right to get the price of the unused operating system refunded. Unfortunately Lenovo had attempted to not live up to their obligations as specified by the end user license presented to the customer, and they were fined the amount of 20,000 Euros.</p>
<p>A decision of this category is apparently not automatically a binding legal precedent in other European countries, and we are aware of decisions in other countries that did not grant the customer the right to treat a computer and its operating system as separate items. As the Norwegian association of Unix and free software users (Norwegian Unix User Group - <a href="https://www.nuug.no/">NUUG</a>) we are now entering in a cooperative effort coordinated by the Free Software Foundation Europe (<a href="https://fsfe.org/">FSFE</a>) to protect and defend your right and mine to privacy, the right to repair and the right to choose the tools we use to manage our digital existence.</p>
<p>If any of the things you just read makes you concerned, confused, angry or just eager to help strengthen our civil rights and human rights in the digital domain, we will be very happy to hear from you.</p>
<p> Peter N. M. Hansteen<br />
Board Chair of the Norwegian Unix User Group (<a href="https://www.nuug.no/">NUUG</a>)</p>
<p>The Italian court decision that offers some hope is described in some detail on the FSFEs web: <a href="https://fsfe.org/news/2021/news-20210302-01.en.html">Refund of pre-installed Windows: Lenovo must pay 20,000 euros in damages</a></p>
<p>
This article originally appeared 2021-05-15 in Norwegian on NUUG's news web <a href="http://www.nuug.no/news/Vet_du_hva_du_mister_n_r_du_bare_klikker_OK_for___komme_i_gang_med___bruke_noe_.shtml">"2021-05-15 - Vet du hva du mister når du bare klikker OK for å komme i gang med å bruke noe?"</a>
</p>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-80161480001543714792021-02-22T22:18:00.004+01:002021-02-23T12:40:21.875+01:00RFC7505 Means Yes, Your Domain Can Refuse to Handle Mail. Please Leave Us a TXT If You Do.<p><i>If you do not want a domain to receive any mail, there is a way to be at last somewhat civil about it. There's a different DNS trick for that.</i></p><p>It used to be that if you went to the trouble of registering a domain, one of the duties that came with it was set up somewhere to receive mail.</p><p>A number of networking professionals, myself included, have been know to insist that not only should a valid domain receive mail, at least a significant subset of the identities listed in <a href="https://tools.ietf.org/html/rfc2142" target="_blank">RFC2142</a> (dated May 1997) should exist and mail sent there should be read at some reasonable interval.</p><p>Then of course we all know that a number of things happened in networking in the years between 1997 and today.</p><p>As regular or returning readers of this column will be aware, one of the phenomena that rose to become a prominent irritation and possible risk factor was spam, otherwise known as unsolicited commercial email, and of course some of the unsolicited traffic carried payloads that were part of various kinds of criminal activity.</p><p>I have written fairly extensively on how to suppress spam and other malicious traffic and have fun doing so, all the while assuming that if you run a domain you will want at least some mail to have a chance of making it to an inbox that is actually read by a person or perhaps processed by your robotic underlings.</p><p>Then there is that other consideration that with the proliferation of top level domains means that organizations that own trademarks and would in the early days see the need only for <span style="font-family: courier;">.com</span> or <span style="font-family: courier;">.net</span> domain (the latter was in fact originally intended for organizations involved in networking) or perhaps a country domain such as a <span style="font-family: courier;">.no</span> or <span style="font-family: courier;">.se</span> one would tend to hoard domains in other top level domains too.</p><p>There are of course those who try to exploit trademark protection too, as we have seen in among other things <a href="https://bsdly.blogspot.com/2016/03/domain-name-scams-are-alive-and-well.html" target="_blank">my brush</a> with a certain Chinese registrar or that time when what could only be seen as an extortion attempt a little too <a href="https://bsdly.blogspot.com/2011/07/sek-1995-for-six-months-worth-of.html" target="_blank">forcefully telemarketed</a> landed me an otherwise white-elephant <span style="font-family: courier;">.se</span> domain.</p><p>Now with the combination of potentially for most practical purposes redundant domains and the likely burden of handling spam for the same, it is understandable that attitudes started to shift. Finally in June 2015 <a href="https://tools.ietf.org/html/rfc7505" target="_blank">RFC7505</a> was issued, with a simple and practical solution, dubbed the <i>NULL MX record</i>. The RFC explains how to set one up, though in language that is not too easy to penetrate.</p><p>For any domain that runs a mail service, there should be at least one <i>MX record</i>. Looking up, say, <b>bsdly.net</b> with <span style="font-family: courier;">dig bsdly.net mx</span> yields a response where the answer section gives</p><div><div><span style="font-family: courier;">;; ANSWER SECTION:</span></div><div><span style="font-family: courier;">bsdly.net.<span style="white-space: pre;"> </span>300<span style="white-space: pre;"> </span>IN<span style="white-space: pre;"> </span>MX<span style="white-space: pre;"> </span>1 skapet.bsdly.net.</span></div><div><span style="font-family: courier;">bsdly.net.<span style="white-space: pre;"> </span>300<span style="white-space: pre;"> </span>IN<span style="white-space: pre;"> </span>MX<span style="white-space: pre;"> </span>5 portal.nuug.no.</span></div><div><br /></div><div>In your zone file, you would probably have similar lines, likely with only the <i>MX <priority> hostname</i> part on the actual line, the rest taken care of by the zone file it's all wrapped in.</div><div><br /></div><div>If you want to make your domain an RFC7505-adherent one, you would remove your current MX records and replace with</div><div><br /></div><div><span style="font-family: courier;"><span style="white-space: pre;"> </span>MX<span style="white-space: pre;"> </span>0 .</span></div><div><br /></div><div>I did that for my little white elephant domain last week, since I did not by then remember when I last received anything sensible via that domain. </div><div><br /></div><div>So if you run <span style="font-family: courier;">dig bsdly.se mx</span> now, it will yield</div><div><br /></div><div><span style="font-family: courier;">;; ANSWER SECTION:</span></div><div><span style="font-family: courier;">bsdly.se.<span style="white-space: pre;"> </span>300<span style="white-space: pre;"> </span>IN<span style="white-space: pre;"> </span>MX<span style="white-space: pre;"> </span>0 .</span></div><div><br /></div><div>Which means nobody will ever see mail you attempt to send to <b>bsdly.se.</b> The delivery will fail immediately and produce a bounce message that likely references the RFC if your mailer is a reasonably recent version.</div><div><br /></div><div>But while I was doing the change it struck me that it would be useful to let the world know why I did not want that domain to handle mail. Fortunately there is already an appropriate DNS record type for the purpose: the <i>TXT record</i>. <br /><br />TXT records are used for some specific purposes such as the <a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework" target="_blank">SPF records</a> used to list allowed outoing SMTP senders for the domain, and a few other variants tied to specific services. But fundamentally a TXT record is simply a string of characters most applications will not actually attempt to handle. This means you have the option of fitting a message on your own in one. Now, if you do a lookup on that white elephant domain's TXT records, you will get</div><div><br /></div><div><span style="font-family: courier;">;; ANSWER SECTION:</span></div><div><span style="font-family: courier;">bsdly.se.<span style="white-space: pre;"> </span>300<span style="white-space: pre;"> </span>IN<span style="white-space: pre;"> </span>TXT<span style="white-space: pre;"> </span>"v=spf1 -all"</span></div><div><span style="font-family: courier;">bsdly.se.<span style="white-space: pre;"> </span>300<span style="white-space: pre;"> </span>IN<span style="white-space: pre;"> </span>TXT<span style="white-space: pre;"> </span>"This exists only because https://bsdly.blogspot.com/2011/07/sek-1995-for-six-months-worth-of.html happened."</span></div><div><span style="font-family: courier;">bsdly.se.<span style="white-space: pre;"> </span>300<span style="white-space: pre;"> </span>IN<span style="white-space: pre;"> </span>TXT<span style="white-space: pre;"> </span>"For actual contact info please check the corresponding net domain."</span></div><div><br /></div><div>Note the first TXT record here, which carries the domain's SPF specification that had been in place for a while already. It says essentially in terse if eloquent SPF speak, <i>"This domain does not send mail".</i></div><div><br /></div><div>So wrapping up, with these simple changes, quick to implement if you are in a position to edit your DNS zones we achieved:</div><div><br /></div><div><ul style="text-align: left;"><li>Ridding ourselves of an entry point that produced only annoyances</li></ul></div><div><ul style="text-align: left;"><li>Letting the world know (or at least the subset that knows how to operate common DNS tools) what the status of the mail service is and why, plus a small hint on how to make contact in case that is actually required.</li></ul></div><div>A little DNS will sometimes go a long way.</div><div><br /></div><div>A big <i>Thank You</i> to Security Evangelist Per Thorsheim (yes, that is his actual title) who brought RFC7505 to my attention again with this somewhat shorter <a href="https://securepractice.co/bulletins/nb-no/null-mx-her-tar-vi-ikke-imot-epost/12" target="_blank">blog post</a> in Norwegian (also in English <a href="https://securepractice.co/bulletins/en-us/null-mx-we-do-not-accept-email-here/12" target="_blank">here</a>).</div></div><div><br /></div><div><b>Update 2021-02-23</b>: After gentle prodding in this tweet (via JP Mens)<br /><blockquote class="twitter-tweet"><p dir="ltr" lang="en">I would add a dmarc with p=reject too</p>— Simon (@sa7sse) <a href="https://twitter.com/sa7sse/status/1364167529455898628?ref_src=twsrc%5Etfw">February 23, 2021</a></blockquote> <script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>-- also preserved as a screenshot - </div><div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://www.bsdly.net/~peter/rfc7505_plus_dmarc.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="493" data-original-width="800" height="197" src="https://www.bsdly.net/~peter/rfc7505_plus_dmarc.jpg" width="320" /></a></div><br /> <br />I added a dmarc record for the domain too (kind of overkill, but can't hurt I suppose).</div><div><br /></div>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-34752601701846826242020-02-28T21:32:00.051+01:002022-11-23T22:11:41.025+01:00The 'sextortion' Scams: The Numbers Show That What We Have Is A Failure Of Education<span style="font-family: "courier new" , "courier" , monospace;">Subject: Your account was under attack! Change your credentials!<br />From: Melissa <chenbin@jw-hw.com><br />To: adnan@bsdly.net<br /><br />Hello!<br /><br />I am a hacker who has access to your operating system.<br /><br />I also have full access to your account.<br /><br />I've been watching you for a few months now.<br /><br />The fact is that you were infected with malware through an adult site that you visited.</span><br />
<br />
Did you receive a message phrased more or less like that, which then went on to say that they have a video of you performing an embarrasing activity while visiting an "adult" site, which they will send to all your contacts unless you buy Bitcoin and send to a specific ID?<br /><p><b>Note:</b> This piece is also available, with more basic formatting but with no trackers, <a href="https://www.bsdly.net/~peter/the_numbers_show_we_have_a_failure.html">here</a>.</p>
The good news is that the video does not exist. I know this, because neither does our friend Adnan here. Despite that fact, whoever operates the account presenting as Melissa appears to believe that Adnan is indeed a person who can be blackmailed. You're probably safe for now. I will provide more detail later in the article, but first a few <i>dos</i> and <i>don't</i>s:<br />
<ul>
<li>Whatever some tempting web site tells you in a popup, unless <i>you know what you are doing</i>, do <b>not</b> install software on your devices from any other sources than the official ones. You do <b>not</b> need to install a new video viewer for that site or update your existing one, neither do you need to enter your administrator user name and password along with your credit card details into an unfamiliar-looking dialog box or web form.<br /> </li>
<li>Unless <i>you know what you are doing</i>, stay away from Bitcoin or other cryptocurrencies. If that message is the first you've heard of Bitcoin, you do <b>not</b> know what you are doing, leave it alone. As assets go, there is not much difference between financial derivatives, toxic waste and cryptocurrencies like Bitcoin, in that they should be handled with equal care and only from a distance unless you are in fact an expert in the field.<br /> </li>
<li>If you are not sure about either of the two bullet points before this one, please forget any shame over what you may or may not have done, and contact somebody you trust and who knows the subject better. This may be an adult such as a parent, teacher, social worker or other, a tech-savvy friend, or for that matter law enforcement such as your local police.</li>
</ul>
<br />
The important point is that you are or were about to be the victim of what I consider a very obvious scam, and for no good or even nearly valid reason. You should not need to become the next victim.<br />
<br />
And this, dear policy makers and tech heads in general is our problem: A large subset of the general public simply do not know their way around the digital world we created for them to live in. We need to do better.<br />
<br />
In that context I find it quite disturbing that people who <i>should</i> know better, such as the <a href="https://www.norsis.no/">Norwegian Center for Information Security</a>, in a <a href="https://norsis.no/download/18286/">recently issued report</a> (also see <a href="https://www.digi.no/artikler/norsis-spar-mer-avansert-pornosvindel-i-2020/484493">Digi.no's article</a> (both in Norwegian only, sorry)) predict that the sextortion attacks will become "more sophisticated and credible". Then again at some level they may technically be right, since this kind of activity starts out with a net negative credibility score.<br />
<br />
<b>A case in point</b>: Some versions of the scam messages I have been able to study went as far as to claim that the perpetrators had not only had taken control of the target's device, they had even sent that very email message from there. That never happened, of course, and it would have been easy for anybody who had learned to interpret <span style="font-family: "courier new" , "courier" , monospace;">Received:</span> headers to verify that the message was in fact sent from the great elsewhere. Unfortunately the skill of reading email headers is rarely, if ever, taught to ordinary users.<br />
<br />
The fact that people do not understand those -- to techies -- obvious facts is a fairly central and burdening problem, and again we need to do better.<br />
<br />
Now let me explain. Things get incrementally more technical from here, so if you came here only for the admonitions or practical advice and have no use for the background, feel free to wander off.<br />
<br />
I know the message I quoted at the beginning here is a scam because I run my own mail service, and looking at just the logs there just now I see that since the last logs archiving rotation early Saturday morning, more than 3000 attempts at delivery of messages like the one for Adnan happened, aimed at approximately 200 non-existent recipients before my logs tell me they finally tried to deliver one to my primary contact address, never actually landing in any inboxes. <br />
<br />
One of the techniques we use to weed out unwanted incoming mail is to maintain and <a href="https://www.bsdly.net/~peter/traplist.shtml">publish</a> a list of known bad and invalid email addresses in our domains. These known bad addresses have then in ways unknown (at least not known to us in any detail) made it into the list of addresses sold to spammers, and we at the receiving end can use the bad addresses as triggers to block traffic from the sending hosts (If you are interested, you can read elsewhere <a href="https://bsdly.blogspot.com/">on this blog</a> for details on how we do this, look for tags such as <i>greylisting</i>, <i>greytrapping</i> or <i>antispam</i>).<br />
<br />
If it was not clear earlier, those numbers tell us something about the messages at hand. It should be fairly obvious that compromising videos of non-existent users could not, in fact, exist.<br />
<br />
Looking back in archived logs from the same system I see that a variant of this message started appearing in late January 2018. The specifics of that message sequence will be interesting to revisit when the full history of <i>sextortion</i> (I still do not like the term, but my preferred alterantive is at risk of being filtered out by polite society-serving robots) will be written, but let us rather turn to the more recent data, as in data recorded earlier this week.<br />
<br />
Mainly because I found the media coverage of the "sextortion" phenomenon generally uninformed and somewhat annoying, I had been been mulling writing an article about it for a while, but I was still looking for a productive angle when on Wednesday evening I noticed a slight swelling in the number of greytrapped hosts. A glance at my <a href="https://man.openbsd.org/spamd">spamd</a> log seemed to indicate that at least one of the delivery attempts had a line like<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> I am a hacker who has access to your operating system.</span><br />
<br />
Which was actually just what I had been pondering writing about. <br />
<br />
So I set about for a little research. I <a href="https://man.openbsd.org/grep">grep</a>ed (searched) in my yet-un<a href="https://man.openbsd.org/newsyslog">rotate</a>d <a href="https://man.openbsd.org/spamd">spamd</a> logs for the word <i>hacker</i>, which yielded lots of lines of the type<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Feb 22 04:04:35 skapet spamd[8716]: 89.22.104.47: Body: I am a hacker who has access to your operating system.<br />Feb 22 04:17:04 skapet spamd[8716]: 5.79.23.92: Body: I am a hacker who has access to your operating system.<br />Feb 22 04:34:03 skapet spamd[8716]: 153.120.146.199: Body: I am a hacker who has access to your operating system.<br />Feb 22 04:40:30 skapet spamd[8716]: 45.181.93.45: Body: I am a hacker who has access to your operating system.<br />Feb 22 04:55:04 skapet spamd[8716]: 93.186.247.18: Body: I am a hacker who has access to your operating system.<br />Feb 22 05:09:39 skapet spamd[8716]: 123.51.190.154: Body: I am a hacker who has access to your operating system.<br />Feb 22 05:13:22 skapet spamd[8716]: 212.52.131.4: Body: I am a hacker who has access to your operating system.<br />Feb 22 05:38:02 skapet spamd[8716]: 5.79.23.92: Body: I am a hacker who has access to your operating system.<br />Feb 22 05:44:39 skapet spamd[8716]: 123.51.190.154: Body: I am a hacker who has access to your operating system.<br />Feb 22 06:00:30 skapet spamd[8716]: 45.181.93.45: Body: I am a hacker who has access to your operating system.</span><br />
(the full result has been preserved <a href="https://www.bsdly.net/~peter/2020-02-26_i_am_a_hacker_initial_search.txt">here</a>). Extracting the source addresses gave a list of 198 IP addresses (preserved <a href="https://www.bsdly.net/~peter/2020-02-26_i_am_a_hacker_ip_addresses.txt">here</a>).<br />
<br />
Extracting the <span style="font-family: "courier new" , "courier" , monospace;">To:</span> addresses from the fuller listing yielded 192 unique email addresses (preserved <a href="https://www.bsdly.net/~peter/2020-02-26_i_am_a_hacker_email_addresses.txt">here</a>). Looking at the extracted target email addresses yielded some interesting insights:<br />
<br />
1) The target email addresses were not exclusively in the domains my system actually serves, and<br />
<br />
2) Some ways down the list of target email addresses, my own primary address turns up.<br />
<br />
Of course 2) made me look a little closer, and only one IP address in the extract had tried delivery to my email address.<br />
<br />
A further <a href="https://man.openbsd.org/grep">grep</a> on that IP address turned up <a href="https://www.bsdly.net/~peter/2020-02-26_i_am_a_hacker_found_a_real_one.txt">this result</a>.<br />
<br />
There are really no surprises to be had here, at least to a large subset of my supposed readers. The sender had first tried to deliver one of the sexstortion video messages to one of the by now more than quarter million <a href="https://www.bsdly.net/~peter/traplist.shtml">spamtraps</a>, and its IP address was still blacklisted by the time it finally tried delivery to a potentially deliverable address.<br />
<br />
Doing a few spot checks on the sender IP addresses in recent and less recent logs it looks like the only two things could be mildly exciting about those messages. One is the degree the content was intended to be embarrasing to the recipient. The other is a possible indicator of the campaign's success: Looking back through the logs for the approximate year of known activity, it even looks like the campaign became multilingual, while retaining the word "<i>hacker</i>" in most if (possibly) not all language versions. <br />
<br />
Other than that it is almost depressing how normal the sextortion campaign is: It uses the <i>same</i> spam sending infrastructure and the <i>same</i> low quality target address lists (the ones containing some subset of my <a href="https://www.bsdly.net/~peter/traplist.shtml">spamtrap</a> addresses) as the regular and likely not too successful spammers of every stripe. Nothing else stands out. <br />
<br />
And as returning readers will notice, the logs indicate that the spambots are naive enough in their SMTP code that they frequently mistake spamd's delaying tactics for a slow, but functional open SMTP relay.<br />
<br />
Now to recap the main points:<br />
<ul>
<li><b>Regular users</b>: The sextortion messages are scams, the videos do not exist. If this quasi-random sample is representative, the scammers are seen to send to 200 non-existing, invalid addresses before lucking on a real one. This alone <b>strongly</b> indicates that no videos exist. There is no reason to send money, bitcoin or otherwise. Look instead to learning how your devices and the networks and services they connect to actually work.<b> </b></li>
<li><b>Competent mail admins</b>: The tools to stop the flow of sextortion messages or at least slow to a manageable trickle are available today. You simply need to keep your antispam game up to speed with best practices and best of breed tools. If you are a user or someone who manages mail admins, check what your mail service does.<b><br /> </b></li>
<li><b>Competent authorities</b>: Please step up to the task of educating the public. Sane, fact based approaches to IT security work. While it is easy to get distracted by the potential presence of porn and users' feelings of shame over accessing that kind of material, assigning much weight to that side of the matter is counterproductive. Work to educate the public and please focus on real threats, not imagined ones like the present topic.</li>
</ul>
Whatever evolves next out of these rather hamfisted attempts at blackmail is unlikely to ever achieve any level of sophistication worthy of the name.<br />
<br />
We would all be much better served by focusing on real threats such as, but not limited to, credential harvesting via deceptive content delivered over advertising networks, which themselves are a major headache security- and privacy-wise, or even harvesting via phishing email.<br />
<br />
Both of the latter have been known to lead to successful compromise with data exfiltration and identity theft as possible-to-probable results.<br />
<br />
To a large extent the damage could could have been significantly limited had the general public been taught sensible security practices such as using multi-factor authentication or at least actually good passwords combined with securely coded password management applications, and insisting that services encourage such practices. <br />
<br />
Yes, I know you have been dying to ask: What is the thing about Adnan? According to my activity log, the address <span style="font-family: "courier new" , "courier" , monospace;">adnan@bsdly.net</span> was added as a spamtrap on July 8th, 2017 after somebot had tried to log on as the user <span style="font-family: "courier new" , "courier" , monospace;">adnan</span>, a user name not seen before at <b>bsdly.net</b>,<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Jul 8 09:40:34 skapet sshd[34794]: Failed password for invalid user adnan from 118.217.181.8 port 41091 ssh2</span><br />
<br />
apparently from a network in South Korea.<br />
<br />
As always, there is more log material available to competent practitioners and researchers with a valid research agenda. Please contact me if you are such a person who could use the collected data productively.<br />
<br />
<hr /><p>
<b>Update 2020-02-29</b>: For completeness and because I felt that an unsophisticated attack like the present one deserves a thorough if unsophisticated analysis, I decided to take a look at the log data for the entire 7 day period, post-rotation.<br />
<br />
So here comes some armchair analysis, using only the tools you will find in the base system of your <a href="https://www.openbsd.org/">OpenBSD</a> machine or any other running a sensibly stocked unix-like operating systen. We start with finding the total number of delivery attempts logged where we have the body text <i>'am a hacker'</i> (this would show up only after a sender has been blacklisted, so the gross number actual delivery attempts will likely be a tad higher), with the command<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><a href="https://man.openbsd.org/zgrep">zgrep</a> "am a hacker" /var/log/spamd.0.gz | <a href="https://man.openbsd.org/awk">awk</a> '{print $6}' | wc -l</span><br />
<br />
which tells us the number is 3372.<br />
<br />
Next up we use a variation of the same command to extract the source IP addresses of the log entries that contain the string <i>'am a hacker'</i>, sort the result while also removing duplicates and store the end result in an environment variable called <span style="font-family: "courier new" , "courier" , monospace;">lastweek</span>:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">export lastweek=`zgrep "am a hacker" /var/log/spamd.0.gz | awk '{print $6}' | tr -d ':' | sort -u `</span><br />
<br />
With our list of IP addresses tucked away in the environment variable go on to: For each IP address in our <span style="font-family: "courier new" , "courier" , monospace;">lastweek</span> set, extract all log entries and store the result (still in crude sort order by IP address), in the file <a href="https://www.bsdly.net/~peter/2020-02-29_i_am_hacker.raw.txt">2020-02-29_i_am_hacker.raw.txt</a>:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> for foo in $lastweek ; do zgrep $foo /var/log/spamd.0.gz | <a href="https://man.openbsd.org/tee">tee</a> -a 2020-02-09_i_am_hacker.raw.txt ; done</span><br />
<br />
For reference I kept the <a href="https://www.bsdly.net/~peter/2020-02-29_i_am_hacker.raw_IP_addresses.txt">list of unique IP addresses</a> (now totalling <i>231</i>) around too.<br />
<br />
Next, we are interested in extracting the target email addresses, so the command<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">grep "To:" 2020-02-29_i_am_hacker.raw.txt | awk '{print substr($0,index($0,$8))}' | sort -u</span><br />
<br />
finds the lines in our original extract containing "<span style="font-family: "courier new" , "courier" , monospace;">To:</span>", and gives us the list of target addresses the sources in our data set tried to deliver mail to.<br />
<br />
The result is preserved as <a href="https://www.bsdly.net/~peter/2020-02-29_i_am_hacker.raw_targets.txt">2020-02-29_i_am_hacker.raw_targets.txt</a>, a total of <i>236</i> addresses, mostly but not all in domains we actually host here. One surprise was that among the target addresses one actually invalid address turned up that was not at that time yet a spamtrap. See the end of the <a href="https://www.bsdly.net/~peter/spamtraps-dateadded.txt">activity log</a> for details (it also turned out to be the last SMTP entry in that log for 2020-02-29).<br />
<br />
This little round of armchair analysis on the static data set confirms the conclusions from the original article: Apart from the possibly titillating aspects of the "adult" web site mentions and the attempt at playing on the target's potential shamefulness over specific actions, as spam campaigns go, this one is ordinary to the point of being a bit boring.<br />
<br />
There may well be other actors preying on higher-value targets through their online clumsiness and known peculiarities of taste in an actually targeted fashion, but this is not it.<br />
<br />
<b>A final note on tools:</b> In this article, like all previous entries, I have exclusively used the tools you will find in the <a href="https://www.openbsd.org/">OpenBSD</a> (or other sensibly put together unixlike operating system) base system or at a stretch as an easily available package.<br />
<br />
For the simpler, preliminary investigations and poking around like we have done here, the basic tools in the base system are fine. But if you will be performing log analysis at scale or with any regularity for purposes that influences your career path, I would encourage you to look into setting up a proper, purpose-built log analysis system.<br />
<br />
Several good options, open source and otherwise, are available. I will not recommend or endorse any specific one, but when you find one that fits your needs and working style you will find that after the initial setup and learning period it will save you significant time.<br />
<br />
As per my practice, only material directly relevant to the article itself has been published via the links. If you are a professional practitioner or researcher with who can state a valid reason to need access to unpublished material, please let me know and we will discuss your project.<br />
<br />
<b>Update 2020-03-02</b>: I knew I had some early samples of messages that did make it to an inbox near me squirreled away somewhere, and after a bit of rummaging I found them, stored <a href="https://www.bsdly.net/~peter/webcam-hack-scam/">here</a> (note the directory name, it seemed so obvious and transparent even back then). It appears that the oldest intact messages I have are from December 2018. I am sure earlier examples can be found if we look a littler harder.<br />
<br />
<b>Update 2020-03-17:</b> A fresh example turned up this morning, addressed to (of all things) the <i>postmaster</i> account of one of our associated <i>.no</i> domains, written in Norwegian (and apparently generated with Microsoft Office software). The preserved message can be downloaded <a href="https://www.bsdly.net/~peter/2020-03-17_vaer_trygg_postmaster_at_nuug.txt">here</a>. <br />
<br />
<b>Update 2020-05-10:</b> While rummaging about (aka 'researching') for something else I noticed that <a href="https://man.openbsd.org/spamd">spamd</a> logs were showing delivery attempts for messages with the subject <span style="font-family: "Courier New", Courier, monospace;">"High level of danger. Your account was under attack."</span> So out of idle curiosity on an early Sunday afternoon, I did the following:<br /><br /><span style="font-family: "Courier New", Courier, monospace;">$ export muggles=`grep " High level of danger." /var/log/spamd | awk '{print $6}' | tr -d ':' | sort -u`<br />$ for foo in $muggles; do grep $foo /var/log/spamd >>20200510-muggles ; done</span><br /><br />and the result is preserved for your entertainment and/or enlightenment <a href="https://www.bsdly.net/~peter/20200510-muggles.txt">here</a>. Not much to see, really other than that they sent the message in two language varieties, and to a small subset of our imaginary friends.<br />
<br />
<b>Update 2020-08-13:</b> Here is another snapshot of activity from August 12 and 13: <a href="https://www.bsdly.net/~peter/2020-08-13_knuckleheads_log.txt">this</a> file preserves the activity of 19 different hosts, and as we can see that since they targeted our imaginary friends first, it is unlikely they reached any inboxes here. Some of these campaigns may have managed to reach users elsewhere, though</p><p><b>Update 2020-09-06</b>: Occasionally these messages manage to hit a mailbox here. Apparently enough Norwegians fall for these scams that Norwegian language versions (not terribly well worded) get aimed at users here. This <a href="https://www.bsdly.net/~peter/2020-09-06_bsdly_vi_ma_diskutere.txt" target="_blank">example</a>, aimed at what has only ever been an email alias made it here, slipping through by a stroke of luck during a time that IP address was briefly not in the <span style="font-family: courier;">spamd-greytrap</span> list here, as can be seen from this <a href="https://www.bsdly.net/~peter/2020-09-06_bsdly_vi_ma_diskutere_logs.txt" target="_blank">log excerpt</a>. It is also worth noting that an identically phrased message was sent from another IP address to <span style="font-family: courier;">mailer-daemon@</span> for one of the domains we run here.<br /></p><p><b>Update 2021-01-06</b>: For some reason, a <a href="https://www.bsdly.net/~peter/20210106_wankstortion_message1c6c5f8dbef75@mail.f-rs.com_bsdly@bsdly.net.txt" target="_blank">new variant</a> turned up here today (with a <a href="https://www.bsdly.net/~peter/20210106_wankstortion_message73e86b03477f2@hireassociation.co.uk_i_recorded_you_bsdly@bsdly.net.txt" target="_blank">second message</a> a few minutes later and then a <a href="https://www.bsdly.net/~peter/20210106_wankstortion_message0aded5f24b@mail.fairway-corp.co.jp_bsdly@bsdly.net.txt" target="_blank">third</a>), addressed to a generic contact address here. A very quick check of logs here only turned up only <a href="https://www.bsdly.net/~peter/20210106_PRONOGRAPHIC.txt" target="_blank">this</a> indication of anything similar (based on a search for the variant spelling <i>PRONOGRAPHIC</i>), but feel free to check your own logs based on these samples if you like.</p><p><b>Update 2021-01-16:</b> One more round, <a href="https://www.bsdly.net/~peter/20210115_ubsoncao@sonha-quangngai.gov.vn_wankstortion_peter@bsdly.se.txt">this</a> for my Swedish alter ego. Apparently sent from a poorly secured Vietnamese system.</p><p><b>Update 2021-01-18:</b> A Norwegian version has surfaced, attempted sent to approximately 115 addresses in .no domains we handle, fortunately the majority of the addresses targeted were in fact spamtraps, as <a href="https://www.bsdly.net/~peter/20210117_wankstortion_spamd_extract.txt" target="_blank">this log extract</a> shows.</p><p><b>Update 2021-03-03</b>: After a few quiet weeks, another campaign started swelling our <a href="https://home.nuug.no/~peter/pf/en/spamd.greytrapping.html">greytrapped</a> hosts <a href="https://www.bsdly.net/~peter/traplist.shtml">collection</a>, as this hourly count of IP addresses in the traplist at dump to file time shows:<br /><br /></p><p>Tue Mar 2 21:10:01 CET 2021 : 2425 <br />Tue Mar 2 22:10:01 CET 2021 : 4014 <br />Tue Mar 2 23:10:01 CET 2021 : 4685 <br />Wed Mar 3 00:10:01 CET 2021 : 4847 <br />Wed Mar 3 01:10:01 CET 2021 : 5759 <br />Wed Mar 3 02:10:01 CET 2021 : 6560 <br />Wed Mar 3 03:10:01 CET 2021 : 6774 <br />Wed Mar 3 04:10:01 CET 2021 : 7997 <br />Wed Mar 3 05:10:01 CET 2021 : 8231 <br />Wed Mar 3 06:10:01 CET 2021 : 8499 <br />Wed Mar 3 07:10:01 CET 2021 : 9910 <br />Wed Mar 3 08:10:01 CET 2021 : 10240 <br />Wed Mar 3 09:10:01 CET 2021 : 11872 <br />Wed Mar 3 10:10:01 CET 2021 : 12255 <br />Wed Mar 3 11:10:01 CET 2021 : 13689 <br />Wed Mar 3 12:10:01 CET 2021 : 14181<br />Wed Mar 3 13:10:01 CET 2021 : 15259<br />Wed Mar 3 14:10:01 CET 2021 : 15881 <br />Wed Mar 3 15:10:02 CET 2021 : 17061<br />Wed Mar 3 16:10:01 CET 2021 : 17625<br />Wed Mar 3 17:10:01 CET 2021 : 18758<br />Wed Mar 3 18:10:01 CET 2021 : 19170<br />Wed Mar 3 19:10:01 CET 2021 : 20028<br />Wed Mar 3 20:10:01 CET 2021 : 20578<br />Wed Mar 3 21:10:01 CET 2021 : 20997</p><div>and they attempted to get to <span style="font-family: courier;">mailer-daemon@</span>, as can be seen from <a href="https://www.bsdly.net/~peter/20210303_wankstortion_for_mailer-daemon.txt">this</a> preserved message as well as <a href="https://www.bsdly.net/~peter/20210303_wankstortion_64.119.192.166_mailer-daemon@nuug.no_mailer-daemon@nuug.no.txt">this</a> one (both of which actually did inbox due to <a href="https://en.wikipedia.org/wiki/Email_alias">aliases</a>).</div><div><br />Stay safe out there.</div><div><br /></div><div><b>Update 2021-04-17</b>: A new variant, somewhat crudely worded, inboxed today. Preserved <a href="https://bsdly.net/~peter/20210417_messagec49907cdf15e12@mackenziesbakery.com_wankstortion_peter@bsdly.net.txt">here</a>, <a href="https://bsdly.net/~peter/20210417_messagee655d756bbadc@synergymedsys.com_wankstortion_peter@bsdly.net.txt">here</a> and <a href="https://bsdly.net/~peter/20210417_messagee655d756bbadc@synergymedsys.com_wankstortion_peter@bsdly.net.txt">here</a>.</div><div><br /></div><div><b>Update 2021-05-15:</b> After swelling the list of trapped hosts considerably over the last few days, a sample of the campaign with the most correctly worded Norwegian text <a href="https://www.bsdly.net/~peter/20210515_hackere_har__wankstortion_peter@nuug.no.txt">inboxed</a> today, and I later found <a href="https://www.bsdly.net/~peter/20210515_hackere_har__wankstortion_bsdly@bsdly.net.txt">other</a> <a href="https://www.bsdly.net/~peter/20210515_hackere_har__wankstortion_peter@nuug.no.2.txt">samples</a>.</div><div><br /></div><div>From the logs it looks like the campaign started on May 13th:<br /><br /> Thu May 13 10:10:01 CEST 2021 : 3998</div><div>Thu May 13 11:10:01 CEST 2021 : 4064</div><div>Thu May 13 12:10:01 CEST 2021 : 7052</div><div>Thu May 13 13:10:01 CEST 2021 : 7297</div><div>Thu May 13 14:10:01 CEST 2021 : 7474</div><div>Thu May 13 15:10:01 CEST 2021 : 10178</div><div>Thu May 13 16:10:01 CEST 2021 : 10251</div><div>Thu May 13 17:10:01 CEST 2021 : 11150</div><div>Thu May 13 18:10:01 CEST 2021 : 12686</div><div>Thu May 13 19:10:01 CEST 2021 : 12866</div><div>Thu May 13 20:10:01 CEST 2021 : 14708</div><div>Thu May 13 21:10:01 CEST 2021 : 14713</div><div>Thu May 13 22:10:01 CEST 2021 : 14907</div><div>Thu May 13 23:10:02 CEST 2021 : 16336</div><div>Fri May 14 00:10:01 CEST 2021 : 16360</div><div>Fri May 14 01:10:01 CEST 2021 : 16473</div><div>Fri May 14 02:10:01 CEST 2021 : 17608</div><div>Fri May 14 03:10:01 CEST 2021 : 17643</div><div>Fri May 14 04:10:01 CEST 2021 : 17671</div><div>Fri May 14 05:10:01 CEST 2021 : 17763</div><div>Fri May 14 06:10:01 CEST 2021 : 18796</div><div>Fri May 14 07:10:01 CEST 2021 : 18950</div><div>Fri May 14 08:10:02 CEST 2021 : 18972</div><div>Fri May 14 09:10:01 CEST 2021 : 18725</div><div>Fri May 14 10:10:01 CEST 2021 : 19929</div><div>Fri May 14 11:10:01 CEST 2021 : 19942</div><div>Fri May 14 12:10:01 CEST 2021 : 17046</div><div>Fri May 14 13:10:01 CEST 2021 : 18068</div><div>Fri May 14 14:10:01 CEST 2021 : 18619</div><div>Fri May 14 15:10:01 CEST 2021 : 16066</div><div>Fri May 14 16:10:01 CEST 2021 : 17468</div><div>Fri May 14 17:10:01 CEST 2021 : 17297</div><div>Fri May 14 18:10:01 CEST 2021 : 15859</div><div>Fri May 14 19:10:01 CEST 2021 : 17395</div><div>Fri May 14 20:10:01 CEST 2021 : 15934</div><div>Fri May 14 21:10:01 CEST 2021 : 15996</div><div>Fri May 14 22:10:01 CEST 2021 : 17120</div><div>Fri May 14 23:10:02 CEST 2021 : 16238</div><div>Sat May 15 00:10:01 CEST 2021 : 16299</div><div>Sat May 15 01:10:01 CEST 2021 : 16362</div><div>Sat May 15 02:10:01 CEST 2021 : 16346</div><div>Sat May 15 03:10:01 CEST 2021 : 16814</div><div>Sat May 15 04:10:01 CEST 2021 : 16812</div><div>Sat May 15 05:10:01 CEST 2021 : 16787</div><div>Sat May 15 06:10:01 CEST 2021 : 16007</div><div>Sat May 15 07:10:01 CEST 2021 : 17093</div><div>Sat May 15 08:10:01 CEST 2021 : 17101</div><div>Sat May 15 09:10:01 CEST 2021 : 17015</div><div>Sat May 15 10:10:01 CEST 2021 : 15702</div><div>Sat May 15 11:10:01 CEST 2021 : 15637<br /><br /></div><div><b>Update 2021-06-16</b>: Another campaign seems to be under way, with <a href="https://www.bsdly.net/~peter/20210616_messageb277c470c50e3eae9180@realdeeden.com.br_wankstortion_bsdly@bsdly.net.txt">this message</a> sent to an address which I can reveal is simply an alias. </div><div><br /></div><div><b>Update 2021-08-16</b>: Thanks to one particular operator being 'too big to block' <a href="https://www.bsdly.net/~peter/wankstortion/20210816_rrhh2@sanatorio-anchorena.com.ar_nyet@dataped.no.txt">this message</a>, apparently part of a campaign that has at least 103 other sending hosts that are currently trapped here, actually inboxed despite being sent to a spamtrap which also corresponded to a forgotten alias for an actual in-use mailbox. </div><div><br /></div><div><b>Update 2021-08-17</b>: By lunchtime the output of <br /><br /><span style="font-family: courier;">grep vellykket /var/log/spamd | awk '{ print $6 }' | sort -u | tr -d ':' | wc -l<br /></span><br />had reached 471, so I did <br /><br /><span style="font-family: courier;">export trash=`grep vellykket /var/log/spamd | awk '{ print $6 }' | sort -u | tr -d ':'`<br />for foo in $trash ; do grep $foo /var/log/spamd >> vellykket.txt ; done<br /></span><br />You can find the result here: <a href="https://www.bsdly.net/~peter/vellykket_20210817T1200.txt">vellykket_20210817T1200.txt</a>. It looks like the campaign is still in progress. <br /></div><div> </div><div>A few hours later, the number was 570 and the new export looks like <a href="https://www.bsdly.net/~peter/vellykket.txt">vellykket.txt</a> while the most up to date list of IP addresses participating in the campaign is in <a href="https://www.bsdly.net/~peter/vellykket_addresses.txt">vellykke_addressest.txt</a></div><div> </div><div>If you're interested in further data, please let me know. <br /></div><div> </div><div><b>Update 2021-09-09:</b> There are signs that another campaign is in progress, with an inboxed sample preserved <a href="https://www.bsdly.net/~peter/wankstortion/20210909_peter_waiting_for_payment_peter%40bsdly.net.txt">here</a>. This particular message appears to have been delivered from a Korean network.</div><div><br /><b>Update 2021-10-27:</b> Another <a href="https://www.bsdly.net/~peter/wankstortion/20211027_wankstortion_peter_233-225-245-190.fibertel.com.ar.txt">sample</a> inboxed overnight, from a campaign that uses a text with only slight edits from eariler.</div><div><br /></div><div><b>Update 2021-11-29:</b> Overnight a collection of trimmed-down messages like <a href="https://www.bsdly.net/~peter/wankstortion/20211129_cespinoza@mdp.gov.PY_wankstortion_postmaster@nuug.no.txt">this one</a> appeared, claiming to have installed a trojan on the supposed victim's phone, but asking the victim to answer the message for further instruction. An attempt to weed out spamtraps from their address lists, perhaps?</div><div><br /></div><div><b>Update 2022-02-02</b>: Another campaign is underway, a sample has been preserved <a href="https://www.bsdly.net/~peter/wankstortion/20220201_peter_dont_forget_to_pay_the_tax_wankstortion_peter%40bsdly.net.txt">here</a>. It makes the usual claims of device takeover. This particular message seems to have been delivered via a Kenyan system.</div><div><br /></div><div><b>Update 2022-03-30:</b> A <a href="https://www.bsdly.net/~peter/wankstortion/20220330_wankstortion_peter_pending_for_a_payment.txt">new</a> entry appeared today, with only minor variations relative to earlier campaigns. As expected a log extract <a href="https://www.bsdly.net/~peter/wankstortion/20220330_wankstortion_peter_pending_for_a_payment.log.txt">shows</a> that the same host had been used in some spamming campaign or other -- possibly even an earlier segment of the same one -- only a few days ago.</div><div><br /></div><div><b>Update 2022-04-08:</b> The number of languages used in those messages received here grew by one with today's <a href="https://bsdly.net/~peter/wankstortion/20220408_Sie_haben_eine_ausstehende_Zahlung_peter_at_bsdly.net.txt">entry</a>, which seems to be in German. I am not qualified to speak to the quality or lack of same of the translation, but I note that the host that was used to send the message seems to belong in an Indonesian network.</div><div><br /></div><div><b>Update 2022-04-09: </b>Yet <a href="https://www.bsdly.net/~peter/wankstortion/20220409_Sie_haben_eine_ausstehende_Zahlung_peter%40bsdly.net.txt">another</a> German language entry, this on also sent from a system apparently in Indonesia.</div><div><br /></div><div><b>Update 2022-08-19:</b> A new Norwegian language campaign is under way, with a handful of new samples available in the <a href="https://www.bsdly.net/~peter/wankstortion/">archive</a>.</div><div><br /></div><div><b>Update 2022-09-18</b>: Another campaign in progress, this time picking up on quasi-recent buzzwords. I offer <a href="https://www.bsdly.net/~peter/wankstortion/20220918_wankstortion_160.154.233.151_to_peter.txt">the evidence so far</a>.</div><div><br /></div><div>U<b>pdate 2022-09-24:</b> Yet another campaign, very similar to the last one. This <a href="https://nxdomain.no/~peter/wankstortion/20220924_wankstortion_outstanding_debt_peter@bsdly.net.txt">message</a> was apparently sent from a (likely compromised) Kuwaiti system.</div><div><br /></div><div><b>Update 2022-10-06</b>: Here we go again. The campaign has been going on for a little while, the first message to inbox (sort of) was <a href="https://nxdomain.no/~peter/wankstortion/20221006_wankstortion_business02%40hibumwoo.co.kr_aktive-owner%40nuug.no_txt">this</a> one, apparently delivered from a host located in Korea. The list of identified spam sources (246 hosts at this point) is <a href="https://nxdomain.no/~peter/wankstortion/20221006_spamsources.txt">here</a>, while a log of activity can be found <a href="https://nxdomain.no/~peter/wankstortion/20221006_campaign_raw_09.55_CEST.txt">here</a>. <b>Warning</b>: that last one is <i>not</i> a small file.</div>
<div><br /></div><div><b>Update 2022-11-23</b>: Another campaign is underway, with a variety of subjects, all with the word <tt>trouble</tt>, a full list preserved <a href="https://nxdomain.no/~peter/wankstortion/20221123/20221123_campaign_subjects.txt">here</a>. In addition, a few extracts, startin with the <a href="https://nxdomain.no/~peter/wankstortion/20221123/20221123_campaign_ip_addresses.txt">list of sending addresses</a> (178 total) the <a href="https://nxdomain.no/~peter/wankstortion/20221123/20221123_campaign_targets.txt">list of target email adddresses</a> (480 total) a full <a href="https://nxdomain.no/~peter/wankstortion/20221123/20221123_campaign_log_extract_all.txt">log extract</a> covering the whole campaign while extract per host can be found in <a href="https://nxdomain.no/~peter/wankstortion/20221123/">this directory</a>.<p>
None of this inboxed, of course, as they could all be found in the list of spamtraps, referenced among other places in my previous article <i><a href="https://bsdly.blogspot.com/2022/09/the-things-spammers-believe-tale-of.html">The Things Spammers Believe - A Tale of 300,000 Imaginary Friends</a></i> (also available <i><a href="spammers_believe_in_300k_imaginary_friends.html">without trackers</a></i>. </div>
<hr />
<p>If you have further data on these or similar incidents that you are able to share or if you want to look further into these and similar incidents, please let me know. </p>
<p>If you find any errors in the material I publish or disagree with my sentiments, or if you find this article interesting, useful or annoying, please let me know, either in comments or via email.</p>
<hr />
Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com5tag:blogger.com,1999:blog-8616610987649128333.post-55618067279242874522019-12-28T19:00:00.005+01:002022-03-31T20:55:06.381+02:00The Year 2019 in Review: This Was, Once Again, Weirder Than the Last One<i>The year is 2019. By now <a href="https://en.wikipedia.org/wiki/Blade_Runner">Blade Runner</a> is a movie about the past, but there are still bots out there trying to guess our passwords. It gets betterworse from here while the dictionaries expand.</i><br />
<br />
The year is coming to an end and events during that year, as they happened, somehow lead to me leaving writing mainly to one side and blog posting only until I saw a bigger picture. <br />
<br />
Now with only a couple of days left to go, we see that this year began much like the previous, with a not too bright set of bots endlessly trying to guess passwords. But on January 2, a new development caught my eye:<br />
<br />
<blockquote class="twitter-tweet">
<div dir="ltr" lang="en">
The the present round of <a href="https://twitter.com/hashtag/hailmary?src=hash&ref_src=twsrc%5Etfw">#hailmary</a> <a href="https://twitter.com/hashtag/ssh?src=hash&ref_src=twsrc%5Etfw">#ssh</a> <a href="https://twitter.com/hashtag/password?src=hash&ref_src=twsrc%5Etfw">#password</a> gropers are not run by are the sharpest knives in the drawer, it seems:<br />
<br />
Jan 2 21:29:48 skapet sshd[8180]: Failed password for invalid user password321 from 5.196.69.109 port 57634 ssh2<br />
<br />
And more where they came from of course.</div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1080587714456678401?ref_src=twsrc%5Etfw">January 2, 2019</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<br />
It was fairly obvious that some bot operator had the columns in their database mixed up, and I found the episode so laughable myself that I did not even bother to include it as the local part of a spamtrap. But as we will see later, it was an early sign of things to come. As you have probably suspected, the ssh password guessing activities have continued at pace, yielding this year so far<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">[Sat Dec 28 17:01:28] peter@skapet:~/website$ grep 2019 spamtraps-dateadded.txt | grep -c SSH<br />51233</span><br />
<br />
that is, the local part of more than fifty thousand spamtraps.<br />
<br />
In addition the early part of the year saw several campaigns of the email scams trying to extort various Bitcoin amounts in return for not publishing supposedly embarrassing videos, one of which I tweeted about:<br />
<br />
<blockquote class="twitter-tweet">
<div dir="ltr" lang="en">
Once more for the <a href="https://twitter.com/hashtag/blooper_reel?src=hash&ref_src=twsrc%5Etfw">#blooper_reel</a>, this time in German, TXT w/headers <a href="https://t.co/xcH0vmBQYs">https://t.co/xcH0vmBQYs</a>, PNG <a href="https://t.co/DnvMZrbCAG">https://t.co/DnvMZrbCAG</a> - note that all those messages come with List-Unsubscribe: headers. How utterly nice of them. <a href="https://t.co/OhYWvsEtVX">pic.twitter.com/OhYWvsEtVX</a></div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1083767335285149697?ref_src=twsrc%5Etfw">January 11, 2019</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
You should be able to find further absurdities of a similar kind by looking for the hashtags <b>#blooper_reel</b> from that tweet as well ast <b>#turbators</b>. With those hashtags you will notice that there is at least anecdotal evidence that messages of the same kind have been directed at a significant subset of our spamtraps here (which for obvious reasons would not have been used in connection with any actual user login anywhere), evidenced by the <a href="https://man.openbsd.org/spamd">spamd(8)</a> log snippet preserved in this tweet:<br />
<br />
<blockquote class="twitter-tweet">
<div dir="ltr" lang="en">
I almost wish they'd sent to actual users' addresses so I could see the full text:<br />
<br />
Feb 11 11:08:17 skapet spamd[79149]: 74.6.131.123: Body: s letter is not a hoax mail and I urge you to treat it serious. This letter=<br />
<br />
(another wankvid-hoax-blackmail attempt most likely).</div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1095000342360662016?ref_src=twsrc%5Etfw">February 11, 2019</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
And as noted in the followup tweet, other weirness was already happening:
<br />
<blockquote class="twitter-tweet">
<div dir="ltr" lang="en">
Also in a parallel development, the list of imaginary friends at <a href="https://t.co/3uthWgKWmL">https://t.co/3uthWgKWmL</a> rolled past 64K today (a pointless statistic to anyone except me, I know), proving that if you do something for long enough, true absurdity is achievable.</div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1095000343862300672?ref_src=twsrc%5Etfw">February 11, 2019</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<br />
More specifically, in the overnight haul on the morning of January 30th, I noticed via my scriptery that reports on such things that a large number of apparent bounce message deliveries to addresses made up of "Western-firstname.Chinese-lastname@mydomain.tld", such as <b>aaron.pu@bsdly.net</b> or <b>abby.na@bsdly.net</b>, had turned up, in addition to a few other varieties with no dot in the middle, possibly indicating separate sources.<br />
<br />
That initial overnight batch only had only a couple of hundred new potential spamtraps in it (as evidenced by the <a href="https://www.bsdly.net/~peter/spamtraps-dateadded.txt">spamtraps added log</a>), but even at that point the greylist data seemed to indicate that the bounces were produced by a relatively small set of IP addresses in Chinese networks. We see such bursts at times, but they rarely last long, so at first I did not think much of it before simply adding those addresses as <a href="https://www.bsdly.net/~peter/traplist.shtml">spamtraps</a>. <br />
<br />
This was one round that kind of exceeded expectations, in that what we can only conclude was the noise generated by one or more phishing campaigns targeting Chinese users lasted well into April of this year and ended up yielding more than 120,000 "imaginary friends", or <a href="https://www.bsdly.net/~peter/traplist.shtml">spamtraps</a> as others would say. It is likely that each of those fake addresses were used more than once, and in this context we only count new ones, so the actual number of messages and users targeted was probably a lot larger than the number of faked email addresses found in our logs here.<br />
<br />
The delivery attempt from this tweet may well have been a product of the same campaigns:<br />
<br />
<blockquote class="twitter-tweet">
<div dir="ltr" lang="en">
Possibly in the spirit of the upcoming holidays, this morning somebot in .cn tried delivering mail to new imaginary friend Jesus Mao:<br />
Apr 17 06:08:51 skapet spamd[32133]: new entry 106.13.14.227 from <ywlxo@funnyconsult.com> to <jesusmao@bsdly.net>, helo <a href="https://t.co/WC2xsKJs5N">https://t.co/WC2xsKJs5N</a></div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1118488932252094464?ref_src=twsrc%5Etfw">April 17, 2019</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<br />
By this time whoever was behind the campaign may have acieved their goals and moved on, or we could hope that they had been shut down by competent authorities.<br />
<br />
But back to the password guessers, sometimes referred to as <a href="https://bsdly.blogspot.com/2013/10/the-hail-mary-cloud-and-lessons-learned.html">The Hail Mary Cloud</a>. We have seen amazing feats of incompetence on their part before, but I seriously thought we had reached peak when some bot tried to log on a system in my care as the user <b>"*"</b> (yes, asterisk):<br />
<br />
<blockquote class="twitter-tweet">
<div dir="ltr" lang="en">
This must mean we have reached peak something:<br />
<br />
May 31 05:48:50 skapet sshd[35089]: Failed password for invalid user * from 178.62.90.135 port 53721 ssh2</div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1134330818648784899?ref_src=twsrc%5Etfw">May 31, 2019</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<br />
It should be noted of course that this confused my very much <a href="https://man.openbsd.org/grep">grep(1)</a>-based script that among other things turns up new candiates for spamtraps. But again it was an early indication that by their incompetence at least some of the bot herders had exposed their methods. Weird things turn up on occasion, but it took until October before it dawned on me that at least some of the password guessing bots could be running with their username and passwords fields swithched around:<br />
<br />
<blockquote class="twitter-tweet">
<div dir="ltr" lang="en">
The overnight haul of new user names attempted for ssh logins looks a lot like somebot switched the usenames and password columns around. Probably what passes for innovation in those parts of the world.</div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1180000032969363462?ref_src=twsrc%5Etfw">October 4, 2019</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<br />
A few days later I stopped trying to write a witty article about the phenomenon:
<br />
<blockquote class="twitter-tweet">
<div dir="ltr" lang="en">
I have tried but probably will keep failing at writing a coherent article about the bot that apparently has its username and passwords fields reversed. To wit, <br />
Oct 25 12:26:11 skapet sshd[78767]: Failed password for invalid user /']\\\\\\\\ from 188.254.0.226 port 51958 ssh2</div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1188039849493909505?ref_src=twsrc%5Etfw">October 26, 2019</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
<br />
but I kept harvesting new entries for the local parts of spamtraps, while noting that my still <a href="https://man.openbsd.org/grep">grep(1)</a>-centric script for detecting candiates would relatively frequently fail while trying to interpret what looked like regular expressions, with messages such as
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">grep: repetition-operator operand invalid<br />-bash: [: ==: unary operator expected</span><br />
<br />
turning up instead.<br />
<br />
Some of these entries (this month's worth so far* can be found in <a href="https://www.bsdly.net/~peter/namestotry.2019-12.txt">this file</a>) were weird enough (would you actually have created a user called <b>!@#$%^&*()dianlut+_ ?</b>) that they had me thinking that the operators of those bots were actually trying to be smart by working from stolen and published collections of password hashes. <br />
<br />
The wrinkle to this could turn to our advantage is that some of these operators managed to get the order of their fields wrong and are throwing either raw password hashes or decoded ones at our systems instead of matching user names. By reversing the process we might be able to see which collections are used, or other weird and creative things.<br />
<br />
If you are interested in doing further research on this, please contact me by email or the comments. I consider the <a href="https://www.bsdly.net/~peter/traplist.shtml">traplist</a> data, the <a href="https://www.bsdly.net/~peter/spamtraps-dateadded.txt">dates added log</a> and the other material mentioned in <a href="https://bsdly.blogspot.com/2018/08/badness-enumerated-by-robots.html">this piece</a> and links therein to be public and the data should be available to anyone. Howerer, some data exists only in some more detailed logs that are preserved here only to be seen by competent eyes and used for valid purposes. If you consider yourself such a person (aka a professional), please feel free to contact me.<br />
<br />
All the while, we see the dictionaries of user names and passwords expanding, and I for one am more than willing to help out in the effort. It helps us all identify the never-do-wells as early as possible in the game.<br />
<br />
The raw numbers for our contributions to the hopefully confusing dictionary as they stand right now are (they will be different when you read this):<br />
<br />
We have a total of 242778 spamtraps, with the numbers added according to the <a href="https://www.bsdly.net/~peter/spamtraps-dateadded.txt">dates added log</a> this year at 131195 from SMTP traffic, 51233 from failed SSH login attempts and 11 innovations from POP3 logon attempts.<br />
<br />
This means our list of spamtraps did not reach a full quarter milllion this year.<br />
<br />
But I already sense that somebody, somewhere is about to say "Hold my beer".<br />
<br />
<br />
<hr />
* <b>Update 2020-01-02</b>: This file now has the complete, or as complete as can be with the current scriptery, list of usernames tried during the whole month of December 2019.<br />
<br />
<b>Update 2020-01-07</b>: You would probably not notice from looking at the <a href="https://www.bsdly.net/~peter/namestotry.txt">raw listing</a> of attempted usernames so far this month, but the theme so far in 2020 among the new arrivals seems to be, of all things, three letter user names (take a peek at 2020 part so far at the end of the <a href="https://www.bsdly.net/~peter/spamtraps-dateadded.txt">spamtraps added log</a>). Go figure. <br />
<br />
<br />Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-17790132909329849402018-11-04T20:59:00.004+01:002024-02-01T12:07:01.494+01:00Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting<i>SMTP email is not going away any time soon. If you run a mail service, when and to whom you present the code signifying a temporary local problem code is well worth your attention.</i><br />
<br />
SMTP email is everywhere and is used by everyone.<br />
<br />
If you are a returning reader, there is a higher probability that you run a mail service yourself than in the general population.<br />
<br />
This in turn means that you will be aware that one of the rather annoying oversights of the original and still-current specifications of the <a href="https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol">SMTP</a> based mail system is that while it's straightforward to announce which systems are supposed to <i>receive</i> mail for a domain, specifying which hosts would be valid email <i>senders</i> was not part or the original specification at all.<br />
<br />
Any functioning domain MUST have at least one <a href="https://en.wikipedia.org/wiki/MX_record">MX</a> (mail exchanger) record published via the <a href="https://en.wikipedia.org/wiki/Domain_Name_System">domain name system</a>, and registrars will generally not even let you register a domain unless you have set up somewhere to receive mail for the domain. <br />
<br />
But email worked most of the time anyway, and while you would occasionally hear about valid mail not getting delivered, it was a rarer occurrence than you might think.<br />
<br />
Then a few years along, the Internet grew out of the pure research arena and became commercial, and <a href="https://en.wikipedia.org/wiki/Email_spam">spam</a> started happening. Even in the early days of spam it seems that a significant subset of the messages, possibly even the majority, was sent with faked sender addresses in domains not connected to the actual senders.<br />
<br />
Over time people have tried a number of approaches to the problems involved in getting rid of unwanted commercial and/or malware carrying email. If you are interested in a deeper dive into the subject, you could jump over to my earlier piece <a href="https://bsdly.blogspot.com/2014/02/effective-spam-and-malware.html">Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools</a>.<br />
<br />
Two very different methods of reducing spam traffic were originally formulated at roughly the same time, and each method's adherents are still duking it out over which approach is the better one.<br />
<br />
One method consists simply of implementing a strict interpretation of a requirement that was already formulated in the SMTP <a href="https://en.wikipedia.org/wiki/Request_for_Comments">RFC</a> at the time.<br />
<br />
The other is a complicated extension of the SMTP-relevant data that is published via <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a>, and full implementation would require reconfiguration of every SMTP email system in the world.<br />
<br />
As you might have guessed, the first is what is commonly referred to as <i><a href="https://en.wikipedia.org/wiki/Greylisting">greylisting</a></i>, where we point to the RFC's requirement that on encountering a temporary error, the sender <i>MUST</i> (RFC language does not get stronger than this) retry delivery at a later time and keep trying for a reasonable amount of time.<br />
<br />
Spammers generally did not retry as per the RFC specifications, and even early greylisting adopters saw huge drop in the volume of spam that actually made it to mailboxes.<br />
<br />
On the other hand, end users would sometimes wonder why their messages were delayed, and some mail administrators did not take well to seeing the volume of data sitting in the mail spool directories grow measurably, if not usually uncontrollably, while successive retries after waiting were in progress. <br />
<br />
In what could almost almost appear as a separate, unconnected universe, other network engineers set out to fix the now glaringly obvious omission in the existing RFCs.<br />
<br />
A way to announce valid senders was needed, and the specification that was to be known as the <i><a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework">Sender Policy Framework</a></i> (SPF for short) was offered to the world. SPF offered a way to specify which IP addresses valid mail from a domain were supposed to come from, and even included ways to specify how strictly the limitations it presented should be enforced at the receiving end.<br />
<br />
The downsides were that all mail handling would need to be upgraded with code that supported the specification, and as it turned out, traditional forwarding such as performed by common mailing list software would not easily be made compatible with SPF.<br />
<br />
<i>The flame wars over both methods.</i> You either remember them or should be able to imagine how they played out.<br />
<br />
And while the flames grew less frequent and generally less fierce over time, mail volumes grew to the level where operators would have a large number of servers for outgoing mail, and while the <i>site</i> would honor the requirement to retry delivery, the retries would not be guaranteed to come from the same IP address as the original attempt.<br />
<br />
It was becoming clear to greylisting practitioners that interpreting published SPF data as <i>known good</i> senders was the most workable way forward. Several of us already had started maintaining <span style="font-family: "courier new" , "courier" , monospace;">nospamd</span> tables (see eg <a href="https://home.nuug.no/~peter/pf/newest/downsides-compensate.html">this</a> slide and <a href="https://home.nuug.no/~peter/pftutorial/#56">this</a>), and using the output of<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ host -ttxt domain.tld</span><br />
<br />
(sometimes many times over because some domains use <span style="font-family: "courier new" , "courier" , monospace;">include</span> statements), we generally made do. I even made a habit of publishing my <span style="font-family: "courier new" , "courier" , monospace;">nospamd</span> file.<br />
<br />
As hinted in this <a href="https://home.nuug.no/~peter/pftutorial/#56">slide</a>, <a href="https://man.openbsd.org/smtpctl">smtpctl</a> (part of the <a href="https://www.opensmtpd.org/">OpenSMTPd</a> system and in your <a href="https://www.openbsd.org/">OpenBSD</a> base system) now since <a href="https://www.openbsd.org/63.html">OpenBSD 6.3</a> is able to retrieve the entire contents of the published SPF information for any domain you feed it.<br />
<br />
Looking over my old <span style="font-family: "courier new" , "courier" , monospace;">nospamd</span> file during the last week or so I found enough sedimentary artifacts there, including IP addresses for which there was no explanation and that lacked a reverse lookup, that I turned instead to deciphering which domains had been problematic and wrote a tiny <a href="https://home.nuug.no/~peter/generate-nospamd">script</a> to generate a fresh <span style="font-family: "courier new" , "courier" , monospace;">nospamd</span> on demand, based on fresh SPF lookups on those domains. The list of domains fed to the script is available <a href="https://home.nuug.no/~peter/thedomains.txt">here</a>, but please do edit to suit your local needs.<br />
<br />
For those wary of clicking links to scripts, it reads like this:<br />
<br />
<pre>#!/bin/sh
domains=`cat thedomains.txt`
outfile=nospamd
generatedate=`date`
operator="Peter Hansteen <peter@bsdly.net>"
locals=local-additions
echo "##############################################################################################">$outfile;
echo "# This is the `hostname` nospamd generated from domains at $generatedate. ">>$outfile;
<pre>echo "# See https://bsdly.blogspot.com/2018/11/goodness-enumerated-by-robots-or.html for some">>$outfile;</pre>
<pre>echo "# background and on why you should generate your own and not use this one.">>$outfile;</pre>
</pre>
<pre>echo "# Any questions should be directed to $operator. ">>$outfile;
echo "##############################################################################################">>$outfile;
echo >>$outfile;
for dom in $domains; do
echo "processing $dom";
echo "# $dom starts #########">>$outfile;
echo >>$outfile;
echo $dom | doas smtpctl spf walk >>$outfile;
echo "# $dom ends ###########">>$outfile;
echo >>$outfile;
done
echo "##############################################################################################">>$outfile;
echo "# processing done at `date`.">>$outfile;
echo "##############################################################################################">>$outfile;
echo "adding local additions from $locals";
echo "# local additions below here ----" >>$outfile;
cat $locals >> $outfile;
</pre>
<br />
If you have been in the habit of fetching my <span style="font-family: "courier new" , "courier" , monospace;">nospamd</span>, you have been fetching the output of this script for the last day or so.<br />
<br />
What it does is simply read a prepared list of domains, run them through <span style="font-family: "courier new" , "courier" , monospace;">smtpctl spf walk</span> and slap the results in a file which you would then load into the <a href="https://man.openbsd.org/pf.conf">pf configuration</a> on your <a href="https://man.openbsd.org/spamd">spamd</a> machine. You can even tack on a few local additions that for whatever reason do not come naturally from the domains list.<br />
<br />
But I would actually recommend you do <b>not</b> fetch my generated data, and rather use this script or a close relative of it (it's a truly trivial script and you probably can create a better version) and your own list of domains to generate a <span style="font-family: "courier new" , "courier" , monospace;">nospamd</span> tailored to your local environment. <br />
<br />
The specific list of domains is derived from more than a decade of maintaining my setup and the specific requests for whitelisting I have received from my users or quick fixes to observed problems in that period. It is conceivable that some domains that were problematic in the past no longer are, and unless we actually live in the same area, some of the domains in my list are probably not relevant to your users. There is even the possibility that some of the larger operators publish different SPF information in specific parts of the world, so the answers I get may not even match yours in all cases.<br />
<br />
So go ahead, script and generate! This is your chance to help the robots generate some <i>goodness</i>, for the benefit of your users.<br />
<br />
In related news, a request from my new colleagues gave me an opportunity to update the sometimes-repeated <a href="https://home.nuug.no/~peter/openbsd_and_you/">OpenBSD and you</a> presentation so it now has at least some information on <a href="https://www.openbsd.org/64.html">OpenBSD 6.4</a>. You could call the presentation a bunch of links in a thin wrapper of advocacy and you would not be very wrong. <br />
<br />
If you have comments or questions on any of the issues raised in this article, please let me know, preferably via the (moderated) comments field, but I have also been known to respond to email and via various social media message services.<br />
<br />
<b>Update 2018-11-11:</b> A few days after I had posted this article, an incident happened that showed the importance of keeping track of both goodness and badness for your services. This tweet is my reaction to a few quick glances at the <a href="http://bsdly.net/">bsdly.net</a> mail server log:<br />
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
The downside of maintaining a 55+ thousand entry spamtrap list and whitelisting by SPF is seeing one of the whitelisted sites apparently trying to spam every one of your spamtraps (see <a href="https://t.co/ulWt1EloRp">https://t.co/ulWt1EloRp</a>). Happening now. Wondering is collecting logs and forwarding worth it?</div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1060794868883177472?ref_src=twsrc%5Etfw">November 9, 2018</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
A little later I'm clearly pondering what to do, including doing another detailed writeup.
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
Then again it is an indication that the collected noise is now a required part of the spammer lexicon. One might want to point sites at throwing away outgoing messages to any address on <a href="https://t.co/3uthWgKWmL">https://t.co/3uthWgKWmL</a> (direct link to list <a href="https://t.co/mTaBpF5ucU">https://t.co/mTaBpF5ucU</a> - beware of html tags!).</div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1060800047812673536?ref_src=twsrc%5Etfw">November 9, 2018</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
Fortunately I had had some interaction with this operator earlier, so I knew roughly how to approach them. I wrote a couple of quick messages to their abuse contacts and made sure to include links to both my spamtrap resources and a fresh log excerpt that indicated clearly that someone or someones in their network was indeed progressing from top to bottom of the spamtraps list.
<br />
<blockquote class="twitter-tweet" data-lang="en">
<div dir="ltr" lang="en">
I ended up contacting their abuse@ with pointers to the logs that showed evidence of several similar campaigns over the last few days (the period I cared to look at) plus pointers to the spamtrap list and articles. About 30m after the second email to abuse@ the activity stopped.</div>
— Peter N. M. Hansteen (@pitrh) <a href="https://twitter.com/pitrh/status/1061182005294129153?ref_src=twsrc%5Etfw">November 10, 2018</a></blockquote>
<script async="" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>
As the last tweet says, delivery attempts stopped after progressing to somewhere into the Cs. The moral might be that a list of spamtraps like the one I publish might be useful for other sites to filtering their outgoing mail. Any activity involving the known-bad addresses would be a strong indication that somebody made a very unwise purchasing decision involving address lists.<br />
<br />
<b>Update 2019-08-07</b>: Gmail seems to be stuck on considering bsdly.net mail spam these days. If you are using a Google-attached mail service and have not received mail you were expecting from me, please check your spam folder and if you find anything, please use the "Report as not spam" feature. <br />
<br />
<b>Update 2019-08-07</b>: Updated script and generated file comment with encouragement to generate your own <b>nospamd</b> based on local needs, included link to the list used for the last <b>generate-nospamd</b> run.Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-15397189669581129572018-08-13T23:38:00.006+02:002023-10-21T15:41:34.356+02:00Badness, Enumerated by Robots<i>A condensed summary of the blocklist data generated from traffic hitting <a href="https://bsdly.net/">bsdly.net</a> and cooperating sites.</i><br />
<br />
After my <a href="https://runbsd.info/people/pitrh.html">runbsd.info entry</a> (previously <i>bsdjobs.com</i>) was posted, there has been an uptick in interest about the security related data generated at the <a href="https://bsdly.net/">bsdly.net</a> site. I have written quite extensively about these issues earlier so I'll keep this piece short. If you want to go deeper, the field note-like articles I reference and links therein will offer some further insights.<br />
<br />
There are three separate sets of downloadable data, all automatically generated and with only very occasional manual intervention.<br />
<br />
<br />
<b>Known spam sources during the last 24 hours</b><br />
<br />
This is the list directly referenced in the <a href="http://runbsd.info/">runbsd.info</a> piece.<br />
<br />
This is a <a href="https://bsdly.blogspot.com/2013/05/keep-smiling-waste-spammers-time.html">greytrapping</a> based list, where the conditions for inclusion are simple: Attempts at delivery to <a href="https://www.bsdly.net/~peter/traplist.shtml">known-bad addresses</a> (download link <a href="https://www.bsdly.net/~peter/sortlist.txt">here</a>) in domains we handle mail for have happened within the last 24 hours. <br />
<br />
In addition there will occasionally be some addresses added by <a href="https://man.openbsd.org/cron">cron</a> jobs I run that pick the IP addresses of hosts that sent mail that made it through <a href="https://en.wikipedia.org/wiki/Greylisting">greylisting</a> performed by our <a href="https://man.openbsd.org/spamd">spamd(8)</a> but did not pass the subsequent <a href="http://spamassassin.apache.org/">spamassassin</a> or <a href="https://www.clamav.net/">clamav</a> treatment. The <a href="https://bsdly.net/">bsdly.net</a> system is part of the <a href="https://www.bgp-spamd.net/">bgp-spamd</a> cooperation.<br />
<br />
The traplist has a <a href="https://www.bsdly.net/~peter/traplist.shtml">home page</a> and at one point was furnished with a set of <a href="https://www.bsdly.net/~peter/traplist_ethics.shtml">guidelines</a>. <br />
<br />
A partial history (the log starts <span style="white-space: pre-wrap;">2017-05-20)</span> of when spamtraps were added and from which sources can be found in <a href="https://www.bsdly.net/~peter/spamtraps-dateadded.txt">this log</a> (or at this <a href="https://home.nuug.no/~peter/spamtraps-dateadded.txt">alternate location</a>). Read on for a bit of information on the alternate sources.<div><br /></div><div><b>Note:</b> The list is generated at ten past every full hour by a script that uses essentially the one-liner<br /><br /><span><span style="font-family: courier;"> spamdb | grep TRAPPED | awk -F\| '{print $2}' >bsdly.net.traplist</span></span></div><div><br /></div><div><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><div style="text-align: left;">to generate the body of the list.</div></blockquote>
<br />
<b>Misc other bots: SSH Password bruteforcing, malicious web activity, POP3 Password Bruteforcing. </b><br />
<br />
The <i><a href="https://www.bsdly.net/~peter/bruteforcers.txt">bruteforcers</a></i> list is really a combination of several things, delivered as one file but with minimal scripting ability you should be able to dig out the distinct elements, described in <a href="https://bsdly.blogspot.com/2017/04/forcing-password-gropers-through.html">this piece</a>.<br />
<br />
The (usually) largest chunk is a list of hosts that hit the rate limit for SSH connections described in the article or that was caught trying to log on as a non-existent user or other undesirable activity aimed at my <a href="https://man.openbsd.org/sshd">sshd(8)</a> service. Some as yet unpublished scriptery helps me feed the miscreants that the automatic processes do not catch into the table after a manual quality check. For a more thorough treatment of <span style="font-family: courier;">ssh</span> bruteforcers, see the <i><a href="https://bsdly.blogspot.com/2013/10/the-hail-mary-cloud-and-lessons-learned.html">The Hail Mary Cloud and the Lessons Learned</a></i> overview article which links to several other articles in the sequence.<br />
<br />
The second part is a list of IP addresses that tried to access our web service in undesirable ways, including trying for specific URLs or files that will never be found at any world-facing part of our site. <br />
<br />
After years of advocating short lifetimes (typically 24 hours) for blocklist entries only to see my logs fill up with attempts made at slightly slower speeds, I set the lifetime for entries in this data set to 28 days (since expanded to 2419200 seconds, or if you will, six weeks). The background including some war stories of monitoring SSH password groping can be found in <a href="https://bsdly.blogspot.com/2013/10/the-hail-mary-cloud-and-lessons-learned.html">this piece</a>, while the more recent piece <a href="https://bsdly.blogspot.com/2017/04/forcing-password-gropers-through.html">here</a> covers some of the weeding out bad web activity.<br />
<br />
The <i><a href="https://www.bsdly.net/~peter/pop3gropers.txt">POP3 gropers</a></i> list comes in two variations. Again lists of IP addresses caught trying to access a service, most of those accesses are to non-existent user names with an almost perfect overlap with the spamtraps list, local-part only (the part before the @ sign). <br />
<br />
The <a href="https://www.bsdly.net/~peter/pop3gropers.txt">big list</a> is a complete corpus of IP addresses that have tried these kinds of accesses since I started recording and trapping them (see <a href="https://bsdly.blogspot.com/2014/08/password-gropers-take-spamtrap-bait.html">this piece</a> for some early experience and <a href="https://bsdly.blogspot.com/2016/08/chinese-hunting-chinese-over-pop3-in.html">this one</a> for the start of the big collection).<br />
<br />
There is also a <a href="https://www.bsdly.net/~peter/pop3_sixweeks.txt">smaller set</a>, produced from the <span style="font-family: "courier new", "courier", monospace;">longterm</span> table described in <a href="https://bsdly.blogspot.com/2017/04/forcing-password-gropers-through.html">this piece</a>. For much the same reason I did not stick to 24-hour expiry for the SSH list, this one has six-week expiry. With some minimal scriptery I run by hand one or two times per day, any invalid POP3 accesses to valid accounts get their IP adresses added to the <span style="font-family: "courier new", "courier", monospace;">longterm</span> table and the exported list.<br />
<br /><b>Note:</b> The lists generated by table exports are generated by variations of <a href="https://man.openbsd.org/pfctl"><span style="font-family: courier;">pfctl</span></a>'s <span style="font-family: courier;">show</span> table subcommand. At ruleset reload such as reboots after a <a href="https://man.openbsd.org/sysupgrade">sysupgrade</a>, the tables are re-initialized from these same exported files.<br /><br />
If you're wondering about the title, the term <i>"enumerating badness"</i> stems from Marcus Ranum's classic piece <i><a href="http://www.ranum.com/security/computer_security/editorials/dumb/index.html">The Six Dumbest Ideas in Computer Security</a></i>. Please do read that one.<br />
<br />
Here are a few other references other than those referenced in the paragraphs above that you might find useful:<br />
<br />
<a href="https://www.nostarch.com/pf3">The Book of PF</a>, 3rd edition<br />
<a href="https://bsdly.blogspot.com/2007/07/hey-spammer-heres-list-for-you.html">Hey, spammer! Here's a list for you!</a> which contains the announcement of the bsdly.net traplist.<br />
<a href="http://bsdly.blogspot.com/2014/02/effective-spam-and-malware.html">Effective Spam and Malware Countermeasures</a>, a more complete treatment of those keywords<br />
<br />
If you're interested in further information on any of this, the most useful contact information is in the comment blocks in the exported lists.<br /><br /><b>Update 2020-07-29:</b> I added a direct link to the complete list of spamtraps, since the web page seemed a bit crowded to at least one visitor. Direct link again <a href="https://www.bsdly.net/~peter/sortlist.txt">here</a> for your convenience.<div><br /></div><div><b>Update 2021-01-15:</b> Note that at some point after the article was written I cranked up expiry for the <span style="font-family: courier;">bruteforce</span> tables to six weeks (sorry, I forgot to note the exact date).<br /><br /><b>Update 2021-03-11:</b> In light of recent Microsoft Exchange exploits it might interest some that any request to <a href="http://bsdly.net">bsdly.net</a> for <span style="font-family: courier;">"GET /owa/"</span> lands the source in the <span style="font-family: courier;">webtrash</span> table, exported as part of the <i><a href="https://www.bsdly.net/~peter/bruteforcers.txt">bruteforcers</a></i> list.</div></div><div><br /></div><div><b>Update 2021-08-03:</b> Added notes about how the lists are generated and table maintenance.</div>Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com0tag:blogger.com,1999:blog-8616610987649128333.post-87301218168840693562018-04-01T12:21:00.000+02:002018-04-01T16:43:30.032+02:00ed(1) mastery is a must for a real Unix person<i><a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> is the standard editor. Now there's a book out to help you master this fundamental Unix tool.</i><br />
<br />
In some circles on the Internet, your choice of text editor is a serious matter.<br />
<br />
We've all seen the threads on mailing lits, USENET news groups and web forums about the relative merits of <i>Emacs</i> vs <i>vi</i>, including endless iterations of flame wars, and sometimes even involving lesser known or non-portable editing environments.<br />
<br />
And then of course, from the Linux newbies we have seen an endless stream of tweeted graphical <i>'memes'</i> about the editor <i>vim</i> (aka <i>'vi Improved'</i>) versus the various apparently friendlier-to-some options such as GNU <i>nano</i>. Apparently even the 'improved' version of the classical and ubiquitous <a href="https://man.openbsd.org/vi" target="_blank">vi(1)</a> editor is a challenge even to exit for a significant subset of the younger generation.<br />
<br />
Yes, your choice of text editor or editing environment is a serious matter. Mainly because text processing is so fundamental to our interactions with computers.<br />
<br />
But for those of us who keep our systems on a real Unix (such as <a href="https://www.openbsd.org/" target="_blank">OpenBSD</a> or <a href="https://www.freebsd.org/" target="_blank">FreeBSD</a>), there is no real contest. The <a href="https://www.openbsd.org/" target="_blank">OpenBSD</a> base system contains several text editors including <a href="https://man.openbsd.org/vi" target="_blank">vi(1)</a> and the almost-emacs <a href="https://man.openbsd.org/mg" target="_blank">mg(1)</a>, but <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> remains the standard editor.<br />
<br />
Now <a href="https://www.michaelwlucas.com/" target="_blank">Michael Lucas</a> has written a book to guide the as yet uninitiated to the fundamentals of the original Unix text editor. It is worth keeping in mind that much of Unix and its original standard text editor written back when the standard output and default user interface was more likely than not a printing terminal.<br />
<br />
To some of us, reading and following the narrative of <a href="https://www.michaelwlucas.com/tools/ed" target="_blank">Ed Mastery</a> is a trip down memory lane. To others, following along the text will illustrate the horror of the world of pre-graphic computer interfaces. For others again, the fact that <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> doesn't use your terminal settings much at all offers hope of fixing things when something or somebody screwed up your system so you don't have a working terminal for that visual editor.<br />
<br />
<a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> is a line editor. And while you may have heard mutters that <i>'vi is just a line editor in drag'</i>, <a href="https://man.openbsd.org/vi" target="_blank">vi(1)</a> does offer a distinctly visual interface that only became possible with the advent of the video terminal, affectionately known as the glass teletype. <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> offers no such luxury, but as the book demonstrates, even <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> is able to display any part of a file's content for when you are unsure what your file looks like.<br />
<br />
The book <a href="https://www.michaelwlucas.com/tools/ed" target="_blank">Ed Mastery</a> starts by walking the reader through a series of editing sessions using the classical <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> line editing interface. To some readers the thought of editing text while not actually seeing at least a few lines at the time onscreen probably sounds scary. This book shows how it is done and while the author never explicitly mentions it, the text aptly demonstrates how the <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> command set is in fact the precursor of of how things are done in many Unix text processing programs.<br />
<br />
As one might expect, the walkthrough of <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> text editing functionality is followed up by a sequence on searching and replacing which ultimately leads to a very readable introduction to regular expressions, which of course are part of the <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> package too. If you know your <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> command set, you are quite far along in the direction of mastering the stream editor <a href="https://man.openbsd.org/sed" target="_blank">sed(1)</a>, as well as a number of other systems where regular expressions play a crucial role.<br />
<br />
After the basic editing functionality and some minor text processing magic has been dealt with, the book then proceeds to demonstrate <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> as a valuable tool in your Unix scripting environment. And once again, if you can do something with ed, you can probably transfer that knowledge pretty much intact to use with other Unix tools.<br />
<br />
The eighty-some text pages of <a href="https://www.michaelwlucas.com/tools/ed" target="_blank">Ed Mastery</a> are a source of solid information on the <a href="https://man.openbsd.org/ed" target="_blank">ed(1)</a> tool itself with a good helping of historical context that will make it clearer to newcomers why certain design choices were made back when the Unix world was new. A number of these choices influence how we interact with the modern descendants of the Unix systems we had back then.<br />
<br />
Your choice of text editor is a serious matter. With this book, you get a better foundation for choosing the proper tool for your text editing and text processing needs. I'm not saying that you have to switch to the standard editor, but after reading <a href="https://www.michaelwlucas.com/tools/ed" target="_blank">Ed Mastery</a> , your choice of text editing and processing tools will be a much better informed one.<br />
<br />
<a href="https://www.michaelwlucas.com/tools/ed" target="_blank">Ed Mastery</a> is available now directly from Michael W. Lucas' books site at <a href="https://www.michaelwlucas.com/tools/ed">https://www.michaelwlucas.com/tools/ed</a>, and will most likely appear in other booksellers' catalogs as soon as their systems are able to digest the new data.<br />
<br />
Do read the book, try out the standard editor and have fun!Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com6tag:blogger.com,1999:blog-8616610987649128333.post-85956613372477864152018-02-17T17:38:00.000+01:002019-07-16T12:45:49.309+02:00A Life Lesson in Mishandling SMTP Sender Verification<i>An attempt to report spam to a mail service provider's abuse address reveals how incompetence is sometimes indistinguishable from malice.</i><br />
<br />
It all started with one of those rare spam mails that got through.<br />
<br />
This one was hawking address lists, much like the ones I occasionally receive to addresses that I can not turn into <a href="https://www.bsdly.net/~peter/traplist_ethics.shtml" target="_blank">spamtraps</a>. The message was addressed to, of all things, <b><span style="font-family: "courier new" , "courier" , monospace;">root@skapet.bsdly.net</span></b>. (The message with full headers has been preserved <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180217_Janet_Farell_outlook.com.txt" target="_blank">here</a> for reference).<br />
<br />
Yes, that's right, they sent their spam to <b>root@</b>. And a quick peek at the headers revealed that like most of those attempts at hawking address lists for spamming that actually make it to a mailbox here, this one had been sent by an <a href="http://outlook.com/">outlook.com</a> customer.<br />
<br />
The problem with spam delivered via <i>outlook.com</i> is that you can't usefully blacklist the sending server, since the largish chunk of the world that uses some sort of Microsoft hosted email solution (Office365 and its ilk) have their usually legitimate mail delivered via the very same infrastructure.<br />
<br />
And since <i>outlook.com</i> is one of the mail providers that doesn't play well with greylisting (it spreads its retries across no less than 81 subnets (the output of '<span style="font-family: "courier new" , "courier" , monospace;">echo outlook.com | <a href="https://man.openbsd.org/doas" target="_blank">doas</a> <a href="https://man.openbsd.org/smtpctl" target="_blank">smtpctl</a> spf walk</span>' is preserved <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180217_smtpctl_spf_walk_outlook.com.txt" target="_blank">here</a>), it's fairly common practice to just whitelist all those networks and avoid the hassle of lost or delayed mail to and from Microsoft customers.<br />
<br />
I was going to just ignore this message too, but we've seen an increasing number of spammy outfits taking advantage of <i>outlook.com</i>'s seeming right of way to innocent third parties' mail boxes.<br />
<br />
So I decided to try both to do my best at demoralizing this particular sender and alert <i>outlook.com</i> to their problem. I wrote a messsage (preserved <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180217_to_farell_and_abuse_at_outlook.com.txt" target="_blank">here</a>) with a <i>Cc:</i> to <i>abuse@outlook.com</i> where the meat is,<br />
<br />
<blockquote class="tr_bq">
Ms Farell,<br />
<br />
The address root@skapet.bsdly.net has never been subscribed to any mailing list, for obvious reasons. Whoever sold you an address list with that address on it are criminals and you should at least demand your money back.<br />
<br />
Whoever handles abuse@outlook.com will appreciate the attachment, which is a copy of the message as it arrived here with all headers intact.<br />
<br />
Yours sincerely,<br />
Peter N. M. Hansteen</blockquote>
<br />
What happened next is quite amazing.<br />
<br />
If my analysis is correct, <i>it may not be possible</i> for senders who are not themselves <i>outlook.com</i> customers to actually reach the <i>outlook.com</i> abuse team.<br />
<br />
Almost immediately after I sent the message to Ms Farell with a Cc: to <i>abuse@outlook.com</i>, two apparently identical messages from <i>staff@hotmail.com</i>, addressed to <i>postmaster@bsdly.net</i> appeared (preserved <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180217_from_staff_at_hotmail.com_1.txt" target="_blank">here</a> and <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180217_from_staff_at_hotmail.com_2.txt" target="_blank">here</a>), with the main content of both stating<br />
<br />
<blockquote class="tr_bq">
This is an email abuse report for an email message received from IP 216.32.180.51 on Sat, 17 Feb 2018 01:59:21 -0800.<br />
The message below did not meet the sending domain's authentication policy.<br />
For more information about this format please see <a href="http://www.ietf.org/rfc/rfc5965.txt">http://www.ietf.org/rfc/rfc5965.txt</a>.</blockquote>
<br />
In order to understand what happened here, it is necessary to look at the mail server log for a time interval of a few seconds (preserved <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180217_mailserverlog.txt" target="_blank">here</a>).<br />
<br />
The first few lines describe the processing of my outgoing message:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">2018-02-17 10:59:14 1emzGs-0009wb-94 <= peter@bsdly.net H=(greyhame.bsdly.net) [192.168.103.164] P=esmtps X=TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=no S=34977 id=31b4ffcf-bf87-de33-b53a-0
ebff4349b94@bsdly.net</span>
<br />
<br />
My server receives the message from my laptop, and we can see that the connection was properly TLS encrypted<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">2018-02-17 10:59:15 1emzGs-0009wb-94 => peter <root@skapet.bsdly.net> R=localuser T=local_delivery</span>
<br />
<br />
I had for some reason kept the original recipient among the <i>To:</i> addresses. Actually useless but also harmless.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">2018-02-17 10:59:16 1emzGs-0009wb-94 [104.47.40.33] SSL verify error: certificate name mismatch: DN="/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/OU=Microsoft Corporation/CN=mail.protection.outlook.com" H="outlook-com.olc.protection.outlook.com"
<br />
2018-02-17 10:59:18 1emzGs-0009wb-94 SMTP error from remote mail server after end of data: 451 4.4.0 Message failed to be made redundant due to A shadow copy was required but failed to be made with an AckStatus of Fail [CO1NAM03HT002.eop-NAM03.prod.protection.outlook.com] [CO1NAM03FT002.eop-NAM03.prod.protection.outlook.com]
<br />
2018-02-17 10:59:19 1emzGs-0009wb-94 [104.47.42.33] SSL verify error: certificate name mismatch: DN="/C=US/ST=WA/L=Redmond/O=Microsoft Corporation/OU=Microsoft Corporation/CN=mail.protection.outlook.com" H="outlook-com.olc.protection.outlook.com"
</span>
<br />
<br />
What we see here is that even a huge corporation like Microsoft does not always handle certificates properly. The certificate they present for setting up the encrypted connection is not actually valid for the host name that the outlook.com server presents.<br />
<br />
There is also what I interpret as a file system related message which I assume is meaningful to someone well versed in Microsoft products, but we see that
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">2018-02-17 10:59:20 1emzGs-0009wb-94 => janet@prospectingsales.net R=dnslookup T=remote_smtp H=prospectingsales-net.mail.protection.outlook.com [23.103.140.138] X=TLSv1.2:ECDHE-RSA-AES256-SHA384:256 CV=yes K C="250 2.6.0 <31b4ffcf-bf87-de33-b53a-0ebff4349b94@bsdly.net> [InternalId=40926743365667, Hostname=BMXPR01MB0934.INDPRD01.PROD.OUTLOOK.COM] 44350 bytes in 0.868, 49.851 KB/sec Queued mail for delivery"</span>
<br />
<br />
even though the certificate fails the verification part, the connection sets up with TLSv1.2 anyway, and the message is accepted with a "Queued mail for delivery" message.<br />
<br />
The message is also delivered to the Cc: recipient:
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">2018-02-17 10:59:21 1emzGs-0009wb-94 => abuse@outlook.com R=dnslookup T=remote_smtp H=outlook-com.olc.protection.outlook.com [104.47.42.33] X=TLSv1.2:ECDHE-RSA-AES256-SHA384:256 CV=no K C="250 2.6.0 <31b4ffcf-bf87-de33-b53a-0ebff4349b94@bsdly.net> [InternalId=3491808500196, Hostname=BY2NAM03HT071.eop-NAM03.prod.protection.outlook.com] 42526 bytes in 0.125, 332.215 KB/sec Queued mail for delivery"
<br />
2018-02-17 10:59:21 1emzGs-0009wb-94 Completed
</span>
<br />
<br />
And the transactions involving my message would normally have been completed.
<br />
<br />
But ten seconds later this happens:
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">2018-02-17 10:59:31 1emzHG-0004w8-0l <= staff@hotmail.com H=bay004-omc1s10.hotmail.com [65.54.190.21] P=esmtps X=TLSv1.2:ECDHE-RSA-AES256-SHA384:256 CV=no K S=43968 id=BAY0-XMR-100m4KrfmH000a51d4@bay0-xmr-100.phx.gbl
<br />
2018-02-17 10:59:31 1emzHG-0004w8-0l => peter <postmaster@bsdly.net> R=localuser T=local_delivery
<br />
2018-02-17 10:59:31 1emzHG-0004w8-0l => peter <postmaster@bsdly.net> R=localuser T=local_delivery
</span>
<br />
<br />
That's the <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180217_from_staff_at_hotmail.com_1.txt" target="_blank">first</a> message to my domain's postmaster@ address, followed two seconds later by
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">2018-02-17 10:59:33 1emzHI-0004w8-Fy <= staff@hotmail.com H=bay004-omc1s10.hotmail.com [65.54.190.21] P=esmtps X=TLSv1.2:ECDHE-RSA-AES256-SHA384:256 CV=no K S=43963 id=BAY0-XMR-100Q2wN0I8000a51d3@bay0-xmr-100.phx.gbl
<br />
2018-02-17 10:59:33 1emzHI-0004w8-Fy => peter <postmaster@bsdly.net> R=localuser T=local_delivery
<br />
2018-02-17 10:59:33 1emzHI-0004w8-Fy Completed
</span>
<br />
<br />
a <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180217_from_staff_at_hotmail.com_2.txt" target="_blank">second</a>, apparently identical message.<br />
<br />
Both of those messages state that the message I sent to abuse@outlook.com had failed <a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework" target="_blank">SPF</a> verification, because the check happened on connections from <b>NAM03-BY2-obe.outbound.protection.outlook.com</b> (<i>216.32.180.51</i>) by whatever handles incoming mail to the <i>staff@hotmail.com</i> address, which apparently is where the system forwards <i>abuse@outlook.com</i>'s mail.<br />
<br />
Reading Microsoft Exchange's variant SMTP headers has never been my forte, and I won't try decoding the exact chain of events here since that would probably also require you to have fairly intimate knowledge of Microsoft's internal mail delivery infrastructure.<br />
<br />
But even a quick glance at the messages reveals that the message passed SPF and other checks on incoming to the <i>outlook.com</i> infrastructure, but may have ended up not getting delivered after all since a second SPF test happened on a connection from a host that is not in the sender domain's SPF record. <br />
<br />
In fact, that second test would only succeed for domains that have<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">include:spf.protection.outlook.com</span></blockquote>
<br />
in their SPF record, and those would presumably be <i>Outlook.com</i> customers.<br />
<br />
Any student or practitioner of SMTP mail delivery should know that <a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework" target="_blank">SPF</a> records should only happen on <i>ingress</i>, that is at the point where the mail traffic enters your infrastructure and the sender IP address is the original one. Leave the check for later when the message may have been forwarded, and you do not have sufficient data to perform the check.<br />
<br />
Whenever I encounter incredibly stupid and functionally destructive configuration errors like this I tend to believe they're down to simple incompetence and not malice.<br />
<br />
But this one has me wondering. If you essentially require incoming mail to include the contents of <i>spf.outlook.com</i> (currently no less than <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180217_smtpctl_spf_walk_outlook.com.txt" target="_blank">81 subnets</a>) as valid senders for the domain, you are essentially saying that only <i>outlook.com</i> customers are allowed to communicate.<br />
<br />
If that restriction is a result of a deliberate choice rather than a simple configuration error, the problem moves out of the technical sphere and could conceivably become a legal matter, depending on what <i>outlook.com</i> have specified in their contracts that they are selling to their customers.<br />
<br />
But let us assume that this is indeed a matter of simple bad luck or incompetence and that the solution is indeed technical.<br />
<br />
I would have liked to report this to whoever does technical things at that domain via email, but unfortunately there are indications that being their customer is a precondition for using that channel of communication to them.<br />
<br />
I hope they fix that, and soon. And then move on to terminating their spamming customers' contracts.<br />
<br />
The main <i>lesson to be learned</i> from this is that when you shop around for email service, please do yourself a favor and make an effort to ensure that your prospective providers actually understand how the modern-ish <a href="https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol" target="_blank">SMTP</a> addons <a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework" target="_blank">SPF</a>, <a href="https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail" target="_blank">DKIM</a> and <a href="https://en.wikipedia.org/wiki/DMARC" target="_blank">DMARC</a> actually work.<br />
<br />
Otherwise you may end up receiving more of the mail you don't want than what you do want, and your own mail may end up not being delivered as intended.<br />
<br />
<b>Update 2018-02-19</b>: Just as I was going to get ready for bed (it's late here in CET) another <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180219_Janet_farell_to_tutorial_at_bsdly.net.txt" target="_blank">message</a> from Ms Farell arrived, this time to an alias I set up in order to make it easier to filter PF tutorial related messages into a separate mailbox. <br />
<br />
I wrote another <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180219_to_farell_and_abuse_at_outlook.com.txt" target="_blank">response</a>, and as the <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180219_exim_log.txt" target="_blank">mail server log</a> will show, despite the fact that a friend with an Office365 contract contacted them quoting this article, <i>outlook.com</i> have still <b>not</b> fixed the problem. Two more messages (preserved <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180219_from_staff_at_hotmail.com_1.txt" target="_blank">here</a> and <a href="https://home.nuug.no/~peter/spf-fail_outlook/20180219_from_staff_at_hotmail.com_2.txt" target="_blank">here</a>) shot back here immediately.<br />
<br />
<b>Update 2018-02-20</b>: <i>A response from Microsoft, with pointers to potentially useful information</i>.<br />
<br />
A message from somebody identifying as working for Microsoft Online Safety arrived, apparently responding to my message dated 2018-02-19, where the main material was,<br />
<br />
<blockquote class="tr_bq">
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Hi,</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Based on the information you provided, it appears to have originated from an Office 365 or Exchange Online tenant account. </div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
To report junk mail from Office 365 tenants, send an email to junk@office365.microsoft.com<span style="mso-spacerun: yes;"> </span>and include the junk mail as an attachment. </div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
This link provides further junk mail education <a href="https://technet.microsoft.com/en-us/library/jj200769%28v=exchg.150%29.aspx">https://technet.microsoft.com/en-us/library/jj200769(v=exchg.150).aspx</a>. </div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<span style="font-size: x-small;"></span></div>
</blockquote>
<br />
<blockquote class="tr_bq">
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<span style="font-size: x-small;">Kindly,</span></div>
</blockquote>
I have asked for clarification of some points, but no response has arrived by this getting close to bedtime in CET. <br />
<br />
However I did take the advice to forward the offending messages as attachment to the <b>junk@</b> message, and put the <i>outlook.com</i> abuse address in the <b>Cc:</b> on that message. My logs indicate that the certificate error had not gone away, but no SPF-generated bounces appeared either.<br />
<br />
If Microsoft responds with further clarifications, I will publish a useful condensate here.<br />
<br />
<b>Update 2019-07-16</b>: If you were wondering how I make the output of <b>smtpctl spf walk</b> useful as mentioned in this article, please see the article <a href="https://bsdly.blogspot.com/2018/11/goodness-enumerated-by-robots-or.html">Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting</a> for some specifics.<br />
<hr />
<br />
In other news, there will be <a href="https://2018.asiabsdcon.org/program.html.en#" target="_blank">PF tutorial</a> at the 2018 <a href="https://2018.asiabsdcon.org/index.html.en" target="_blank">AsiaBSDCon</a> in Tokyo. Follow the links for the most up to date information.Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com4tag:blogger.com,1999:blog-8616610987649128333.post-90159874501024250472017-08-27T14:44:00.000+02:002017-12-29T16:52:43.645+01:00Twenty-plus years on, SMTP callbacks are still pointless and need to die<div class="tr_bq">
<i>A rarely used legacy misfeature of the main Internet email protocol creeps back from irrelevance as a minor annoyance. You should ask your mail and antispam provider about their approach to 'SMTP callbacks'. Be wary of any assertion that is not backed by evidence.</i></div>
<br />
Even if you are an IT professional and run an email system, you could be forgiven for not being immediately aware that there is such a thing as <i>SMTP callbacks</i>, also referred to as <a href="https://en.wikipedia.org/wiki/Callback_verification">callback verification</a>. As you will see from the Wikipedia article, the feature was never widely adopted, and for all too understandable reasons.<br />
<br />
If you do run a mail system, you have probably heard about that feature's predecessor, the still-required but rarely used <a href="https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol">SMTP</a> <span style="font-family: "courier new" , "courier" , monospace;">VRFY</span> and <span style="font-family: "courier new" , "courier" , monospace;">EXPN</span> commands. Those commands offer a way to verify whether an address is valid and to show the component addresses that a mailing list resolves to, respectively.<br />
<br />
Back when all things inter-networking were considered experimental and it was generally thought that information should flow freely in and between those experimental networks, it was quite common for mail servers to offer <span style="font-family: "courier new" , "courier" , monospace;">VRFY</span> and <span style="font-family: "courier new" , "courier" , monospace;">EXPN</span> service to all comers. <br />
<br />
I'm old enough to remember using <span style="font-family: "courier new" , "courier" , monospace;">VRFY</span> by hand, <span style="font-family: "courier new" , "courier" , monospace;">telnet</span>-ing to port 25 on a mail server and running <span style="font-family: "courier new" , "courier" , monospace;">VRFY <i>$user@$domain.$tld</i></span> commands to check whether an email address was indeed valid. I've forgotten which domains and persons were involved, but I imagine the reason why was that I wanted to contact somebody who had said something interesting in a post to a <a href="https://en.wikipedia.org/wiki/Usenet">USENET</a> news group.<br />
<br />
But networkers trying to make contact with each other were not the only ones who discovered the <i>VRFY</i> and <i>EXPN</i> commands. Soon spammers were using those commands to actively harvest <i><b>actually! valid! deliverable!</b></i> addresses, and by 1999 the <a href="https://tools.ietf.org/html/rfc2505">RFC2505</a> best practices document recommended disabling the features altogether. After all, there would usually be some other way available to find somebody's email address (there was even a FAQ, a longish Frequently Asked Questions document with apparent USENET origins written and maintained on the subject, a copy of which can be found <a href="https://www.pimall.com/nais/n.emailfind.html">here</a>).<br />
<br />
In roughly the same time frame, somebody came up with the idea of SMTP callbacks. The idea was that all domains on the Internet need to publish the address of their mail exchangers via <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> <a href="https://en.wikipedia.org/wiki/MX_record">MX</a> (mail exchanger) records. The logical next step is then that when a piece of mail arrives over SMTP, the receiving end should be able to contact the sender domain's known mail exchanger to check that the sender address is indeed valid. If you by now hear the echoes of VRFY and EXPN, you're right. There are indications that some early implementations did in fact use VRFY for that purpose.<br />
<br />
But then the world changed, and you could not rely on VRFY being available in the post-RFC2505 world.<br />
<br />
In the post-RFC2505 world, the other side would most likely not offer up any useful information in response to VRFY commands, and you would most likely be limited to the short interchange that the Wikipedia entry <a href="https://en.wikipedia.org/wiki/Callback_verification#Process">quotes</a>,<br />
<div>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">HELO <verifier host name><br />
MAIL FROM:<><br />
RCPT TO:<the address to be tested><br />
QUIT
</span></blockquote>
</div>
<div>
which a perceptive reader would identify as only verifying in a very limited sense that the domain's mail exchanger was indeed equipped with a functional SMTP service.</div>
<div>
<br />
It is worth noting, as many have over the years, that the MX records only specify where a domain expects to <i>receive</i> mail, not where valid mail from the domain is supposed to originate. Several mechanisms to help identify valid mail senders for a domain have been devised in the intervening years, but none existed at the time SMTP callbacks were considered even remotely useful. </div>
<div>
<br />
<div>
For reasons that are not entirely clear, some developers kept working on SMTP callback code and several mail server implementations available today (2017) still contain code that looks like it was intended to support information-rich callbacks, if the system was configured to enable the feature at all. The default configurations in general do not enable the SMTP callback feature, and mail admins rarely bother to even learn much about the largely disused and (in my opinion at least) not too well thought out feature.</div>
<div>
<br /></div>
<div>
This all happened back in the 1990s, but quite recently an incident occurred that indicates that in some pockets of the Internet, SMTP callbacks are still in use, and in at least some cases data from the callbacks are used for generating blacklists and block mail delivery. The last part should raise a few eyebrows at least.</div>
<div>
<br /></div>
<div>
Jumping forward from the distant 1990s to the present day, regular readers of this column will be aware that <b>bsdly.net </b>and cooperating domains run SMTP service with <a href="https://www.openbsd.org/">OpenBSD</a> <a href="https://man.openbsd.org/spamd">spamd(8)</a> doing <a href="https://en.wikipedia.org/wiki/Greylisting">greylisting</a> service, and that the <a href="https://man.openbsd.org/spamd">spamd(8)</a> setup produces a greytrapping-based <a href="https://www.bsdly.net/~peter/traplist.html">blacklist</a> which is available for download, dumped to a file (available <a href="https://home.nuug.no/~peter/bsdly.net.traplist">here</a> and <a href="https://www.bsdly.net/~peter/bsdly.net.traplist">here</a>) once per hour.</div>
<div>
<br /></div>
<div>
Maintaining the mail system and the blacklist also involves keeping an eye on mail-related activities, and invalid addresses in our domains that turn up in the greylist are usually added to the list of spamtrap addresses within not too many hours after they first appear. The process of actually adding spamtrap addresses is a manual one, but based on the output of pathetically simple shell scripts that run as <a href="https://man.openbsd.org/cron">cron</a> jobs.</div>
<div>
<br /></div>
<div>
The list of spamtraps has grown over the years to more than 38 000 entries. Most of the entries have local parts that are pure generated gibberish, some entries are probably degraded versions of earlier spamtrap addresses and some again seem to conform with specific patterns, including but not limited to SMTP or <a href="https://en.wikipedia.org/wiki/Network_News_Transfer_Protocol">NNTP</a> message IDs.</div>
</div>
<div>
<br /></div>
<div>
<div>
On August 19th and 20th 2017 I noticed a different, but yet familiar pattern in some of the new entries.</div>
<div>
<br /></div>
<div>
The entry that caught my eye had the <span style="font-family: "courier new" , "courier" , monospace;">MAIL FROM:</span> part as</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">mx42.antispamcloud.com-1503146097-testing@bsdly.com</span></div>
<div>
<br /></div>
<div>
The local part pattern was somewhat familiar, and breaks down to</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">$localhostname-$epochtime-testing</span></div>
<div>
<br /></div>
<div>
with <span style="font-family: "courier new" , "courier" , monospace;">@targetdomain.$tld</span> (in our case, <b>bsdly.com</b>) appended. I had at this point totally forgotten about SMTP callbacks, but I decided to check the logs for any traces of activity involving that host. The only trace I could find in the logs was at the spamd-serving firewall in front of the <b>bsdly.com</b> domain's secondary mail exchanger:</div>
</div>
<div>
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Aug 19 14:35:27 delilah spamd[26915]: 207.244.64.181: connected (25/24)
<br />Aug 19 14:35:38 delilah spamd[26915]: (GREY) 207.244.64.181: <> ->
<mx42.antispamcloud.com-1503146097-testing@bsdly.com>
<br />Aug 19 14:35:38 delilah spamd[15291]: new entry 207.244.64.181 from <>
to <mx42.antispamcloud.com-1503146097-testing@bsdly.com>, helo
mx18-12.smtp.antispamcloud.com
<br />Aug 19 14:35:38 delilah spamd[26915]: 207.244.64.181: disconnected after
11 seconds.
</span></div>
<div>
<br /></div>
<div>
Essentially a normal first contact: spamd at our end answers slowly, one byte per second, but the greylist entry is created in the expectation that any caller with a valid message to deliver would try again within a reasonable time. The spamd synchronization between the hosts in our group of greylisting hosts would see to that an entry matching this sequence would appear in the greylist on all participating hosts.<br />
<br />
But the retry never happened, and even if it had, that particular local-part would anyway have produced an "<span style="font-family: "courier new" , "courier" , monospace;">Unknown user</span>" bounce. But at that point I decided to do a bit of investigation and dug out what seemed to be a reasonable point of contact for the <b>antispamcloud.com</b> domain and sent an email with a question about the activity.<br />
<br />
That message bounced, with the following explanation in the bounce message body:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> DOMAINS@ANTISPAMCLOUD.COM</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> host filter10.antispamcloud.com [31.204.155.103]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> SMTP error from remote mail server after end of data:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> 550 The sending IP (213.187.179.198) is listed on https://spamrl.com as a source of dictionary attacks.</span></div>
<div>
<br /></div>
<div>
As you have probably guessed, <span style="font-family: "courier new" , "courier" , monospace;">213.187.179.198</span> is the IPv4 address of the primary mail exchanger for <b>bsdly.net</b>, <b>bsdly.com</b> and a few other domains under my care.</div>
<div>
<br /></div>
<div>
If you go to the URL quoted in the bounce, you will notice that the only point of contact is via an email adress in an unrelated domain.</div>
<div>
<br /></div>
<div>
I did fire off a message to that address from an alternate site, but before the answer to that one arrived, I had managed to contact another of their customers and got confirmation that they were indeed running with an <a href="https://en.wikipedia.org/wiki/Exim">exim</a> setup that used SMTP callbacks.</div>
<div>
<br /></div>
<div>
The s<a href="http://pamrl.com/">pamrl.com</a> web site states clearly that they will not supply any evidence in support of their decision to blacklist. Somebody claiming to represent <b>spamrl.com</b> did respond to my message, but as could be expected from their published policy was not willing to supply any evidence to support the claim stated in the bounce.</div>
<div>
<br /></div>
<div>
In my last message to <b>spamrl.com</b> before starting to write this piece, I advised</div>
</div>
<blockquote>
<br />
<span style="font-family: "courier new" , "courier" , monospace;">I remain unconvinced that the description of that problem is accurate, but investigation at this end can not proceed without at least some supporting evidence such as times of incidents, addresses or even networks affected.</span></blockquote>
<blockquote>
<span style="font-family: "courier new" , "courier" , monospace;">If there is a problem at this end, it will be fixed. But that will not happen as a result of handwaving and insults. Actual evidence to support further investigation is needed.</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">Until verifiable evidence of some sort materializes, I will assume that your end is misinterpreting normal greylisting behavior or acting on unfounded or low-quality reports from less than competent sources.</span></blockquote>
<br />
The domain <b>bsdly.com</b> was one I registered some years back mainly to fend off somebody who offered to help the owner of the <b>bsdly.net</b> domain acquire the very similar <b>bsdly.com</b> domain at the price of a mere few hundred dollars. <br />
<br />
My response was to spend something like ten dollars (or was it twenty?) to register the domain via my regular registrar. I may even have sent back a reply about trying to sell me what I already owned, but I have not bothered to dig that far back into my archives.<br />
<br />
The domain does receive mail, but is otherwise not actively used. However, as the list of <a href="https://www.bsdly.net/~peter/traplist.html">spamtraps</a> can attest (the full list does not display in a regular browser, since some of the traps are interpreted as html tags, if you want to see it all, fetch the <a href="https://www.bsdly.net/~peter/sortlist.txt">text file</a> instead), others have at times tried to pass off something or other with from addresses in that domain.<br />
<br />
But with the knowledge that this outfit's customers are believers in SMTP callbacks as a way to identify spam, here is my hypothesis on what actually happened:<br />
<br />
On August 19th 2017, my greylist scanner identified the following new entries referencing the <b>bsdly.com</b> domain:<br />
<div>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">anecowuutp@bsdly.com<br />pkgreewaa@bsdly.com<br />eemioiyv@bsdly.com<br />keerheior@bsdly.com<br />mx42.antispamcloud.com-1503146097-testing@bsdly.com<br />vbehmonmin@bsdly.com<br />euiosvob@bsdly.com<br />otjllo@bsdly.com<br />akuolsymwt@bsdly.com</span></blockquote>
<div>
<br /></div>
<div>
I'll go out on a limb and guess that <span style="font-family: "courier new" , "courier" , monospace;">mx42.antispamcloud.com</span> was contacted by any of the roughly 5000 hosts blacklisted at <b>bsdly.net</b> at the time, with an attempt to deliver a message with a <span style="font-family: "courier new" , "courier" , monospace;">MAIL FROM:</span> of either <b>anecowuutp@bsdly.com</b>, <b>pkgreewaa@bsdly.com</b>, <b>eemioiyv@bsdly.com</b> or perhaps most likely <b>keerheior@bsdly.com</b>, which appears as a bounce-to address in the same hourly greylist dump where <b>mx42.antispamcloud.com-1503146097-testing@bsdly.com</b> first appears as a <b>To:</b> address.</div>
<div>
<br /></div>
<div>
The first seen time in epoch notation for <b>keerheior@bsdly.com</b> is</div>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">1503143365</span></blockquote>
<div>
which translates via <a href="https://man.openbsd.org/date"><span style="font-family: "courier new" , "courier" , monospace;">date -r</span></a> to</div>
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">Sat Aug 19 13:49:25 CEST 2017</span></blockquote>
<div>
while <b>mx42.antispamcloud.com-1503146097-testing@bsdly.com</b> is first seen here at epoch <span style="font-family: "courier new" , "courier" , monospace;">1503146138</span>, which translates to <span style="font-family: "courier new" , "courier" , monospace;">Sat Aug 19 14:35:38 CEST 2017</span>.</div>
<div>
<br /></div>
<div>
The data indicate that this initial (and only) attempt to contact was aimed at the <b>bsdly.com</b> domain's secondary mail exchanger, and was intercepted by the greylisting <a href="https://man.openbsd.org/spamd">spamd</a> that sits in the incoming signal path to there. The other epoch-tagged callbacks follow the same pattern, as can be seen from the data preserved <a href="https://home.nuug.no/~peter/smtp-callback-logs.bsdly.com.txt">here</a>.</div>
<div>
<br /></div>
<div>
Whatever action or address triggered the callback, the callback appears to have followed the familiar script:</div>
</div>
<div>
<ol>
<li>register attempt to deliver mail</li>
<li>look up the domain stated in the <b>MAIL FROM:</b> or perhaps even the <b>HELO</b> or <b>EHLO</b></li>
<li>contact the domain's mail exchangers with the rump SMTP dialog quoted earlier</li>
<li>with no confirmation or otherwise of anything other than the fact that the domain's mail exchangers do listen on the expected port, proceed to whatever the next step is.</li>
</ol>
<div>
The known facts at this point are:</div>
</div>
<div>
<ol>
<li>a mail system that is set up for SMTP callbacks received a request to deliver mail from <b>keerheior@bsdly.com</b></li>
<li>the primary mail exchanger for <b>bsdly.com</b> has the IPv4 address <b>213.187.179.198</b></li>
</ol>
<div>
<div>
Both of these are likely true. The second we know for sure, and the first is quite likely. What is missing here is any consideration of where the request to deliver came from.</div>
<div>
<br /></div>
<div>
From the data we have here, we do not have any indication of what host contacted the system that initiated the callback. In a modern configuration, it is reasonable to expect that a receiving system checks for sender validity via any <a href="https://en.wikipedia.org/wiki/Sender_Policy_Framework">SPF</a>, <a href="https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail">DKIM</a> or <a href="https://en.wikipedia.org/wiki/DMARC">DMARC</a> records available, or for that matter, greylist and wait for the next attempt (in fact, greylisting before performing any other checks - as an <a href="https://www.openbsd.org/">OpenBSD</a> <a href="https://man.openbsd.org/spamd">spamd(8)</a> setup would do by default - is likely to be the least resource intensive approach). <br />
<br />
We have no indication that the system performing the SMTP callout used any such mechanism to find an indication as to whether the communication partner was in fact in any way connected to the domain it was trying to deliver mail for.</div>
<div>
<br /></div>
<div>
My hypothesis is that whatever code is running on the SMTP callback adherents' systems does not check the actual sending IP address, but assumes that any message claiming to be from a domain must in fact involve the primary mail exchanger of that domain and since the code likely predates the SPF, DKIM and DMARC specifications by at least a decade, it will not even try to check those types of information. Given the context it is a little odd but perhaps within reason that in all cases we see here, the callback is attempted not to the domain's primary mail exchanger, but the secondary. </div>
</div>
</div>
<div>
<br /></div>
<div>
With somebody or perhaps even several somebodies generating nonsense addresses in the <b>bsdly.com</b> domain at an appreciable rate (see the <a href="https://home.nuug.no/~peter/spamtraps-dateadded.txt">record of new spamtraps</a> starting May 20th, 2017, alternate location <a href="https://www.bsdly.net/~peter/spamtraps-dateadded.txt">here</a>) and trying to deliver using those fake <b>From:</b> addresses to somewhere doing SMTP callback, it's not much of a stretch to assume that the code was naive enough to conclude that the purported sender domain's primary mail exchanger was indeed performing a dictionary attack.<br />
<br />
<div>
The most useful lesson to take home from this sorry affair is likely to be that you need to kill SMTP callback setups in any system where you may find them. In today's environment, SMTP callbacks do not in fact provide useful information that is not available from other public sources, and naive use of results from those lookups is likely to harm unsuspecting third parties. <br />
<br />
So,</div>
</div>
<div>
<ul>
<li>If you are involved in selling or operating a system that behaves like the one described here and are in fact <i>generating blacklists</i> based on those very naive assumptions, you need to stop doing so right away. <br /><br />Your mistaken assumptions help produce bad data which could lead to hard to debug problems for innocent third parties. <br /><br />Or as we say in the trade, <i>you are part of the problem</i>.<br /><br /></li>
<li>If you are operating a system that does SMTP callbacks but doesn't do much else, you are part of a small problem and likely only inconveniencing yourself and your users. <br /><br />The fossil record (aka the accumulated collection of spamtrap addresses at <b>bsdly.net</b>) indicates that the callback variant that includes epoch times is rare enough (approximately 100 unique hosts registered over a decade) that callback activity in total volume probably does not rise above the level of random background noise. <br /><br />There may of course be callback variants that have other characteristics, and if you see a way to identify those from the data we have, I would very much like to hear from you.<br /><br /></li>
<li>If you are a customer of somebody selling antispam products, you have reason to demand an answer to just how, if at all, your antispam supplier utilizes SMTP callbacks. If they think it's a fine and current feature, you have probably been buying <a href="https://en.wikipedia.org/wiki/Snake_oil">snake oil</a> for years.<br /><br /></li>
<li>If you are the developer or maintainer of mail server code that contains the SMTP callbacks feature, please remove the code. Leaving it disabled by default is not sufficient. Removing the code is the only way to make sure the misfeature will never again be a source of confusing problems.</li>
</ul>
<div>
For some hints on what I consider a reasonable and necessary level of transparency in blacklist maintenance, please see my April 2013 piece <a href="https://bsdly.blogspot.no/2013/04/maintaining-publicly-available.html">Maintaining A Publicly Available Blacklist - Mechanisms And Principles</a>.<br />
<br />
The data this article is based on still exists and will be available for further study as long as the request comes wit a reasonable justification. I welcome comments in the comment field or via email (do factor in any possible greylist delay, though). <br />
<br />
Any corrections or updates that I find necessary based on your responses will be appended to the article.<br />
<br /></div>
</div>
<br />
<hr />
<b>Update 2017-09-05</b>: Since the article was originally published, we've seen a handful of further SMTP callback incidents. The last few we've handled by sending the following to the addresses that could be gleaned from <a href="https://man.openbsd.org/whois">whois</a> on the domain name and source IP address (with mx.nxdomain.nx and 192.0.2.74 inserted as placeholders here to protect the ignorant):<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<br />
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Hi,</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">I see from my greylist dumps that the host identifying as </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">mx.nxdomain.nx, IP address 192.0.2.74</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">is performing what looks like SMTP callbacks, with the (non-existent </span><span style="font-family: "courier new" , "courier" , monospace;">of course) address</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">mx.nxdomain.nx-1504629949-testing@bsdly.com</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">as the RCPT TO: address.</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">It is likely that this activity has been triggered by spam campaigns </span><span style="font-family: "courier new" , "courier" , monospace;">using made up addresses in one of our little-used domains as from: </span><span style="font-family: "courier new" , "courier" , monospace;">addresses.</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">A series of recent incidents here following the same pattern are </span><span style="font-family: "courier new" , "courier" , monospace;">summarized in the article</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">http://bsdly.blogspot.com/2017/08/twenty-plus-years-on-smtp-callbacks-are.html</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Briefly, the callbacks do not work as you expect. Please read the </span><span style="font-family: "courier new" , "courier" , monospace;">article and then disable that misfeature. Otherwise you will be </span><span style="font-family: "courier new" , "courier" , monospace;">complicit in generating false positives for your SMTP blacklist.</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">If you have any questions or concerns, please let me know.</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Yours sincerely,</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Peter N. M. Hansteen</span></div>
<div>
<br /></div>
If you've received a similarly-worded notice recently, you know why and may be closer to having a sanely run mail service.<br />
<br />
<br class="Apple-interchange-newline" />
<hr />
<b>Update 2017-11-02</b>: It looks like the <a href="http://spamrl.com/">spamrl.com</a> service still has customers that believe in the same claim made in the bounce messages quoted earlier in this article and perform SMTP callbacks exactly like they did when this article was first written. If you have any verifiable information on that outfit and their activities, I would very much like to hear from you.Peter N. M. Hansteenhttp://www.blogger.com/profile/12852746787621165833noreply@blogger.com9