Sunday, August 22, 2021

Recent and not so recent changes in OpenBSD that make life better (and may turn up elsewhere too)

Known to be "functional, free and secure by default", the OpenBSD operating system has played an important role in open source for more than a quarter century. It has also been fairly central to what I have done for the last two decades and some. What follows is my personal view of what life with OpenBSD has been like, with an emphasis on moments and developments that I feel made life, or at least my life, better.

I will assume that you know already that one of the signature features of the OpenBSD project is the continuous code audit and the sharp focus on secure and correct code. The audit by itself has produced a number of improvements, including a stream of bugfixes with bugs of a similar kind fixed in the whole tree and even the occasional subsystem rewrite. In addition, even for a free operating system project, life just happens. The world changes around us and drives the developers to take up fresh approaches to both new and well known problems and in the process develop code in ways that improves life for us all.

The Norwegian tech news site took this article in as a three part series, see parts 1, 2 and 3 if you want to read this in my native language.  

If you are not that familiar with OpenBSD the system or project, my "OpenBSD and you" talk, which I update occasionally, might be a not too bad place to start. But in this article I will focus on some specific moments when I felt that changes in OpenBSD made my life better. These are the things that made me start and go on advocating the system.

So who am I and what can I offer?

My name is Peter Hansteen. I have worked in information technology and information technology related things like documentation since the late 1980s. I am in the process of transitioning from a "Security Engineer" role into a "Cloud Expert" one, and across several other roles and titles I have always done a bit, or a lot of Unix system and network administration. At most times you will find me on The Other West Coast, specifically in Bergen, Norway.

Note: If you are more of a slides person than a fulltext person, you may be relieved to hear that you can find the slides for the talk this article is based on (and vice versa) here.

The installer was always good, got better

When I found OpenBSD more than twenty years ago, my main Unix exposure was from working with Linuxes and FreeBSD. What attracted me to OpenBSD and finally had me buy an OpenBSD 2.5 CD set was the strong focus on security and code correctness. When the CD set and the classic wireframe daemon T-shirt finally arrived in the mail, I set about at first to install it on whatever spare hardware I had lying around.

OpenBSD wireframe daemon head

If I remember correctly, the first machine I tried installing OpenBSD on was an 80386/33MHz with 8MB RAM and I think a 100MB IDE hard disk. Which I can report sounded pretty crappy even then, but the thing did work.

The initial install was fairly straightforward, and when I started poking around I found two things about myself and the new system: Everyting made sense, and everything I could think of had a readable man page. So the first change I am aware of that made the world better with OpenBSD was the decision to enforce the "No commit without documentation" rule, which came into being early in the project's life, probably roughly at the same time the OpenBSD developers gave us a real-time view of development via anonymous CVS. You can see things happening in almost real time.

It is worth mentioning that the installer has remained famously non-graphical, text only. The reason the installer remains text-only is that this is a major advantage that enables the developers and the users to handle the fairly diverse collection of hardware platforms that OpenBSD runs on with the same portable, familiar and compact code everywhere.

The installer was always scriptable and extensible, and over the years the installer has added automatic, repeatable and scriptable installs (dubbed autoinstall(8) which appeared in OpenBSD 5.5 in 2014) and the sysupgrade(8) extension (first found in OpenBSD 6.6 in 2019) that automates snapshot to snapshot or release to subsequent release upgrades for all not too hacked-up configurations. Each of these moments, or more specifically when the new code started appearing in snapshots, had me appreciate the OpenBSD system a bit more, and made me feel quality of life had improved.

Now something for your laptop - hardware support

Fast forward some twenty-plus years and the last article I published, and even got into Norwegian mainstream IT news site, centers on a few moments involving new OpenBSD developments. It took some interaction with OpenBSD developers, but those interactions lead to my new laptop with an 11th generation Intel Core chipset working even better with OpenBSD. Yes, OpenBSD developers and a significant subset of their user base actually run OpenBSD on their laptops. I do use a Mac and a work-issued Thinkpad with Ubuntu Linux too, but life is not complete without an OpenBSD laptop.

Now to be honest, what I saw within the space of a few days was development that had me going from "Oh, sh*t, the SSD isn't recognized" -- the controller was set to a RAID-ish mode by default -- through this kernel panic:

OpenBSD 6.9-current panic message

-- to seeing it all fully supported.

The SSD problem turned out to be simple to fix: Simply find the "Advanced" BIOS option that turned the pseudo-raid feature off and let the operating system speak directly to the storage device.

For the rest there was a period of a couple of weeks I had to run with not yet commited patches in a home baked kernel built from checkouts from Jonathan Grey's git repo. When the code was committed to -current, I could resume my normal sysupgrade(8) routine, going from one development snapshot to the next.

The process, even with the need to build custom kernels for a while, was actually quite pleasant, and when the support code went into the main development branch, that too was a a moment when I felt my life had been improved by changes in OpenBSD. The hardware support is now in snapshots and will be in OpenBSD 7.0 which is set to be released approximately early November 2021.

Living the life dynamic

Now that we're talking about laptops, there is another recent development that makes your OpenBSD on laptop experience even better. Laptops and other equipment that uses dynamic network configuration became easier to operate with dhcpleased(8) now enabled by default in OpenBSD 6.9-current after it was first introduced in OpenBSD 6.9. That change marked the completion of a five year cycle of incremental development which involved writing several new daemons. Each of those programs was designed with the Unix philosophy that a program should do one thing and do it well.

The first piece of the puzzle was slaacd(8) - the stateless IPv6 address autoconfiguration daemon - which appeared in OpenBSD 6.2 to handle IPv6 automatic configuration, listening for route advertisements.

The corresponding router advertisement daemon rad(8) arrived in OpenBSD 6.4. That got most of the things involved in IPv6 autoconfiguration in order.

Next up was the arrival in OpenBSD 6.5 of unwind(8), a validating DNS resolver which learns which resolvers to query from DHCP and other sources.

To complete the set, OpenBSD 6.9 brought us resolvd(8) to manage and edit /etc/resolv.conf, updating the file with information learned from other sources, and dhcpleased(8) now serves as the client for IPv4 DHCP client information which is then fed into the configuration.

Combined with setting your laptop to join networks as they become available, moving between networks can now be an non-disruptive, even unremarkable event.

This all comes into place if you edit your /etc/hostname.$if for (for example hostname.iwx0) to something like

join adipose wpakey thedoctorknows
join tardis  wpakey biggerontheinside
join cybermen wpakey nowedont
inet autoconf
inet6 autoconf

you should expect minimal efforts needed when moving between those networks. As usual, as soon as a new feature is trusted, it is on by default in OpenBSD-current, and OpenBSD 7.0 will ship with this behavior enabled by default for interfaces set to autoconf for either inet or inet6.

But that is the modern day and for some in the future. OpenBSD on a laptop is a good experience. On the other hand, most of the world sees the BSDs and OpenBSD in particular as mainly server or even network device operating systems, despite the fairly high BSD code content in such things as Apple systems.

The thing that lured me in

But I hear you ask, what made me turn into an almost all-in OpenBSD user?

Back in 2001 I was still only experimenting with OpenBSD, but my experience with Linux and iptables had made me long for a switch to a saner firewall. I had done some small experiments with the IPF firewall that was in OpenBSD until the 2.9 release. Then, as some of us will remember, the it was discovered that IPF's license was in fact not free, so it needed to be replaced.

There was a distinct rush, not quite a stampede, to replace IPF over the months that followed. Fortunately, the new code that replaced the previous packet filter proved to perform better. The OpenBSD Packet filter, dubbed PF for short, had been born and made its debut in OpenBSD 3.0 in December 2001. The release had originally been planned for November, but was pushed out a month to hack the "working prototype" packet filter into something usable.

Almost needless to say, this turn of events finally pushed me to take the final steps to replace the Linux gateways I had in place with OpenBSD ones. I was pleasantly surprised to find that not only did they perform well, but they also came with complete and reasonably well documented tools so I could understand what was going on. That's how I got started on the process that lead to among other things writing The Book of PF and taking that text through three editions so far. But more about that later.

It is worth noting that the IPFilter copyright episode spurred the OpenBSD developers to perform a license audit of the entire source tree and ports in order to avoid similar situations in the future. This activity ran for some months and uncovered a number of potential problems. Theo de Raadt summed up the effort in a message to the openbsd-misc mailing list on February 20th, 2003.

What they found when they started looking was that there was a significant number of files that were in fact not under a free license, much like the entire IPF subsystem had been. Those needed to be replaced. Other parts had either no license or no copyright stated. In some cases the developers gave explicit permission to continuing use, but quite a few things needed to be rewritten with a free license so OpenBSD and other free software would be able to move forward without copyright problems.

I later heard in a rather informal setting that among the no copyright and/or no license cases, it was usually possible to track down the developers via version control system logs or mailing list archives. In a large number of those cases, the initial reaction was along the lines "Say what? Are people still using that?".

SSH, open and better

PF was written from scratch to replace a subsystem that it turned out was illegal to use in an open source context. But it was not the first time the OpenBSD project had performed a nonlibreectomy, that is, taken on the task of replacing code for license reasons.

A few years earlier it had become clear that the original developer of the secure shell system ssh had commercial ambitions and the license for the software had changed in a proprietary direction. After a bit of deliberation on how to resolve the situation, the OpenBSD developers started digging around for earlier versions of the code that had been published with an acceptable license. Then they forked their version from the last version they found that still had free license. Next came an intensive period of re-introducing the features that were missing in the old code.

The result was introduced as OpenSSH in OpenBSD 2.6 in 1999. Over the next few years OpenSSH grew a portable version that started grabbing market share rapidly. The last I heard OpenSSH's market share is somewhere in the high nineties percent.

With a state of the art secure shell subsystem in place and growing all sorts of useful features, the time finally came to end unencrypted shell login sessions on OpenBSD. OpenBSD's telnetd was moved to the CVS attic in time for OpenBSD 3.8, which was released November 2005.

One other notable thing about OpenSSH is that it was the first daemon to be properly privilege separated, a model practice that debuted with the overhauled OpenSSH in OpenBSD 3.2 in March 2002. Since then privilege separation has been put in place in all daemons where it made sense to do so, and it is now a signature part of the secure by default stance of all newer OpenBSD daemons.

And yes, that packet filter

I mentioned PF, the OpenBSD packet filter, earlier. I must confess that PF has been an important part of my life in various context since the early noughties. Over the years, things I have written have contributed to creating the popular but actually wrong perception that OpenBSD was primarily a firewall operating system. There are a lot of useful and fun features that turned up in or in connection with PF over the years and were pioneered by OpenBSD. Some features were ported to or imitated in other systems, while others remain stubbornly OpenBSD only.

So I will touch on some of my favorite PF and PF-attached features, in quasi-random but almost chronological order.

Beating up spammers with OpenBSD spamd(8) since OpenBSD 3.3

When I started playing with OpenBSD in general and PF in particular way back when, I was already responsible for the SMTP mail service for my colleagues. My gateways by then ran OpenBSD, while the mail server rosalita, named after a Springsteen song, was not too badly specced server running FreeBSD with exim as the mail transfer agent that fed the incoming messages to spamassassin and clamav for content filtering before handing off to user mailboxes.

So when it dawned on me that I could set up spamd(8) the spam deferral daemon on the internet-facing gateway and save load on the poor suffering rosalita that was running hot with content filtering, I was quick to implement a setup that sucked in well known block lists.

Going grey, then trapping

The effect was obvious and immediate, the mail server's fans grew noticeably quieter. When greylisting was introduced in spamd soon after, I implemented that too, and witnessed yet another drop in pitch and intensity of the sound from rosalita's fans. Then a couple of releases later greytrapping -- the practice of adding IP addresses of incoming SMTP connections to blocklists if the attempted delivery is aimed at a known-bad address in the target domain -- was introduced, and that sounded like enough fun that I just went ahead and did it.

The idea of detecting spam senders by the bogus addresses they were already trying to deliver to just sounded too good to not try. And we knew that getting started would be pretty easy too. We had seen rejects for addresses that had never existed in our domains in our mail server logs for quite a while, so it was simply a matter of harvesting from a fairly bountiful source and adding stuff that we were sure would never ever be actually deliverable here to the spamtrap list. I think the first setup had only a couple of hundred entries in it, but I did not note the exact number at the time.

By July 2007 I had decided to publish both the list of spamtrap addresses and an hourly dump of the greytrapped addresses. Both remain free to download. The list of spamtraps, harvested from various log sources, by now numbers just over 270,000 imaginary friends, while the number of trapped hosts is typically in the 3000 to 5000 range. We occasionally see the list swell to 20,000 or more when high volume campaigns run with bad address lists fed to them. I am pretty sure it went over 100,000 at one point.

It's fun to watch, and it looks like a significant subset of the spamtraps have made it into the address lists of active spam operations. I frankly never thought I would still be collecting spam traps from logs all these years later. Yes, it all sounds a bit absurd, but it is effective for keeping our mailboxes largely spam free, even though it feels at times like running a weird found object-ish art project. Anyway, a summary of the lists we publish can be found in this article.

The brutes, the password gropers and the state tracking options

If you run an SSH service or really any kind of listening service with the option to log in, you will see some number of failed authentication attempts that generate noise in the logs. The password guessing, or as some of us say, password groping, turned out to be annoying enough that OpenBSD 3.6-current and later OpenBSD 3.7 introduced a set of features to use data that would anyway be available in the state table, to track the state of active connections, and to act on limits you define such as number of connections from a single host over a set number of seconds.

The action could be to add the source IP that tripped the limit to a table. Additional rules could then subject the members of that table to special treatment. Since that time, my internet-facing rule sets have tended to include variations on

table <bruteforce> persist
block quick from <bruteforce>
pass inet proto tcp from any to $localnet port $tcp_services \
        flags S/SA keep state \
	(max-src-conn 100, max-src-conn-rate 15/5, \
         overload <bruteforce> flush global)

which means that any host that tries more than 100 simultaneous connections or more than 15 new connections over 5 seconds are added to the table and blocked, with any existing connections terminated.

It is a good practice to let table entries in such setups expire eventually. At first I followed the spamd(8) defaults' example and set expiry at 24 hours, but with password gropers like those caught by this rule being what they are, I switched a few years ago to at four weeks at first, then upped again a few months later to six weeks. Groperbots tend to stay broken for that long. And since they target any service you may be running, state tracking options with overload tables can be useful in a lot of non-SSH contexts as well.

It is also worth noting that state tracking actions are useful for essentially all services. The article Forcing the password gropers through a smaller hole with OpenBSD's PF queues has a few suggestions on how to handle noise sources with various other services.

One final point I would like to make about the state tracking and actions is that much like the greytrapping feature of spamd, this feature gives you the tools to build a configuration that adapts to network conditions and learns from the traffic it sees. 

While this does not rise to the level of being an actual Artificial Intelligence or AI, this has enough buzzwordability potential that I remain to this day extremely puzzled that none of the other big names at least imitated those features in their own products and marketed for all it would be worth. 

I certainly know what I would have done in their position. But then I am more engineer than marketer and in the contexts where I call the shots, the best option is just to keep running OpenBSD.

NAT's guts ripped out

When the OpenBSD 4.7 release cycle came around, Henning Brauer had been hard at work for a while maintaining a diff of several thousand lines -- which when applied actually shrunk the code -- that contained a total rewrite of the IPv4 network address translation code.

Previous PF versions had 'nat on interface' and 'rdr on interface' rules, while the new code introduced nat-to and rdr-to as options on pass or match rules.

The match keyword had been introduced in the previous release to act on packets and connections without affecting pass or block state, such as applying specific options or adding tags for processing later in the rule set. Now with the major NAT rewrite in place, it became even clearer why match was in fact a useful keyword and feature.

The NAT rewrite added a lot of flexibility to how you can write PF rule sets, and of course for my own part that rewrite made it necessary to write the second edition of The Book of PF, timed to hit bookshelves as close as possible to the OpenBSD 4.7 release. And yes, the rewrite improved the performance too.

We went to modern queueing

OpenBSD has had traffic shaping available in the ALTQ subsystem since the very early days. ALTQ was rolled into PF at some point, but the code was still marked experimental 15 years after it was written, and most people who tried to use it in anger at the time found the syntax inelegant at best, infuriating or worse at most times.

So Henning Brauer took a keen interest in the problem, and reached the conclusion that all the various traffic shaping algorithms were not in fact needed. They could all except one be reduced to mere configuration options, either as setting priorities on pass or match rules or as variations of the theme of the mother algorithm Hierarchical Fair Service Curve (HFSC for short).

Soon after, another not-small diff was making the rounds. The patch was applied early in the OpenBSD 5.5 cycle, and for the lifetime of that release older ALTQ setups were possible side by side with the new queueing system.

The feedback I get is that the saner syntax in the new queueing system lead to more users taking up traffic shaping. Here is the queue setup that I came up with for one of my sites:

queue rootq on $ext_if bandwidth 20M
        queue main parent rootq bandwidth 20479K min 1M \
                                    max 20479K qlimit 100
             queue qdef parent main bandwidth 9600K min 6000K \ 
                                    max 18M default
             queue qweb parent main bandwidth 9600K min 6000K \
                                    max 18M
             queue qpri parent main bandwidth 700K min 100K \
                                    max 1200K
             queue qdns parent main bandwidth 200K min 12K \
                                    burst 600K for 3000ms
        queue spamd parent rootq bandwidth 1K min 0K max 1K \
                                    qlimit 300

while tying the queues into the subsequent rules with a set of match rules just following that block.

This is what triggered the need to write the third edition of The Book of PF. The book includes descriptions of both the new and the old system as well as tips on how to make a smooth transition. The ALTQ code was removed from OpenBSD during the OpenBSD 5.6 cycle, but continues to live on in some form in FreeBSD and NetBSD.

And yes, if you think my queues setup punishes spammers a bit more in addtion to being subjected to spamd(8), you're right.

pflow(4) offers network insights lite

Everybody who has been tasked with looking after a network has at some point been at least a little curious about what actually moves around there. At times we will see situations where it is essential for troubleshooting purposes to see the traffic flows with data about endpoints, packets and bytes transferred, protocol and so forth.

If you do not need to see the data itself, but rather the metadata, the NetFlow standard and its close cousin IPFIX offers just that. Netflow tools existed as packages on OpenBSD already, but from OpenBSD 4.5 PF has the pflow state tracking option, paired with the pflow(4) virtual network interface which together offer a full netflow sensor package.

Set up one or more pflow interfaces to send data to one or more collectors, and add the pflow option to specific rules or as a state default and you have started your collecting. You can even have metadata for traffic matching specific rules going to separate pflow devices and collectors.

My field notes in Yes, You Too Can Be An Evil Network Overlord - On The Cheap With OpenBSD, pflow And nfsen offers some practical examples and insights, including how we used a pflow setup to track down a noisy machine on a somewhat critical network as well as some pointers to valueable further reading.

LibreSSL, the great deobfuscation

People tell me they think that the reason LibreSSL was created was the Heartbleed bug, but no, actually not, just damn close.

The LibreSSL project was in fact started a few weeks before heartbleed became common knowledge. LibreSSL is the result of a group of OpenBSD developers taking the existing OpenSSL code and starting to fix it.

This time it was not a matter of a bad license. No, this was the result of the number of OpenBSD developers who took a look at the OpenSSL code that had been part of the OpenBSD base system since quite early on, and turned away in disgust and with symptoms of physical pain, reached a critical mass of sorts. I had heard OpenBSD developers complain about the absolute horror of the OpenSSL code for at least ten years. The code quality was just that bad.

What happened next was that a group of hardened OpenBSD developers grabbed the OpenSSL code and started two activities in parallel. One was looking in the OpenSSL request tracker for bugs that had not been addressed. The other was reformatting the OpenSSL code into something resembling the OpenBSD style of readable and maintainable C.

With the code in more readable form, discovering what it did became easier. In addition to a few obvious eye-stinging bugs the LibreSSL developers found a number of oddities, including, but not limited to

  • Code was never deleted even when it became irrelevant or obsolete

  • OpenSSL did not use the system memory allocation system, but rather opted for their own which never actually deallocated memory, but rather used LIFO recycling, and could easily be made to put private info into logs

  • all written in "OpenSSL C", which according to beck@ is a dialect of the "worst common denominator"

It is worth digging out the various articles and presentations made by LibreSSL developers over the years, with specific emphasis on Bob Beck's BSDCan talk on the first 30 days of LibreSSL (available on youtube), which is the original source of the term code flensing.

Since the OpenBSD 5.6 release in 2014, LibreSSL has been the default TLS library in OpenBSD. LibreSSL has been ported elsewhere based on the -portable variant.

For my own part I can only attest to not ever running into a TLS problem that was LibreSSL's fault. It probably still has bugs, but it is a lot more of a healthy choice than its predecessor.

This was my list of life improving OpenBSD events - I'd love to hear yours

As I warned earlier, this has been about my personal list of OpenBSD events that I remember fondly.

I am sure your list is at least a little different. I am sure there are things from the innovations page that I have simply forgotten about.

Each release comes with a detailed list of changes, such as this one for OpenBSD 6.9, and the page has pointers back to the equivalent pages for previous releases.

I would love to hear about your favorite OpenBSD moments.

More items for your OpenBSD reading is the official OpenBSD web site. If you want to donate, go to the donations page and find the most appropriate option. Corporate entities may prefer to donate via The OpenBSD Foundation, which is a Canadian non-profit corporation. is the OpenBSD Journal news site. My rant^H^H^H^Hblog posts Ted Unangst (tedu@) on developments

Michael W Lucas: Absolute OpenBSD, 2nd edition

Peter N. M. Hansteen: The Book of PF, 3rd edition

Henning Brauer: OpenBSD sucks (… least)

Thursday, July 1, 2021

The Impending Doom of Your Operating System Going to or Past 11, Versus the Lush Oasis of Open Source Systems

Will the uncertainty over forced obsolescence of fairly recent hardware force Microsoft and Apple users to switch to open source alternatives?

During the last few weeks several items of computing hardware in our household had reached the point in their lifetime when it made sense to trade in for upgrades. published a Norwegian version of this articleEn skummel fremtid med operativ­system som går til 11 eller forbi, eller en rolig oase med fri programvare?

I've written articles about my last two major laptop upgrades and each time detailed (in 2010 and 2017, respectively) how to deal with hardware that was new enough that I had no way to be certain it would work optimally with my chosen operating system, OpenBSD

I have tended to jump from snapshot to snapshot, generally following whatever was -current on OpenBSD/amd64. There were other upgrades during that time, but those were straightforward enough that I did not see a need to write about them.

This time around, even though the process involved interactions with OpenBSD developers via the bugs@ mailing list and even trying two separate models from the same manufacturer before settling on what I wanted, I considered just letting this upgrade round just pass relatively undocumented. There was simply not enough drama involved in the process to make for interesting reading or an inspired writing process. 

But then came the announcements from Apple and Microsoft of their operating systems going past 11 or to 11 respectively, spaced not too many weeks apart. In both cases, the announcements indicated that the new operating system versions would not work with older hardware.

At their WWDC event in early June 2021, Apple announced new versions of their system with somewhat vague but only thinly veiled formulations that specific new features of the upcoming system would only be available on the newer ARM architecture "Apple silicon" hardware.

Then a few weeks later into June 2021, Microsoft announced their Windows 11, and the announcement included some fairly confusing statements that seemed to indicate at first that Windows 11 would only work well or at all on hardware based on Intel's 8th generation Core processors or equivalent.

Apple is almost a year into their announced two year transition from Intel-supplied processors, with a base architecture generally known as AMD64, to their own Apple-designed ARM64-based system on a chip cores. Apple has generally kept some level of support for Macs for seven years after release, and with a transition to a new architecture underway, it becomes even less surprising that support for older devices will gradually erode and that some new system features will only be available on newer model hardware.

This contrasts sharply with Microsoft's situation, with the company not really dependent on hardware sales and not with any announced or unannounced but apparent move to a different architecture. Whatever the reason for the cutback in support, the initial response from the public seemed to indicate that there now was a real fear that on installing the new software, upgrading Windows users would be faced with something like

(which is in fact an OpenBSD panic) unless they upgrade to newer hardware before trying the new software release.

The fear of abandonment seemed real and echoed the feelings I have had myself over the years when getting new hardware to run a free operating system on.

The previous articles chronicle some of the experimenting that was needed in the past to make OpenBSD work when the hardware was newer than what yet had time to reach the developers. But in the end we could always be quite certain that we could make what we were interested in work, given time and perhaps some interaction with developers, or if you were up to it, becoming a developer yourself.

Anyway, over time the chance that things would just work increased, and your sweet spot for some time was buying hardware that was released within the last couple of years before the operating system release you were installing.

Hardware drivers would generally be kept in and maintained as long as they appeared to be useful. In general a driver would only be retired from the tree if it was useful only to an architecture that was going out of support such as OpenBSD/vax which went to the attic after the OpenBSD 5.9 release in 2016.

The major lesson here is that the free systems like OpenBSD, Linux or others would keep hardware support around as long as it appears to be useful to somebody, somewhere. 

If major players like Microsoft choose to simply abandon users who do not have the latest hardware to stagnation plus only security updates, moving to a free software alternative may very well be a viable option for users who are not willing to abandon not very outdated hardware as long as their typical use case allows.

In my own experience, with hardware that has been on the market for about a year or possibly more you will encounter few to no problems making things work. My most recent Linux experience on laptops is with 9th and 11th generation Intel Core hardware, both of which will serve you well, including multimedia setups, excluding only those that explicitly tell you that you are on your own (Netflix being a case in point).

Now for an incrementally geekier part. If you are not that interested in OpenBSD, please feel free to skip.

But if you were waiting for the promised OpenBSD on newer hardware runthrough, you will get the fuller picture by reading the following and by looking up the details in the mailing list archives via the links and links in those messages.

The thread AMD Ryzen based Asus ZENBOOK 14 UM433DA-PURE4 14" panic at first boot post install - how to debug chronicles the interactions from "machine installs but does not survive first boot" through finding that the machine's BIOS announced but did not actually implement some features, and the subsequent changes that went in to the mainstream OpenBSD kernel, if I remember correctly just in time to be included in OpenBSD 6.9.

However, as can be seen in ASUS ZenBook X freezes, there were problems in the DRM/xorg area that would prove too hard to debug. Do read the whole thread, it contains useful debug info for when you get into a similar situation yourself.

Returning that system to the shop for a refund while I was still fiddling with the finer points of the next system was an interesting experience in itself.

I tried to restore the system to its pre-OpenBSD state before returning it, but as it turns out the Windows 10 install image Microsoft supplies will not be able to complete an install by itself.

Rather, it will prompt you for hardware driver you are supposed to have to hand for this system.

As a result of this, the machine still had OpenBSD installed -- with my user and home directory removed and only root as an active user -- when I handed the machine in for the refund, and it was immediately clear that the support techs had never seen anything more Unixy than macOS before. Fortunately this only lead to a short delay in the issuing the refund (but I now have a 1 year PC and Mac Support contract which I do not know that I actually need).

Anyway, I had already discovered an offer for a slightly more expensive model with better features, so ordered and took delivery of the machine described in ASUS ZenBook S: SSD unrecognized, possible new iwx variant, which chronicles the relatively light debugging needed to get the system in shape.

In short, after receiving the package with the new machine late in the afternoon, I spent a few hours trying to work around a few items that lead to rather puzzling failures at first, but fortunately they were all relatively easy to fix with a little help from OpenBSD developers who read the bugs@ list.

The first hurdle was that the system apparently did not recognize the built in SSD. This turned out to a matter of finding the BIOS option for turning off RAID controller functionality, which anyway does not make a whole lot of sense in a system where it is physically impossible to fit more than one storage device on a permanent basis.

The option turned out to live in the BIOS' Advanced menu, labeled VMD setup menu, where you set the Enable VMD controller option to Disabled. Once that is done, the SSD shows up as a regular NVMe device:

nvme0 at pci3 dev 0 function 0 "Intel NVMe" rev 0x03: msix, NVMe 1.3
nvme0: INTEL SSDPEKNW010T8, firmware 004C, serial BTNH03460GYE1P0B
scsibus1 at nvme0: 2 targets, initiator 0
sd0 at scsibus1 targ 1 lun 0: <NVMe, INTEL SSDPEKNW01, 004C>
sd0: 976762MB, 512 bytes/sector, 2000409264 sectors

This made it possible to install on the internal SSD proper, and the next issue was that this 11th generation Intel Core system needed a newer revision (version 5.10) of the Linux-derived DRM code. At the time (and still at the time of writing) Jonathan Gray maintained an as-not-yet-committed branch of the OpenBSD kernel with the code I needed in. The reason this DRM code version was not committed to the main tree was that the newer code caused some regressions on older hardware.

On my system, it looked like the stock kernel would panic when loading the iwx(4) driver, but booting the test kernel Jonathan supplied cured that problem, and I have been running once a week checkouts of the drm510 kernel on top of sysupgraded snapshots since.

However, even with the iwx(4) driver now loading, the wireless network device did not work. 

Running doas fw_update -v revealed that several of the relevant firmware files had been corrupted, and after doas fw_update -d iwx and re-installing (doas fw_update iwx), doas /etc/netstart iwx0 worked as expected and with excellent performance.

In the meantime it had turned out that not only was the audio parts of the system in fact supported (it only needed a one line patch to enable it), only minor manipulation to configuration files would make the audio output signal switch correctly between the internal speaker and my headphones, and that for video conferencing a low cost full duplex USB headset was the better choice.

So now I have a gorgeous, lightweight 13.9 inch laptop running OpenBSD with Xorg running with a 3300x2200 pixel resolution and everything I care about working. With a little attention to proper testing, we have reason to believe that all of this will be properly supported without regression for older hardware versions in the upcoming OpenBSD 7.0 release.

As I had hinted earlier, you may very well find yourself better served and supported by the open source operating system of your choice and its developers and users than you can reasonably expect from the commercial, proprietary options.

If you have questions about anything in this article, OpenBSD or other free systems, please let me know in comments here, seek out a local-to-you user group (the ones I am most involved in are NUUG, the national Norwegian Unix User Group, and BLUG, the Bergen (BSD and) Linux User Group), or drop me an email. If you choose the last option, please read my read me first document before sending a second message.

Update 2021-07-07: As reported in the following tweet, the DRM 5.10 update is now in, and I can go back to quiet sysupgrade(8) from snapshot to snapshot:

Which also means OpenBSD 7.0 will seriously rock on this and similar machines.

Saturday, May 15, 2021

Are you aware what you lose by just clicking OK to get started using something?

The right to privacy, the right to repair and the right to choose your tools for tasks at hand are aspects of the same. A new court ruling in Italy could help us regain righs that we were manipulated into giving up.

It's likely you do not spend much time thinking about the fact that if you are an ordinary IT user in an industrialized country, you have most likely have been tricked into giving up rights. This happens on a scale that should be worrying to anyone concerned about human rights in general.

Consider the situation when you want to start using something you are interested in, either a computer of some sort such as a PC, tablet or phone, or a network based service.

First, look at what happens when you get get your new computer, tablet or telephone and start unboxing. One of the very first things after you have powered the device on, and certainly before you get any opportunity to use the thing for whatever you want to do, is that you are required to accept a legally binding agreement that has been designed by and for those who manufactured the equipment. In order to be able to use the thing you bought, you are required to accept an agreement that governs what you are able to do with the device.

With some devices you will be presented with several separate agreements, each with its own registration of whether you accept the terms or not.

Some of the agreements set limits on what you can use the device for, while others grant the supplier or cooperating parties permission to collect information about you and what you do with the device.

Most of those yes/no style questions will give the impression that you have a real choice to not agree to the terms, but you will find that you probably will not go on to getting a device that is in fact usable for the intended use until you have agreed to the terms of all of the agreements.

One of the more visible consequences of the COVID 19 crisis is that a larger subset of us were forced into an almost totally digital existence, where communication for work and school happens via digital devices and via services that are provided according to terms dictated by the service providers. Some of us have led a mostly digital existence for years already, but for a large chunk of the population this is a totally new set of circumstances and it is slowly dawning on an increasing number of people that important freedoms and rights may be on a path towards extinction.

This is not a new set of problems. Among IT professionals, many of us have for years been warning that crucial human rights or civil rights are being slowly worn away, largely to the benefit of a few corporations and their owners.

When you turn on a new computer or phone for the first time, most likely you will be asked right away to accept an "end user license" for the operating system, that is, the software that controls the device. In its simplest form, a license is a document that specifies the terms that govern granting other someone other than a work's author (here the software developers) permission to produce copies of the work. 

However, in many cases the license document contains far more wide reaching terms and permissions. We often see that the license agreement grants you a right to not accept the terms for using the operating system and delete or return any copies delivered with the device and get a refund, but you retain the right to use the physical device. 

Some of us have bought PCs or other devices and managed to install an operating system that was not supplied with the device, choosing to live our digital lives using free alternatives such as Linux or OpenBSD. Some of us do this in order to gain more direct control of the tools we use.

If we have tried to get a refund for an unused operating system license, most of us have not been sucessful. But we will return to that matter shortly.

If you have successfully installed a free alternative to the operating system that came with your device, you have contributed to strengthening the right to choose your tools, the right to repair and to make your own decisions about your possessions. But unfortunately this is not the only part of your digital life where your rights are in grave danger.

Regardless of whether you accepted the end user license earlier or not, you will soon encounter software or network based services that present end user agreements of their own. There is a considerable chance that you will just click OK without reading the conditions of that agreement.

Please take a break from reading this and go check what conditions you have actually agreed to. More likely than not, you will find that both operating system suppliers and social media services have had you give them permission to record what you do when you use the system or the service. For good measure, please take the time to check the conditions for all products and services you have registered for. Most likely not just one, but a large majority of the services and products you use on a network connected device have granted themselves the right to record and store data on your behavior. If you use the device to anything privacy relevant or involving sensitive information it is well worth checking how consequences those agreements bear ouut for your right to privacy.

On paper (yes, I'm sounding old fashioned on purpose) residents of the EU and EEA attached countries have a right to get a copy of data stored about us and if needed get any errors corrected or even have data deleted accordign to the EU General Data Protection Regulation, known as the GDPR. I

f you found something while checking the agreements on your break from reading this feels concerning or makes you unsure, you would to well to exercise your right to viewing, copying, correction or deletion. If you do not get any meaningful response, your best path of action is to contact the local-to-you Data Protection Authority (in Norway, that is Datatilsynet) or the local-to-you Consumer Protection Agency (again in Norway we have Forbrukertilsynet), both should be able and willing to offer assistance.

But what then, of the right to repair or the right to choose one's own tools? The good news is that there is reason to hope. After a complex and long winded process an Italian court recently decided not only did a Linux enthusiast have the right to install Linux on a new Lenovo computer, the customer also had the right to get the price of the unused operating system refunded. Unfortunately Lenovo had attempted to not live up to their obligations as specified by the end user license presented to the customer, and they were fined the amount of 20,000 Euros.

A decision of this category is apparently not automatically a binding legal precedent in other European countries, and we are aware of decisions in other countries that did not grant the customer the right to treat a computer and its operating system as separate items. As the Norwegian association of Unix and free software users (Norwegian Unix User Group - NUUG) we are now entering in a cooperative effort coordinated by the Free Software Foundation Europe (FSFE) to protect and defend your right and mine to privacy, the right to repair and the right to choose the tools we use to manage our digital existence.

If any of the things you just read makes you concerned, confused, angry or just eager to help strengthen our civil rights and human rights in the digital domain, we will be very happy to hear from you.

Peter N. M. Hansteen
Board Chair of the Norwegian Unix User Group (NUUG)

The Italian court decision that offers some hope is described in some detail on the FSFEs web: Refund of pre-installed Windows: Lenovo must pay 20,000 euros in damages

This article originally appeared 2021-05-15 in Norwegian on NUUG's news web "2021-05-15 - Vet du hva du mister når du bare klikker OK for å komme i gang med å bruke noe?"

Monday, February 22, 2021

RFC7505 Means Yes, Your Domain Can Refuse to Handle Mail. Please Leave Us a TXT If You Do.

If you do not want a domain to receive any mail, there is a way to be at last somewhat civil about it. There's a different DNS trick for that.

It used to be that if you went to the trouble of registering a domain, one of the duties that came with it was set up somewhere to receive mail.

A number of networking professionals, myself included, have been know to insist that not only should a valid domain receive mail, at least a significant subset of the identities listed in RFC2142 (dated May 1997) should exist and mail sent there should be read at some reasonable interval.

Then of course we all know that a number of things happened in networking in the years between 1997 and today.

As regular or returning readers of this column will be aware, one of the phenomena that rose to become a prominent irritation and possible risk factor was spam, otherwise known as unsolicited commercial email, and of course some of the unsolicited traffic carried payloads that were part of various kinds of criminal activity.

I have written fairly extensively on how to suppress spam and other malicious traffic and have fun doing so, all the while assuming that if you run a domain you will want at least some mail to have a chance of making it to an inbox that is actually read by a person or perhaps processed by your robotic underlings.

Then there is that other consideration that with the proliferation of top level domains means that organizations that own trademarks and would in the early days see the need only for .com or .net domain (the latter was in fact originally intended for organizations involved in networking) or perhaps a country domain such as a .no or .se one would tend to hoard domains in other top level domains too.

There are of course those who try to exploit trademark protection too, as we have seen in among other things my brush with a certain Chinese registrar or that time when what could only be seen as an extortion attempt a little too forcefully telemarketed landed me an otherwise white-elephant .se domain.

Now with the combination of potentially for most practical purposes redundant domains and the likely burden of handling spam for the same, it is understandable that attitudes started to shift. Finally in June 2015 RFC7505 was issued, with a simple and practical solution, dubbed the NULL MX record. The RFC explains how to set one up, though in language that is not too easy to penetrate.

For any domain that runs a mail service, there should be at least one MX record. Looking up, say, with dig mx yields a response where the answer section gives

;; ANSWER SECTION: 300 IN MX 1 300 IN MX 5

In your zone file, you would probably have similar lines, likely with only the MX <priority> hostname part on the actual line, the rest taken care of by the zone file it's all wrapped in.

If you want to make your domain an RFC7505-adherent one, you would remove your current MX records and replace with

MX 0 .

I did that for my little white elephant domain last week, since I did not by then remember when I last received anything sensible via that domain. 

So if you run dig mx now, it will yield


Which means nobody will ever see mail you attempt to send to The delivery will fail immediately and produce a bounce message that likely references the RFC if your mailer is a reasonably recent version.

But while I was doing the change it struck me that it would be useful to let the world know why I did not want that domain to handle mail. Fortunately there is already an appropriate DNS record type for the purpose: the TXT record.

TXT records are used for some specific purposes such as the SPF records used to list allowed outoing SMTP senders for the domain, and a few other variants tied to specific services. But fundamentally a TXT record is simply a string of characters most applications will not actually attempt to handle. This means you have the option of fitting a message on your own in one. Now, if you do a lookup on that white elephant domain's TXT records, you will get

;; ANSWER SECTION: 300 IN TXT "v=spf1 -all" 300 IN TXT "This exists only because happened." 300 IN TXT "For actual contact info please check the corresponding net domain."

Note the first TXT record here, which carries the domain's SPF specification that had been in place for a while already. It says essentially in terse if eloquent SPF speak, "This domain does not send mail".

So wrapping up, with these simple changes, quick to implement if you are in a position to edit your DNS zones we achieved:

  • Ridding ourselves of an entry point that produced only annoyances
  • Letting the world know (or at least the subset that knows how to operate common DNS tools) what the status of the mail service is and why, plus a small hint on how to make contact in case that is actually required.
A little DNS will sometimes go a long way.

A big Thank You to Security Evangelist Per Thorsheim (yes, that is his actual title) who brought RFC7505 to my attention again with this somewhat shorter blog post in Norwegian (also in English here).

Update 2021-02-23: After gentle prodding in this tweet (via JP Mens)
-- also preserved as a screenshot - 

I added a dmarc record for the domain too (kind of overkill, but can't hurt I suppose).

Friday, February 28, 2020

The 'sextortion' Scams: The Numbers Show That What We Have Is A Failure Of Education

Subject: Your account was under attack! Change your credentials!
From: Melissa <>


I am a hacker who has access to your operating system.

I also have full access to your account.

I've been watching you for a few months now.

The fact is that you were infected with malware through an adult site that you visited.

Did you receive a message phrased more or less like that, which then went on to say that they have a video of you performing an embarrasing activity while visiting an "adult" site, which they will send to all your contacts unless you buy Bitcoin and send to a specific ID?

The good news is that the video does not exist. I know this, because neither does our friend Adnan here. Despite that fact, whoever operates the account presenting as Melissa appears to believe that Adnan is indeed a person who can be blackmailed. You're probably safe for now. I will provide more detail later in the article, but first a few dos and don'ts:
  • Whatever some tempting web site tells you in a popup, unless you know what you are doing, do not install software on your devices from any other sources than the official ones. You do not need to install a new video viewer for that site or update your existing one, neither do you need to enter your administrator user name and password along with your credit card details into an unfamiliar-looking dialog box or web form.
  • Unless you know what you are doing, stay away from Bitcoin or other cryptocurrencies. If that message is the first you've heard of Bitcoin, you do not know what you are doing, leave it alone. As assets go, there is not much difference between financial derivatives, toxic waste and cryptocurrencies like Bitcoin, in that they should be handled with equal care and only from a distance unless you are in fact an expert in the field.
  • If you are not sure about either of the two bullet points before this one, please forget any shame over what you may or may not have done, and contact somebody you trust and who knows the subject better. This may be an adult such as a parent, teacher, social worker or other, a tech-savvy friend, or for that matter law enforcement such as your local police.

The important point is that you are or were about to be the victim of what I consider a very obvious scam, and for no good or even nearly valid reason. You should not need to become the next victim.

And this, dear policy makers and tech heads in general is our problem: A large subset of the general public simply do not know their way around the digital world we created for them to live in. We need to do better.

In that context I find it quite disturbing that people who should know better, such as the Norwegian Center for Information Security, in a recently issued report (also see's article (both in Norwegian only, sorry)) predict that the sextortion attacks will become "more sophisticated and credible". Then again at some level they may technically be right, since this kind of activity starts out with a net negative credibility score.

A case in point: Some versions of the scam messages I have been able to study went as far as to claim that the perpetrators had not only had taken control of the target's device, they had even sent that very email message from there. That never happened, of course, and it would have been easy for anybody who had learned to interpret Received: headers to verify that the message was in fact sent from the great elsewhere. Unfortunately the skill of reading email headers is rarely, if ever, taught to ordinary users.

The fact that people do not understand those -- to techies -- obvious facts is a fairly central and burdening problem, and again we need to do better.

Now let me explain. Things get incrementally more technical from here, so if you came here only for the admonitions or practical advice and have no use for the background, feel free to wander off.

I know the message I quoted at the beginning here is a scam because I run my own mail service, and looking at just the logs there just now I see that since the last logs archiving rotation early Saturday morning, more than 3000 attempts at delivery of messages like the one for Adnan happened, aimed at approximately 200 non-existent recipients before my logs tell me they finally tried to deliver one to my primary contact address, never actually landing in any inboxes.

One of the techniques we use to weed out unwanted incoming mail is to maintain and publish a list of known bad and invalid email addresses in our domains. These known bad addresses have then in ways unknown (at least not known to us in any detail) made it into the list of addresses sold to spammers, and we at the receiving end can use the bad addresses as triggers to block traffic from the sending hosts (If you are interested, you can read elsewhere on this blog for details on how we do this, look for tags such as greylisting, greytrapping or antispam).

If it was not clear earlier, those numbers tell us something about the messages at hand. It should be fairly obvious that compromising videos of non-existent users could not, in fact, exist.

Looking back in archived logs from the same system I see that a variant of this message started appearing in late January 2018. The specifics of that message sequence will be interesting to revisit when the full history of sextortion (I still do not like the term, but my preferred alterantive is at risk of being filtered out by polite society-serving robots) will be written, but let us rather turn to the more recent data, as in data recorded earlier this week.

Mainly because I found the media coverage of the "sextortion" phenomenon generally uninformed and somewhat annoying, I had been been mulling writing an article about it for a while, but I was still looking for a productive angle when on Wednesday evening I noticed a slight swelling in the number of greytrapped hosts. A glance at my spamd log seemed to indicate that at least one of the delivery attempts had a line like

       I am a hacker who has access to your operating system.

Which was actually just what I had been pondering writing about.  

So I set about for a little research. I greped (searched) in my yet-unrotated spamd logs for the word hacker, which yielded lots of lines of the type

Feb 22 04:04:35 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.
Feb 22 04:17:04 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.
Feb 22 04:34:03 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.
Feb 22 04:40:30 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.
Feb 22 04:55:04 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.
Feb 22 05:09:39 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.
Feb 22 05:13:22 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.
Feb 22 05:38:02 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.
Feb 22 05:44:39 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.
Feb 22 06:00:30 skapet spamd[8716]: Body: I am a hacker who has access to your operating system.

(the full result has been preserved here). Extracting the source addresses gave a list of 198 IP addresses (preserved here).

Extracting the To: addresses from the fuller listing yielded 192 unique email addresses (preserved here). Looking at the extracted target email addresses yielded some interesting insights:

1) The target email addresses were not exclusively in the domains my system actually serves, and

2) Some ways down the list of target email addresses, my own primary address turns up.

Of course 2) made me look a little closer, and only one IP address in the extract had tried delivery to my email address.

A further grep on that IP address turned up this result.

There are really no surprises to be had here, at least to a large subset of my supposed readers. The sender had first tried to deliver one of the sexstortion video messages to one of the by now more than quarter million spamtraps, and its IP address was still blacklisted by the time it finally tried delivery to a potentially deliverable address.

Doing a few spot checks on the sender IP addresses in recent and less recent logs it looks like the only two things could be mildly exciting about those messages. One is the degree the content was intended to be embarrasing to the recipient. The other is a possible indicator of the campaign's success: Looking back through the logs for the approximate year of known activity, it even looks like the campaign became multilingual, while retaining the word "hacker" in most if (possibly) not all language versions.

Other than that it is almost depressing how normal the sextortion campaign is: It uses the same spam sending infrastructure and the same low quality target address lists (the ones containing some subset of my spamtrap addresses) as the regular and likely not too successful spammers of every stripe. Nothing else stands out.

And as returning readers will notice, the logs indicate that the spambots are naive enough in their SMTP code that they frequently mistake spamd's delaying tactics for a slow, but functional open SMTP relay.

Now to recap the main points:
  • Regular users: The sextortion messages are scams, the videos do not exist. If this quasi-random sample is representative, the scammers are seen to send to 200 non-existing, invalid addresses before lucking on a real one. This alone strongly indicates that no videos exist. There is no reason to send money, bitcoin or otherwise. Look instead to learning how your devices and the networks and services they connect to actually work.
  • Competent mail admins: The tools to stop the flow of sextortion messages or at least slow to a manageable trickle are available today. You simply need to keep your antispam game up to speed with best practices and best of breed tools. If you are a user or someone who manages mail admins, check what your mail service does.
  • Competent authorities: Please step up to the task of educating the public. Sane, fact based approaches to IT security work. While it is easy to get distracted by the potential presence of porn and users' feelings of shame over accessing that kind of material, assigning much weight to that side of the matter is counterproductive. Work to educate the public and please focus on real threats, not imagined ones like the present topic.
Whatever evolves next out of these rather hamfisted attempts at blackmail is unlikely to ever achieve any level of sophistication worthy of the name.

We would all be much better served by focusing on real threats such as, but not limited to, credential harvesting via deceptive content delivered over advertising networks, which themselves are a major headache security- and privacy-wise, or even harvesting via phishing email.

Both of the latter have been known to lead to successful compromise with data exfiltration and identity theft as possible-to-probable results.

To a large extent the damage could could have been significantly limited had the general public been taught sensible security practices such as using multi-factor authentication or at least actually good passwords combined with securely coded password management applications, and insisting that services encourage such practices.

Yes, I know you have been dying to ask: What is the thing about Adnan? According to my activity log, the address was added as a spamtrap on July 8th, 2017 after somebot had tried to log on as the user adnan, a user name not seen before at,

Jul  8 09:40:34 skapet sshd[34794]: Failed password for invalid user adnan from port 41091 ssh2

apparently from a network in South Korea.

As always, there is more log material available to competent practitioners and researchers with a valid research agenda. Please contact me if you are such a person who could use the collected data productively.

Update 2020-02-29: For completeness and because I felt that an unsophisticated attack like the present one deserves a thorough if unsophisticated analysis, I decided to take a look at the log data for the entire 7 day period, post-rotation.

So here comes some armchair analysis, using only the tools you will find in the base system of your OpenBSD machine or any other running a sensibly stocked unix-like operating systen. We start with finding the total number of delivery attempts logged where we have the body text 'am a hacker' (this would show up only after a sender has been blacklisted, so the gross number actual delivery attempts will likely be a tad higher), with the command

zgrep "am a hacker" /var/log/spamd.0.gz | awk '{print $6}' | wc -l

which tells us the number is 3372.

Next up we use a variation of the same command to extract the source IP addresses of the log entries that contain the string 'am a hacker', sort the result while also removing duplicates and store the end result in an environment variable called lastweek:

 export lastweek=`zgrep "am a hacker" /var/log/spamd.0.gz | awk '{print $6}' | tr -d ':' | sort -u `

With our list of IP addresses tucked away in the environment variable go on to: For each IP address in our lastweek set, extract all log entries and store the result (still in crude sort order by IP address), in the file 2020-02-29_i_am_hacker.raw.txt:

 for foo in $lastweek ; do zgrep $foo /var/log/spamd.0.gz | tee -a 2020-02-09_i_am_hacker.raw.txt ; done

For reference I kept the list of unique IP addresses (now totalling 231) around too.

Next, we are interested in extracting the target email addresses, so the command

grep "To:" 2020-02-29_i_am_hacker.raw.txt | awk '{print substr($0,index($0,$8))}' | sort -u

finds the lines in our original extract containing "To:", and gives us the list of target addresses the sources in our data set tried to deliver mail to.

The result is preserved as 2020-02-29_i_am_hacker.raw_targets.txt, a total of 236 addresses, mostly but not all in domains we actually host here. One surprise was that among the target addresses one actually invalid address turned up that was not at that time yet a spamtrap. See the end of the activity log for details (it also turned out to be the last SMTP entry in that log for 2020-02-29).

This little round of armchair analysis on the static data set confirms the conclusions from the original article: Apart from the possibly titillating aspects of the "adult" web site mentions and the attempt at playing on the target's potential shamefulness over specific actions, as spam campaigns go, this one is ordinary to the point of being a bit boring.

There may well be other actors preying on higher-value targets through their online clumsiness and known peculiarities of taste in an actually targeted fashion, but this is not it.

A final note on tools: In this article, like all previous entries, I have exclusively used the tools you will find in the OpenBSD (or other sensibly put together unixlike operating system) base system or at a stretch as an easily available package.

For the simpler, preliminary investigations and poking around like we have done here, the basic tools in the base system are fine. But if you will be performing log analysis at scale or with any regularity for purposes that influences your career path, I would encourage you to look into setting up a proper, purpose-built log analysis system.

Several good options, open source and otherwise, are available. I will not recommend or endorse any specific one, but when you find one that fits your needs and working style you will find that after the initial setup and learning period it will save you significant time.

As per my practice, only material directly relevant to the article itself has been published via the links. If you are a professional practitioner or researcher with who can state a valid reason to need access to unpublished material, please let me know and we will discuss your project.

Update 2020-03-02: I knew I had some early samples of messages that did make it to an inbox near me squirreled away somewhere, and after a bit of rummaging I found them, stored here (note the directory name, it seemed so obvious and transparent even back then). It appears that the oldest intact messages I have are from December 2018. I am sure earlier examples can be found if we look a littler harder.

Update 2020-03-17: A fresh example turned up this morning, addressed to (of all things) the postmaster account of one of our associated .no domains, written in Norwegian (and apparently generated with Microsoft Office software). The preserved message can be downloaded here

Update 2020-05-10: While rummaging about (aka 'researching') for something else I noticed that spamd logs were showing delivery attempts for messages with the subject "High level of danger. Your account was under attack."  So out of idle curiosity on an early Sunday afternoon, I did the following:

$ export muggles=`grep " High level of danger." /var/log/spamd | awk '{print $6}' | tr -d ':' | sort -u`
$ for foo in $muggles; do grep $foo /var/log/spamd >>20200510-muggles ; done

and the result is preserved for your entertainment and/or enlightenment here. Not much to see, really other than that they sent the message in two language varieties, and to a small subset of our imaginary friends.

Update 2020-08-13: Here is another snapshot of activity from August 12 and 13: this file preserves the activity of 19 different hosts, and as we can see that since they targeted our imaginary friends first, it is unlikely they reached any inboxes here. Some of these campaigns may have managed to reach users elsewhere, though

Update 2020-09-06: Occasionally these messages manage to hit a mailbox here. Apparently enough Norwegians fall for these scams that Norwegian language versions (not terribly well worded) get aimed at users here. This example, aimed at what has only ever been an email alias made it here, slipping through by a stroke of luck during a time that IP address was briefly not in the spamd-greytrap list here, as can be seen from this log excerpt. It is also worth noting that an identically phrased message was sent from another IP address to mailer-daemon@ for one of the domains we run here.

Update 2021-01-06: For some reason, a new variant turned up here today (with a second message a few minutes later and then a third), addressed to a generic contact address here. A very quick check of logs here only turned up only this indication of anything similar (based on a search for the variant spelling PRONOGRAPHIC), but feel free to check your own logs based on these samples if you like.

Update 2021-01-16: One more round, this for my Swedish alter ego. Apparently sent from a poorly secured Vietnamese system.

Update 2021-01-18: A Norwegian version has surfaced, attempted sent to approximately 115 addresses in .no domains we handle, fortunately the majority of the addresses targeted were in fact spamtraps, as this log extract shows.

Update 2021-03-03: After a few quiet weeks, another campaign started swelling our greytrapped hosts collection, as this hourly count of IP addresses in the traplist at dump to file time shows:

Tue Mar  2 21:10:01 CET 2021 : 2425
Tue Mar  2 22:10:01 CET 2021 : 4014
Tue Mar  2 23:10:01 CET 2021 : 4685
Wed Mar  3 00:10:01 CET 2021 : 4847
Wed Mar  3 01:10:01 CET 2021 : 5759
Wed Mar  3 02:10:01 CET 2021 : 6560
Wed Mar  3 03:10:01 CET 2021 : 6774
Wed Mar  3 04:10:01 CET 2021 : 7997
Wed Mar  3 05:10:01 CET 2021 : 8231
Wed Mar  3 06:10:01 CET 2021 : 8499
Wed Mar  3 07:10:01 CET 2021 : 9910
Wed Mar  3 08:10:01 CET 2021 : 10240
Wed Mar  3 09:10:01 CET 2021 : 11872
Wed Mar  3 10:10:01 CET 2021 : 12255
Wed Mar  3 11:10:01 CET 2021 : 13689 
Wed Mar  3 12:10:01 CET 2021 : 14181
Wed Mar  3 13:10:01 CET 2021 : 15259
Wed Mar  3 14:10:01 CET 2021 : 15881
Wed Mar  3 15:10:02 CET 2021 : 17061
Wed Mar  3 16:10:01 CET 2021 : 17625
Wed Mar  3 17:10:01 CET 2021 : 18758
Wed Mar  3 18:10:01 CET 2021 : 19170
Wed Mar  3 19:10:01 CET 2021 : 20028
Wed Mar  3 20:10:01 CET 2021 : 20578
Wed Mar  3 21:10:01 CET 2021 : 20997

and they attempted to get to mailer-daemon@, as can be seen from this preserved message as well as this one (both of which actually did inbox due to aliases).

Stay safe out there.

Update 2021-04-17: A new variant, somewhat crudely worded, inboxed today. Preserved here, here and here.

Update 2021-05-15: After swelling the list of trapped hosts considerably over the last few days, a sample of the campaign with the most correctly worded Norwegian text inboxed today, and I later found other samples.

From the logs it looks like the campaign started on May 13th:

 Thu May 13 10:10:01 CEST 2021 : 3998
Thu May 13 11:10:01 CEST 2021 : 4064
Thu May 13 12:10:01 CEST 2021 : 7052
Thu May 13 13:10:01 CEST 2021 : 7297
Thu May 13 14:10:01 CEST 2021 : 7474
Thu May 13 15:10:01 CEST 2021 : 10178
Thu May 13 16:10:01 CEST 2021 : 10251
Thu May 13 17:10:01 CEST 2021 : 11150
Thu May 13 18:10:01 CEST 2021 : 12686
Thu May 13 19:10:01 CEST 2021 : 12866
Thu May 13 20:10:01 CEST 2021 : 14708
Thu May 13 21:10:01 CEST 2021 : 14713
Thu May 13 22:10:01 CEST 2021 : 14907
Thu May 13 23:10:02 CEST 2021 : 16336
Fri May 14 00:10:01 CEST 2021 : 16360
Fri May 14 01:10:01 CEST 2021 : 16473
Fri May 14 02:10:01 CEST 2021 : 17608
Fri May 14 03:10:01 CEST 2021 : 17643
Fri May 14 04:10:01 CEST 2021 : 17671
Fri May 14 05:10:01 CEST 2021 : 17763
Fri May 14 06:10:01 CEST 2021 : 18796
Fri May 14 07:10:01 CEST 2021 : 18950
Fri May 14 08:10:02 CEST 2021 : 18972
Fri May 14 09:10:01 CEST 2021 : 18725
Fri May 14 10:10:01 CEST 2021 : 19929
Fri May 14 11:10:01 CEST 2021 : 19942
Fri May 14 12:10:01 CEST 2021 : 17046
Fri May 14 13:10:01 CEST 2021 : 18068
Fri May 14 14:10:01 CEST 2021 : 18619
Fri May 14 15:10:01 CEST 2021 : 16066
Fri May 14 16:10:01 CEST 2021 : 17468
Fri May 14 17:10:01 CEST 2021 : 17297
Fri May 14 18:10:01 CEST 2021 : 15859
Fri May 14 19:10:01 CEST 2021 : 17395
Fri May 14 20:10:01 CEST 2021 : 15934
Fri May 14 21:10:01 CEST 2021 : 15996
Fri May 14 22:10:01 CEST 2021 : 17120
Fri May 14 23:10:02 CEST 2021 : 16238
Sat May 15 00:10:01 CEST 2021 : 16299
Sat May 15 01:10:01 CEST 2021 : 16362
Sat May 15 02:10:01 CEST 2021 : 16346
Sat May 15 03:10:01 CEST 2021 : 16814
Sat May 15 04:10:01 CEST 2021 : 16812
Sat May 15 05:10:01 CEST 2021 : 16787
Sat May 15 06:10:01 CEST 2021 : 16007
Sat May 15 07:10:01 CEST 2021 : 17093
Sat May 15 08:10:01 CEST 2021 : 17101
Sat May 15 09:10:01 CEST 2021 : 17015
Sat May 15 10:10:01 CEST 2021 : 15702
Sat May 15 11:10:01 CEST 2021 : 15637

Update 2021-06-16: Another campaign seems to be under way, with this message sent to an address which I can reveal is simply an alias. 

Update 2021-08-16: Thanks to one particular operator being 'too big to block' this message, apparently part of a campaign that has at least 103 other sending hosts that are currently trapped here, actually inboxed despite being sent to a spamtrap which also corresponded to a forgotten alias for an actual in-use mailbox. 

Update 2021-08-17: By lunchtime the output of 

grep vellykket /var/log/spamd | awk '{ print $6 }' | sort -u | tr -d ':' | wc -l

had reached 471, so I did 

export trash=`grep vellykket /var/log/spamd | awk '{ print $6 }' | sort -u | tr -d ':'`
for foo in $trash ; do grep $foo /var/log/spamd >> vellykket.txt ; done

You can find the result here: vellykket_20210817T1200.txt. It looks like the campaign is still in progress.
A few hours later, the number was 570 and the new export looks like vellykket.txt while the most up to date list of IP addresses participating in the campaign is in vellykke_addressest.txt
If you're interested in further data, please let me know.
Update 2021-09-09: There are signs that another campaign is in progress, with an inboxed sample preserved here. This particular message appears to have been delivered from a Korean network.