Sunday, April 3, 2022

Spammers in the Public Cloud, Protected by SPF; Intensified Password Groping Still Ongoing; Spamware Hawked to Spamtraps

This past week was truly one for the blooper reel. A public cloud service provider let the great unwashed into the address ranges published as safe mailers via their SPF records, with hilarious if rather predictable results. Next up, we find an intensive advertising campaign for spamware aimed at our imaginary friends. And the password guessing aimed at an ever-expanding dictionary of non-existing users continues.

To the rest of the world, bsdly.net is known variously as a honeypot, a source of various kinds of blocklists or a frequent target of domain joejobs that contribute to the ever-expanding list of imaginary friends also know as spamtraps.

To me and a very small set of other people, it's home on the net, providing a set of services we need fairly painlessly on an OpenBSD platform that rarely requires much work besides the odd pkg_add -u -D snap followed by sysupgrade -s (yes, we jump from snapshot to snapshot on this one).

Then the past week served us with three separate events that, while actually harmless to our side, together serve to show that a certain subset of humans would perhaps be better diverted to activities that do not involve computers.

Note: This piece is also available, with more basic formatting but with no trackers, here.

A Public Cloud Provider Let Spammers Set Up Within Their Published SPF Ranges

The first event started on Tuesday. While looking for something else entirely in my mail server logs, I noticed an unusually high number of delivery attempts to what definitely looked like spamtraps reaching the actual mail server. The earliest entry was from that morning:

2022-03-29 06:28:31 H=ministeriodesanidad34.yomevacunoseguro.com [20.104.226.220] X=TLS1.2:ADH-AES256-GCM-SHA384:256 CV=no F=<root@ministeriodesanidad34.yomevacunoseguro.com> rejected RCPT <claus-leba@ehtrib.org>: Unknown user

There were several thousand entries of that type (full log here, extracted source IP addresses here). My initial impulse was of course to check the logs to see how they got past spamd in the first place. Oddly, I found no trace of any activity involving spamd and a random sampling of those IP addresses. That would in turn indicate that they had been pass-listed, most likely by being included in the permanent nospamd pass list, which we generate here mainly based on the published SPF records of domains we need to communicate with.

Again taking a random subset of the extracted IP addresses and using whois identified the IP address range owner as a large cloud services provider that among other things also provide hosted or hybrid on premise plus hosted email service. This in turn means that domains that use those services also include large segments of that provider's IP address ranges in their SPF records. Not quite knowing what to do I tweeted,

In addition to tweeting and looking for feedback (which was not huge but dominated by Y answers), I notified the relevant abuse@ address by mail, including links to the log data and the IP addresses.

I also tweaked the log reader hinted at in this earlier piece so any attempt at delivering mail from that domain in the future will put the sending IP address safely away both in the spamd blocklist and in the safety of a table that is subject to block drop quick from for six weeks after the activity stops, and exported to downloadable blocklists as described in the article I referenced earlier.

The abuse@ handlers at the company I am not naming explicitly here (yes, there's a unix command you can use to find out who if it matters to you) were quite responsive and said that the activity seemed to be coming from their public cloud section, and yes they were forwarding my data to their internal CERT. As a followup I suggested to them that using our slowly expanding list of spamtraps in their outbound filtering might be a good idea if they intend to offer SMTP-for-hire in the future.

What seems to have happened is that the miscreants here set up using a range of the provider's services including domain registration, DNS hosting, and judging from the consistent use of root@ as the sender addess, set up some number of Linux virtual machines to do the spamming.

Before the activity stopped later in the week, we identified two more campaigns that fit the pattern. The data can be found here: log entries and IP addresses for the second wave, log entries and IP addresses for the third. Each of the campaigns appear to have stopped shortly after their domains were de-registered. I never saw the contents of the messages since not a single one appears to have inboxed here.

The episode has a few teachable items: First, that some subset of our list of spamtraps is indeed incorporated in the address lists used by gullible spammers and their customers, and second, that if you run a public cloud service, you need to pay attention to what your customers do and be wary of letting them use IP address ranges that have been announced as being really safe to accept mail from.

I notified the cloud provider that I would be writing an article about the events and asked them for any and all useful input they could provide. No such information has surfaced by the time of writing. If any useful information turns up from them after publication, I will of course update this piece accordingly.

A Separate Set of Spammers Market Their Spamming Software, Intensively Targeting Our Imaginary Friends

While the public cloud spammers thing was developing, I noticed another campaign that was actively targeting our spamtraps.

Sent from as far as I could tell only three IP addresses and with a total of 58 different subject lines all about spamming tools. It is possible that the campaigns did not target our spamtraps exclusively in our domains, but the log archive (1.5M compressed but expands to 40M raw) serves as testament that our imaginary friends are definitely targeted by some subset of the online marketing community, and they are pouring resources into doing that, one byte per second.

And once again, I have not seen the actual contents of the messages beyond what turns up in the logs after greytrapping kicks in. Not one of those messages found its way to an actual mailbox here.

SSH Password Groping Still High, But Bursty

As noted in a previous article, SSH password guessing activity went up significantly in the days leading up to the Russian invasion of Ukraine in February.

In addition to the data referenced directly in the article, the archived logs and the summaries of numbers of attempts per day (March and subsequent month summaries are archived along with other SSH log data, while the data for the current month so far gets updates several times a day) the number of attempts per day are consistently on a higher level than before the Ukraine war started, with a new higher intensity episode ongoing as I type.

One interesting feature of the password guessing attempts during the last few days is that they feature a much larger number of new user names attempted than usual. This means that the list of spamtraps here is now growing at the highest rate since the episode involving what was likely one or more phishing campaigns targeting Chinese users during early 2019 as mentioned in my 2019 summed up piece published at the end of that year.

If you have further data on these or similar incidents that you are able to share or if you want to look further into these and similar incidents, please let me know.

If you find any errors in the material I publish or disagree with my sentiments, or if you find this article interesting, useful or annoying, please let me know, either in comments or via email.



Sunday, February 27, 2022

Predicting developments in real world conflict from patterns of failed logins

Anecdotal evidence indicates that it may be possible to predict developments in real world conflicts from certain indicators in cybercrime traffic.

Is it possible to glean useful information about international developments or even predict real world attacks from the activity that we record in the logs of Internet-facing systems? 

Note: A trackerless (other than my webserver log) version of the article is now available on my web site.

Looking at data I collect for other, quite pragmatic, reasons I see a clear correlation between the run-up to the Russian invasion of Ukraine earlier this month and the password guessing activity targeting non-classified systems in my care.

I'll be backing up that assertion with data later, but first, a bit of background. 

As returning readers already know, I have been running Internet facing systems for a select group of friends and family for decades. In the late noughties I noticed a pattern of slow, distributed password guessing that I dubbed The Hail Mary Cloud, summed up in the summary article linked here and links therein. The data I collect from those failed logins make it into a set of blocklists, along with data from a few other sources. And yes, this is also one source of new spamtraps, as noted in the blocklists article.

A few years after the original Hail Mary Cloud events, in January 2016, I started seeing Hail Mary-like activity again, and started collecting data (available in the raw here), but failing to see any new patterns worth writing about, never started a new article based on the data. Until now, that is.

The table here has the totals for number of attempts per month since then (the table as a .csv file is available here):

Failed SSH login attempts per month, 2016 - 2022

 

2016

2017

2018

2019

2020

2021

2022

Jan

27015

348020

17738

35143

34882

42866

2355

Feb

121675

329074

2115

32053

60605

39029

24218

Mar

62254

498613

4648

29839

37477

29575

 

Apr

94335

271992

9588

38310

29941

27876

 

May

26428

106688

4782

55485

46207

24455

 

Jun

71321

65966

10831

75515

21947

36292

 

Jul

39088

49675

5865

47619

57082

20225

 

Aug

162529

65899

7631

59421

14030

62002

 

Sep

183196

26007

5804

85336

17814

31179

 

Oct

165295

16109

8211

82020

38185

6812

 

Nov

184660

28234

5395

58547

20734

3814

 

Dec

127288

15049

38320

82739

33650

5509

 

It is worth noting that these attempts in almost all cases lead to getting the host blocked and published in the blocklist, some immediately if they come in very quickly, others after a short time, such as if they try logging on as root, admin or a few other obvious ones.

Anyway, the data shows that the typical number of attempts per month is at least in five digit numbers, with seasonal variations. But at the end of 2021 we saw a marked decrease in activity from October 2021 onwards, only to see a sharp rise again in February of 2022. 

The development becomes even more striking if we look at the per day data for February, complete up to about mid-day CET on the 27th:

Failed SSH login attempts February 2022

Feb  1: 66
Feb  2: 13
Feb  3: 50
Feb  4: 31
Feb  5: 35
Feb  6: 85
Feb  7: 13
Feb  8: 70
Feb  9: 28
Feb 10: 13
Feb 11: 32
Feb 12: 13
Feb 13: 48
Feb 14: 28
Feb 15: 30
Feb 16: 337
Feb 17: 2006
Feb 18: 1906
Feb 19: 1608
Feb 20: 2113
Feb 21: 2207
Feb 22: 2424
Feb 23: 1978
Feb 24: 2976
Feb 25: 3044
Feb 26: 2071
Feb 27: 992

The developments stand out even clearer when presented as a graph:



Again, it is worth noting that the data is complete only up to mid-day on February 27th. 

But we see clearly that the number of attempts (effectively attempts by new hosts, quickly blocked) increased slightly on the 15th, only to essentially explode into levels equal to or slightly higher than the previous normal roughly a week before the Russian invasion of Ukraine started in the early hours of Febuary 24.

On Febuary 15th, activity that was obviously preparations for an attack on Ukraine had been underway for weeks, and it was obvious to me that the activity was part of a campaign to acquire new nodes for botnets to be used for any kind of cybercrime or disinformation bot that can be run from things that run a SSH service, typically Unix-ish systems. 

Even before the current Ukraine episode I had a vague feeling that similar correlations would be visible in the data collected around earlier international events, but by Febuary 17th I was fairly confident that the attack was imminent.

Then on the day of the attack, I tweeted:
I will of course be monitoring the situation for developments and may follow up with further analysis.

If you do not want to wait for those write-ups, you are welcome to go ahead and analyse the data (as long as any reuse or resulting analysis credits the source) -- the archive is here while the data for the month so far can be found here (note: that link always points to freshest collection for the current month).

If you too collect logs on similar activity and you are able to share data or analysis, I would like to hear from you.

Update 2022-03-01: The month ended, and the final data for February 2022 are in. The counts for February 27 and 28 were 1509 and 1138 respectively, continuing the decline in new hosts attempting that we had been seeing over the previous few days. The total number of failed attempts for that month ended up at 25873. 

The final month graph looks like this:


The full data for February can be found in this file as well as in the current year's .zip archive in the archive directory.

Update 2022-03-05 11:50 CET: We may be seeing another uptick in expectation of a propaganda-needing event. 

The numbers for failed ssh logon attempts in March so far are:

Mar  595
Mar  1003
Mar  699
Mar  657
Mar  727

Notice that the the number of attempts before noon today already exceeds the total for the previous day. We may be seeing the preparation for another offensive or other event in need of propaganda or other cybercrime support. Stay tuned for updates.

Update 2022-03-09: The month to date daily numbers for failed attempts are now available, updated at quasi-random intervals a few times a day as ssh_attempts_month_to_date.txt

Update 2022-04-02: As can be seen from the month to date numbers (now covering April 2022) we are now seeing another busy day on the password guessing front with almost 2000 attempts already by only a little past noon CEST. For reference, the numbers for March can be found along with previous data in the archive directory.


Thursday, September 30, 2021

What every IT person needs to know about OpenBSD

How to have fun with the world’s most important free software project

"Functional, free and secure by default", OpenBSD remains a crucial yet largely unacknowledged player in the open source field. This talk aims to highlight the project's signature security features and development practices -- razor sharp focus on correct and secure code coupled with continuing code audit -- as well as the project's role as source of innovation in security practices and 'upstream' source for numerous widely used components such as OpenSSH, PF, LibreSSL and others.

If you only have a few minutes to spare, the highlights are:

  • OpenBSD has been around for more than 25 years (started October 1995)
  • OpenBSD is proactively secure with only 2 remote holes in default install in all those years
  • OpenBSD pioneered use of strong cryptography, the first free system to ship with IPSec (entangling itself in US export regulations in the process)
  • OpenBSD pioneered and is still leading in code audit, fixing similar bugs tree-wide when found
  • OpenBSD has all security enhancements enabled by default and are hard going on impossible to disable
  • OpenBSD is open source, free software and the project actively encourages independent verification of code quality and security.
  • Today OpenBSD is in use in many network-centric roles, even though it is a general purpose operating system albeit with a particular emphasis on security.
  • OpenBSD has a high profile quality image based on actual code quality and proven performance in real world use
  • OpenBSD is upstream (origin) for several widely used pieces of software such as OpenSSH, OpenBGPD, PF, OpenSMTPd, LibreSSL, iked, mandoc and a number of others. For a complete list, please see the OpenBSD Innovations page on the OpenBSD website.
  • OpenBSD has been ‘growing up in public’ with code generally accessible via anonymous CVS (the first of its kind) since 1995 – transparent process, development discussions on public tech@ mailing list
  • Developers would do well to study high quality (mainly) C source and how the project runs a 6 month release cycle like clockwork (with only a few notable exceptions).

Note: If you are more of a slides person you will be happy to hear that indeed the presentation I used for this when given as a talk is available here with the main highlights and little to no geek jokes.

Now with that out of the way, let's step back to  where it all started.


OpenBSD: How it all started

OpenBSD's history is to a large extent the history of the Internet itself. You may have heard of the time back in the 1980s when the likes of IBM and Digital were slugging it out in the corporate IT sphere and the US department of defence paid for experiments in distributed, device independent networking.

That's when a loosely organized group of hackers somewhat coordinated by researchers at University of California's Berkeley campus rose to prominence with "BSD Unix", which by a sequence of happy accidents became the home of the reference implementation of the TCP/IP internet protocols.

By the early 1990s, commercialization of the Internet had started, and the Berkeley Computer Science Research Group (CSRG) that had coordinated the efforts was set to be disbanded. In addition to the net itself, the main tangible product out of Berkeley was the Berkeley Software Distribution (BSD), often distributed on tapes in the mail but also available on the net itself, which had started as a collection of software for AT & T's Unix but had over the years been extended become a full featured Unix operating system.

Several different groups wanted BSD to go on even if the CSRG did not, and several things happened in fairly rapid succession:
 

  • Lynne and Bill Jolitz ported BSD to Intel x86 (actually 80386sx), creating 386BSD. This was chronicled in a series of articles in Dr Dobbs' Journal (also see a more condensed summary over at salon.com)
  • Next up, hackers started sharing improvements to the 386BSD code as "patchkits", and two separate groups took the work further to form their projects: The FreeBSD group would be working on bringing the best possible BSD to PC-style hardware, while the NetBSD group's ambition was to make BSD run on any hardware they could get their hands on. [See Addendum at the end of the article.]
  • A group of former CSRG employees formed BSDi Inc. and marketed their product BSD/386 with among other things a contact phone number "1-800-ITS-UNIX". The activities of an actual corporation in turn triggered a lawsuit from the owners of the UNIX trademark over code copyrights.


The lawsuit was eventually settled -- only six files of several thousand in the tree were 'potentially encumbered' and had to be replaced, leaving both NetBSD and FreeBSD with a rush to replace the code which if I remember correctly was at least in part fairly central to the virtual memory subsystem.

OpenBSD came into existence a couple of years later, from a fork of the NetBSD code base in October 1995, with the initial release in July 1996.

From the very start, the OpenBSD project has been running a code audit of the entire tree, focusing on code correctness and security. We want secure, correct code that makes up a usable system with sane defaults and complete and readable documentation. And we want that code to be available under a free license and actually available to the world as soon as it is committed to the project's version control.

One of the early achievements of the OpenBSD project was anonymous CVS, which makes it possible for anyone on the internet to get the code with changes in near real time. This was a major break with the normal practice of most projects of the time, which would typically work in relative isolation on private mailing lists and at quasi-random intervals issue a release as a tarball on an FTP server somewhere.

You are already an OpenBSD user!

It is probably useful at this point to reveal that even if you do not know it, you are more likely than not using code with an OpenBSD origin right now. Your Apple product, be it iPad, iPhone or Mac, your Android device, your Cisco router, Solaris, Linux or other Unix or even your Microsoft product has some or a lot of OpenBSD originated code in it. We will get back some detail on that later.
 

OpenBSD: Code audit and security evolution

But about the code audit. The activity runs roughly like this: 

Read the code, understand what it does.

Look for unsafe behaviors, assume a hostile environment.

When you find a bug and fix it, look for similar code elsewhere in the tree and fix everywhere.

You will be amazed how much different programmers think alike and make the same mistakes. Wash, rinse, repeat.

That may sound somewhat unexciting, but careful study of how the code actually performs in real life situations lead to a number of innovations over the years, with a strong slant for being proactively secure, making it harder for bugs to actually do damage:

  • W^X -- memory can be writeable XOR executable
  • Address space randomization (ASLR) so the jump targets and gaps will vary for each execution
  • random-sized gaps inserted in the stack, again catching fixed-sized returns
  • unreadable, unwriteable guard pages at the end of malloc()ed chunks to catch overruns
  • privilege separation -- daemons run the bulk of their code as a non-privileged user, more likely than not in a shell-less chroot, coupled with privilege revocation, which means that daemons drop privilege as soon as possible
  • the pledge(2) system call to declare a profile to restrict program behavior to only specified operations and resources
  • the unveil(2) system call to restrict file system access to specified paths and permissions only
  • A fairly user visible change came when OpenBSD 6.2 introduced kernel address randomized link, or KARL, which sees to it that the kernel is relinked to a new, randomized layout for each boot. Once again, introducing randomness where none had been before is seen as a way to mitigate possible exploits based on code loading at predictable addresses.


All of those features have been integrated in the OpenBSD source tree, and with the developers admonished to adhere to the rule

"where it is possible to spot damage, fail hard".


-which means that poorly written software will crash a lot more often on OpenBSD than elsewhere. That in itself should make the platform attractive to developers. Exposing your code to a hostile environment and see it perform or fail can be quite entertaining and enlightening.

Usable, portable and secure


To complete the picture, it is useful to keep in mind that OpenBSD runs on a total of fourteen platforms. All platforms are self-hosting. Cross-compiling is only used in the early phase of porting to a new platform.

And of course, in addition to purely maintaining existing code to run on diverse platforms, users and developers have real world needs that are addressed by developing new software, extending the features of existing programs, adding new functionality or even replacing programs or entire subsystems.

Security is a many-faceted topic. Early on, OpenBSD stood out as the system that included real crypto in the base system, to the extent that exporting OpenBSD source code from the United States was technically illegal under that country's munitions export restrictions as they were defined at the time. 

Fortunately for us, the project was always coordinated from Canada by undisputed project leader Theo de Raadt who lives in Canada. There is anecdotal evidence that US based developers would trek across the border for hackathons with clean slate equipment to install OpenBSD while in Canada and hack, that is, work on the system and would then legally bring the result back with them.

One early application of crypto in OpenBSD was when a full IPSEC stack was included in the system in 1997. OpenBSD was the first free system to include IPSec by default in its base install.

In a prime example of hacker humor of the time, a T-shirt featuring one of the early appearances of Puffy the blowfish that would become the project mascot touted the Blowfish password hashing algorithm which remains the default on OpenBSD both with the picture caption "So long and thanks for all the passwords" just below Puffy on the front, along with the full source code of the blowfish function on the back. 

The expectation was that the T-shirt would be illegal to re-export from the US.

In addition to attention to security and code correctness, one other important feature of OpenBSD is attention to intellectual integrity and insisting on clearly worded and unambiguous license to use and modify code and documentation that forms part of the system.

So why use OpenBSD? What is it like?


So what is OpenBSD like for a user or developer, and why is it better?

I'd say the short version is that it's a real Unix. Unlike the Linuxes of the world that spent years muddling through an evolutionary succession of init systems and have ended up more or less settling on the ever expanding systemd which seems to have tentacles into everything and is on a clear course to replacing most of what we have traditionally thought of as the base system, OpenBSD has stayed with and refined the traditional BSD init so can have both uncluttered services management and a base system that consists of programs that for the most part adhere to the classical Unix philosophy that every program should do exactly one thing, and do that thing well.

If you are a developer, you will also appreciate hearing that the base system of well designed programs that all have a readable and useful man page already contains basic Unix developer tools along with a C and C++ compiler -- clang where supported and gcc where necessary -- plus perl and a host of tools. Basically, everything that is needed to build the base system from a fresh checkout of the source code is contained in the base system on a default install.


Ported software goes under /usr/local


Once you have the thing installed on whatever hardware you have, keeping in mind that you can run a selection of 14 platforms ranging from fairly ancient kit to modern hardware, you will likely turn to installing ported third party software from packages, using pkg_add(8) which will suck in whatever you tell it to fetch from the same mirror you installed from or what appears to be the most local one.

More software is available on the more popular platforms than on the more, dare we say exotic ones.

For the OpenBSD 6.9 release, the most mainstream platform amd64 came with 11310 prebuilt and installable packages, while mips64 had only 8182 and the mips64el platform is marked as (still building).

Installing pre-built packages is almost always more convenient and is recommended in most cases, but if you for one reason or the other want to build your own from a cvs checkout of the ports tree, you are free to do so at the cost of your own time watching the process.

Whichever route you choose to go, you will see that installing packages does not land you with any files in the directories used by the base system outside of a few that drop their configuration files in subdirectories under /etc and add their combined startup/shutdown scripts to the collection in /etc/rc.d. Anything else ends up under /usr/local, and you can see why the installer by default sets up that file system on a separate and fairly roomy partition. My previous article You've installed it. Now what? Packages! is a few years old, but gives you the main motivations and some background along with some pointers on packages practice.

Note: If you are more of a slides person than a fulltext person, you may be relieved to hear that you can find the slides for the talk this article is based on (and vice versa) here.

The installer was always good, got better

When I found OpenBSD more than twenty years ago, my main Unix exposure was from working with Linuxes and FreeBSD. What attracted me to OpenBSD and finally had me buy an OpenBSD 2.5 CD set was the strong focus on security and code correctness. When the CD set and the classic wireframe daemon T-shirt finally arrived in the mail, I set about at first to install it on whatever spare hardware I had lying around.

OpenBSD wireframe daemon head


If I remember correctly, the first machine I tried installing OpenBSD on was an 80386/33MHz with 8MB RAM and I think a 100MB IDE hard disk. Which I can report sounded pretty crappy even then, but the thing did work.

The initial install was fairly straightforward, and when I started poking around I found two things about myself and the new system: Everything made sense, and everything I could think of had a readable man page. So the first change I am aware of that made the world better with OpenBSD was the decision to enforce the "No commit without documentation" rule, which came into being early in the project's life, probably roughly at the same time the OpenBSD developers gave us a real-time view of development via anonymous CVS. You can see things happening in almost real time.

It is worth mentioning that the installer has remained famously non-graphical, text only. The reason the installer remains text-only is that this is a major advantage that enables the developers and the users to handle the fairly diverse collection of hardware platforms that OpenBSD runs on with the same portable, familiar and compact code everywhere.

The installer was always scriptable and extensible, and over the years the installer has added automatic, repeatable and scriptable installs (dubbed autoinstall(8) which appeared in OpenBSD 5.5 in 2014) and the sysupgrade(8) extension (first found in OpenBSD 6.6 in 2019) that automates snapshot to snapshot or release to subsequent release upgrades for all not too hacked-up configurations. Each of these moments, or more specifically when the new code started appearing in snapshots, had me appreciate the OpenBSD system a bit more, and made me feel quality of life had improved.

Now something for your laptop - hardware support

Fast forward some twenty-plus years and the last article I published, and even got into Norwegian mainstream IT news site Digi.no, centers on a few moments involving new OpenBSD developments. It took some interaction with OpenBSD developers, but those interactions lead to my new laptop with an 11th generation Intel Core chipset working even better with OpenBSD. Yes, OpenBSD developers and a significant subset of their user base actually run OpenBSD on their laptops. I do use a Mac and a work-issued Thinkpad with Ubuntu Linux too, but life is not complete without an OpenBSD laptop.

Now to be honest, what I saw within the space of a few days was development that had me going from "Oh, sh*t, the SSD isn't recognized" -- the controller was set to a RAID-ish mode by default -- through this kernel panic:

OpenBSD 6.9-current panic message

-- to seeing it all fully supported.

The SSD problem turned out to be simple to fix: Simply find the "Advanced" BIOS option that turned the pseudo-raid feature off and let the operating system speak directly to the storage device.

For the rest there was a period of a couple of weeks I had to run with not yet commited patches in a home baked kernel built from checkouts from Jonathan Grey's git repo. When the code was committed to -current, I could resume my normal sysupgrade(8) routine, going from one development snapshot to the next.

The process, even with the need to build custom kernels for a while, was actually quite pleasant, and when the support code went into the main development branch, that too was a a moment when I felt my life had been improved by changes in OpenBSD. The hardware support is now in snapshots and will be in OpenBSD 7.0 which is set to be released on November 1st, 2021.

Why use OpenBSD? IPSEC

As I mentioned earlier, OpenBSD was the first free system to ship with IPSEC -- the tools for enabling encrypted network traffic -- in its base system. The first OpenBSD release with IPSEC was OpenBSD 2.1, which was released in 1997.

The tools worked of course, but in the early days the complaint was that IPSEC was hard and near impossible to debug from an almost-working to a fully working setup.

Further development took a while, but the tools got a major usuability upgrade in OpenBSD 3.8 with ipsecctl and its human-readable configuration file /etc/ipsec.conf to serve as a friendlier front end to the IPSEC tools.

A complete configuration for a minimal setup could look like this:

# Set up two flows:
# First between the machines 192.168.3.14 and 192.168.3.100
# Second between the networks 192.168.7.0/24 and 192.168.8.0/24
flow esp from 192.168.3.14 to 192.168.3.100
flow esp from 192.168.7.0/24 to 192.168.8.0/24 peer 192.168.3.12

This was a major step compared to other platforms. One famous example is preserved in a presentation by Mathieu Sauve-Frankel at AsiaBSDCon 2007, where he demonstrated that setting up the equivalent of the config shown here with Microsoft tools took the user through a sequence of no less than 36 dialog boxes, and still there was a distinct possibility that the configuration was not actually a working one.

The problem in both the Microsoft implementation and others was that the developers had not really given any thought to the user experience. The majority of the options the user was required to set had sensible defaults and could easily have been hidden from view.

The standards documents the developers had worked from were fairly unclear, so it is not entirely unreasonable that "it was hard to write, so it should be hard to use" was a factor in how the products ended up from a user experience perspective.

What defaults would be actually sensible would perhaps not be clear to the developer from the specification, and the only way to see what would be the sensible default would be experience with actual use in the field. The OpenBSD developers who wrote ipsecctl came with extensive experience of using IPSEC and decided that IPSEC did not in fact need to be so hard. The defaults should make sense.

The next milestone in IPSEC development on OpenBSD came with Internet Key Exchange (IKE) protocol support in OpenIKED in OpenBSD 3.8 with iked and ikectl. For convenience, ikectl is able to generate configurations for Windows and macOS clients too. That might save you the agony of clicking through dozens of dialog boxes at the client end.

The thing that lured me in

But I hear you ask, what made me turn into an almost all-in OpenBSD user?

Back in 2001 I was still only experimenting with OpenBSD, but my experience with Linux and iptables had made me long for a switch to a saner firewall. I had done some small experiments with the IPF firewall that was in OpenBSD until the 2.9 release. Then, as some of us will remember, the it was discovered that IPF's license was in fact not free, so it needed to be replaced.

There was a distinct rush, not quite a stampede, to replace IPF over the months that followed. Fortunately, the new code that replaced the previous packet filter proved to perform better. The OpenBSD Packet filter, dubbed PF for short, had been born and made its debut in OpenBSD 3.0 in December 2001. The release had originally been planned for November, but was pushed out a month to hack the "working prototype" packet filter into something usable.

Almost needless to say, this turn of events finally pushed me to take the final steps to replace the Linux gateways I had in place with OpenBSD ones. I was pleasantly surprised to find that not only did they perform well, but they also came with complete and reasonably well documented tools so I could understand what was going on. That's how I got started on the process that lead to among other things writing The Book of PF and taking that text through three editions so far. But more about that later.

It is worth noting that the IPFilter copyright episode spurred the OpenBSD developers to perform a license audit of the entire source tree and ports in order to avoid similar situations in the future. This activity ran for some months and uncovered a number of potential problems. Theo de Raadt summed up the effort in a message to the openbsd-misc mailing list on February 20th, 2003.

What they found when they started looking was that there was a significant number of files that were in fact not under a free license, much like the entire IPF subsystem had been. Those needed to be replaced. Other parts had either no license or no copyright stated. In some cases the developers gave explicit permission to continuing use, but quite a few things needed to be rewritten with a free license so OpenBSD and other free software would be able to move forward without copyright problems.

I later heard in a rather informal setting that among the no copyright and/or no license cases, it was usually possible to track down the developers via version control system logs or mailing list archives. In a large number of those cases, the initial reaction was along the lines "Say what? Are people still using that?".

SSH, open and better

PF was written from scratch to replace a subsystem that it turned out was illegal to use in an open source context. But it was not the first time the OpenBSD project had performed a nonlibreectomy, that is, taken on the task of replacing code for license reasons.

A few years earlier it had become clear that the original developer of the secure shell system ssh had commercial ambitions and the license for the software had changed in a proprietary direction. After a bit of deliberation on how to resolve the situation, the OpenBSD developers started digging around for earlier versions of the code that had been published with an acceptable license. Then they forked their version from the last version they found that still had free license. Next came an intensive period of re-introducing the features that were missing in the old code.

The result was introduced as OpenSSH in OpenBSD 2.6 in 1999. Over the next few years OpenSSH grew a portable version that started grabbing market share rapidly. The last I heard OpenSSH's market share is somewhere in the high nineties percent.

With a state of the art secure shell subsystem in place and growing all sorts of useful features, the time finally came to end unencrypted shell login sessions on OpenBSD. OpenBSD's telnetd was moved to the CVS attic in time for OpenBSD 3.8, which was released November 2005.

One other notable thing about OpenSSH is that it was the first daemon to be properly privilege separated, a model practice that debuted with the overhauled OpenSSH in OpenBSD 3.2 in March 2002. Since then privilege separation has been put in place in all daemons where it made sense to do so, and it is now a signature part of the secure by default stance of all newer OpenBSD daemons.

And yes, that packet filter

I mentioned PF, the OpenBSD packet filter, earlier. I must confess that PF has been an important part of my life in various contexts since the early noughties. Over the years, things I have written have contributed to creating the popular but actually wrong perception that OpenBSD was primarily a firewall operating system. There are a lot of useful and fun features that turned up in or in connection with PF over the years and were pioneered by OpenBSD. Some features were ported to or imitated in other systems, while others remain stubbornly OpenBSD only.

So I will touch on some of my favorite PF and PF-attached features, in quasi-random but almost chronological order.

Beating up spammers with OpenBSD spamd(8) since OpenBSD 3.3

When I started playing with OpenBSD in general and PF in particular way back when, I was already responsible for the SMTP mail service for my colleagues. My gateways by then ran OpenBSD, while the mail server rosalita, named after a Springsteen song, was not too badly specced server running FreeBSD with exim as the mail transfer agent that fed the incoming messages to spamassassin and clamav for content filtering before handing off to user mailboxes.

So when it dawned on me that I could set up spamd(8) the spam deferral daemon on the internet-facing gateway and save load on the poor suffering rosalita that was running hot with content filtering, I was quick to implement a setup that sucked in well known block lists.

Going grey, then trapping

The effect was obvious and immediate, the mail server's fans grew noticeably quieter. When greylisting was introduced in spamd soon after, I implemented that too, and witnessed yet another drop in pitch and intensity of the sound from rosalita's fans. Then a couple of releases later greytrapping -- the practice of adding IP addresses of incoming SMTP connections to blocklists if the attempted delivery is aimed at a known-bad address in the target domain -- was introduced, and that sounded like enough fun that I just went ahead and did it.

The idea of detecting spam senders by the bogus addresses they were already trying to deliver to just sounded too good to not try. And we knew that getting started would be pretty easy too. We had seen rejects for addresses that had never existed in our domains in our mail server logs for quite a while, so it was simply a matter of harvesting from a fairly bountiful source and adding stuff that we were sure would never ever be actually deliverable here to the spamtrap list. I think the first setup had only a couple of hundred entries in it, but I did not note the exact number at the time.

By July 2007 I had decided to publish both the list of spamtrap addresses and an hourly dump of the greytrapped addresses. Both remain free to download. The list of spamtraps, harvested from various log sources, by now numbers just over 270,000 imaginary friends, while the number of trapped hosts is typically in the 3000 to 5000 range. We occasionally see the list swell to 20,000 or more when high volume campaigns run with bad address lists fed to them. I am pretty sure it went over 100,000 at one point.

It's fun to watch, and it looks like a significant subset of the spamtraps have made it into the address lists of active spam operations. I frankly never thought I would still be collecting spam traps from logs all these years later. Yes, it all sounds a bit absurd, but it is effective for keeping our mailboxes largely spam free, even though it feels at times like running a weird found object-ish art project. Anyway, a summary of the lists we publish can be found in this article.

The brutes, the password gropers and the state tracking options

If you run an SSH service or really any kind of listening service with the option to log in, you will see some number of failed authentication attempts that generate noise in the logs. The password guessing, or as some of us say, password groping, turned out to be annoying enough that OpenBSD 3.6-current and later OpenBSD 3.7 introduced a set of features to use data that would anyway be available in the state table, to track the state of active connections, and to act on limits you define such as number of connections from a single host over a set number of seconds.

The action could be to add the source IP that tripped the limit to a table. Additional rules could then subject the members of that table to special treatment. Since that time, my internet-facing rule sets have tended to include variations on

table <bruteforce> persist
block quick from <bruteforce>
pass inet proto tcp from any to $localnet port $tcp_services \
        flags S/SA keep state \
	(max-src-conn 100, max-src-conn-rate 15/5, \
         overload <bruteforce> flush global)

which means that any host that tries more than 100 simultaneous connections or more than 15 new connections over 5 seconds are added to the table and blocked, with any existing connections terminated.

It is a good practice to let table entries in such setups expire eventually. At first I followed the spamd(8) defaults' example and set expiry at 24 hours, but with password gropers like those caught by this rule being what they are, I switched a few years ago to at four weeks at first, then upped again a few months later to six weeks. Groperbots tend to stay broken for that long. And since they target any service you may be running, state tracking options with overload tables can be useful in a lot of non-SSH contexts as well.

It is also worth noting that state tracking actions are useful for essentially all services. The article Forcing the password gropers through a smaller hole with OpenBSD's PF queues has a few suggestions on how to handle noise sources with various other services.

One final point I would like to make about the state tracking and actions is that much like the greytrapping feature of spamd, this feature gives you the tools to build a configuration that adapts to network conditions and learns from the traffic it sees. 

While this does not rise to the level of being an actual Artificial Intelligence or AI, this has enough buzzwordability potential that I remain to this day extremely puzzled that none of the other big names at least imitated those features in their own products and marketed for all it would be worth. 

I certainly know what I would have done in their position. But then I am more engineer than marketer and in the contexts where I call the shots, the best option is just to keep running OpenBSD.

We went to modern queueing

OpenBSD has had traffic shaping available in the ALTQ subsystem since the very early days. ALTQ was rolled into PF at some point, but the code was still marked experimental 15 years after it was written, and most people who tried to use it in anger at the time found the syntax inelegant at best, infuriating or worse at most times.

So Henning Brauer took a keen interest in the problem, and reached the conclusion that all the various traffic shaping algorithms were not in fact needed. They could all except one be reduced to mere configuration options, either as setting priorities on pass or match rules or as variations of the theme of the mother algorithm Hierarchical Fair Service Curve (HFSC for short).

Soon after, another not-small diff was making the rounds. The patch was applied early in the OpenBSD 5.5 cycle, and for the lifetime of that release older ALTQ setups were possible side by side with the new queueing system.

The feedback I get is that the saner syntax in the new queueing system lead to more users taking up traffic shaping. Here is the queue setup that I came up with for one of my sites:

queue rootq on $ext_if bandwidth 20M
        queue main parent rootq bandwidth 20479K min 1M \
                                    max 20479K qlimit 100
             queue qdef parent main bandwidth 9600K min 6000K \ 
                                    max 18M default
             queue qweb parent main bandwidth 9600K min 6000K \
                                    max 18M
             queue qpri parent main bandwidth 700K min 100K \
                                    max 1200K
             queue qdns parent main bandwidth 200K min 12K \
                                    burst 600K for 3000ms
        queue spamd parent rootq bandwidth 1K min 0K max 1K \
                                    qlimit 300

while tying the queues into the subsequent rules with a set of match rules just following that block.

This is what triggered the need to write the third edition of The Book of PF. The book includes descriptions of both the new and the old system as well as tips on how to make a smooth transition. The ALTQ code was removed from OpenBSD during the OpenBSD 5.6 cycle, but continues to live on in some form in FreeBSD and NetBSD.

And yes, if you think my queues setup punishes spammers a bit more in addtion to being subjected to spamd(8), you're right.

pflow(4) offers network insights lite

Everybody who has been tasked with looking after a network has at some point been at least a little curious about what actually moves around there. At times we will see situations where it is essential for troubleshooting purposes to see the traffic flows with data about endpoints, packets and bytes transferred, protocol and so forth.

If you do not need to see the data itself, but rather the metadata, the NetFlow standard and its close cousin IPFIX offers just that. Netflow tools existed as packages on OpenBSD already, but from OpenBSD 4.5 PF has the pflow state tracking option, paired with the pflow(4) virtual network interface which together offer a full netflow sensor package.

Set up one or more pflow interfaces to send data to one or more collectors, and add the pflow option to specific rules or as a state default and you have started your collecting. You can even have metadata for traffic matching specific rules going to separate pflow devices and collectors.

My field notes in Yes, You Too Can Be An Evil Network Overlord - On The Cheap With OpenBSD, pflow And nfsen offers some practical examples and insights, including how we used a pflow setup to track down a noisy machine on a somewhat critical network as well as some pointers to valueable further reading.

LibreSSL, the great deobfuscation

People tell me they think that the reason LibreSSL was created was the Heartbleed bug, but no, actually not, just damn close.

The LibreSSL project was in fact started a few weeks before heartbleed became common knowledge. LibreSSL is the result of a group of OpenBSD developers taking the existing OpenSSL code and starting to fix it.

This time it was not a matter of a bad license. No, this was the result of the number of OpenBSD developers who took a look at the OpenSSL code that had been part of the OpenBSD base system since quite early on, and turned away in disgust and with symptoms of physical pain, reached a critical mass of sorts. I had heard OpenBSD developers complain about the absolute horror of the OpenSSL code for at least ten years. The code quality was just that bad.

What happened next was that a group of hardened OpenBSD developers grabbed the OpenSSL code and started two activities in parallel. One was looking in the OpenSSL request tracker for bugs that had not been addressed. The other was reformatting the OpenSSL code into something resembling the OpenBSD style of readable and maintainable C.

With the code in more readable form, discovering what it did became easier. In addition to a few obvious eye-stinging bugs the LibreSSL developers found a number of oddities, including, but not limited to

  • Code was never deleted even when it became irrelevant or obsolete
  • OpenSSL did not use the system memory allocation system, but rather opted for their own which never actually deallocated memory, but rather used LIFO recycling, and could easily be made to put private info into logs
  • all written in "OpenSSL C", which according to beck@ is a dialect of the "worst common denominator"

It is worth digging out the various articles and presentations made by LibreSSL developers over the years, with specific emphasis on Bob Beck's BSDCan talk on the first 30 days of LibreSSL (available on youtube), which is the original source of the term code flensing.

Since the OpenBSD 5.6 release in 2014, LibreSSL has been the default TLS library in OpenBSD. LibreSSL has been ported elsewhere based on the -portable variant.

For my own part I can only attest to not ever running into a TLS problem that was LibreSSL's fault. It probably still has bugs, but it is a lot more of a healthy choice than its predecessor.

This was my list of life improving OpenBSD events - I'd love to hear yours

As I warned earlier, this has been about my personal list of OpenBSD events that I remember fondly.

I am sure your list is at least a little different. I am sure there are things from the innovations page that I have simply forgotten about.

Each release comes with a detailed list of changes, such as this one for OpenBSD 6.9, and the page has pointers back to the equivalent pages for previous releases.

I would love to hear about your favorite OpenBSD moments.


More items for your OpenBSD reading

www.openbsd.org is the official OpenBSD web site. If you want to donate, go to the donations page and find the most appropriate option. Corporate entities may prefer to donate via The OpenBSD Foundation, which is a Canadian non-profit corporation.

undeadly.org is the OpenBSD Journal news site.

bsdly.blogspot.com My rant^H^H^H^Hblog posts

https://flak.tedunangst.com/ Ted Unangst (tedu@) on developments

Michael W Lucas: Absolute OpenBSD, 2nd edition

Peter N. M. Hansteen: The Book of PF, 3rd edition

Henning Brauer: OpenBSD sucks (… least)


Addendum 2021-11-06

The original statement in the article that the two groups (NetBSD and FreeBSD) were only vaguely aware of each other in the early days has been disputed by at least one patchkit era participant, Tom Ivar Helbekkmo, who wrote in to say,
"That is not entirely true.  When Bill Jolitz didn't include patches from the Internet community in 386bsd 0.1, and then again not in 0.2, Chris Demetriou took the initiative to fork the project, and call ours NetBSD.

 It soon became apparent, however, that there were divergent goals for further development.  This led to the creation of FreeBSD.  So we had one community that amicably divided itself into two separate groups."

In my own experience, however, those who joined in the post-patchkit era of both projects frequently seem to be unaware of this aspect of the early days. 


Sunday, August 22, 2021

Recent and not so recent changes in OpenBSD that make life better (and may turn up elsewhere too)

Known to be "functional, free and secure by default", the OpenBSD operating system has played an important role in open source for more than a quarter century. It has also been fairly central to what I have done for the last two decades and some. What follows is my personal view of what life with OpenBSD has been like, with an emphasis on moments and developments that I feel made life, or at least my life, better.

I will assume that you know already that one of the signature features of the OpenBSD project is the continuous code audit and the sharp focus on secure and correct code. The audit by itself has produced a number of improvements, including a stream of bugfixes with bugs of a similar kind fixed in the whole tree and even the occasional subsystem rewrite. In addition, even for a free operating system project, life just happens. The world changes around us and drives the developers to take up fresh approaches to both new and well known problems and in the process develop code in ways that improves life for us all.

The Norwegian tech news site digi.no took this article in as a three part series, see parts 1, 2 and 3 if you want to read this in my native language.  

If you are not that familiar with OpenBSD the system or project, my "OpenBSD and you" talk, which I update occasionally, might be a not too bad place to start. But in this article I will focus on some specific moments when I felt that changes in OpenBSD made my life better. These are the things that made me start and go on advocating the system.

So who am I and what can I offer?

My name is Peter Hansteen. I have worked in information technology and information technology related things like documentation since the late 1980s. I am in the process of transitioning from a "Security Engineer" role into a "Cloud Expert" one, and across several other roles and titles I have always done a bit, or a lot of Unix system and network administration. At most times you will find me on The Other West Coast, specifically in Bergen, Norway.

Note: If you are more of a slides person than a fulltext person, you may be relieved to hear that you can find the slides for the talk this article is based on (and vice versa) here.

The installer was always good, got better

When I found OpenBSD more than twenty years ago, my main Unix exposure was from working with Linuxes and FreeBSD. What attracted me to OpenBSD and finally had me buy an OpenBSD 2.5 CD set was the strong focus on security and code correctness. When the CD set and the classic wireframe daemon T-shirt finally arrived in the mail, I set about at first to install it on whatever spare hardware I had lying around.

OpenBSD wireframe daemon head


If I remember correctly, the first machine I tried installing OpenBSD on was an 80386/33MHz with 8MB RAM and I think a 100MB IDE hard disk. Which I can report sounded pretty crappy even then, but the thing did work.

The initial install was fairly straightforward, and when I started poking around I found two things about myself and the new system: Everyting made sense, and everything I could think of had a readable man page. So the first change I am aware of that made the world better with OpenBSD was the decision to enforce the "No commit without documentation" rule, which came into being early in the project's life, probably roughly at the same time the OpenBSD developers gave us a real-time view of development via anonymous CVS. You can see things happening in almost real time.

It is worth mentioning that the installer has remained famously non-graphical, text only. The reason the installer remains text-only is that this is a major advantage that enables the developers and the users to handle the fairly diverse collection of hardware platforms that OpenBSD runs on with the same portable, familiar and compact code everywhere.

The installer was always scriptable and extensible, and over the years the installer has added automatic, repeatable and scriptable installs (dubbed autoinstall(8) which appeared in OpenBSD 5.5 in 2014) and the sysupgrade(8) extension (first found in OpenBSD 6.6 in 2019) that automates snapshot to snapshot or release to subsequent release upgrades for all not too hacked-up configurations. Each of these moments, or more specifically when the new code started appearing in snapshots, had me appreciate the OpenBSD system a bit more, and made me feel quality of life had improved.

Now something for your laptop - hardware support

Fast forward some twenty-plus years and the last article I published, and even got into Norwegian mainstream IT news site Digi.no, centers on a few moments involving new OpenBSD developments. It took some interaction with OpenBSD developers, but those interactions lead to my new laptop with an 11th generation Intel Core chipset working even better with OpenBSD. Yes, OpenBSD developers and a significant subset of their user base actually run OpenBSD on their laptops. I do use a Mac and a work-issued Thinkpad with Ubuntu Linux too, but life is not complete without an OpenBSD laptop.

Now to be honest, what I saw within the space of a few days was development that had me going from "Oh, sh*t, the SSD isn't recognized" -- the controller was set to a RAID-ish mode by default -- through this kernel panic:

OpenBSD 6.9-current panic message


-- to seeing it all fully supported.

The SSD problem turned out to be simple to fix: Simply find the "Advanced" BIOS option that turned the pseudo-raid feature off and let the operating system speak directly to the storage device.

For the rest there was a period of a couple of weeks I had to run with not yet commited patches in a home baked kernel built from checkouts from Jonathan Grey's git repo. When the code was committed to -current, I could resume my normal sysupgrade(8) routine, going from one development snapshot to the next.

The process, even with the need to build custom kernels for a while, was actually quite pleasant, and when the support code went into the main development branch, that too was a a moment when I felt my life had been improved by changes in OpenBSD. The hardware support is now in snapshots and will be in OpenBSD 7.0 which is set to be released approximately early November 2021.

Living the life dynamic

Now that we're talking about laptops, there is another recent development that makes your OpenBSD on laptop experience even better. Laptops and other equipment that uses dynamic network configuration became easier to operate with dhcpleased(8) now enabled by default in OpenBSD 6.9-current after it was first introduced in OpenBSD 6.9. That change marked the completion of a five year cycle of incremental development which involved writing several new daemons. Each of those programs was designed with the Unix philosophy that a program should do one thing and do it well.

The first piece of the puzzle was slaacd(8) - the stateless IPv6 address autoconfiguration daemon - which appeared in OpenBSD 6.2 to handle IPv6 automatic configuration, listening for route advertisements.

The corresponding router advertisement daemon rad(8) arrived in OpenBSD 6.4. That got most of the things involved in IPv6 autoconfiguration in order.

Next up was the arrival in OpenBSD 6.5 of unwind(8), a validating DNS resolver which learns which resolvers to query from DHCP and other sources.

To complete the set, OpenBSD 6.9 brought us resolvd(8) to manage and edit /etc/resolv.conf, updating the file with information learned from other sources, and dhcpleased(8) now serves as the client for IPv4 DHCP client information which is then fed into the configuration.

Combined with setting your laptop to join networks as they become available, moving between networks can now be an non-disruptive, even unremarkable event.

This all comes into place if you edit your /etc/hostname.$if for (for example hostname.iwx0) to something like

join adipose wpakey thedoctorknows
join tardis  wpakey biggerontheinside
join cybermen wpakey nowedont
inet autoconf
inet6 autoconf

you should expect minimal efforts needed when moving between those networks. As usual, as soon as a new feature is trusted, it is on by default in OpenBSD-current, and OpenBSD 7.0 will ship with this behavior enabled by default for interfaces set to autoconf for either inet or inet6.

But that is the modern day and for some in the future. OpenBSD on a laptop is a good experience. On the other hand, most of the world sees the BSDs and OpenBSD in particular as mainly server or even network device operating systems, despite the fairly high BSD code content in such things as Apple systems.

The thing that lured me in

But I hear you ask, what made me turn into an almost all-in OpenBSD user?

Back in 2001 I was still only experimenting with OpenBSD, but my experience with Linux and iptables had made me long for a switch to a saner firewall. I had done some small experiments with the IPF firewall that was in OpenBSD until the 2.9 release. Then, as some of us will remember, the it was discovered that IPF's license was in fact not free, so it needed to be replaced.

There was a distinct rush, not quite a stampede, to replace IPF over the months that followed. Fortunately, the new code that replaced the previous packet filter proved to perform better. The OpenBSD Packet filter, dubbed PF for short, had been born and made its debut in OpenBSD 3.0 in December 2001. The release had originally been planned for November, but was pushed out a month to hack the "working prototype" packet filter into something usable.

Almost needless to say, this turn of events finally pushed me to take the final steps to replace the Linux gateways I had in place with OpenBSD ones. I was pleasantly surprised to find that not only did they perform well, but they also came with complete and reasonably well documented tools so I could understand what was going on. That's how I got started on the process that lead to among other things writing The Book of PF and taking that text through three editions so far. But more about that later.

It is worth noting that the IPFilter copyright episode spurred the OpenBSD developers to perform a license audit of the entire source tree and ports in order to avoid similar situations in the future. This activity ran for some months and uncovered a number of potential problems. Theo de Raadt summed up the effort in a message to the openbsd-misc mailing list on February 20th, 2003.

What they found when they started looking was that there was a significant number of files that were in fact not under a free license, much like the entire IPF subsystem had been. Those needed to be replaced. Other parts had either no license or no copyright stated. In some cases the developers gave explicit permission to continuing use, but quite a few things needed to be rewritten with a free license so OpenBSD and other free software would be able to move forward without copyright problems.

I later heard in a rather informal setting that among the no copyright and/or no license cases, it was usually possible to track down the developers via version control system logs or mailing list archives. In a large number of those cases, the initial reaction was along the lines "Say what? Are people still using that?".

SSH, open and better

PF was written from scratch to replace a subsystem that it turned out was illegal to use in an open source context. But it was not the first time the OpenBSD project had performed a nonlibreectomy, that is, taken on the task of replacing code for license reasons.

A few years earlier it had become clear that the original developer of the secure shell system ssh had commercial ambitions and the license for the software had changed in a proprietary direction. After a bit of deliberation on how to resolve the situation, the OpenBSD developers started digging around for earlier versions of the code that had been published with an acceptable license. Then they forked their version from the last version they found that still had free license. Next came an intensive period of re-introducing the features that were missing in the old code.

The result was introduced as OpenSSH in OpenBSD 2.6 in 1999. Over the next few years OpenSSH grew a portable version that started grabbing market share rapidly. The last I heard OpenSSH's market share is somewhere in the high nineties percent.

With a state of the art secure shell subsystem in place and growing all sorts of useful features, the time finally came to end unencrypted shell login sessions on OpenBSD. OpenBSD's telnetd was moved to the CVS attic in time for OpenBSD 3.8, which was released November 2005.

One other notable thing about OpenSSH is that it was the first daemon to be properly privilege separated, a model practice that debuted with the overhauled OpenSSH in OpenBSD 3.2 in March 2002. Since then privilege separation has been put in place in all daemons where it made sense to do so, and it is now a signature part of the secure by default stance of all newer OpenBSD daemons.

And yes, that packet filter

I mentioned PF, the OpenBSD packet filter, earlier. I must confess that PF has been an important part of my life in various context since the early noughties. Over the years, things I have written have contributed to creating the popular but actually wrong perception that OpenBSD was primarily a firewall operating system. There are a lot of useful and fun features that turned up in or in connection with PF over the years and were pioneered by OpenBSD. Some features were ported to or imitated in other systems, while others remain stubbornly OpenBSD only.

So I will touch on some of my favorite PF and PF-attached features, in quasi-random but almost chronological order.

Beating up spammers with OpenBSD spamd(8) since OpenBSD 3.3

When I started playing with OpenBSD in general and PF in particular way back when, I was already responsible for the SMTP mail service for my colleagues. My gateways by then ran OpenBSD, while the mail server rosalita, named after a Springsteen song, was not too badly specced server running FreeBSD with exim as the mail transfer agent that fed the incoming messages to spamassassin and clamav for content filtering before handing off to user mailboxes.

So when it dawned on me that I could set up spamd(8) the spam deferral daemon on the internet-facing gateway and save load on the poor suffering rosalita that was running hot with content filtering, I was quick to implement a setup that sucked in well known block lists.

Going grey, then trapping

The effect was obvious and immediate, the mail server's fans grew noticeably quieter. When greylisting was introduced in spamd soon after, I implemented that too, and witnessed yet another drop in pitch and intensity of the sound from rosalita's fans. Then a couple of releases later greytrapping -- the practice of adding IP addresses of incoming SMTP connections to blocklists if the attempted delivery is aimed at a known-bad address in the target domain -- was introduced, and that sounded like enough fun that I just went ahead and did it.

The idea of detecting spam senders by the bogus addresses they were already trying to deliver to just sounded too good to not try. And we knew that getting started would be pretty easy too. We had seen rejects for addresses that had never existed in our domains in our mail server logs for quite a while, so it was simply a matter of harvesting from a fairly bountiful source and adding stuff that we were sure would never ever be actually deliverable here to the spamtrap list. I think the first setup had only a couple of hundred entries in it, but I did not note the exact number at the time.

By July 2007 I had decided to publish both the list of spamtrap addresses and an hourly dump of the greytrapped addresses. Both remain free to download. The list of spamtraps, harvested from various log sources, by now numbers just over 270,000 imaginary friends, while the number of trapped hosts is typically in the 3000 to 5000 range. We occasionally see the list swell to 20,000 or more when high volume campaigns run with bad address lists fed to them. I am pretty sure it went over 100,000 at one point.

It's fun to watch, and it looks like a significant subset of the spamtraps have made it into the address lists of active spam operations. I frankly never thought I would still be collecting spam traps from logs all these years later. Yes, it all sounds a bit absurd, but it is effective for keeping our mailboxes largely spam free, even though it feels at times like running a weird found object-ish art project. Anyway, a summary of the lists we publish can be found in this article.

The brutes, the password gropers and the state tracking options

If you run an SSH service or really any kind of listening service with the option to log in, you will see some number of failed authentication attempts that generate noise in the logs. The password guessing, or as some of us say, password groping, turned out to be annoying enough that OpenBSD 3.6-current and later OpenBSD 3.7 introduced a set of features to use data that would anyway be available in the state table, to track the state of active connections, and to act on limits you define such as number of connections from a single host over a set number of seconds.

The action could be to add the source IP that tripped the limit to a table. Additional rules could then subject the members of that table to special treatment. Since that time, my internet-facing rule sets have tended to include variations on

table <bruteforce> persist
block quick from <bruteforce>
pass inet proto tcp from any to $localnet port $tcp_services \
        flags S/SA keep state \
	(max-src-conn 100, max-src-conn-rate 15/5, \
         overload <bruteforce> flush global)

which means that any host that tries more than 100 simultaneous connections or more than 15 new connections over 5 seconds are added to the table and blocked, with any existing connections terminated.

It is a good practice to let table entries in such setups expire eventually. At first I followed the spamd(8) defaults' example and set expiry at 24 hours, but with password gropers like those caught by this rule being what they are, I switched a few years ago to at four weeks at first, then upped again a few months later to six weeks. Groperbots tend to stay broken for that long. And since they target any service you may be running, state tracking options with overload tables can be useful in a lot of non-SSH contexts as well.

It is also worth noting that state tracking actions are useful for essentially all services. The article Forcing the password gropers through a smaller hole with OpenBSD's PF queues has a few suggestions on how to handle noise sources with various other services.

One final point I would like to make about the state tracking and actions is that much like the greytrapping feature of spamd, this feature gives you the tools to build a configuration that adapts to network conditions and learns from the traffic it sees. 

While this does not rise to the level of being an actual Artificial Intelligence or AI, this has enough buzzwordability potential that I remain to this day extremely puzzled that none of the other big names at least imitated those features in their own products and marketed for all it would be worth. 

I certainly know what I would have done in their position. But then I am more engineer than marketer and in the contexts where I call the shots, the best option is just to keep running OpenBSD.

NAT's guts ripped out

When the OpenBSD 4.7 release cycle came around, Henning Brauer had been hard at work for a while maintaining a diff of several thousand lines -- which when applied actually shrunk the code -- that contained a total rewrite of the IPv4 network address translation code.

Previous PF versions had 'nat on interface' and 'rdr on interface' rules, while the new code introduced nat-to and rdr-to as options on pass or match rules.

The match keyword had been introduced in the previous release to act on packets and connections without affecting pass or block state, such as applying specific options or adding tags for processing later in the rule set. Now with the major NAT rewrite in place, it became even clearer why match was in fact a useful keyword and feature.

The NAT rewrite added a lot of flexibility to how you can write PF rule sets, and of course for my own part that rewrite made it necessary to write the second edition of The Book of PF, timed to hit bookshelves as close as possible to the OpenBSD 4.7 release. And yes, the rewrite improved the performance too.

We went to modern queueing

OpenBSD has had traffic shaping available in the ALTQ subsystem since the very early days. ALTQ was rolled into PF at some point, but the code was still marked experimental 15 years after it was written, and most people who tried to use it in anger at the time found the syntax inelegant at best, infuriating or worse at most times.

So Henning Brauer took a keen interest in the problem, and reached the conclusion that all the various traffic shaping algorithms were not in fact needed. They could all except one be reduced to mere configuration options, either as setting priorities on pass or match rules or as variations of the theme of the mother algorithm Hierarchical Fair Service Curve (HFSC for short).

Soon after, another not-small diff was making the rounds. The patch was applied early in the OpenBSD 5.5 cycle, and for the lifetime of that release older ALTQ setups were possible side by side with the new queueing system.

The feedback I get is that the saner syntax in the new queueing system lead to more users taking up traffic shaping. Here is the queue setup that I came up with for one of my sites:

queue rootq on $ext_if bandwidth 20M
        queue main parent rootq bandwidth 20479K min 1M \
                                    max 20479K qlimit 100
             queue qdef parent main bandwidth 9600K min 6000K \ 
                                    max 18M default
             queue qweb parent main bandwidth 9600K min 6000K \
                                    max 18M
             queue qpri parent main bandwidth 700K min 100K \
                                    max 1200K
             queue qdns parent main bandwidth 200K min 12K \
                                    burst 600K for 3000ms
        queue spamd parent rootq bandwidth 1K min 0K max 1K \
                                    qlimit 300

while tying the queues into the subsequent rules with a set of match rules just following that block.

This is what triggered the need to write the third edition of The Book of PF. The book includes descriptions of both the new and the old system as well as tips on how to make a smooth transition. The ALTQ code was removed from OpenBSD during the OpenBSD 5.6 cycle, but continues to live on in some form in FreeBSD and NetBSD.

And yes, if you think my queues setup punishes spammers a bit more in addtion to being subjected to spamd(8), you're right.

pflow(4) offers network insights lite

Everybody who has been tasked with looking after a network has at some point been at least a little curious about what actually moves around there. At times we will see situations where it is essential for troubleshooting purposes to see the traffic flows with data about endpoints, packets and bytes transferred, protocol and so forth.

If you do not need to see the data itself, but rather the metadata, the NetFlow standard and its close cousin IPFIX offers just that. Netflow tools existed as packages on OpenBSD already, but from OpenBSD 4.5 PF has the pflow state tracking option, paired with the pflow(4) virtual network interface which together offer a full netflow sensor package.

Set up one or more pflow interfaces to send data to one or more collectors, and add the pflow option to specific rules or as a state default and you have started your collecting. You can even have metadata for traffic matching specific rules going to separate pflow devices and collectors.

My field notes in Yes, You Too Can Be An Evil Network Overlord - On The Cheap With OpenBSD, pflow And nfsen offers some practical examples and insights, including how we used a pflow setup to track down a noisy machine on a somewhat critical network as well as some pointers to valueable further reading.

LibreSSL, the great deobfuscation

People tell me they think that the reason LibreSSL was created was the Heartbleed bug, but no, actually not, just damn close.

The LibreSSL project was in fact started a few weeks before heartbleed became common knowledge. LibreSSL is the result of a group of OpenBSD developers taking the existing OpenSSL code and starting to fix it.

This time it was not a matter of a bad license. No, this was the result of the number of OpenBSD developers who took a look at the OpenSSL code that had been part of the OpenBSD base system since quite early on, and turned away in disgust and with symptoms of physical pain, reached a critical mass of sorts. I had heard OpenBSD developers complain about the absolute horror of the OpenSSL code for at least ten years. The code quality was just that bad.

What happened next was that a group of hardened OpenBSD developers grabbed the OpenSSL code and started two activities in parallel. One was looking in the OpenSSL request tracker for bugs that had not been addressed. The other was reformatting the OpenSSL code into something resembling the OpenBSD style of readable and maintainable C.

With the code in more readable form, discovering what it did became easier. In addition to a few obvious eye-stinging bugs the LibreSSL developers found a number of oddities, including, but not limited to

  • Code was never deleted even when it became irrelevant or obsolete

  • OpenSSL did not use the system memory allocation system, but rather opted for their own which never actually deallocated memory, but rather used LIFO recycling, and could easily be made to put private info into logs

  • all written in "OpenSSL C", which according to beck@ is a dialect of the "worst common denominator"

It is worth digging out the various articles and presentations made by LibreSSL developers over the years, with specific emphasis on Bob Beck's BSDCan talk on the first 30 days of LibreSSL (available on youtube), which is the original source of the term code flensing.

Since the OpenBSD 5.6 release in 2014, LibreSSL has been the default TLS library in OpenBSD. LibreSSL has been ported elsewhere based on the -portable variant.

For my own part I can only attest to not ever running into a TLS problem that was LibreSSL's fault. It probably still has bugs, but it is a lot more of a healthy choice than its predecessor.

This was my list of life improving OpenBSD events - I'd love to hear yours

As I warned earlier, this has been about my personal list of OpenBSD events that I remember fondly.

I am sure your list is at least a little different. I am sure there are things from the innovations page that I have simply forgotten about.

Each release comes with a detailed list of changes, such as this one for OpenBSD 6.9, and the page has pointers back to the equivalent pages for previous releases.

I would love to hear about your favorite OpenBSD moments.


More items for your OpenBSD reading

www.openbsd.org is the official OpenBSD web site. If you want to donate, go to the donations page and find the most appropriate option. Corporate entities may prefer to donate via The OpenBSD Foundation, which is a Canadian non-profit corporation.

undeadly.org is the OpenBSD Journal news site.

bsdly.blogspot.com My rant^H^H^H^Hblog posts

https://flak.tedunangst.com/ Ted Unangst (tedu@) on developments

Michael W Lucas: Absolute OpenBSD, 2nd edition

Peter N. M. Hansteen: The Book of PF, 3rd edition

Henning Brauer: OpenBSD sucks (… least)