Friday, March 6, 2026

The Book of PF, 4th Edition: It's Here, It's Real

© 2026 Peter N. M. Hansteen

The Book of PF 4th edition, posed on the author's laptop

The long wait is over. Fresh copies of The Book of PF, 4th Edition arrived here today. Which means: I'll bring some to upcoming conferences!

When the email message with package tracking information turned up, I thought at first that this was yet another dirty, scammy, spammy phishing thing.


Note: This piece is also available without trackers but classic formatting only here.

It had been years since the last time I had had a UPS delivery, and since then I had only seen obvious scammy phishing things pretending to delivery messages that all in fact looked very much like the crummy fakes they turned out to be.

But this one looked subtly different, in that it had both my address and my somewhat unusual name correctly spelled.

In addition, the URLs were on close inspection all for actual UPS.com resources.

And finally, the shipment had a total weight of 10.4 kg, which sounded about right for the stack of The Book of PF, 4th Edition author copies I was expecting to receive soon after the print run was ready to ship.

Now, a few days later, I have just taken delivery of package, this time delivered to my door by a delivery man who was very excited to be delivering a box of books to their author. The box was slightly heavier than my regular backpack:

The box containing the Book of PF 4th edition author copies

and after taking the box inside, I was able to extract the 15 copies, as expected.

This is me, holding my first physical copy of the fourth edition to come out of the box:

Peter Hansteen and the first of the The Book of PF 4th edition author copies to come out of the box

This means that the process that started almost two and a half years ago is finally, tangibly complete. I have the shipment of author copies of The Book of PF, 4th Edition in hand.

Regular readers of this column as well people on relevant mailing lists will be aware, as will my followers on various social media, that people who ordered the book early have already started receiving their copies of the refreshed The Book of PF. And some have posted their (so far only positive as far as I am aware) reactions to their media of choice.

But for one reason or the other, my expected stack of author copies were not among the very early deliveries.

Among the likely reasons are the fact that I am located in Bergen, Norway, which, at roughly 62 degrees North and in The Other West Coast fjords country, is somewhat off the beaten track for high speed deliveries.

And as any good business would do, No Starch Press will put a priority on delivering the product to customers who put in preorders or were on the spot once the thing had been declared ready.

Now I can tell you all yet again, the thing exists, and I am delighted to see the actual physical copies.

I know some readers were expecting there to be an auction for the first signed copy like there was for the third edition.

I have to disappoint you there. Unfortunately, side effects of decisions made by politicians on the other side of the Atlantic have made it impractical to set up an equivalent auction this time around.

Instead, I would encourage readers who can afford to do so to direct a donation to the OpenBSD Foundation or to the OpenBSD Project directly, in order to support the OpenBSD project and the immensely valuable work they do for free software.

If you're more of a FreeBSD person, the FreeBSD Foundation will be happy to hear from you as well.

Once you have done the rounds of your favorite free operating system supporting foundation and have your (or your bosses') credit card back in its regular location, you can think forward to one or more of the upcoming regional BSD conferences.

The main ones this year are,

  • AsiaBSDCon, March timeframe, alternates between Tokyo (JP) and Taipei (TW). AsiaBSDCon 2026 will be March 19-22, 2026 in Taipei, Taiwan.

    This will include the first scheduled Network Management with the OpenBSD Packet Filter Toolset session of the year, on March 19th, 2026.

  • BSDCan, Mid May to mid June, Ottawa (CA). BSDCan 2026 will be June 17-20 in Ottawa, Canada.

    This, too, will have a Network Management with the OpenBSD Packet Filter Toolset session, on June 18th. Details to follow closer to the date of the conference.

  • EuroBSDcon, September timeframe, each year in a new European city. EuroBSDCon 2026 will be September 10-13 in Brussels, Belgium. The call for papers is open from March 1st, 2026 and runs until June 20th.

All three conferences will welcome submissions during their Call for Papers period for talks, tutorials and other types of sessions as well as general participation by people regardless of geographic or other origin.

And unless truly unexpected events take place, I will be at all three this year, bringing copies of The Book of PF, 4th edition.

I look forward to seeing you at these events.

If you are interested in coming to a regional BSD conference as a speaker, volunteer or other participant, see What is BSD? Come to a conference to find out! (also here) for some information.

Through all of those events, our resident philosopher took it all with her usual calm:

Our residen philosopher, taking all that book stuff with her usual calm

If you want to get the book, and using the link to buy from No Starch Press is not practical, I would encourage you to support your local bookstore and see if they can order it in.

The Book of PF, 4th Edition: It's Here, It's Real is © 2026 Peter N. M. Hansteen (published 2026-03-06)
You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

Sunday, February 1, 2026

The Rest Is Trash

© 2026 Peter N. M. Hansteen

We are now halfway through the nineteenth year of greytrapping, still tracking and collecting from the wealth of imbecility out there

My logs tell me that in early September of 2025, I added the following entries to the list of spamtraps (or raw version)

!!!!!!!!der!rest!ist!muell!@bsdly.net
!!!!!!!!el!resto!es!basura!@bsdly.net
!!!!!!!!the!rest!is!trash!@bsdly.net
    
on a whim and to emphasize to anyone who would actually care to read any part of the file that the content is, for most purposes, utterly worthless.

Note: This piece is also available without trackers but classic formatting only here.

Planting a clue that obvious may or may not have helped somebody out there, but it felt somehow appropriate. And while writing this article, I added a French equivalent for those who would feel that one was needed.

As I have mentioned earlier, there are a number of ways to find potential spamtrap material. In the early days, I simply extracted entries from the mail server logs with some very crude regular expression match using grep on the log files.

Later I set up to extract from the spamd greylist instead. That turned out to be a little shortsighted, since it would not then catch the attempted deliveries from a host that had already entered the blocklist. That thought only struck me again some years later, and in the 2022 piece Harvesting the Noise While it's Fresh, Revisited (also here) I run through how I amended the collection method a little.

The retrospective piece Eighteen Years of Greytrapping - Is the Weirdness Finally Paying Off? (also here) offers descriptions of a few other methods of creating spamtrap entries.

Then in the early days of January 2026, another episode involving a mail operator you will have heard about led me to write A Major Mail Provider Demonstrate They Likely Do Not Understand Mail At All (also here).

That episode also led me to realize that with a significant part of the traffic from at least that operator coming in over IPv6 or from a set of pre-cleared hosts such as the mail exchangers of the too big to block operators, there would be nothing logged by spamd anyway, so I set about collecting yet again candidate spamtraps from the real mailserver logs.

It turned out that by not looking at delivery attempts via IPv6, I had indeed been missing quite a significant number of potential imaginary friends.

As a result of the expanded scope of my noise harvesting, combined with some light massaging of the finds using tools such as rot13 and rev, the total number of added spamtraps for the month of January 2026 was

5665771

a total of five million, six hundred and sixty-five thousand, seven hundred and seventy-one added, so the total by end of January 2026 was more than twenty-two and a half million.

I will keep updating the retrospective piece Eighteen Years of Greytrapping - Is the Weirdness Finally Paying Off? (also here) with monthly numbers, and those who are interested can even follow the progress by monitoring the files in the log directory.

And since I know some of my returning readers feel that a greytrapping article is not complete without a graph, I have made one for the year so far,

Line graph of hosts in spamd-greytrap 2026 up to publication date

The data that went into producing the graph is available as 2026-traplistcounts.txt.

I take the numbers so far as an indication that the trend towards fewer routable IPv4 addresses are in play continues, with some traffic moving to IPv6 while an increasing number of spam senders are likely behind NAT, carrier grade or otherwise.

But there is another annoying phenomenon that turns up occasionally. From this perch it looks like one or more somebots or somebodies mistake a greylisting and greytrapping spamd for an open relay, and something like this happens.

A host somewhere (in this case apparently in China) tries to relay via my setup, in this case using a significant subset of the spamtraps as their From address.

Some of these episodes have come with a rate of new connections and sheer number of attempts that it leads me to think there may be more than one host actually active, likely behind a NAT device.

One recent episode annoyed me enough that it had me toy with appending state tracking with overload to the pf.conf rule that feeds our spamd, so the rule would be


pass in on egress inet proto tcp from any to any port smtp \
divert-to 127.0.0.1 port spamd flags S/SA keep state \
(max-src-conn 1200, max-src-conn-rate 255/10, overload <webtrash> flush global, pflow)

which in means that any spam source that insists on trying to fill our state table will earn themselves an entry in the <webtrash> table, to be blocked by us and consumers of the exported blocklist for the next six weeks.

This has the slight disadvantage of letting the spamd-greytrap entry expire 24 hours later, but on the other hand the rules in place here and possibly other places mean that the host will not be able to communicate at all with any system that uses our exported blocklists for at least six weeks.

But of course, our log files will be incrementally quieter for that period of time.

If you have any comments you want to share, please let me know either in comments to this piece (where available) or followups to the post where you found this article referenced.

Update 2026-02-07: I added a few more translations of the phrase "the rest is trash", prepended with !!!!!!!! and interspersed with further !s where I it seemed appropriate. If you feel more languages need to be covered, please let me know in comments or send me an email, with the desired translation.


In other news, The Book of PF, 4th edition is inching nearer to delivery in physical form. See the earlier piece Yes, The Book of PF, 4th Edition Is Coming Soon (also here) for the story of why that edition needed to exist.

The next scheduled Network Management with the OpenBSD Packet Filter Toolset session will be March 19, 2026 at AsiaBSDCon 2026 in Taipei, Taiwan.

Papers and other sessions selection for BSDCan 2026 should be complete very soon after this piece has been published, so if you submitted, look for incoming BSDCan mail.

The call for papers be for EuroBSDcon 2026 will open March 1st, 2026 and run until June 20th.

If you are interested in coming to a regional BSD conference as a speaker, volunteer or other participant, see What is BSD? Come to a conference to find out! (also here) for some information.


The Rest Is Trash is © 2026 Peter N. M. Hansteen (published 2026-02-01)
You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

Thursday, January 8, 2026

A Major Mail Provider Demonstrate They Likely Do Not Understand Mail At All

© 2026 Peter N. M. Hansteen

Screenshot of part of a mail bounce message, saying '  peter.hansteen@gmail.com
    host gmail-smtp-in.l.google.com [2a00:1450:4025:402::1b]
    SMTP error from remote mail server after pipelined end of data:
    550-5.7.1 [2a03:94e0:182c::1      19] Gmail has detected that this message is
    550-5.7.1 likely suspicious due to the very low reputation of the sending
    550-5.7.1 domain. To best protect our users from spam, the message has been
    550-5.7.1 blocked. For more information, go to
    550 5.7.1  https://support.google.com/mail/answer/188131 a640c23a62f3a-b842a23243csi674652166b.80 - gsmtp'

So it happened again. A major mail provider proved that they do not, in fact, understand how modern email works.

I've been running mail services for longer than I care to remember. It started out back when I was running a small business on the edge of tech, mainly dealing with software localization and documentation writing.


Note: This piece is also available without trackers but classic formatting only here.

The company I had started with a few colleagues were in close cooperation with another business that worked in similar, but not primarily overlapping fields of interest.

Then during the early to mid 1990s, Internet and proper SMTP Internet email became available, and as one did at the time, we set up with a mail service, running at first on an early Red Hat Linux.

After a while we moved to a Debian setup, and over time, unlike most small businesses of the period, chose to not go for a Microsoft solution but rather moved to a setup based on the other free alternatives, with a combination of OpenBSD and FreeBSD for services.

When spam became annoying enough, we configured content filtering of various kinds, which lowered the noise level for a while. Later we discovered that our OpenBSD firewalls could also handle greylisting and tarpitting via spamd, and immediately saw that our mail feed became cleaner yet again.

Those of us who were in the server room when the greylisting was turned on, could not help noticing that the fan noise from the mail server just disappeared.

But one thing we did not get entirely rid of was bounce messages from other sites, directed to user names that had never existed in any of the domains we served. Clearly, one or more groups out there were sending messages with faked addresses in our domains.

So I was very happy when I found that in the OpenBSD 3.3 release, spamd offered a new feature called greytrapping, which meant we could actually put those fake addresses to good use.

From that point on, a high level view of mail delivery to our systems work like this:

  1. When a message arrives, spamd checks whether we have received mail from that host in recent times. Mail from known sending hosts is sent on to the mail server.

    If the message comes from a host we have not seen SMTP traffic from earlier, spamd answers one byte per second, finally presenting a Temporary local error, please retry later code and message. The host is greylisted. If the host returns with the same set of sender IP address, from and to addresses, it will be let through. Except in one circumstance which we will come back to very soon.

    If the message comes from a known bad sender IP address, spamd answers in a subset of SMTP at a rate of one byte per second, until the sending side gives up.

  2. When a message comes from a greylisted host, spamd checks whether the to address is in the list of spamtraps. If if the message matches this criterion, the sending IP address is added to teh the list of hosts with a TRAPPED status and will receive the one byte per second treatment until the sending party gives up. Addresses that enter the spamd-greytrap set stay there until 24 hours after the time of last contact.

    If the to address is not in the list of spamtraps, spamd adds the sending IP address to the set of likely valid senders, and passes the message on the real mail server.

  3. The real mail server performs various checks on incoming mail, including whether the sending domain has valid SPF, DKIM and DMARC information and several kinds checks of message contents against known indicators of spam or malware.

  4. If the message passes all those validity and content checks, the mail server check whether the to address matches a valid recipient in the domains we handle.

  5. If the to address does not match a valid recipient, the message is rejected with a bounce message back to the sending party.

  6. If the to address does match a valid recipient, the message is delivered according to the configuration that is in place for that recipient.

The main difference between this setup and any mail server you will have heard about, is that we have a list of spamtraps. The source for spamtraps was originally the invalid addresses that turned up in our mail server logs.

Later, with greylisting in place, the obvious selection criterion was checking the greylist for addresses that did not match any local recipient or an existing spamtrap. Over time the number and kinds of sources expanded a bit. You can read all you want about that and more in the retrospective article Eighteen Years of Greytrapping - Is the Weirdness Finally Paying Off?

I have an hourly cron job that runs a script that exports the list of trapped IP addresses for use elsewhere and produces a report of various useful items, including a list of possible new spamtraps, extracted from the greylist.

The overnight batch on January 8th, 2026 looked like this:

fuchigamikamogawa522@bsdly.net
itsukiishibashimitaka@bsdly.net
kinoshitario0310@bsdly.net
kuromotoasuka_1007@bsdly.net
kyokomatsudakita@bsdly.net
motohashi_katsuyama@bsdly.net
namikiakari-funabashi@bsdly.net
namikidaisukeyamagata@bsdly.net
nomurajunichi_1958@bsdly.net
numanoenergy@bsdly.net
otaniasami@bsdly.net
rikoizawa2004@bsdly.net
sakakihifumi@bsdly.net
seoreina-1991@bsdly.net
shimamura_asahikawa@bsdly.net
sugimototaro.0321@bsdly.net
teshigawaraishikawa@bsdly.net
tokitakota.1968@bsdly.net
tsuboneyuji_mori@bsdly.net
ueki-miyagi633@bsdly.net
yamabenoboru@bsdly.net

The local parts or user names are not what you would expect to find in a business based in Norway.

But these Japanese-sounding names are quite typical of the backscatter we have been seeing here during the last two or three years. Most likely somebody is running spam or phishing campaigns aimed at Japanese users.

The bounce messages do not ever reach an inbox, but they do turn up in the greylist dump in the hourly message. There, the <> in the fourth column reveals that the messages were indeed bounce messages.

The afternoon batch looked similar:

aoki-1990600@bsdly.net
fumiakihachiya0827@bsdly.net
hachiyaakira-stormchaser@bsdly.net
hanamurachihiro2000@bsdly.net
isaacclark.sarahclark@www.bsdly.net
izawanaoki0819@bsdly.net
junanzai0902@bsdly.net
kagawa.1965@bsdly.net
kashiwagi1967@bsdly.net
kawamurasoma1971@bsdly.net
kenmiyagawa1953@bsdly.net
kodama_1985@bsdly.net
kusunokiotsu@bsdly.net
liammartin.norakumar@lfja.org
machiasuka_1977@bsdly.net
masuda1955@bsdly.net
matsuoka-1976@bsdly.net
monmagenji654@bsdly.net
mutojunichi-noon@bsdly.net
nakaya-2017@bsdly.net
oscartanaka.zoeevans@lfja.org
rukanakagome1981@bsdly.net
ryuseiterasawa2021@bsdly.net
shinjiohno.futtsu@bsdly.net
tabatayusuke_0302@bsdly.net
tamuraryuji1995@bsdly.net
tsukuda2016@bsdly.net
usuda.ishikawa.star@bsdly.net
yoshidakatsuo@bsdly.net
yukitakahagi94770@bsdly.net

The addresses in both of these batches were added to our spamtraps, affectionately known as our imaginary friends, along with a number of synthetic entries as described in the longer article Eighteen Years of Greytrapping - Is the Weirdness Finally Paying Off?.

But that day, after processing new spamtraps and a bit of overnight mail, I sent a message to a business contact of mine that uses a Google as their mail service provider. That produced a bounce message, some of which is quoted in this graphic:

Screenshot of part of a mail bounce message, saying '  peter.hansteen@gmail.com
    host gmail-smtp-in.l.google.com [2a00:1450:4025:402::1b]
    SMTP error from remote mail server after pipelined end of data:
    550-5.7.1 [2a03:94e0:182c::1      19] Gmail has detected that this message is
    550-5.7.1 likely suspicious due to the very low reputation of the sending
    550-5.7.1 domain. To best protect our users from spam, the message has been
    550-5.7.1 blocked. For more information, go to
    550 5.7.1  https://support.google.com/mail/answer/188131 a640c23a62f3a-b842a23243csi674652166b.80 - gsmtp'

I had put my own gmail.com address on Cc:, in part due to various earlier episodes with that provider. The diagnostic was the same for the other recipients.

This means that after several years of mostly managing to deliver messages sent from our systems to their intended recipients in Google managed domains (at random times deciding to put mail from here in their users' Spam folders), somebody decided it was time to disregard our domains' published SPF and DMARC information.

Their "very low reputation of the sending domain" is is likely down to a very poor understanding of how modern mail delivery works.

More likely than not, the volume of messages sent with faked sender addresses claiming to be from our domains and a source IP address in the great elsewhere is vastly larger than the actually valid messages sent from valid users, all of which will come from the hosts listed in our published records.

The existence of spamtraps should not be a surprise either, after all we have been doing greytrapping for more than eighteen years.

I would posit that this is a mail services provider that has demonstrated that they do not, in fact, understand SMTP mail at all.

Fortunately, posting the data and a description of the incident to a mailing list for mail administrators indicates that persons who work for that operator read that list, since the problem lessened a bit a few hours later. My messages now only land in the recipients' Spam folders.

I would like to invite a debate about incidents of this type. The big operators can be quite nasty to smaller players, as we can see from this episode and the earlier one you can find by following links in this article.

What are the sensible standards of behavior (aka netizenship) to expect from mail service providers?

Should, for example, we consider making the large operators (or smaller ones, for that matter) liable for damages for mishandling their service offerings like this?

Followup in comments to this article (where possible) or to the social media post that lead you to find this article.


Update 2026-01-09: One small batch of data might be of interest to my core readership. The output of a grep for "Unknown user" in the as-yet-unrotated mail server log is preserved in this file. My reading of this is that even the big names do not actually value their SPF/DMARC checks much at all.
Update 2026-01-10: A few hours after I had posted to the limited-distribution mailing lists for mail operators, and soon enough after I had published the original version of this article that it likely they did not read it yet, I was contacted by people who were plausibly claiming to be close to the actual developement and maintenance of the gmail codebase.

My impression, or at least the way I read what was in that communication, is that

  • Gmail is a large code base that has been evolving over a long time

  • The system is complicated enough and with enough factors (in the hundreds) involved that feeding the same message through the system several times is likely to produce different results each time

  • The messages the system produces for external parties to see are unspecific at best and may in fact point to factors other than those that actually determined the pass/no pass decision

  • The code has passed through many hands, and I at least get the impression that nobody currently there can honestly say they understand all aspects of the system

Despite all of this, they trust the system absolutely, claiming that it has a negligible false positive rate.

The last bit I at least think is a delusion that is sustained by the fact that they have made it pretty much impossible to file a problem report. It likely is easier for paying customers, but the only way in I have found is to post my gripes in public.

And yes, for every incident (there have been quite a few over the years) I have used side channels to contact my GOOG-using connections and ask them to file a problem report with as much details as possible. That seems to help, sometimes.

I had thought up a really snappy and harsh one-liner to characterize all of this, but I think I'll save that for another occasion.


A Major Mail Provider Demonstrate They Likely Do Not Understand Mail At All is © 2026 Peter N. M. Hansteen (published 2026-01-08)
Max Stucchi and I will be giving a PF tutorial at AsiaBSDC0n 2026, and I welcome your questions now that I'm revising the material for that session. See this blog post for some ideas.

For more information about the BSD conferences, see What is BSD? Come to a conference to find out! (also tracked).

For a broader overview and retrospective of mail and greytrapping, you may be interested in reading Eighteen Years of Greytrapping - Is the Weirdness Finally Paying Off?, which links to this piece and a number of other related resources.

You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

Separately, pre-orders of The Book of PF, 4th edition are now open. For a little background, see the blog post Yes, The Book of PF, 4th Edition Is Coming Soon (also here). The latest information I have is that physical copies should be ready to ship by the end of January 2026.


Sunday, December 28, 2025

Why 451 is Good for You - Greylisting Perspectives From the Early Noughties

© 2025 Peter N. M. Hansteen

A software vendor was using SMTP spamware to send license keys customers had paid thousands for. A measured rant was in order, and turned out to be quite enlightening.

While looking through directories of old material, I stumbled upon what was most likely the main part of a complaint sent to a software vendor for failing to deliver a license key file the company I worked for then had paid some thousands of dollars for.

The file as I found it was dated August 2010, but was likely a preserved copy of a draft that was written some time before the first edition of The Book of PF (now in its fourth edition) was published, in response to the non-delivery incident. A quick investigation had me conclude from my spamd logs that their side did not play well with greylisting.


Note: This piece is also available without trackers but classic formatting only here.
I have revisited the handling sites that do not play well with greylisting theme a number of times, such as the 2018 piece Goodness, Enumerated by Robots. Or, Handling Those Who Do Not Play Well With Greylisting (also here).

But I found these early notes interesting enough that I include them here, with only minor redactions to protect the (relatively) innocent:

SWCrafters' reaction to finding out that their messages do not get through, essentially blaming "inaccurate spam filtering" was not unexpected, but I will take the opportunity to explain a few things about how Internet email works and how this makes their position at odds with reality.

Even though Internet services are offered with no guarantees, usually described as 'best effort' services, a significant amount of effort has been put into making essential services such as SMTP email transmission fault tolerant, making the 'best effort' one with as close as does not matter to having a perfect record for delivering messages.

The EXECUTIVE SUMMARY of this message is that the matter which trips up the delivery of SWCrafters' license-carrying emails is the fact that their email sending software's best effort at delivery falls significantly short of what current Internet standards require.

The current standard for Internet email transmission is defined in RFC2821, which in section 4.5.4.1, "Sending Strategy", states

"In a typical system, the program that composes a message has some method for requesting immediate attention for a new piece of outgoing mail, while mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender."
and
"The sender MUST delay retrying a particular destination after one attempt has failed. In general, the retry interval SHOULD be at least 30 minutes; however, more sophisticated and variable strategies will be beneficial when the SMTP client can determine the reason for non-delivery."
Contrast this with the application which sends the SWCrafters license information messages, which according to the data I have avaliable opens two SMTP sessions within a second of each other (the time resolution I have in my logs at the moment), apparently discarding the message without delivery afterwards.

RFC2821 goes on to state that

"Retries continue until the message is transmitted or the sender gives up; the give-up time generally needs to be at least 4-5 days."
After all, delivering email is a collaborative, best effort thing, and he RFC states clearly that if the site you are trying to send mail to reports it can't receive anything at the moment, it is your DUTY (a MUST requirement) to try again later, after an interval which is long enough that your unfortunate communication partner has had a chance to clear up whatever was the problem.

A sending strategy which relies on every receiver to be receptive at all times, discarding undelivered messages after only one unsuccessful attempt, possibly makes sense if the data you are sending is unimportant or if your intended targets are unlikely read or even want to receive the messages you send.

If on the other hand the data you are sending matter to either you or the intended recipient, it is in everybody's interest that you use the fault tolerance features which a compliant SMTP mail system offers.

To put this in context, you need to remember that the SWCrafters license messages are the result of some SWCrafters customer ordering at least a thousand dollars' worth or software licenses, with no real upper limit on the dollar value of a single message. The system used to send these messages apparently does not understand SMTP status messages, discarding undelivered messages without a trace. Essentially, the system you are using treats the data your customers expect to receive in exchange for thousands of dollars paid as discardable.

The "greylisting" technique which is in use at justgottahave.faith and other sites means that our systems expect any SMTP sender to understand SMTP status codes and to respect "451 temporary error please try again later" messages. The hows and whys are detailed at https://www.greylisting.org/, with a tutorial which contains a 'close enough' description of how it's done at justgottahave to be found at https://home.nuug.no/~peter/pf/, with the particulars starting at https://home.nuug.no/~peter/pf/en/spamd.html.

We do content filtering as well, but this particular application never managed to get its data sent far enough to encounter content filtering until its IP address got whitelisted (listed as 'known good', or if you will, not having to conform to normal criteria).

Greylisting works extremely well, and since it is both standards compliant (essentially insisting on compliance) and simple to implement you should expect it to be deployed at the next site you are trying to send email to.

If I remember correctly, the other side found a way to send the missing license codes with something that did handle SMTP status codes correctly a short time after the mail that included some version of these notes was sent.

I had originally intended to make the URLs in the text here clickable, but changed my mind when I discovered that the current operators of greylisting.org have decided that a large language model (the current iteration of what passes for artificial intelligence needed to be included in the processing. That will perhaps serve as a sign that the world does move on, if not necessarily in useful directions at all times.

If you want to explore the ins and outs of greylisting and the related phenomenon greytrapping, my recent piece Eighteen Years of Greytrapping - Is the Weirdness Finally Paying Off? (also here) is a way to start. For the greylisting part, the notes above capture the main points.


Why 451 is Good for You - Greylisting Perspectives From the Early Noughties is © 2025 Peter N. M. Hansteen (published 2025-12-28)
You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

Separately, pre-orders of The Book of PF, 4th edition are now open. For a little background, see the blog post Yes, The Book of PF, 4th Edition Is Coming Soon (also here). The latest information I have is that physical copies should be ready to ship by the end of January 2026.


Friday, November 28, 2025

A Bus Ride and the (At Least) 3x UX FAILs

© 2025 Peter N. M. Hansteen

Norway is digital to a fault. That is why attempting to buy the ticket for a bus ride can reveal a cascade of user experience (UX) failures.

Most days, I either take a half hour stroll to get to my main customer's offices, or work from home. But occasionally, I need to visit my employer's offices. On those days, I take the bus for an easy 20-ish minutes ride.


Note: This piece is also available without trackers but classic formatting only here.
This week, a few meetings and an internal session on UX Design were scheduled at my employer's site, so after my usual morning routine of making coffee, feeding the cat and going over overnight mail and news, I got ready to head out to the office.

On the way out the door, I opened the tickets app from our local bus company Skyss on my Android phone, selected the single ticket option, went on as usual to select Vipps as the payment method and cleared the authentication steps before locking and putting the phone away.

Some unrelated alert buzzed on the phone and had me unlock it again, only to see that the payment had failed with an Unknown error message.

I had been on the way out the door when the transaction was initiated, so I suspected that perhaps the network change from my home WiFi to Telia 5G had somehow disrupted connectivity. That would be a rare occurence, but has happened.

So I tried completing the transaction again, only to get the same result. After a couple of more tries, the bus turned up and I got on my way.

So yes, I had technically taken a bus ride without paying. That means I in principle owe Skyss something like NOK 41.32 and would be at risk of getting fined something like NOK 950 if caught by the ticket inspectors without a proof of payment.

No inspectors turned up, however, so my day went on to some customer work performed remotely from the office, meetings and finally the main item of the day which was a short, compact, intensive but also quite interesting and inspiring session on UX design work.

The UX session concluded, and we went on to some socializing over pizza and refreshments.

Then, naturally, came the time for my bus ride back home. Once more I tried to purchase a single ride ticket via the app, only to be presented with the exact same error.

Unknown error.
And no way to get any details on what the actual error was.

So I got on the bus, again without completing a transaction, so my debt to Skyss would now have roughly doubled, and again I ran the risk of getting fined, should the inspectors turn up.

At this point my main suspect for the source of the failure was the Vipps app.

For context, the Vipps smartphone app is very close to being the default payment method in Norway, even more so for transactions involving online payments. Any failures or problems of any kind involving the Vipps service are almost guaranteed to make headlines with strongly worded articles and aggressively ugly comment threads.

So when I got back home, I opened the Vipps app on my phone, only to find that instead of its usual transaction UI I was presented with a question about whether I was a politically exposed person, with the options to answer basically, "Yes", "I was one previously", and "No".

But no way to bypass the prompt and perform a payment or other transaction.

The answer was obvious, but once I entered the answer, I was only taken to a screen with a single option, Update, presumably to update the app to a newer version.

Pushing the Update button took me and my Android phone to the Play store entry for the Vipps app, which offered the option to Open the app or to install it on my Android tablet in addition to my phone.

Choosing to to Open the app only took me back to the same single-option Update screen, in a perfectly circular progession.

So after failing to find any other option, I ended up uninstalling, then reinstalling the Vipps app.

Which of course involves a completely new setup. Fortunately (or perhaps worryingly from a privacy perspective), the app managed to connect itself to my main bank account, inferred from my national ID number, which is a required bit of information in the sign-up process.

So UX fail #1 was in the Skyss app, where the developers had apparently trusted the Vipps app to either never fail or at least fail in some obvious way, so displaying any information from Vipps was deemed not necessary.

UX fail #2 would likely go to the developers of the Vipps app, who seem to have assumed that users will only ever interact with their system directly, never through a third party app that uses Vipps as the payment back end. Or perhaps the Skyss developers screwed up their app's API interaction with the Vipps app, possibly hooking in the app when they really should have been talking to the Vipps back end instead.

Finally, UX fail #3 goes clearly to the Vipps team, who appear to have failed to test the sequence of events that will be triggered by their Update button in the app. Whatever they did test apparently did not involve any recent-ish Android phone from those too-big-to-fail Koreans.

While an Internet greybeard like myself was able to figure out that the app needed to be dealt some minor violence, I can only imagine the utter puzzlement any less (Internet) digital native senior citizen of actually pretty much the same age as myself would have experienced when met with this exact scenario.

Bonus Track: Adobe Does This Too, With AI

For the developers I have just chided for not doing their jobs properly UX-wise, there might be some consolation in knowing that they are not alone in producing UX failures.

Returning readers will be aware that The Book of PF, 4th edition is coming soon (also here), and we have reached the time when the thing is in the last rounds of proofing.

For reasons probably best explained by the publishers' production team, the application we use for final proofing and related annotations is Adobe's Acrobat. A few years ago I decided that macOS is BSDish enough that I will use it quite a bit, so installing the no direct cost version of the app on a system within reach was a fairly painless excercise. As was the initial proofing round and an intermediate one.

Then when the PDF for the final proofing cycle arrived, and I loaded the two hundred and fifty-some pages PDF, I discovered that Acrobat had acquired an AI Assistant component.

When the progress indicator showed that the file was ready to display for my final proofing round, the Acrobat AI Assistant oh-so-helpfully prompted me with

This looks like a long document. Would you like to see a summary instead?

Granted, my use case here is possibly not the typical one for a user of the gratis version of Acrobat.

But I will award Adobe the UX fail #4 bonus prize here, a UX FAIL because AI, for failing utterly to consider that some people do, in fact, create long-ish documents and prefer to see them in the full.


A Bus Ride and the (At Least) 3x UX FAILs is © 2025 Peter N. M. Hansteen (published 2025-11-28)
You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

Friday, November 7, 2025

What is BSD? Come to a conference to find out!

© 2025 Peter N. M. Hansteen

What is BSD? It's where the Internet comes from!

Berkeley Software Distribution (BSD) is a family of computer operating systems derived from the software developed at the University of California at Berkeley from the late 1970s through the early 1990s.

You may or may not be aware that the BSD code still powers a lot of things, and we meet up regularly for conferences. More about conferences later, first a little history to set the context.

A short history of the BSD operating systems

The history of the BSD family of operating systems is to a large extent the history of the Internet itself. You may have heard of the time back in the 1980s when the likes of IBM and Digital were slugging it out in the corporate IT sphere and the US department of defence paid for experiments in distributed, device independent networking.

That's when a loosely organized group of hackers somewhat coordinated by researchers at University of California's Berkeley campus rose to prominence with "BSD Unix", which by a sequence of happy accidents became the home of the reference implementation of the TCP/IP internet protocols.

By the early 1990s, commercialization of the Internet had started, and the Berkeley Computer Science Research Group (CSRG) that had coordinated the efforts was set to be disbanded. In addition to the net itself, the main tangible product out of Berkeley was the Berkeley Software Distribution (BSD), often distributed on tapes in the mail but also available on the net itself, which had started as a collection of software for AT & T's Unix but had over the years been extended become a full featured Unix operating system.

Several different groups wanted BSD to go on even if the CSRG did not, and several things happened in fairly rapid succession:

  • Lynne and Bill Jolitz ported BSD to Intel x86 (actually 80386sx), creating 386BSD. This was chronicled in a series of articles in Dr Dobbs' Journal (also see a more condensed summary over at salon.com)
  • Next up, hackers started sharing improvements to the 386BSD code as "patchkits", eventually forming two separate groups that took the work further to form their projects: The FreeBSD group would be working on bringing the best possible BSD to PC-style hardware, while the NetBSD group's ambition was to make BSD run on any hardware they could get their hands on.
  • A group of former CSRG employees formed BSDi Inc. and marketed their product BSD/386 with among other things a contact phone number "1-800-ITS-UNIX". The activities of an actual corporation in turn triggered a lawsuit from the owners of the UNIX trademark over code copyrights.

The lawsuit was eventually settled -- only six files of several thousand in the tree were 'potentially encumbered' and had to be replaced, leaving both NetBSD and FreeBSD with a rush to replace the code which was at least in part fairly central to the virtual memory subsystem.

That episode was however just a temporary setback, and by 1996 we also had OpenBSD, which forked off the NetBSD code base and formed the third main member of the BSD family, with a stated purpose to focus on security and correct code. Finally in 2003, the DragonFly BSD project forked off the FreeBSD code and became the fourth member of the family of open source BSD operating systems.

Code from the BSDs is widely used in Internet infrastructure and in numerous not too obvious contexts. In fact, all devices with TCP/IP Internet capability ran some derivative of the BSD code until alternative implementations started appearing during the early 2000s.

The likely most popular BSD variant is Apple's macOS, which shares a huge amount of code with the FreeBSD project. Modern BSD systems include DragonFly BSD, FreeBSD, NetBSD, OpenBSD, and, by some counts, Apple's macOS.

BSD code continues to be the base, however largely unsung, of significant technology development wherever the Internet is relevant. And you can even meet developers and practitioners at regional conferences every year!

The annual, regional BSD conferences

Most of the time, the development of the BSD systems is done by developers working by themselves or in small groups, in locations all over all inhabited time zones. However, by the early 2000s, a number of individuals in the various BSD communities started seeing the need for in-person meetups.

In addition to some projects calling up developers for hackathons, pioneered by the OpenBSD project, or developer summits, groups of interested parties including individual users and organizations started meeting up for conferences. The main regularly arranged BSD conferences are,

  • AsiaBSDCon, March timeframe, alternates between Tokyo (JP) and Taipei (TW). AsiaBSDCon 2026 will be March 19-22, 2026 in Taipei, Taiwan.
  • BSDCan, Mid May to mid June, Ottawa (CA). BSDCan 2026 will be June 17-20 in Ottawa, Canada.
  • EuroBSDcon, September timeframe, each year in a new European city. EuroBSDCon 2026 will be September 10-13 in Brussels, Belgium.

All three conferences will welcome submissions for talks, tutorials and other types of sessions as well as general participation by people regardless of geographic or other origin.

For further information, browse the conference websites.

We hope to see you there at future events!

Further reading

Explaining BSD on the FreeBSD documentation site

What every IT person needs to know about OpenBSD (part 1) at the APNIC blog site, continued in part 2 and part 3

DragonFly BSD project website

FreeBSD project website

NetBSD project website

OpenBSD project website


Historic project art follows: Left to right: OpenBSD, FreeBSD and NetBSD

puffy with so long and thanks for all the fish FreeBSD mascot Beastie as drawn by Poul-Henning Kamp The original NetBSD logo


Thursday, September 18, 2025

EU CRA: It's Later Than You Think, Time to Engineer Up!

© 2025 Peter N. M. Hansteen

On December 12 2027, it's already too late. The day before, the European Union Cyber Resilience Act (CRA) will have fully entered into force.

On December 11 2027, the Cyber Resilience Act is fully in force in the European Union member states and associated countries and territories.

From that date onward, suppliers of any "product with digital elements" are required to present those products along with a full overview and insight into all components and dependencies that went into making that product.

Unless, of course, you are a supplier that is fine with being considered at best second rate, or even being ineligible for lucrative contracts. Selling product that has not qualified for the CE mark for its product category will simply not do.

The European timeline for phased implementation of the CRA is outlined here, among other places.

Even if you are on the other side of the pond, you're not out of the woods. But more on that later.


Note: This piece is also available without trackers but classic formatting only here.

Upping Your Engineering Game

For individual developers, the question becomes something more along the lines of "Do you know what your code does?", or even "Do you know everything your code does?".

To put it bluntly, whether you answer to either of these is a clear yes or no determines whether you are just a coder or an engineer who codes.

The purpose of this session is to help you move towards becoming the latter. To start you upping your engineering game.

To set the stage for what real engineers (should) do and to keep focus on the importance of doing things right, the anecdote of the Canadian engineers' iron ring is a useful reference.

This all sounds a bit harsh, I know. So we will go a little softer at first, much like I did in my earlier article No Project Is an Island: Why You Need SBOMs and Dependency Management.

And yes, some of this will sound familiar if you have taken in that piece or participated in the live sessions based on the text.

Dear Developer, do you know what your code does?

So let's ask the question,

Dear developer, do you know what your code does?

Your answer is likely to be along the lines of

Sure, I wrote it all. I know what it does.

Unless you vibe coded the thing, that is. But let's leave that particular set of circumstances for another time.

The answer I wrote it all. I know what it does is, however, unlikely to be totally accurate. Unless you are doing extremely low level stuff and your code speaks directly to the hardware, your code more likely than not also pulls in and utilizes dependencies such as system calls and library functions that provide the foundation of functionality that makes the code you wrote work.

Knowing your dependencies and what role they plain in making your code work is a significant part of delivering proper quality. More on that later. First, we turn to a little history of software.

Just a Bit of Typing

Software is a relatively recent phenomenon. For a long time, you could credibly say most of its existence, software was poorly understood by society and industry at large.

There was a time -- and I am old enough to remember that time -- when software was considered a minor, somewhat irritating but necessary, component in IT deliveries.

On the more extreme end of things, you would occasionally hear that software was not at all important, literally just a bit of typing.

All the while it was ever more clear to developers and practitioners that the software was what made all that expensive hardware useful. But software was all ephemereal to most and in almost all cases the source code was secret, and the customer was expected to just accept whatever came you way as-is.

That perception changed over time, and during recent decades it is no longer in doubt that the software industry is just that, an industry in its own right.

But Then Suddenly Software Turned Important

Then, as some of us still remember, the Internet happened.

Few people realized it at the time, but this was the time in history when two important things happened at roughly the same time.

For one, it became obvious to developers at least that the infrastructure we all have come to rely upon owes its strength and resilience to the fact that it consists mainly of software that was built on standards built on rough consensus and working code, code that was open source.

The other thing was that software faced the full force of the entire world banging away at their keyboards.

Some of those keyboards were operated by people who intended to do bad things.

And eventually, bad things started happening.

Over the years, eventually enough episodes piled up that software security, sometimes discussed under other labels, started becoming an issue.

During the twenty-tens and -teens, we had several incidents where software bugs were tickled enough to lead to costly and embarrasing episodes. Some of these episodes were grave enough that the powers that be (the kind wearing suits) discovered that software was indeed something they needed to care about.

These episodes spurred several things, one being memes like

(XKCD #2347, please also read the explainer), which lead to the common belief that supply chain management and the subtopic dependency management is mainly a problem that concerns open source software.

This assertion is simply not true, in that no project is an island.

Whether you let others see the code you wrote or not, the software does not exist in isolation.

The XKCD comic struck a chord with open source developers, who at the time were a lot more in tune with the world of software dependencies than most other people.

Dependencies Became A Thing

There were several high profile and scary security incidents during the twenty-tens and twenty-teens. Some were due to exploitable and exploited bugs in open source code and dependencies, such as the log4shell incident involving a very popular logging library. This incident served to make it clear to C-level executives that dependencies were indeed a thing, and that their infrastructure was to a large extent made up of open source software.

At roughly the same time, the SUNBURST supply chain incident, which involved a popular piece of proprietary network management software that had been backdoored, demonstrated that even when the source code is kept secret, that is not sufficient protection against skilled adversaries.

These and other grave incidents made supply chain security an important new addition to our software security vocabulary.

No Project Is an Island

As I mentioned earlier, no project is an island.

Whether you let others see the code you wrote or not, the software does not exist in isolation.

Summing up so far,

  • We write software
  • Which depends on other software
  • Which interacts with other software
  • Which again interacts with other components (hardware, humans)
  • To run important stuff
  • Nothing exists in actual isolation – No project is an island

So what we do is important. What do we do about that?

Learn From Those Who Build Important Things

One way to handle the situation is to look at what other people who build important things do.

In other fields, the term Bill of Materials, or BOM for short, is a familiar term. The Bill of Materials is a document or set of documents that lists all component parts of a delivery.

This is the kind of document that becomes crucial in contexts where the procuring organization is geared toward accounting for everything and auditing when the supplier least expects it.

One such context could be when your organization has landed a contract to supply a backhoe, an armored personnel carrier or even a ship, and the contract requires you to specify component materials used, down to the nuts and bolts level.

For an example of the scale of things we are talking about, consider this ultra high level view of an item that was delivered to the UK Royal Navy, one aircraft carrier HMS Queen Elizabeth:

Aircraft carrier HMS Queen Elizabeth, exploded view

Your delivery would not be considered complete without the Bill of Materials or Manifest, even for a thing this size.

In practice, the BOM for the HMS QE and similar-sized projects would be a collection of BOMs with specifications for each of the multitude of component deliveries that make up the whole. Each supplier would be required to come up with a Bill of Materials for their delivery.

For physical deliveries to organizations of some stature, a Bill of Materials has been a standard part of the process across industries as an important part of quality assurance and a fundamental part of maintenance processes.

Software, on the other hand, has traditionally not been subject to that kind of scrutiny.

What Do Engineers Do?

In other fields of engineering, the process runs roughly like this:

You design your product, make detailed plans and descriptions of how to build the thing.

While planning and building, you keep track of all parts and components.

A Bill of Materials (BOM) for a pump that could well be a part of the HMS Queen Elizabeth could look like

Screenshot of a Bill of Materials (BOM) foar a boat pump, possibly part of a larger delivery

Your plans and design documents will likely undergo changes during product development and assembly.

For each delivery, you create a Bill of Materials that is a required and essential part of the delivery.

The Bill of Materials (BOM) lists all component parts, to the detail level required for running maintenance.

The BOM typically also references and serves as reference for maintenance documentation.

As an aside, it is likely worth noting that the US Department of Defense's need for structured text markup in processing inventory information such as bills of materials was one of the more important drivers, albeit not the only one, behind the creation of SGML, the direct precursor to HTML and XML.

Again, for a long time, this kind of engineering practice was not seen as a requirement for software deliveries.

Libre Software Has Package Management Already

Handling dependencies in software is not a new thing. You probably poke around for dependencies yourself when you start looking into a new project.

You will start looking into the source code files in your project, any libraries or tools needed to build the thing would be nice-to-knows. Once you have the thing built, it becomes interesting to know what other things -- libraries, suites of utilities, services that are required to be running or other software frameworks of any kind -- that are required in order to have the thing run.

So basically, any item your code would need comes out as a dependency, and you will find that your code has both build time and run time dependencies.

Those terms will be quite familiar to users and the developers of the package manager systems for the various open source operating systems. The very same items you would recognize from a listing of package dependencies in a package management tool will turn up in our Software Bill of Materials too. Depending on the specific tool and options you use, the SBOM could contain additional information that may not be entirely relevant in a package manager context.

Under any circumstances, with package systems in place, and even vulnerability scanners available to scan for unsecure code at rest or while running, the free and open source software communities were in fact well positioned for the legal requirements when they hit. Even more, the lessons learned from package management came in quite useful in meeting and satisfying the updated requirements.

Every free operating system, and in fact most modern-ish programming languages come with a package system to install software and to track and handle the web of depenencies. You are supposed to use the corresponding package manager for the bulk of maintenance tasks.

So when the security relevant incidents hit, the open source world was fairly well stocked with code that did almost all the things that were needed for producing what became known as Software Bill of Materials, or SBOM for short.

Introducing: A Software Bill of Materials (SBOM)

So what would a Software Bill of Materials even look like?

Obviously nuts and bolts would not be involved, but items such as the source code files in your project, any libraries or tools needed to build the thing would be nice-to-knows. And once you have the thing built, it becomes interesting to know what other things -- libraries, suites of utilities, services that are required to be running or other software frameworks of any kind -- that are required in order to have the thing run.

The information is there in our code, and with development tools and code scanners a developer is well placed to poke around.

The next challenge it to take that information and present it in a way that conforms with the legal specification and is presented in a way that is usable for stakeholders that are not developers.

In addition to module or package names and versions, the expected SBOM product will typically include information on any identified security problems such as CVEs and a specification of the licenses that apply to each of the identified dependencies.

Thanks in large measure to the open source heritage of the specifications and tools, both of the commonly used SBOM specifications (SPDX and CycloneDX) consider information on licenses used in a file or project as tagging and tracking relevant items.

The tools we describe have some measure of support for tracking and reporting on licenses in use. This can be useful for flagging licenses that may be mutually incompatible or even incompatible with your organization's business goals.

Several pieces of legislation emerged from the at times panic flavored fallout from the security incidents. Which ones are more relevant to you will become clear as we move on.

Depending on what parts of the world you care more about, the emphasis will either be on

So that's our backdrop for now.

Mainly (I think) due to coordinated lobbying by major players, both have rougly the same time frames for becoming formal requirements, with the EU Cyber Resilience Act (CRA) entering fully into force, with a CE mark scheme for digital products to be in place with the same deadline.

The name of the SBOM game is compliance with those legal requirements, and to not only generate the information -- that's the relatively easy part -- but also to present the information in a way that is understandable and actionable to stakeholders who are not themselves software developers.

We're Real Engineers Now, Sparky! We Have Tools

As I hinted at earlier, there are tools available for all of this. If you want to go on and explore for yourself, I would recommend going to the awesome-sbom site, which offers a curated collection of SBOM resources and tools hosted as a Github repo.

There are a large number of tools available, with varying feature sets. In addition to the free tools you find via that collection, several tool suites exist that are exclusively commercial or with free trial or reduced features set versions out with full features available only to paying customers.

The tool set I found the most accessible for my poking around was the combination of syft for generating SBOMs and bomber for display and presentation. The home pages for both are linked from the awesome-sbom collection.

As you can see from that page, there are several SBOM formats around, and to some extent standardization and interoperability efforts are under way. But enough of that, let's look at the actual tools in use.

Tools and How To Use Them

The tool set I found the most accessible for my poking around was the combination of syft for generating SBOMs and bomber for display and presentation. The home pages for both are linked from the awesome-sbom collection.

As you can see from that page, there are several SBOM formats around, and to some extent standardization and interoperability efforts are under way. But enough of that, let's look at the actual tools in use.

As a first step, it is instructive to point syft at the base directory of your project and see if it can tell you something you did not know already. syft supports a number of output formats, so if XML is the more readable format to you,

$ syft . -s all-layers -o cyclonedx-xml | xq

will give you pretty-printed XML (assuming you have xq installed) output of what syft found out. Do explore the various command line options for extracting various information about your project.

If you prefer JSON over XML, something like

$ syft . -s all-layers -o cyclonedx-json | jq

will give you readable JSON of the same information. Again, there are a number of options to explore.

If you want an html report with known vulnerabilities

$ syft . -o cyclonedx-json | bomber scan --provider ossindex --output=html

Note: For this particular command to work, you also need to supply provider login credentials (available with free registration), see the Bomber provider documentation.

Your SBOM, The Build Artifact

When you have explored a bit, you may want to look into how you incorporate these tools in your project and make the SBOM a build artifact.

The bomber documentation has this example suggestion for inclusion in a CI/CD pipeline:

# Make sure you include the - character at the end of the command.
# This triggers bomber to read from STDIN
syft packages . -o cyclonedx-json | bomber scan --provider ossindex --output json -

Note: For this particular command to work, you also need to supply provider login credentials (available with a free registration), see the Bomber provider documentation.

For your own projects you will tweak to taste, of course.

Your Tools May Already Have (Some of) This

More SBOM-savvy co-stakeholders in your project may even be capable of processing your json or xml formatted SBOMs themselves, using tools of their choice.

Your project and customer may already have chosen a different toolset, or you may find that some other SBOM generating and presentation tool set are better matches for your requirements.

It is in fact conceivable that you have SBOM-capable tools within reach in your environment already. The fairly popular images-and-sundry repository system Harbor supports automatic SBOM generation on image push by hooking in trivy for image scanning duty, should you choose to enable that feature for your Harbor hosted projects.

Track Your Dependencies On The Fly

In a real world scenario, I could imagine that non-developers would appreciate it if you supplement that line with one using the --output=html option. The HTML output provides a report that lists licenses involved before listing know vulnerabilites by severity and assigned CVE.

While I was writing this article, a colleague who had been reviewing it told me of an episode that shows that even extremely basic use of the SBOM tools can be useful. A customer had called, saying they needed a complete list of tools and dependencies involved in a project, and right away. As a first step, my colleague cd'ed in to the main directory of one of the subprojects for that customer, and issued the command

$ cdxgen .

and was rewarded with a bom.json file that listed somewhere in excess of three hundred dependencies for that relatively minor subproject alone. The customer was suitably impressed and granted my colleague a more realistic and less immediate time frame for submitting the full dependency tree.

There Is More

If you want to explore further, please dive into the resource references at the end here.

For the more Bill of Materials savvy developers who want to explore even more, it may be of interest that the OWASP and SPDX teams are working on more specialized BOM variants, including

  • OBOM (Operating system Bill of Materials)
  • SaaSBOM (Software as a Service Bill of Materials)
  • CBOM (Cryptography Bill of Materials)
  • AISBOM (Artificial Intelligence Bill of Materials)
and several more. Again, see the referenced resources at the end here and follow the breadcrumbs.

Now It's Your Turn: Get The Tools

Now it's your turn to go exploring. The first item is to get the tools installed.

If you haven't already, go to the home pages of each:

And follow the instructions on how to install for your environment.

The exact steps to install depends, of course, on your platform.

If you are running a recent Linux distribution, you more likely than not have them within reach via your package system. Failing that, or if you happen to be on macOS or a supported Linux, the command

$ brew install $toolname

where the value of toolname expands to the name of the tool you want will get you there.

There are even instructions for Microsoft systems at some of the tools' home pages.

If none of these methods work, do a git clone of the tool source code (you were going to do that anyway, right?) and follow the build instructions.

If necessary, tweak to get the thing to work. If you find you need to do something non-trivial to make the tool build and run on your system, consider submitting a pull request to the project.

Tools in Hand, Dig Into a New Project

Now that you have to tools installed, it is time to put them and your own skills to work on some actual source code.

If you have the source for a project you are already familiar with available, or a project you are interested in exploring, choose that. Otherwise, find something you're interested in on Github or somewhere else public.

Once you have a local copy of the codebase, go to that directory.

Once there, start with

$ cdxgen .

then watch the output (it may be useful to run commands like these in a script(1) session so you can look up what happened in the script file later), and act upon it.

Be prepared that there may be issues in the code that needs fixing or some dependency that you were not aware of.

Then look up the cdxgen, syft and bomber documentation to find out the following about your chosen code base:

  • What is the number of dependencies for this code base? How many direct dependencies? How many indirect ones (dependencies of dependencies)?
  • Does the code base itself have any known problems, reported as CVEs? How many for the dependencies?

If you are feeling a bit more ambitious, you could try checking out the tools themselves, and run the tools on those codebases:

Fetching cdxgen source code is as easy as

$ git clone git@github.com:CycloneDX/cdxgen.git

There may be some challenges ahead. If the result of your first session looks like this (an actual script session of cdxgen run on its own source), please do not let that discourage you. Those are problems to be fixed, and you are developer enough to do that, right?

for syft, the command is

$ git clone git@github.com:anchore/syft.git

and for bomber,

$ git clone git@github.com:devops-kung-fu/bomber.git

You may find other tools, via awesome-sbom or elsewhere, that fit your tastes or your projects better than those.

This is when the fun part starts.

Resources for Further Reading

Linux Foundation Training:
Automating Supply Chain Security: SBOMs and Signatures (LFEL1007) a short but information- and reference-filled introduction (free, requires registration, gives you a badge at the end)
Understanding the EU Cyber Resilience Act (CRA) (LFEL1001) Focused on the EU CRA, gives an overview with lots of useful references, nominally a 1 hour course worth taking

The Software Bill of Materials home page at NTIA is the mother ship of SBOM documentation

Browse OWASP CycloneDX for all things about the CycloneDX specification and related tools, also their CycloneDX tool center

Browse the System Package Data Exchange specification (SPDX) for all things SPDX (supported by the Linux Foundation), including copious linked reference material

awesome-sbom is a curated list of SBOM tools and resources

EU residents will want to poke around the Cyber Resilience Act site for reference

Brewing Transparency: How OWASP's TEA Is Revolutionizing Software Supply Chains is a summary of recent work on OWASP Transparency Exchange API (TEA)

SBOM buyer’s guide: 8 top software bill of materials tools to consider is a readable overview of (some) SBOM tools

Olle Johansson's FOSDEM presentations are among several good SBOM talks at that conference (search the site for more)

Peter N. M. Hansteen: Open Source in Enterprise Environments - Where Are We Now and What Is Our Way Forward? (2022, also here) has some insights on how open source software plays a crucial role in enterprise environments and elsewhere

Peter N. M. Hansteen: No Project Is an Island: Why You Need SBOMs and Dependency Management (also here)

Peter N. M. Hansteen: EU CRA: It's Later Than You Think, Time to Engineer Up! (this article) (also here)

Peter N. M. Hansteen: EU CRA: It's Later Than You Think, Time to Engineer Up! (slides)