Showing posts with label The Book of PF. Show all posts
Showing posts with label The Book of PF. Show all posts

Friday, March 6, 2026

The Book of PF, 4th Edition: It's Here, It's Real

© 2026 Peter N. M. Hansteen

The Book of PF 4th edition, posed on the author's laptop

The long wait is over. Fresh copies of The Book of PF, 4th Edition arrived here today. Which means: I'll bring some to upcoming conferences!

When the email message with package tracking information turned up, I thought at first that this was yet another dirty, scammy, spammy phishing thing.


Note: This piece is also available without trackers but classic formatting only here.

It had been years since the last time I had had a UPS delivery, and since then I had only seen obvious scammy phishing things pretending to delivery messages that all in fact looked very much like the crummy fakes they turned out to be.

But this one looked subtly different, in that it had both my address and my somewhat unusual name correctly spelled.

In addition, the URLs were on close inspection all for actual UPS.com resources.

And finally, the shipment had a total weight of 10.4 kg, which sounded about right for the stack of The Book of PF, 4th Edition author copies I was expecting to receive soon after the print run was ready to ship.

Now, a few days later, I have just taken delivery of package, this time delivered to my door by a delivery man who was very excited to be delivering a box of books to their author. The box was slightly heavier than my regular backpack:

The box containing the Book of PF 4th edition author copies

and after taking the box inside, I was able to extract the 15 copies, as expected.

This is me, holding my first physical copy of the fourth edition to come out of the box:

Peter Hansteen and the first of the The Book of PF 4th edition author copies to come out of the box

This means that the process that started almost two and a half years ago is finally, tangibly complete. I have the shipment of author copies of The Book of PF, 4th Edition in hand.

Regular readers of this column as well people on relevant mailing lists will be aware, as will my followers on various social media, that people who ordered the book early have already started receiving their copies of the refreshed The Book of PF. And some have posted their (so far only positive as far as I am aware) reactions to their media of choice.

But for one reason or the other, my expected stack of author copies were not among the very early deliveries.

Among the likely reasons are the fact that I am located in Bergen, Norway, which, at roughly 62 degrees North and in The Other West Coast fjords country, is somewhat off the beaten track for high speed deliveries.

And as any good business would do, No Starch Press will put a priority on delivering the product to customers who put in preorders or were on the spot once the thing had been declared ready.

Now I can tell you all yet again, the thing exists, and I am delighted to see the actual physical copies.

I know some readers were expecting there to be an auction for the first signed copy like there was for the third edition.

I have to disappoint you there. Unfortunately, side effects of decisions made by politicians on the other side of the Atlantic have made it impractical to set up an equivalent auction this time around.

Instead, I would encourage readers who can afford to do so to make a direct a donation to the OpenBSD Foundation or to the OpenBSD Project directly, in order to support the OpenBSD project and the immensely valuable work they do for free software.

If you're more of a FreeBSD person, the FreeBSD Foundation will be happy to hear from you as well.

Once you have done the rounds of your favorite free operating system supporting foundation and have your (or your bosses') credit card back in its regular location, you can think forward to one or more of the upcoming regional BSD conferences.

The main ones this year are,

  • AsiaBSDCon, March timeframe, alternates between Tokyo (JP) and Taipei (TW). AsiaBSDCon 2026 will be March 19-22, 2026 in Taipei, Taiwan.

    This will include the first scheduled Network Management with the OpenBSD Packet Filter Toolset session of the year, on March 19th, 2026.

  • BSDCan, Mid May to mid June, Ottawa (CA). BSDCan 2026 will be June 17-20 in Ottawa, Canada.

    This, too, will have a Network Management with the OpenBSD Packet Filter Toolset session, on June 18th. Details to follow closer to the date of the conference.

  • EuroBSDcon, September timeframe, each year in a new European city. EuroBSDCon 2026 will be September 10-13 in Brussels, Belgium. The call for papers is open from March 1st, 2026 and runs until June 20th.

All three conferences will welcome submissions during their Call for Papers period for talks, tutorials and other types of sessions as well as general participation by people regardless of geographic or other origin.

And unless truly unexpected events take place, I will be at all three this year, bringing copies of The Book of PF, 4th edition.

I look forward to seeing you at these events.

If you are interested in coming to a regional BSD conference as a speaker, volunteer or other participant, see What is BSD? Come to a conference to find out! (also here) for some information.

Through all of those events, our resident philosopher took it all with her usual calm:

Our residen philosopher, taking all that book stuff with her usual calm

If you want to get the book, and using the link to buy from No Starch Press is not practical, I would encourage you to support your local bookstore and see if they can order it in.

The Book of PF, 4th Edition: It's Here, It's Real is © 2026 Peter N. M. Hansteen (published 2026-03-06)
You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

Friday, November 28, 2025

A Bus Ride and the (At Least) 3x UX FAILs

© 2025 Peter N. M. Hansteen

Norway is digital to a fault. That is why attempting to buy the ticket for a bus ride can reveal a cascade of user experience (UX) failures.

Most days, I either take a half hour stroll to get to my main customer's offices, or work from home. But occasionally, I need to visit my employer's offices. On those days, I take the bus for an easy 20-ish minutes ride.


Note: This piece is also available without trackers but classic formatting only here.
This week, a few meetings and an internal session on UX Design were scheduled at my employer's site, so after my usual morning routine of making coffee, feeding the cat and going over overnight mail and news, I got ready to head out to the office.

On the way out the door, I opened the tickets app from our local bus company Skyss on my Android phone, selected the single ticket option, went on as usual to select Vipps as the payment method and cleared the authentication steps before locking and putting the phone away.

Some unrelated alert buzzed on the phone and had me unlock it again, only to see that the payment had failed with an Unknown error message.

I had been on the way out the door when the transaction was initiated, so I suspected that perhaps the network change from my home WiFi to Telia 5G had somehow disrupted connectivity. That would be a rare occurence, but has happened.

So I tried completing the transaction again, only to get the same result. After a couple of more tries, the bus turned up and I got on my way.

So yes, I had technically taken a bus ride without paying. That means I in principle owe Skyss something like NOK 41.32 and would be at risk of getting fined something like NOK 950 if caught by the ticket inspectors without a proof of payment.

No inspectors turned up, however, so my day went on to some customer work performed remotely from the office, meetings and finally the main item of the day which was a short, compact, intensive but also quite interesting and inspiring session on UX design work.

The UX session concluded, and we went on to some socializing over pizza and refreshments.

Then, naturally, came the time for my bus ride back home. Once more I tried to purchase a single ride ticket via the app, only to be presented with the exact same error.

Unknown error.
And no way to get any details on what the actual error was.

So I got on the bus, again without completing a transaction, so my debt to Skyss would now have roughly doubled, and again I ran the risk of getting fined, should the inspectors turn up.

At this point my main suspect for the source of the failure was the Vipps app.

For context, the Vipps smartphone app is very close to being the default payment method in Norway, even more so for transactions involving online payments. Any failures or problems of any kind involving the Vipps service are almost guaranteed to make headlines with strongly worded articles and aggressively ugly comment threads.

So when I got back home, I opened the Vipps app on my phone, only to find that instead of its usual transaction UI I was presented with a question about whether I was a politically exposed person, with the options to answer basically, "Yes", "I was one previously", and "No".

But no way to bypass the prompt and perform a payment or other transaction.

The answer was obvious, but once I entered the answer, I was only taken to a screen with a single option, Update, presumably to update the app to a newer version.

Pushing the Update button took me and my Android phone to the Play store entry for the Vipps app, which offered the option to Open the app or to install it on my Android tablet in addition to my phone.

Choosing to to Open the app only took me back to the same single-option Update screen, in a perfectly circular progession.

So after failing to find any other option, I ended up uninstalling, then reinstalling the Vipps app.

Which of course involves a completely new setup. Fortunately (or perhaps worryingly from a privacy perspective), the app managed to connect itself to my main bank account, inferred from my national ID number, which is a required bit of information in the sign-up process.

So UX fail #1 was in the Skyss app, where the developers had apparently trusted the Vipps app to either never fail or at least fail in some obvious way, so displaying any information from Vipps was deemed not necessary.

UX fail #2 would likely go to the developers of the Vipps app, who seem to have assumed that users will only ever interact with their system directly, never through a third party app that uses Vipps as the payment back end. Or perhaps the Skyss developers screwed up their app's API interaction with the Vipps app, possibly hooking in the app when they really should have been talking to the Vipps back end instead.

Finally, UX fail #3 goes clearly to the Vipps team, who appear to have failed to test the sequence of events that will be triggered by their Update button in the app. Whatever they did test apparently did not involve any recent-ish Android phone from those too-big-to-fail Koreans.

While an Internet greybeard like myself was able to figure out that the app needed to be dealt some minor violence, I can only imagine the utter puzzlement any less (Internet) digital native senior citizen of actually pretty much the same age as myself would have experienced when met with this exact scenario.

Bonus Track: Adobe Does This Too, With AI

For the developers I have just chided for not doing their jobs properly UX-wise, there might be some consolation in knowing that they are not alone in producing UX failures.

Returning readers will be aware that The Book of PF, 4th edition is coming soon (also here), and we have reached the time when the thing is in the last rounds of proofing.

For reasons probably best explained by the publishers' production team, the application we use for final proofing and related annotations is Adobe's Acrobat. A few years ago I decided that macOS is BSDish enough that I will use it quite a bit, so installing the no direct cost version of the app on a system within reach was a fairly painless excercise. As was the initial proofing round and an intermediate one.

Then when the PDF for the final proofing cycle arrived, and I loaded the two hundred and fifty-some pages PDF, I discovered that Acrobat had acquired an AI Assistant component.

When the progress indicator showed that the file was ready to display for my final proofing round, the Acrobat AI Assistant oh-so-helpfully prompted me with

This looks like a long document. Would you like to see a summary instead?

Granted, my use case here is possibly not the typical one for a user of the gratis version of Acrobat.

But I will award Adobe the UX fail #4 bonus prize here, a UX FAIL because AI, for failing utterly to consider that some people do, in fact, create long-ish documents and prefer to see them in the full.


A Bus Ride and the (At Least) 3x UX FAILs is © 2025 Peter N. M. Hansteen (published 2025-11-28)
You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

Friday, July 11, 2025

Yes, The Book of PF, 4th Edition Is Coming Soon

© 2025 Peter N. M. Hansteen

Long rumored and eagerly anticipated by some, the fourth edition of The Book of PF is now available for preorder

The Book of PF, 4th edition front cover, placeholder

This week it was finally time to announce, to the fediverse and to mailing lists, that there is a new edition of The Book of PF in the works, and preordering is now enabled.


Note: This piece is also available without trackers but classic formatting only here.

A few questions immediately pop into readers' minds on hearing this news. The ones I get most often are,

Why now? What took you so long?

which quite frequently combines with

What changed? Are previous editions now useless?

I'll address both after repeating what I said in the email announcements:

The fourth edition was written to bring the text into sync with the realities of the modern Internet, seen from the perspective of someone working with OpenBSD 7.8 or FreeBSD 14-STABLE.

The structure and chapter titles will be recognizable to readers of the previous edition, with the content updated to reflect the realities of the modern Internet.

What happened was, for quite some time after the third edition was finished, there were essentially no user visible changes such as syntax changes in the configuration for OpenBSD PF.

The code was definitely being worked on, developers fixed bugs, introduced optimizations such as network stack wide improvements in multicore support. But user-visible changes other than likely performance improvements did not appear, so I saw no urgent need to make updates to the book.

During the years following the late 2014 publication of third edition, I went on giving talks and tutorials, and at some point I welcomed input and help from my present co-presenters of the Network Management with the OpenBSD Packet Filter Toolset tutorials, Max Stucchi and Tom Smyth.

Over a few revisions, the tutorial sessions became ever more OpenBSD centered, possibly because we were all focusing more on that system than the others. And of course, over time we made tweaks to the material we present at the tutorials in response to our own real world experiences and feedback from attendees and others.

This went on for some years, with the still moderately popular conference sessions each yielding incremental changes to the material. Then, COVID-19 put all physical conference activities on hold for the years 2020 and 2021. EuroBSDCon 2022 in Vienna (originally planned as the 2020 conference), was our first post-lockdown presentation and a well attended one at that.

We went on to do more sessions at BSDCan 2023 and EuroBSDCon 2023. During the post-lockdown period, one question started popping up ever more frequently in email, social media (direct messages, even) and in connection with the sessions themselves,

Are you working on a fourth edition?
more often than not accompanied by
I'd love to get this in a FreeBSD version, can you do that too?

My answers would be roughly, "No tech book writer will ever reveal what their current project is until they have a specific publication date set", and "Sure, I will look into what I can do about that for sometime later"

On my way back from EuroBSDCon 2023 in Coimbra, where these questions had of course come up again, even from the quite small group of attendees at our session, I decided I that after eight years it might be worth at least looking into what, if anything beyond incrementing version numbers, would make sense to do to produce a new edition.

So I set out to assess. The book had seen some light touch updates for its second printing that were not in my pre-production .odt files, so I set out with the freshest .pdf and started making annotations. After a little while the volume of annotations had grown enough that I found it more useful to transfer those annotations to a normal text file. That file was becoming something like an outline of what a fourth edition would look like.

So testing my own 8 year old work against modern OpenBSD and FreeBSD, and poking around for PF material in general, I noticed several things.

Since the third edition was written, NetBSD, prominently featured in that edition, had developed their own NPF packet filter subsystem and deprecated their PF port. While DragonFly BSD still had PF in their tree, it looked like their version was seriously out of date (as far as I could tell equal to roughly OpenBSD 3.6, released in 2004).

So concentrating on the two free systems I was anyway in daily contact with -- OpenBSD and FreeBSD -- made sense.

My notes of things that needed to be done swelled over the next few weeks. The revision notes work became my main activity on evenings and weekends for a while, and by late November I sent off that file as an attachment to a mail message to Bill at No Starch that started with,

Dear Bill,
I think a fourth edition of The Book of PF might have a reason to exist soonish.

I went on to explain that while we had not had major announcements in the packet filtering space during the past few years, quite a few incremental and larger changes had indeed happened. A lot more had happened in user visible PF matters on the FreeBSD side than on the OpenBSD side, but incremental changes had happened there too.

And as you could reasonably expect, the world around us had changed enough that in addition to introducing some new features, existing examples and the way we present the issues to the reader needed a refresh in order to be relevant to anyone working in or starting out with modern TCP/IP networks.

It took some weeks before the yes, we're on board for a fourth edition message came back. It is entirely possible that making an important business pitch just before Thanksgiving weekend is not an optimal thing to do, timing-wise.

But when the go-ahead came, I asked Henning Brauer, who had been very much involved with technical editing for the previous versions, and Kristof Provost, who does the major PF things on the FreeBSD side, to be my tech reviewers. Both accepted immediately.

Over the next few months intense editing and revising followed -- yes, I do make mistakes, and Henning and Kristof proved to be very good at catching them.

Now we are very close to having the final result. The fourth edition of The Book of PF focuses on PF on modern versions of OpenBSD and FreeBSD, with only minor mention of other platforms. The ports to Apple systems and Oracle Solaris are mentioned, but I decided early on to focus on the free systems for all examples. The FreeBSD parts have received significantly more attention than in previous versions, to the extent that we jokingly referred to the fourth edition as the be nice to FreeBSD edition.

NEWS FLASH 2025-12-05: Final proofs done 2025-11-30, the thing is now in production,

The editing process has taken longer than I had anticipated, but we are on track now to have copies in readers' hands some time in the second half of 2025. I hope I will be able to bring physical copies of the fourth edition to Unfortunately physical copies will not be ready for us to bring to EuroBSDcon 2025 in Zagreb in September. There will be a revised and updated version of the Network Management with the OpenBSD Packet Filter Toolset session there, a full day tutorial starting at 2025-09-25 10:30 CET. You can register for the conference and tutorial by following the links from the conference Registration and Prices page.

So summing up,

The fourth edition was written to bring the text into sync with the realities of the modern Internet, seen from the perspective of someone working with OpenBSD 7.8 or FreeBSD 14-STABLE.

The structure and chapter titles will be recognizable to readers of the previous edition, with the content updated to reflect the realities of the modern Internet.

If you have actually read this far, there is a good chance you might be interested in the book, the tutorial, or both. I welcome your comments and input.


If you found this piece to be useful, informative, annoying or would for some other reason like to contact me or comment, please do.

You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).


Wednesday, May 7, 2025

For Upcoming PF Tutorials, We Welcome Your Questions

© 2025 Peter N. M. Hansteen

A good tutorial should sound to passersby much like an intense but amicable discussion between colleagues.

In a little over a month, I'll be heading out to Ottawa to attend BSDCan 2025, to help run the conference and to give this year's Network Management with the OpenBSD Packet Filter Toolset tutorial, with my co-presenters Max Stucchi and Tom Smyth.

I've been a regular at BSDCan since 2006, attending every year since except 2008 -- I wanted to go that year too, but other business (actually the business of getting out of a company I'd helped build) kept blocking my preparations even though I had a fresh book out with the first edition of The Book of PF published late 2007. The most recent edition, the third fourth, is available from here and from other good book sources.

Note: This piece is also available without trackers but classic formatting only here.

But I've kept coming back after that, and I've almost always given the PF tutorial at BSDCan, this year again there is a PF tutorial, Network Management with the OpenBSD Packet Filter Toolset, which is likely to follow the format from the previous years.

The format of the session is a sequence of short, intensive lectures interspersed with lab excercises. You can get a reasonable idea of what to expect by looking at the slides from the last sesssion. It can be worth noting that future updated versions of the slides will be available in the same location.

But we're still weeks away from the session, and we are in the period of time when we are making revisions and adjustments. So this is a good time to tell us what topics you would like to see us cover in the next session.

If you have one or more questions or suggestions, please do send them on to us at questions@pftutorial.net, preferably after taking a peek at the slides from the last sesssion.

We will consider each carefully, and if your question or suggestion turns into a new or changed topic, we will give credit where credit is due.

We are looking forward to seeing you in Ottawa in June!

Check out the BSDCan 2025 site for more information on the conference.

If you are unable to attend BSDCan, all is not lost: the EuroBSDcon 2025 conference is still accepting submissions for papers and tutorials, so if you have an interesting BSD-related topic you want the world to know about, your submissions will be welcome at the EuroBSDcon submissions system, where the deadline is 2025-06-21, or June 21st, 2025 (full disclosure: I'm on the program committee). This year's conference is set in beautiful Zagreb, Croatia in late September.


You might also enjoy

A Few of My Favorite Things About The OpenBSD Packet Filter Tools (2022)
The Hail Mary Cloud And The Lessons Learned (2013 - 2024)
What every IT person needs to know about OpenBSD (2021)

Saturday, October 25, 2014

The Book of PF, 3rd Edition is Here, First Signed Copy Can Be Yours

Continuing the tradition started by Michael Lucas with the Absolute OpenBSD, 2nd edition auction, I will be auctioning off the first signed copy of the Book of PF, 3rd edition.

Updated - the ebay auction has concluded, final bid was US $3,050.00 - see below
 
Today I took delivery of two boxes full of my The Book of PF, 3rd edition author copies. They are likely the first to arrive in Norway as well (a few North Americans received their copies early last week), but of course this is somewhere in the range hard to impossible to verify.

Anyway, here is the long anticipated with book selfie:


(larger size available here)

The writing process and the subsequent editing and proofing steps that you, dear reader, will know to appreciate took significantly longer than I had expected, but this edition of the book has the good luck to become available just before the release of OpenBSD that it targets. My original plan was to be in sync with the OpenBSD 5.5 release, but to nobody's surprise but mine the process took longer than I had wanted it to.

As regular readers will know already, the main reason this edition exists is that from OpenBSD 5.5 on, we have a new traffic shaping system to replace the more than 15 years old experimental ALTQ code. The book is up to date with OpenBSD 5.6 (early preorderers have received their disks already, I hear) and while it gives some hints on how to migrate to the new queues and priorities system, it also notes that ALTQ is no longer part of OpenBSD as of version 5.6.

And of course there have been various improvements in OpenBSD since 2010 and version 4.8, which were the year and version referenced in the second edition. You will see updates reflecting at least some of those changes in various parts of the book.

Even if you're not on OpenBSD at all, this edition is an improvement over previous versions, we've taken some care to include information relevant to FreeBSD and NetBSD as well, and where there are significant differences between the systems, it's noted in the text and examples.

It could have been tempting to include specific references to Apple's operating system as well, but I made a decision early on to stay with the free systems. I have written something about PF and Apple, but not in the book -- see my Call for Testing article How Apple Treats The Gift Of Open Source: The OpenBSD PF Example for a few field notes.

But now for the main item. For this edition, for a limited time only, there will be a

Book of PF Auction

You have a chance to own the first author signed copy of The Book of PF, 3rd edition.

The auction is up at http://www.ebay.com/itm/The-Book-of-PF-3rd-ed-signed-by-the-author-First-Copy-signed-/321563281902? - I'll look into extending the auction period, for some odd reason the max offered was 10 days. If your bid is not the successful one, I strongly urge you to make a direct donation of the same amount to the OpenBSD Foundation instead.

I've signed the book, and will fill in the missing spaces once we have the name and amount:




UPDATE 2014-10-26 01:00 CEST: Whatever it was that stopped ebay from listing the auction was resolved. The auction is up at http://www.ebay.com/itm/The-Book-of-PF-3rd-ed-signed-by-the-author-First-Copy-signed-/321563281902? - I'll look into extending the auction period, for some odd reason the max offered was 10 days. If your bid is not the successful one, I strongly urge you to make a direct donation of the same amount.to the OpenBSD foundation instead.

The first signed copy, and incidentally also the first copy my wife picked out of the first box we opened, will come with this inscribed in my handwriting on the title page:

FOR (your name)
Winner of the 2014 Book of PF Auction
Thank you for Supporting OpenBSD with your
(CAD, USD or EUR amount) donation

Bergen, (date), (my signature)

That's just for your reference. My handwriting is not a pretty sight at the best of times, and when you, the lucky winner, receive the book it's entirely reasonable that you will not be able to dechipher the scrawls at all.

If you think your chances of actually winning are not worth considering, please head over to the OpenBSD donations or orders page and spend some of your (or your boss') hard earned cash!

My speaking schedule has not been set for the upcoming months, but there is a reasonable chance I'll attend at least a few BSD events in the near future. See you there!

UPDATE 2014-11-26: The auction concluded on November 4th, with Bill Allaire as the successful bidder. He paid via PayPal (as you almost inevitably will on an Ebay auction) immediately, and I sent the signed book to him two days later.

As the lady at the post office said, the package took about a week to turn up in Bill's mailbox. But it took 21 days before PayPal finally made the funds available to me, and after a bit of wrestling with the possibly very intuitive (to someone else) PayPal interface, I transferrred the amount to the OpenBSD Foundation today. PayPal of course racked up fees both incoming and outgoing, to a degree that I think if the fees we paid there ar considered at all competitive, whoever coined the phrase "the giant vampire squid" to describe US banks must have been trying desperately to make the traditional US banks sound all nice and cuddly.

For the unsuccessful bidders, I urge you to head over to the OpenBSD Foundation's Donations page and make a donation equal to your highest bid.

Thursday, April 3, 2014

BSDCan Tutorials: Please Help Me Improve Your Experience

A good tutorial should sound to passersby much like an intense but amicable discussion between colleagues.

In a little over a month, I'll be heading out to Ottawa to attend BSDCan 2014. I've been a regular at BSDCan since 2006, attending every year since except 2008 -- I wanted to go that year too, but other business (actually the business of getting out of a company I'd helped build) kept blocking my preparations even though I had a fresh book out with the first edition of The Book of PF published late 2007.

Note: This piece is also available without trackers but classic formatting only here.

But I've kept coming back after that, and I've almost always given the PF tutorial at BSDCan, this year I'm branching out a bit to give two separate sessions:

Note: While this piece refers to past events, some points may apply to upcoming events too.

Building the Network you need with PF, the OpenBSD Packet filter

and

Transitioning to OpenBSD 5.5

Both sessions have been allocated 3 hour slots, and they also share another characteristic: I've invited my attendees to send me an email about what they're interested in learing during the tutorial. The main reason I do that is that I want to improve the experience for you, my prospective tutorial attendee.

Let me give you a bit of background. I've been giving the PF tutorial in various forms quite a few times over the years, and at one point I'd accumulated enough useful PF material that writing a book about PF seemed to be a natural next step. There was always some material that did not quite fit the book format, but a lot stayed at least for a while in my tutorial slides.

Over time I ran into a bit of trouble with the fact that BSDCan tutorials are always 'half day' or 3 hours. My collection of slides and notes have tended to expand over time, partly as a function of more experience, and partly due to the sad fact that the other BSDs have been slow to adopt any post-OpenBSD 4.5 syntax changes and other innovations. At some conferences and events I've done the PF session as a sometimes bit overfull full day event, depending on the number of questions and amount of other interaction.

But this time around it's three hours only, and I think that's quite an opportunity to improve the experience. I have more than enough material, but I've found that I usually know next to nothing about the people who will attend the sessions, and people's backgrounds vary enough that it's sometimes hard to find common ground or even pick out at short notice which parts of the material actually fits the group. I've had groups where some attendees had barely used any BSD at all along with OpenBSD developers who committed updates to man pages during the session in response to my slides and remarks, and most levels in between.

So with a strict limit on time, I would very much like to tailor the event specifically to the people who will be attending and who have something of an idea of what they want to learn. So please send me that email (to tutorial@bsdly.net), and I probably will end up updating the dense mass of slides anyway, and after the session they will be put in the usual place for browsing at your leisure.

If the format works, it's likely I'll try the short and tailored approach again at future events.

But there's more. As it says in the Transitioning to OpenBSD 5.5 tutorial description, OpenBSD has been the source of a number of BSD innovations over the years, and OpenBSD 5.5 has several noteworty improvements: time_t is now a 64-bit value so time will not wrap anytime soon, we have a new traffic shaping system wrapped into PF and a clear path to replacing the once-experimental but now aging ALTQ, signify(1)-signed install sets and packages, and quite a few more bits. The relase page is filling out nicely at the moment.

It's likely that those changes alone cold be made to fill a 3 hour tutorial slot nicely, but once again I would very much like to shape the session to fit the needs of the people who are planning to attend, so it's likely that more general what to look out for when switching to OpenBSD style material will be useful too. And if you haven't already, your experience will be much improved if you prepare a bit. The OpenBSD FAQ and the website in general is a valuable resource, and Michael W. Lucas' Absolute OpenBSD, 2nd edition is a very good source of information.

This year's BSDCan will be the first time I do the Transition to OpenBSD N.m session (unless I do get a rehearsal run organized with some locals), but it's likely I'll try again at later events for whatever is the just released or soon to be released OpenBSD version. Things are shaping up nicely for OpenBSD 5.6 at the moment, but the details of that future release will be out of scope for the Ottawa session. So please do send me an email (to transition@bsdly.net) if you plan to attend the session, and I will do my best to tailor the tutorial to your needs.

For both sessions, my ambition is to have the tutorial sound like an intense, but amicable discussion among colleagues. I look forward to seeing you in Ottawa.

Update 29 April 2014: A few people have asked, and I answer: Even if you're not able to attend the BSDCan session, you're of course welcome to send me questions to indicate what you would like to see covered in the tutorials and in the slides I'll publish afterwards. I won't give a firm promise to cover every question, but I'm happy to hear from you.

In other things, the manuscript for the third edition; of The Book of PF -- which main reason to exist is the new traffic shaping system -- is complete, going through the various editing steps and will be available at a yet to be determined date in 2014. I will be updating here and through twitter, G+ and other channels once more detailed information is available.<

If you are unable to attend BSDCan, all is not lost: the EuroBSDCon 2014 conference is still accepting submissions for papers and tutorials, so if you have an interesting BSD-related topic you want the world to know about, please drop us a line (or even better a title, abstract and short biographical description) at submission@eurobsdcon.org (full disclosure: I'm on the program committee). This year's conference is set in beautiful Sofia, Bulgaria in late September.

You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).

In particular, returning readers may be interested in the new The Book of PF, 4th edition which is available now. See The Book of PF, 4th Edition: It's Here, It's Real and links in the article for more about the new edition.

Friday, April 6, 2012

If We Go One Attempt Every Ten Seconds, We're Under The Radar

The Slime Also Evolves: New bruteforce ssh attempts come in at 10 second intervals, and they keep going even if we block them. Is this the warmup to a new iteration of the Hail Mary Cloud?

Note: This post has been updated with a correction, see the end of the article.

Regular readers will remember the activities of the Hail Mary Cloud, which turned up in authentication logs with large numbers of unsuccessful ssh login attempts, apparently coordinated across a large number of source IP addresses and with any individual host in the attacker set making a new attempts at intervals of anything from several seconds to several minutes.


At the time, commentators took these activites either as an indication of a truly inspired idea from a brilliant mind (after all, avoiding detection is essential) or a token of almost unimaginable ineptitude or perhaps just an overdose of faith that if you keep going long enough, even extremely unlikely things will happen.

It's been a litte while now since we last saw the slow, distributed bruteforce attacks at work here at the BSDly labs (we've kept collecting data here), but one curious incident during the last week indicates that somebody, somewhere is still working on ssh cracking scripts that operate on fairly similar methods.

Bruteforce attacks can be fairly easy to detect and head off. In most cases the attacker comes in with a larger than usual number of login attempts in rapid succession from a single IP address, and with modern tools such as OpenBSD's PF packet filter, you can set up rules that use state tracking options to intercept. The phenomenon is common enough that the bruteforce avoidance section is one of the more popular parts of my online PF tutorial (and of course, a slightly expanded version is avavailable in The Book of PF).

I wouldn't publish or recommend anything that I haven't at least tried myself, so just to illustrate,

[Fri Apr 06 14:48:21] peter@skapet:~$ doas grep bruteforce /etc/pf.conf
table <bruteforce> persist counters
block log (all) quick from <bruteforce>
pass log (all) proto { tcp, udp } to port ssh keep state (max-src-conn 15, max-src-conn-rate 7/4, overload <bruteforce>

The PF rules on BSDly.net's gateway have something much like the published example. This means that a traditional bruteforce attempt will end up something like this:

[Fri Apr 06 15:30:38] peter@skapet:~$ grep 203.34.37.62 /var/log/authlog
Apr 5 17:42:36 skapet sshd[32722]: Failed password for root from 203.34.37.62 port 44936 ssh2
Apr 5 17:42:36 skapet sshd[32722]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:38 skapet sshd[26527]: Failed password for root from 203.34.37.62 port 45679 ssh2
Apr 5 17:42:38 skapet sshd[26527]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:41 skapet sshd[29912]: Invalid user db2inst1 from 203.34.37.62
Apr 5 17:42:41 skapet sshd[29912]: Failed password for invalid user db2inst1 from 203.34.37.62 port 46283 ssh2
Apr 5 17:42:41 skapet sshd[29912]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:43 skapet sshd[30349]: Failed password for root from 203.34.37.62 port 46898 ssh2
Apr 5 17:42:43 skapet sshd[30349]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:46 skapet sshd[25557]: Invalid user prueba from 203.34.37.62
Apr 5 17:42:46 skapet sshd[25557]: Failed password for invalid user prueba from 203.34.37.62 port 47495 ssh2
Apr 5 17:42:46 skapet sshd[25557]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:48 skapet sshd[5380]: Failed password for bin from 203.34.37.62 port 48087 ssh2
Apr 5 17:42:48 skapet sshd[5380]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:51 skapet sshd[23635]: Invalid user postgres from 203.34.37.62
Apr 5 17:42:51 skapet sshd[23635]: Failed password for invalid user postgres from 203.34.37.62 port 48658 ssh2
Apr 5 17:42:51 skapet sshd[23635]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:54 skapet sshd[2450]: Failed password for root from 203.34.37.62 port 49307 ssh2
Apr 5 17:42:54 skapet sshd[2450]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:56 skapet sshd[16673]: Failed password for root from 203.34.37.62 port 49910 ssh2
Apr 5 17:42:57 skapet sshd[16673]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:42:59 skapet sshd[17522]: Failed password for root from 203.34.37.62 port 50503 ssh2
Apr 5 17:42:59 skapet sshd[17522]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:02 skapet sshd[4633]: Invalid user mythtv from 203.34.37.62
Apr 5 17:43:02 skapet sshd[4633]: Failed password for invalid user mythtv from 203.34.37.62 port 51218 ssh2
Apr 5 17:43:02 skapet sshd[4633]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:05 skapet sshd[25728]: Failed password for root from 203.34.37.62 port 51849 ssh2
Apr 5 17:43:05 skapet sshd[25728]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:08 skapet sshd[10487]: Failed password for root from 203.34.37.62 port 52565 ssh2
Apr 5 17:43:08 skapet sshd[10487]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:10 skapet sshd[31156]: Failed password for root from 203.34.37.62 port 53264 ssh2
Apr 5 17:43:11 skapet sshd[31156]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]
Apr 5 17:43:13 skapet sshd[31956]: Invalid user mmroot from 203.34.37.62
Apr 5 17:43:13 skapet sshd[31956]: Failed password for invalid user mmroot from 203.34.37.62 port 53958 ssh2
Apr 5 17:43:13 skapet sshd[31956]: Received disconnect from 203.34.37.62: 11: Bye Bye [preauth]


And looking up the current contents of the table shows our new perpetrator has indeed been caught:

[Fri Apr 06 15:34:23] peter@skapet:~$ sudo pfctl -t bruteforce -vT show
91.197.131.24
  Cleared: ; Thu Apr 5 20:22:29 2012
In/Block: [ Packets: 1 Bytes: 52 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]
200.11.174.131 
Cleared Thu Apr 5 19:09:30 2012  
In/Block: [ Packets: 1 Bytes: 52 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]
203.34.37.62
Cleared: Thu Apr 5 17:43:13 2012
In/Block: [ Packets: 1 Bytes: 52 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]


The table data show us one more thing worth noting: All of these bruteforcers sent exactly one packet after they were blocked, and gave up right away when they noticed they were blocked.

On Sunday, April 1st 2012, I noticed an unusually high number of ssh login attempts coming from two Chinese addresses (58.214.5.51 and 61.160.76.123), amazingly persistent and for some reason they had not been caught by my bruteforce avoidance rules. Thinking I'd simply adjust my rate settings, I simply added those addresses to the table by hand and started looking at the authentication log versus my rule set. Then a little while later, I noticed that instead of just bowing out after blocking, these two kept going. (I also tweeted about this, however not accurate in all details, at the time)

A little later that same evening, the <bruteforce> table looked like this:

[Sun Apr 01 22:58:02] peter@skapet:~$ sudo pfctl -t bruteforce -vT show
58.51.95.75
Cleared: Sun Apr 1 22:05:29 2012
In/Block: [ Packets: 1 Bytes: 52 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]
58.214.5.51
Cleared: Sun Apr 1 14:06:21 2012
In/Block: [ Packets: 3324 Bytes: 199440 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]
61.91.125.115
Cleared: Sun Apr 1 03:10:05 2012
In/Block: [ Packets: 1 Bytes: 52 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]
61.160.76.123
Cleared: Sun Apr 1 14:07:08 2012
In/Block: [ Packets: 3262 Bytes: 195720 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]


The two hosts kept coming, at a rate of roughly one attempt every ten seconds, and apparently ignored the fact that they were blocked in the packet filter rules and would be getting connection refused errors for each attempt.

Looking at the log data (preserved here along with data from various other attempts from other sources in the relevant period), both hosts were busy trying to guess root's password from the time they started until they were blocked. When the block expired after 24 hours, they had both apparently proceeded down similiar lists of user names and were busy with rooter ):

Apr 2 14:10:06 skapet sshd[13332]: Invalid user rooter from 61.160.76.123
Apr 2 14:10:06 skapet sshd[13332]: input_userauth_request: invalid user rooter [preauth]
Apr 2 14:10:06 skapet sshd[13332]: Failed password for invalid user rooter from 61.160.76.123 port 46578 ssh2
Apr 2 14:10:06 skapet sshd[13332]: Received disconnect from 61.160.76.123: 11: Bye Bye [preauth]
Apr 2 14:10:14 skapet sshd[30888]: Invalid user rooter from 58.214.5.51
Apr 2 14:10:14 skapet sshd[30888]: input_userauth_request: invalid user rooter [preauth]
Apr 2 14:10:14 skapet sshd[30888]: Failed password for invalid user rooter from 58.214.5.51 port 47587 ssh2
Apr 2 14:10:14 skapet sshd[30888]: Received disconnect from 58.214.5.51: 11: Bye Bye [preauth]

They both kept going afterwards, at roughly the same rates as before. The host at 61.160.76.123 kept varying its rate and at one point sped up enough that it triggered the automatic bruteforce blocking.

After running a fairly familiar alphabetic progression through a list of supposed user names, the remaining host finally gave up during the first hour of April 3rd, by CEST time:

Apr 3 00:36:24 skapet sshd[30287]: Received disconnect from 58.214.5.51: 11: Bye Bye [preauth]
Apr 3 00:36:33 skapet sshd[27318]: Invalid user clodia from 58.214.5.51
Apr 3 00:36:33 skapet sshd[27318]: input_userauth_request: invalid user clodia [preauth]
Apr 3 00:36:33 skapet sshd[27318]: Failed password for invalid user clodia from 58.214.5.51 port 58185 ssh2
Apr 3 00:36:33 skapet sshd[27318]: Received disconnect from 58.214.5.51: 11: Bye Bye [preauth]

Before we go into further details, I have a question for you, dear reader: Did anything like this turn up in your authentication logs during the same rough time frame? If your logs show something similar, please drop me a line at (lightly obfuscated) peter at bsdly dot se.

It could be instructive to compare this last batch with the previous samples. The log format differs slightly, since the previous attempts were aimed at FreeBSD machines, while this last round was aimed at a single OpenBSD host.

The whois information for the two hosts (58.214.5.51 and 61.160.76.123) both point to Chinese networks, as far as I can tell in the same provice and possibly in the same city, Wuxi, which appears to be one of several Chinese tech cities.

The slow rate of the login attempts and the sequence of user names attempted are both similar enough to the earlier distributed attempts that it's possible this is a limited experiment by the developers of the previous bruteforcing malware. The rate of roughly one attempt per host per 10 seconds is a significant speedup compared to the previous attempts, and it fits in the interval where blocking due to the rate of connections would most likely produce an unacceptably high number of false positives.



It will be interesting to see what rate of incoming connection the next full scale attempts will be using. It is possible that the source addresses are somewhere close to the actual whereabouts of the malware developers, but at this point it's pure speculation.

At this point we can only keep watching our logs and make sure that our sshd configurations are the best possible shape. If you need up to date advice on how to configure and use SSH safely, you could do significantly worse than grabbing Michael W. Lucas' recent SSH book SSH Mastery.

Update 2013-04-25: Revisiting the data in preparation for my BSDCan 2013 talk (also to be featured or rather previewed at tonight's BLUG meeting), I realized that a trivial scripting error had lead me to draw false conclusions.

The total number of attempts is correct, but both the number of hosts involved and the number of user names attempted were seriously off. The two hosts I mentioned in the article were the most active, but actually a total of 23 hosts participated, trying for a total of 1081 user names. Full data available here.

It seems the Hail Mary Cloud had shrunk, but not completely vanished as I thought at the time.


Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

Sunday, June 5, 2011

How Do We Appropriately Celebrate The Arrival Of The 100,000th PF Tutorial Visitor?

A nice surprise may be in line for a new visitor, and you (yes, you) can help me pick the surprise.

In late 2004, I started working on the text for a user group lecture for the BLUG meeting scheduled for the the following January.

The original manuscript was in Norwegian, but after a rather successful and surprisingly well attended user group meeting, I wrote up an English version and posted both online. With some encouragement from Greg Lehey (I'd participated in the group of volunteer reviewers for his third edition of The Complete FreeBSD), I submitted a proposal to give a half day tutorial for the 2005 AUUG conference in Sidney.

The proposal was accepted, as were several of the followup submissions to other conferences, and via a sequence of conferences and some private sessions, the document kept developing. In early 2007 I started working on turning the manuscript into a usable book. As regular readers will be aware, the much revised second edition was completed during the second half of 2010, and even that version has recently been subjected to its first update thanks to the ongoing development of the OpenBSD operating system.

The original tutorial has kept attracting a relatively steady stream of new visitors from all over the world, even though I have not added any new material to the document since I started working on the book version. New material will, rather, find its way into slides for the next session (such as the most recent one at BSDCan 2011), or will be put in the queue for possible upcoming book material.

During periods when I have had little visible output to offer, it has been interesting to see that the documents attract visitors and the occasional comment or suggestion for improvement. Then a little while back, I realized that in a not too distant future, the number of unique host names or IP addresses that have visited the tutorial tree will roll past a hundred thousand (100,000).

That particular number is possibly only significant to me, I keep the count of unique hosts accessing mainly to get an idea how many people have looked at my work. The raw number of page hits for the same location (we don't have any numbers for the early days when it was hosted at a now-defunct ISP) is fairly close to one and a half million, but I feel that number is a rather pointless statistic.

But when visitor number one hundred thousand arrives, how should we celebrate? I'm inclined to try to identify and contact the lucky visitor and offer a prize of sorts, but I have not quite made up my mind what and how. I'll welcome suggestions sent to via email (to peter@bsdly.net) with a recognizable subject.

It is worth mentioning that neither the tutorial nor this blog directly generates any revenue for me. I did for a short time have Google-supplied ads running on both sites, but for reasons that have never been quite clear to me, Google chose to terminate my AdSense account a few days before my second USD 100 transfer was due.

Tuesday, November 9, 2010

The Book of PF, 2nd ed: It's Here!



Yes, it's that time of the year again -- we missed both Halloween and the OpenBSD 4.8 release, but hot on the heels of both, here it is:

The Book of PF, 2nd Edition is here, a box of author's copies turned up here just after lunchtime, and were taken well care of by Nora and Birthe.

This means, of course, that those of you who preordered will be receiving your copies shortly (mod the usual factors eloquently described by Michael Lucas here, the printer in my case is in Louiseville, Quebec), those who have reason to expect copies from my hoard here can rest assured that I'm taking them to the post office right after this. There's an illegible scrawl on some early pages, sorry 'bout that.

Better bookstores online and elsewhere will have it, or you could make it part of a bundle by ordering from the OpenBSD orders page. You will be going there for your six monthly fix anyway, won't you?

Upcoming events: The plans are not fixed yet, but you should expect me to turn up at BSD-themed events over the next few months. Look for announcements here, tweeted, or via the usual mailing lists.

NOTE: This article refers to the now outdated second edition, which has been superseded by The Book of PF, 3rd Edition, which covers changes up to and including OpenBSD 5.6. For the purpose of learning network technology in general and PF in particular, the significantly updated third edition is a better choice than the second edition. Also see the October 25, 2015 post about the arrival of my third edition author copies.

Saturday, December 6, 2008

A Small Update About The Slow Brutes

Slow and steady might actually do it, eventually.

The reactions to my December 2nd column hit me with a bit of surprise. The column was taken on by slashdot and Linux Today both, producing a largish number of page views, but only two clicks on my featured ads. But while my clickthrough rate is not particularly interesting to others, the comments to the columns sometimes are.

If you look at the comments at slashdot and elsewhere, most of the commenters most likely did not actually read the column in full or did not take the time to digest what it actually said, with some notable exceptions. And yes, there were others, some also wrote in via email with informed comment - thanks!

For the benefit of those who did not get the point the first time around, I'll try once more to explain what the observations are and what they may in fact mean.

A number of commenters offered well meant advice to use packages like fail2ban, denyhosts or a few others.

The common denominator for all of them is that they track single hosts that make a larger than usual number of connections or are the source of a number of failed logins higher than a certain threshold value over a set time period. I appreciate your concerns, but the subject of the column did not fit well with the way those the packages work.

In fact, a similar scheme was already in place at the site that provided the data. The machines that provided the ssh logs are FreeBSD ones (as the sharper ones have observed already, and the reasons may possibly be revealed over beer sometime), but any gateway under my control will run OpenBSD, and by extension, PF (and yes, there is a book you might want to order from one (North America) or the other (Europe and elsewhere) of the OpenBSD project's sites). For a quick fix of background, the online PF tutorial may be worth a look.

Anyway, the /etc/pf.conf at that site's gateway contains the lines

table <abusive_hosts> persist
block log quick from <abusive_hosts>

and

pass log (all) quick proto { tcp, udp } from any to any port ssh flags S/SA keep state \
(max-src-conn 15, max-src-conn-rate 7/3, overload <abusive_hosts> flush global)


Those lines provide a variation on the logic that those posters recommended. Essentially, any host that tries 15 or more simultaneous ssh connections, or come in at a rate of more than seven over the span of three seconds, will be added to the table <abusive_hosts>, and the block quick rule blocks any further access from those hosts. Yes, flush global means what you think it does.

This works at the network level. For a gateway with a potentially large number of hosts on either side, the success or otherwise of eventual authentication may not be relevant and may be better dealt with elsewhere. Anyway, at the time I started working on this column, the table <abusive_hosts> on the gateway contained only two hosts:

:~$ sudo pfctl -t abusive_hosts -v show
194.204.37.93
201.57.187.114

I keep offenders in that table for 24 hours only, I do not believe in the permanent bans that some commenters advocate. After all, there is such a thing as DHCP, and entire netblocks are reallocated with amazingly short intervals.

Anyway, looking at the authentication log on the gateway reveals how those hosts got added to the the table in the first place:

:~$ grep 194.204.37.93 /var/log/authlog
Dec 5 22:50:30 delilah sshd[15266]: Did not receive identification string from 194.204.37.93
Dec 5 22:50:37 delilah sshd[6106]: Did not receive identification string from 194.204.37.93
Dec 6 01:29:58 delilah sshd[30359]: Failed password for root from 194.204.37.93 port 47071 ssh2
Dec 6 01:29:58 delilah sshd[7293]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:29:59 delilah sshd[4395]: Failed password for root from 194.204.37.93 port 47296 ssh2
Dec 6 01:29:59 delilah sshd[27615]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:00 delilah sshd[12248]: Failed password for root from 194.204.37.93 port 47330 ssh2
Dec 6 01:30:00 delilah sshd[24579]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:01 delilah sshd[3434]: Failed password for root from 194.204.37.93 port 47380 ssh2
Dec 6 01:30:01 delilah sshd[32737]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:02 delilah sshd[11984]: Failed password for root from 194.204.37.93 port 47425 ssh2
Dec 6 01:30:02 delilah sshd[27059]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:03 delilah sshd[13345]: Failed password for root from 194.204.37.93 port 47459 ssh2
Dec 6 01:30:03 delilah sshd[1858]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:04 delilah sshd[12739]: Failed password for root from 194.204.37.93 port 47516 ssh2
Dec 6 01:30:04 delilah sshd[16843]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:05 delilah sshd[13796]: Failed password for root from 194.204.37.93 port 47564 ssh2
Dec 6 01:30:05 delilah sshd[16789]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:06 delilah sshd[628]: Failed password for root from 194.204.37.93 port 47602 ssh2
Dec 6 01:30:06 delilah sshd[6162]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:07 delilah sshd[2579]: Failed password for root from 194.204.37.93 port 47646 ssh2
Dec 6 01:30:07 delilah sshd[12461]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:08 delilah sshd[12725]: Failed password for root from 194.204.37.93 port 47685 ssh2
Dec 6 01:30:08 delilah sshd[29909]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:09 delilah sshd[16560]: Failed password for root from 194.204.37.93 port 47724 ssh2
Dec 6 01:30:09 delilah sshd[1690]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:09 delilah sshd[1600]: Failed password for root from 194.204.37.93 port 47771 ssh2
Dec 6 01:30:09 delilah sshd[28882]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:10 delilah sshd[29953]: Failed password for root from 194.204.37.93 port 47807 ssh2
Dec 6 01:30:10 delilah sshd[15349]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:11 delilah sshd[2962]: Failed password for root from 194.204.37.93 port 47845 ssh2
Dec 6 01:30:11 delilah sshd[27557]: Received disconnect from 194.204.37.93: 11: Bye Bye

and

:~$ grep 201.57.187.114 /var/log/authlog
Dec 5 23:55:30 delilah sshd[24338]: Did not receive identification string from 201.57.187.114
Dec 5 23:55:35 delilah sshd[23570]: Did not receive identification string from 201.57.187.114
Dec 5 23:59:58 delilah sshd[10216]: Invalid user raimundo from 201.57.187.114
Dec 5 23:59:58 delilah sshd[10216]: Failed password for invalid user raimundo from 201.57.187.114 port 35776 ssh2
Dec 5 23:59:58 delilah sshd[18515]: Received disconnect from 201.57.187.114: 11: Bye Bye
Dec 6 00:00:01 delilah sshd[17353]: Invalid user joan from 201.57.187.114
Dec 6 00:00:01 delilah sshd[17353]: Failed password for invalid user joan from 201.57.187.114 port 37570 ssh2
Dec 6 00:00:02 delilah sshd[30314]: Received disconnect from 201.57.187.114: 11: Bye Bye


which again shows that these were the old-fashioned, rapid-fire kind of bots. Looking a bit closer even reveals that they kept trying after they were put in the doghouse:

:~$ sudo pfctl -t abusive_hosts -vT show
194.204.37.93
Cleared: Sat Dec 6 01:30:11 2008
In/Block: [ Packets: 18985 Bytes: 835336 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]
201.57.187.114
Cleared: Sat Dec 6 00:00:02 2008
In/Block: [ Packets: 800 Bytes: 48268 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]


And of course there were no traces of those IP addresses or corresponding host names in the authentication logs in the machines where I have collected the data about slow bots. My best guess at why is that the gateway's IP address is at the low end of the routable range for that site. Note to bruteforcers: try working the Internet in reverse next time. (Not that it would help much here, but that's another story.)

The reason why I don't see much activity for other services is simply that those machines do not run all that many services, and only the services they actually run for the world's benefit are in fact available to the outside.

My log data shows a definite pattern, and the alphabetic progression points to a degree of coordination. The slow bots are, I theorize, operated by a botnet herder who has a large pool of compromised hosts available and who also believes that given enough time, sooner or later you will find a correct combination of user names and passwords for a given host. Statisticians tell me that assumption is in fact valid, at least to some extent.

By setting up the necessarily large number of attempts to come from a sizable number of hosts in either round-robin or pseudo-random order and intervals for each individual host's attempts in the several minutes to hours range, there is a very real possibility that the slow but determined campaign for control of any single system will drown in the noise.

It is useful to keep in mind that malware moved out of the hands of pranksters and vandals years ago. Mass destruction of systems or data might still make the occasional headline, but staying out of the limelight is likely to be a lot more profitable. Modern malware masters want their creations and charges to stay undetected. What we may be seeing right at this moment is that they have realized the herd may only be sustainable if they grow it slowly.


If you are interested in researching the phenomena I've blogged about, you're welcome to contact me directly for more information or raw data.


Note:
 A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

Wednesday, May 21, 2008

Fake Address Round Trip Time: 13 days

The results are in. Our adversaries really are mindless automata.

Regular readers will have noticed that I've been running a small scale experiment over the last few months, feeding one spammer byproduct back to them via a reasonably accessible web page. The hope was that I would learn a few things about spammer behavior in the process.

After collecting fake addresses in my domains generated elsewhere for a while, I started noticing that a short time after I'd put addresses on the traplist page, they would start appearing as To: addresses on what appeared to be spam message entries in my logs. After a while more, I was certain that the round trip time was down to a few days, but my notes did not include exact dates for when each individual harvested address made it into my spamtrap list. Meaning, of course, I had no way of telling just how fast the process is.

Time to cheat slightly, or, as they say in the trade, perform a controlled experiment. Instead of sticking to the original plan of strictly collecting addresses generated elsewhere and feed them back to the harvesters via the web page, I decided to deviate slightly and plant an address at a specified date, and maybe add another few for data points later. I did the obvious thing and on October 11th, 2007, I slipped put-here-2007-10-11@datadok.no into one of that day's batches of collected addresses.

Needless to say, my attention wandered from the project in the meantime. After all, in October there was still that book to finish, and after the book was done, other developments had my attention or at least drained my energy for quite a while. So it was only last Sunday it struck me that by now I should have at least some data on what actually happened. One other reason it was suddenly quite appropriate to sum up the data was that the datadok.no domain had been turned over to new owners, and it will likely move out of my sphere of responsibility in the near future.

So here's the result, the fake address round trip time:

$ grep put-here-2007-10-11@datadok.no /var/log/spamd
Oct 24 03:40:40 skapet spamd[20795]: (BLACK) 60.50.174.129:
<pepgyoygq@boisdelan.com> -> <put-here-2007-10-11@datadok.no>


That is, the first time my artificially inserted address was used as a spam target was thirteen days after I put it on the traplist page. Since then, something, somewhere, has tried

$ grep -c put-here-2007-10-11@datadok.no /var/log/spamd
300


to deliver email to our imaginary friend a total of 300 times. Data taken from the spamd in front of that domain's secondary mail exchanger, of course. As always, I would love to hear from you about any related experiences.

In upcoming columns we will see, er, actually I find myself with such a selection of tempting topics to choose from, it is really hard to decide what to cover next. But the next one will appear here shortly.


Update 2015-08-01: In a totally unrelated article, posted on the afternoon of July 24th, 2015, the string razz@skapet.bsdly.net appeared. it took only three days, two hours and forty-eight minutes (approximately) before my spamd(8) logged this attempt at delivering mail to that address:

Jul 27 20:16:01 skapet spamd[1520]: (GREY) 183.79.28.71: <esther1jomkoma1ej@yahoo.co.jp> -> <razz@skapet.bsdly.net>

Thanks to the script that also prepares the downloadable list of trapped IP addresses, I was alerted of this happening, and the address was duly added to the spamtraps list and the accompanying web page as part of the batch that took the list to its current count of 29135 entries.

From this accidental anecdotal evidence, we can conclude that the time from when random string containing an at sign appears on a web site to the time it's used as spam target has now shrunk to about three days.

The attempts seem to be a little less energetic this time though: greping the relevant logs turns up only 27 attempts at delivery. It's possible this is down to the fact that there are now so many more imaginary friends to choose from in that long list.



Note: The @datadok.no addresses were removed from the traplist in late 2008 when that domain was handed over to the care of a different organization.

Wednesday, May 14, 2008

Network devices that lie

Do some network devices lie about their config, or are they just talking past each other?

One of my tasks recently has been to start integrating the networks of two companies, because one of the companies bought the other.

At my end you would have guessed already that anything exposed to the Internet would be running OpenBSD, while at the other end they were strangely attached to products from a certain software marketer based in the north-western part of the USA. But for the IPSEC part, they had decided that a certain proprietary "Internet Security Appliance" would fill their needs, and at each of their locations they had identical appliances running for "VPN" purposes.

Therein lies a story. After a few working days of debugging my way to an almost working configuration, my main conclusion is, that it is quite likely that proprietary security appliance lies. That is, it will let its administrator choose a wide range of very specific configuation options, but will actually override them, at least if the device at the other end does not do the "I'm one of your kind" secret handshake.

Here's what it looks like in the logs on the OpenBSD end, the part that is supposed to be a negotiation but actually isn't and the options on the OpenBSD side matched the options given in the screendumps of the other side's GUI that I had to work with:
May 13 14:00:28 delilah isakmpd[13547]: attribute_unacceptable: 
ENCRYPTION_ALGORITHM: got DES_CBC, expected 3DES_CBC
May 13 14:00:28 delilah isakmpd[13547]: attribute_unacceptable: 
HASH_ALGORITHM: got MD5, expected SHA

and so on, through the various options, each time expecting what matched the screendumps, getting something else entirely, up until the inevitable
May 13 14:00:28 delilah isakmpd[13547]: 
message_negotiate_sa: no compatible proposal found

or as they say elsewhere, Do not pass GO.

Moving on from there to actual tunnel setup reveals troublesome vendor IDs and other strangeness. This could very well be an attempt at vendor lock-in, but then the conventional wisdom anyway seems to be that the only way to set up a VPN is to have identical devices at every endpoint. There is no good reason why it should be like that, but I may be discovering the less good ones.

The main reason I'm not naming names or stating flat-out that "this device lies about its configuration" is that my analysis is not complete yet. If you have data you want to share with me on this or similar experiences, I would love to hear from you, at peter@bsdly.net. It's likely I'll write a followup based on what I hear and what happens.

During my silent period blogging wise, some notable events did occur, and I may return to them in later posts.

UKUUG Spring 2008
In late March I went to Birmingham to attend the UKUUG Sping 2008 conference - with a PF tutorial much like the one in Riga in February and a refreshed malware talk. A number of very good talks, and a number of Perl Mongers there, notably Barbie, who told me about his "Understanding Malware" talk that actually overlaps quite a bit with my paper. The social parts were good too, and yes, in Birmingham it is possible to find Real Ale.

BSD Magazine #1 is out
Not too long after the UKUUG Spring Conference, the first issue of BSD Magazine was published. Readers of this blog may find one article to be slightly familiar, but there's a nice selection of articles and reviews, covering all the BSD systems and a wide range of topics. And I should mention, they are looking for articles for upcoming issues.

OpenBSD 4.3 is out
OpenBSD 4.3 was released on schedule on May 1st, and as usual those who pre-ordered got their copies early. As always, exciting new stuff as well as a number of incremental improvements. There's interesting stuff happening post-4.3 as well, watch undeadly for updates.

Next up: FOSS Aalborg, June 4th
Our Danish friends are kind enough to give us yet another excuse to come visit. A group of BSD and Linux people have been hosting good conferences in Copenhagen for years, last year's EuroBSDCon being one.

This time it's Aalborg's turn, with a one day conference on June 4th. Yours truly will be on the speaker list. I hope to see you there, I will be bringing a small stack of books.

Saturday, February 16, 2008

Riga, here we come, OpenBSD 4.3 on the horizon

On Wednesday, Riga is the place to be, if attending a PF tutorial session for free is how you want to spend a day. You may have missed the announcement (and the update), but no matter - on Wednesday, February 20th I will be giving a full day tutorial in Riga, Latvia. bsdly.net has a Riga event page which is the space to watch for updates about the event.

A few things about the event is settled, though: The content will have quite a bit in common with the book, and I will be bringing a some copies of in case you need to pick up one (and promo postcards if you just want a souvenir). And time allowing, we will be previewing some of the things we can expect to see in the upcoming OpenBSD 4.3 release.

OpenBSD 4.3-RELEASE is likely to be cut in a few weeks' time (with a likely release date May 1st), so if you want to get an idea what it is going to look like or even want to help track down and squash any remaining bugs, now is the time to grab a snapshot and start putting it through the paces. If you're a little bit like me and can spare some capacity for testing, you will enjoy the process.

The next chance to catch a PF tutorial by me will be at the UKUUG Spring 2008 conference (not a free as in beer event), unless you arrange something yourself. This year's events look like they'll be a lot more ad hoc than earlier, so there may be other events announced later. Keep an eye on the OpenBSD Events page and any updates here.

Also keep an eye out for the upcoming premiere issue of BSD magazine. It promises to have about 60 pages of BSD related material in the first issue. I have an article in there that I hope you will enjoy.

See you in Riga or some other time,

Thursday, December 27, 2007

A year ends; what to do next?

It's the end of a year already. The end of the year is among other things the traditional time for tallying up totals to see what the year brought and looking forward to the fresh year ahead. Now at this point the inner geek in me and probably you too rebels with Why should any arbitrarily chosen point in time be assigned that much significance, huh?, but let's face it, it's one convention we will just have to live with.

The past year included a number of events, some entirely expected like the formal beginning of the end of the corporation once known as Caldera (yes, I know, it's not quite over yet, and I've written about that earlier), some rather surprising like the recent EU brokered patents and specifications deal which apparently means that the Samba team and other interested parties will not only be given access to usable protocol specs, they will even be furnished with a list of what Microsoft believes to be their relevant patents. That at least puts a serious dent in that corporation's patent FUD capability.

Any pundit in the Microsoft/Linux/FOSS "watcher" crowd who left that one out of their year end summary pieces should consider themselves cautioned: You were not paying attention to what could be this year's most important single piece of news in our field.

The general picture of the IT field is rather one of vast crowds of users who simply want to get on with their lives. The typical user is weary of the seventeen and a half times a week ritual Microsoft malware scare, and doesn't really see any benefit in getting a new computer with Vista to slow it down, now that they've finally weeded out or gotten used to all the annoyances of Windows XP and the background noise of unwanted popups and spam.

Rather more depressingly for us in the FOSS field, the typical user wants to just get on with his or her life and is weary, too, of the constantly overhyped "solutions" IT types are peddling. Faced with < insert your favorite product selling point here > , the stock answer now is, "gimme a break, that's what the last one said too".

This goes for almost any selling point, including vastly improved security along any measurable axis, efficient spam killing (including avoidance techniques like greylisting), the lightweight while useable desktop, and during the past year Microsoft even made a credible attempt at taking "open standards" prisoner. There is clearly a lot of work to be done, and we need to find ways to do that work better and present it in ways that actually add to FOSS people's credibility.

That includes, in my view, finding better ways to handle the periodic squabbles over licenses such as the GPL vs BSD shouting matches. It is likely that I will return to that topic in a future column, if and when I find the time to write it properly.

In my own little corner of the world, the publication of The Book of PF, marked here by the arrival of the author copies, marked the end of a long process that consumed rather more time and resources than I had anticipated. Before those copies arrived I had some copies made for OpenCon which were auctioned off for amazing sums that were subsequently donated to the OpenBSD project (see undeadly.org for details). Even though Amazon.com now lists the book as due for release January 11th, I have confirmation that No Starch shipped all preorders before they closed for the holidays, and I know at least one correspondent who got a message from the UK arm of Amazon that his copy was on its way. I'm interested in hearing from you about the book, of course, even reports that it has arrived safely in your mailbox.

Now other opportunities beckon, and I promise that in the coming year I will be writing about developments, confidentiality agreements allowing. If there is anything specific you want me to write about, please let me know.

I give you all my best wishes for the new year.

PS I almost neglected to mention that the PF tutorial (the forerunner of the Book of PF) saw its visitor (unique IP address or host name) number 27,000 for the period we have log data for on December 24th.

Update 2015-04-02: The Book of PF is now in its third edition, and the link in this article has been changed to point to the more recent edition.