That grumpy BSD guy

SSH Mastery: A Very Welcome Addition to Any Unix User's Bookshelf

2012-01-22T18:30:00.005+01:00

The first paragraph of this book's afterword reads:

"You now know more about SSH, OpenSSH and Putty than the vast majority of IT professionals! Congratulations".

That claim will be true for any reader of SSH Mastery who has read the book up to that point and has incorporated at least some of the elements of the configurations it describes into their own environments.

"But why a book dedicated to a single command?", you might ask. Almost all Unixes and Unix-likes have incorporated OpenSSH, the free SSH that is developed as part of the OpenBSD project, and OpenSSH comes with excellent documentation in the form of several extensive man pages.

Well, that question in itself justifies this title's existence (there are in fact several programs in the OpenSSH suite), and readers familiar with Michael Lucas' work will appreciate hearing that his latest work is task-oriented and well written, covering anything from the basic secure shell access through to the peculiarities of setting up a virtual private network (VPN) using OpenSSH. An enterprising reader would be able to find all the information in this book or close equivalents using the OpenSSH man pages or other online sources, but this book provides a very concise guide to both the basics and some rather advanced concepts and provides you with the vocabulary and understanding that you will need in order to successfully navigate the man pages.

This book has several highlights, such as the very sensible and useful discussion of key based authentication and how to set things up for a passwordless existence, a number of suggestions on how to distribute and maintain both host keys and user keys as well as very readable and useful introductions to various kinds of tunneling, forwarding and proxying available using the OpenSSH tools.

In particular I enjoyed reading the description of SSH-based virtual private networks (VPNs) in Chapter 13. This is one of the most clearly written and useful treatments I've seen of that subject, and for many readers this chapter alone will be worth the price of the book or even considerably more.

The book very sensibly covers OpenSSH on OpenBSD, FreeBSD and Ubuntu Linux, and users who are compelled to use Microsoft Windows desktops will be pleased to hear that configuration and use information for Putty, the most popular and free SSH client for their environment, is included too everywhere it's relevant to the task at hand.

Before Michael W. Lucas' new title was released in January 2012, the most recent widely available book about the Secure Shell protocol (SSH) and applications that support it was an O'Reilly title dated 2005. So even with high quality documentation available via the manual pages, it was time for a new title on the subject.

This title conveniently grew out of one of Michael W. Lucas' other technical writing projects, the second edition of Absolute OpenBSD. The SSH chapter of that manuscript simply kept growing until it made sense to branch the text off to a separate book. This probably means that the treatment of SSH in the upcoming OpenBSD title will be slimmer again, but separating out the OpenSSH parts as a separate book with information for several different environments added makes sense because it makes high-quality information about important tools available to a larger audience.

I am convinced SSH Mastery is a title that Unix users and system administrators like myself will want to keep within reach on their Kindles or other ebook readers for a quick and convenient refresh of important concepts. If you're a student or learning your Unix skills, you will certainly find this to be a very handy guide that helps you
grasp both the basics of SSH and several advanced concepts that are hard to find well described elsewhere.

The ebook is available in several formats via Amazon and other ebook outlets, a printed version is planned but was not yet available at the time of writing (January 22, 2012).

Title: SSH Mastery: OpenSSH, PuTTY, Tunnels and Keys
Author: Michael W. Lucas
Publisher: Tilted Windmill Press (January 18, 2012)

You're Doing It Wrong, Returning Scoundrels

2011-10-29T16:02:00.009+02:00

The numbers are in. The slow dunces still don't get it.

After five days of activity and no wins on my machines, the Hail Mary Cloud moved on. That means we have yet another complete set of data to summarize and analyze. The numbers are:

A total of 4773 attempts, none of them successful, involving 338 distinctive source addresses, the most active host (109.237.210.147, according to whois located somewhere in the Netherlands, made 109, while at the other end of the scale 30 hosts made only a single attempt). The wannabe attackers attempted to access 944 different user names, the most frequently attempted user name by far was root, with several blocks of root-only accesses even during the otherwise purely alphabetical stage.

The current sample is too small to support any far reaching conclusions, but it is tempting to speculate that with only 338 hosts participating we are seeing an indication that their success rate is sinking (previous attempts counted a cople of thousand hosts), even though they may be at least partially succeeding in their secondary goal: avoiding detection. That success is partial at best, this blog post and the earlier ones pluss varied commentary at Slashdot are indications that at least some of us are paying attention to our logs.

Another few observations worth making: 1) I have still not seen any of these sequences aimed at my Internet-facing OpenBSD systems, only Linux and FreeBSD ones. 2) It's likely that the miscreants are directing their attempts at several targets at the same time, so this sample is only a tiny fraction of the whole.

Reports of similar activity are surfacing from elsewhere, but very few people appear to be willing to share their data. It is of course even possible that the earlier episodes generated enough noise that better password policies (or preferably key logins only policies) are now in place, frustrating the random password guessers' attempts.

Whether or not you have been seeing these sequences in you authentication logs, please do yourself a favor and study your logs every now and then. It might even be worth the trouble to set up some kind of log collection and analysis infrastructure. Europeans may have to consider the legal implications of storing logs in light of the Data Retention Directive, denizens of the great elsewhere would do well to check if any similar legislation applies.

Good night and good luck.

Broken link fixed, sorry. Also, of course this has been discussed earlier, most recently in this post, also in this one as well as A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009). Read those for further info.

Update 2011-11-06: Another round of attempts has started, see the data aggregation page for the November 2011 entries. Of particular interest, perhaps is the List of participating hosts, sorted by number of attempts.

Update 2011-11-06 part 2: A note over at the ISC, "New, odd SSH brute force behavior" linked here, generating some additional traffic. Commenting over there requires a login and the confirmation email appears to be delayed by greylisting, so I'll comment here instead: I would not call this a particularly new approach. We've been seeing these attempts on and off since we started noticing them sometime in 2008, and it's entirely possible that there have been earlier attempts that did slip in under our radars. Analyses based on data from other sites beside mine would be very welcome indeed.

Update 2011-11-20: They keep coming back, now again after taking a 9 day breather (or possibly poking elsewhere in the meantime). Data accumulating again at the Hail Mary Cloud Data Page, with notes on the most recent activity at the very end. Please do play with the data, there's hope yet that some useful insights are to be found.

You're Doing It Wrong, Or, The Return Of The Son Of The Hail Mary Cloud

2011-10-23T22:36:00.006+02:00

Do Linux system administrators still in this day and age run with PermitRootLogins yes in their sshd configurations? Do they also allow password logins? Do they ever attempt to keep their systems up to date and reasonably secure?

Apparently the answers are yes, yes, and no, at least for some. The evidence is slowly accumulating in the authentication logs on one of my servers, published via the The Hail Mary Cloud Data Page. There are several reasons why these attempts stand out, but it kind of helps that the number of users with sensible or indeed legitimate reasons for shell access to this particular server is quite limited.

I've ranted about this before, famously but not exclusively in a series of slashdotted and much-syndicated blog posts such as this one. For the TL;DR crowd, here's the summary:

If you're allowing root logins from the great elsewhere, you're doing it wrong.

If you've been allowing root logins from the great elsewhere, I wouldn't be surprised it's one or more of your boxes doing the distributed password guessing.

If you can't remember the last time you checked that your system is up to date and properly configured, you're doing it wrong.

So nothing really new to see here, it's only yours truly seeing his hope of never seeing this silliness repeated dashed, again.

If you're interested in background information about the Hail Mary Cloud phenomenon, please do read the previous posts (A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009) as well as the one referenced earlier.

Good night and good luck.

Update 2011-10-27: The alphabetic stage has started, see refreshed data for details.

Practical Packet Analysis Is Good Fun [Book Review]

2011-08-03T23:07:00.006+02:00

If you've wanted to take a peek at what network traffic really looks like, but put it off because it sounded all too complicated, a new book may be just what you need to get started. Practical Packet Analysis is an excellent network analysis introduction.

When Chris Sanders' Practical Packet Analysis (with the subtitle Using Wireshark to Solve Real-World Network Problems) was originally published in mid-2007, to generally very favorable reviews and quite respectable sales for a network book, the memory of Wireshark's predecessor Ethereal's eviction from the OpenBSD package system for too many unfixed security bugs in a short time was fresh enough that most of us in the OpenBSD contingent generally shrugged and moved on. The criticism of Ethereal centered on the fact that there was next to no separation between the packet capture part of the system (which needs to run with elevated privilege) and the analysis-related parts (that do not in fact need to). At the time I was also rather busy working on a book of my own (full disclosure: also a No Starch title), so I mentally filed Wireshark under 'things to get back to', possibly to look into reviving the OpenBSD port.

Nevertheless I was quite pleasantly surprised when No Starch Press contacted me a little while back and asked if I would like to have a review copy of Practical Packet Analysis, second edition. The book is a pleasant read and good fun, and in fact the program compiles from source fine on OpenBSD, but more about that later.

After a brief introduction that contains summaries of the numbered chapters and practical information such as where to get the sample packet capture files, Practical Packet Analysis leads in with a chapter called Packet Analysis and Network Basics, which introduces the reader through an introduction to network protocols in general and how packet sniffers fit in the general picture.

The chapter then goes on to present the classical ISO seven layer model and makes a credible attempt at mapping those to the slightly fewer layers of the TCP/IP stack before going through a discussion of common variants of networking hardware and touches briefly on classes of network traffic (multicast, broadcast and unicast).

The second chapter, Tapping into the Wire focuses mainly on how and where to position your network tap, introduces a couple of different hardware devices for the purpose and even touches on ARP spoofing as a tap technique before giving a first glimpse of how to display and interpret captured traffic using the Windows utility Cain & Abel, and ends with a summary that includes a flowchart to help decide on the most useful tapping technique for the task at hand.

The third chapter, Introduction to Wireshark, introduces the book's main tool, with installation instructions for the more common operating systems and an initial walkthrough of the graphical user interface. While the install instructions are not correct in all details (when installing from source, you do not actually need elevated privileges until you get to the concluding make install step), they should be sufficient to get most readers started with minimal fuss.

Finishing up the chapter with a section that moves quickly to Your First Packet Capture is a nice touch that emphasizes the practical approach that's typical of this book.

Chapter 4, Working with Captured Packets, walks the user through some highlights of the filtering, analysis and presentation features that makes working with the likes of Wireshark fun. Much of this functionality would be available or at least fairly familiar to a seasoned tcpdump user, but this walkthrough does illustrate that sometimes a graphical interface can be fun too.

The chapter also leads off with fairly weakly worded advice that packet capture and analysis are likely to be separate activities. Considering that Chapter 3 introduced Wireshark's ability to choose interfaces by point and click in a dialog box (indicating that the program runs with elevated privileges), I for one would have found a stronger admonition very helpful.

An inexperienced reader will likely want to view the packet capture of her own network traffic animated and in full color, so at this point or earlier it would have been useful to include a note that on Unixish operating systems, either of these three commands
$ sudo tshark -w - | wireshark -k -i - $ sudo dumpcap -w - | wireshark -k -i - $ sudo tcpdump -e -s 65535 -i <interfacename> -w - | wireshark -k -i -
will at least buy token security in that only the packet capture runs with elevated privileges, the analysis tools run as your regular user (and yes, <interfacename> is where the actual interface name goes).

Fortunately, for the rest of the book, the activities are firmly centered around the packet capture files collected by the author himself. Chapters 5 through 11 present a variety of specific traffic scenarios that showcase the analysis and presentation features (including various graphing options) that make Wireshark a useful tool.

There are several memorable moments to be found here, including a packet capture that demonstrates how a successful compromise of a Microsoft Windows system (getting a privileged command shell) could look like at the network level. There is a wide variety of examples, and most of them are clearly designed to nudge the reader into exploring further. A good selection of protocols and protocol features are explained with a learning by doing approach, but there is enough that's only hinted at to let an interested reader look up the Further Reading appendix to dive into the rest.

All in all Practical Packet Analysis, second edition stands out as a book that's a very useful learning resource, and one that makes the learning process a lot of fun. Seasoned network professionals will most likely not find much new material here, but the book is a good read for anyone with a networking interest and I'm pretty sure you'll enjoy the hours you spend leafing through it before you hand it over to your junior network admin or your students.

Title: Practical Packet Analysis, 2nd Edition - Using Wireshark to Solve Real-World Network Problems
Author: Chris Sanders
Publisher: No Starch Press, San Francisco
Published: July 2011
Pages: 280
ISBN: 978-1-59327-266-1
Price: USD 49.95 for print + ebook, USD 39.95 for ebook (PDF, Mobi, and ePub formats)

What to expect in OpenBSD 5.0 onwards

2011-07-19T16:59:00.013+02:00

With OpenBSD-current tagged as 5.0-beta it's time to take a closer look at the upcoming release and the processes that make the OpenBSD project work.

Before you start getting all worked up about an upcoming dot-zero release, I'll tell you right away: Don't. Or maybe just a little bit. Release numbering in OpenBSD simply does not work the way most people expect. For this upcoming release, slated for general availability via FTP and other download methods on November 1, 2011 (and CD in pre-orderers' hands up to about two weeks earlier), 5.0 was simply the next available version number increment. OpenBSD releases every six months like clockwork, with the version number incremented by exactly 0.1 each time.

That does not mean that there is nothing to be excited about this time around, only that the OpenBSD approach is about guided and well planned evolution rather than revolutionary changes where large chunks of code are thrown away and replaced with new, untested code with bugs to be explored and exploited until a future dot-something-else release is finally considered stable.

The snapshots, as always available from your friendly local mirror, turned 5.0-beta during the early hours of today (July 19th 2011) CEST, and upgraders will notice two significant improvements before they're done running the installer (which in most other respects is fairly identical to what I described in an earlier article).

The most visible set of changes are near the end of the process:

As the illustration shows, once the install sets are in place, the upgrade program will offer to run sysmerge(8), the program that's specifically designed to help you merge any required changes into your configuration files with minimum fuss and disruption. Once you have chosen to either run the merge or skip it, the upgrade program will offer to fetch updated versions of any non-free firmware detected on your system at first boot.

When that first boot is finished, you will find a number of improvements are in place. You can find an overview of the changes at on the Daily changelog page over at the OpenBSD web site. We'll get into some of the more significant ones in this article.

But before we go into details of those changes, let's take a look at how the OpenBSD development process works. The Goals page on the project web site lays out the main project goals, while the Security page goes into somewhat more detail about the OpenBSD approach to security (including code audits and general all-tree bug sqashing), with a very worthwhile Further reading section at the bottom of the page. If you want to go into even more detail (short of reading source code and commit logs), the Papers page has a large selection of presentations and papers by various OpenBSD developers.

There's a lot of good material linked from that page; among the more recent favorites are Damien Miller's Recent developments in OpenSSH, Henning Brauer and Sven Dehmlow's Puffy At Work - getting code right and secure, the OpenBSD way and Theo de Raadt's The OpenBSD release process: A success story.

The last one is Theo's own first-hand account of how the project has been able to consistently deliver high quality releases every six months since the project started more than a decade ago. It's well worth your time flipping through the entire presentation, but for an outsider the possibly most enlightening slides are the ones contrasting the traditional release process with the OpenBSD one.

In particular, pay attention to the two slides describing the OpenBSD release calendar, slide 16 and slide 17. If you do the math, you see every six-month cycle is divided into roughly four months of intense development followed by roughly two months of stabilization before the release is cut.

Anything that will not fit within four months of hacking without breaking the tree (leaving the system in an unusable state) will not make it in. Any large changes need to be carefully planned and introduced via a number of substeps, some purely preparatory, others introducing user-visible changes. One visible effect of this is that truly incompatible changes such as the NAT and redirection syntax change that occured in OpenBSD 4.7 (prompting among other things an extensive rewrite of a certain book) are exceedingly rare, and if you're capable of reading the commit logs, you will see that it took several years of preparation before the switch happened.

It's all about clear thinking and proper planning, and changes will be introduced gradually and as they fit into the overall system. With all this as the background, it makes sense that 5.0 is just another number. Now let's take a peek at what changed since OpenBSD 4.9 and why it makes sense for both enthusiasts and others to start testing snapshots.

The changes listed here are my particular favorites, what you yourself will consider important may be different, depending on your specific use case:

BIGMEM on by default on amd64 - this is literally a big one. The amd64 architecture has true 64-bit capability, but the ghosts of hardware designs past keep hauting us, with legacy devices that continue to require a lot of dark magic to be handled correctly and safely. Excpect Ariane van der Steldt (ariane@) to be presenting at EuroBSDCon with the gory details. This commit happened fairly early in the OpenBSD 5.0 cycle.

Disk UID (DUID) support in all storage related parts such as mount, and by extension fstab (and DUID-style fstab enabled by default for new installs), so disk device renumbering will not be such a headache the next time you add or remove disks.

The proxies (ftp-proxy(8), tftp-proxy(8)) now use divert(4) sockets for performance, meaning that your rdr-to rules for those proxies need to be rewritten to divert-to rules.

Rewrite of the old /etc/security (a shell script) to the new security(8) by Andrew Fresh and Ingo Schwartze, a much needed refresh of a crucial component.

Various IPv6-related improvements in PF and other parts of the networking code, meaning that traffic using the newer generation protocol is now a bit more manageable. Just like everywhere else, the IPv6-related work is still in the early stages of seeing full scale use, and more likely than not future releases will have more news in this area.

The beginning of the end for ALTQ signalled by the introduction of always-on priority queues available as PF per-rule options, as noted in a previous article. This is an example of the longer term, incremental and well planned change like the ones I mentioned earlier. The internals of queueing and traffic shaping in OpenBSD is about to change in the long run, an it's even conceivable that the current ALTQ grammar will at one point use the newer code, but existing ALTQ setups will continue to work in OpenBSD 5.0.

Further package system improvements Marc Espie (espie@) continues his refinement of the package handling with an overhauled pkg_delete(8) that has several important improvements, including the -a option (click the link to read the man page online).

The Daily changelog page contains a lot more changes than the ones I've highlighted here. Among the things I've mostly skipped are added support for new hardware and a host of platform-specific fixes and enhancements. And of course it's likely that the list of changes will grow visibly over the next few weeks via commits caused by you testing and user feedback.

The exact point in time when a release is cut and shipped off to production is never pre-announced. The best indicator is to look for the commit message that changes the -current version string back to N.m.-current again after a brief period as N.m. A little later, the OpenBSD Orders page will allow preorders, and if you get your order in soon enough, you'll have your CDs and other swag before the official release date.

In about six months' time, you will see blog posts and other news items announcing the change to OpenBSD 5.1-beta, and we will be gearing up for yet another OpenBSD release. In any case, the best way to support the project (that produces, among other widely used software, OpenSSH, more likely than not by a wide margin, your remote login system) in addition to contributing code, testing and direct donations is to go to the OpenBSD Orders page and order one or more items.

Update: The sysmerge run from upgrade feature was backed out in a last minute commit by deraadt@, but it's possible it will return in time for the 5.0 to 5.1 upgrade cycle.

Update 2011-09-09: Pre-orders have started, the 5.0 release now has both a release page and a detailed list of changes since previous release. Expect more details to emerge over the coming days and weeks, and please do go order some items to help fund further OpenBSD development!

Anticipating the Post-ALTQ World

2011-07-09T18:45:00.008+02:00

A peek into exciting new features in the upcoming OpenBSD 5.0 release and beyond, plus how to teach our favorite operating system better?

One of the more exciting pieces of news to come out of the Edmonton OpenBSD hackathon this week was that the face of OpenBSD traffic shaping is about to change.

This change is by no means the only news to come out of Edmonton (read source-changes@, either by subscribing or via public archives such as marc.info and, if you're at all BSD geekish, feel your excitement levels rise), but the new prio keyword that was added to PF grammar and announced via a message to the tech@ mailing list titled new small, fast, always on priority queueing is both a major new feature and a prime example of how the OpenBSD project works, avoiding sudden "revolutionary" steps in favor of well planned evolutionary steps. The stepwise approach ensures, among other things, that changes are properly tested, and is visible in the fact that the OpenBSD source tree is as close as doesn't matter to always being in a buildable state, even during hackathons.

ALTQ, the ALTernate Queueing subsystem, was integrated into the PF codebase for OpenBSD version 3.3 (mainly by the efforts of the very same Henning Brauer who posted the message linked in the previous paragraph), and from that release onward, managing your filtering and your traffic shaping via your pf.conf file has been a highly appreciated feature of OpenBSD and other systems that have adopted the PF networking toolset.

Although much appreciated by its users everywhere, the integration was never quite an easy fit and the developers have been privately discussing how the queuing should be rewritten to avoid the separateness of ALTQ compared to the rest of the PF features.

There have been other concerns too. Not all of the features outlined in Cho's original USENIX paper were ever fully implemented, and the code had begun showing its age in several respects. By amazing coincidence another mailing list thread appeared in the same week as the new queueing system was announced, pointing out that ALTQ's total bandwidth parameter is a 32-bit value, meaning in practical terms that the maximum bandwidth ALTQ will be able to manage is in the 4Gbit range. Just changing the relevant variables to a wider type apparently breaks the code in other interesting ways (if you're adventurous, you could try playing with FreeBSD developer Ermal Luci's patch (available among other places here and see what happens. The results could be hackishly interesting, but with the new queueing system set for gradual introduction, this may not be the optimal time to submit ALTQ patches.

Once again, the 32 bit value is a sign of the code's age. At the time ALTQ was originally written, a 32-bit value for bandwidth, and by extension an absolute cap at four gigabits per second, was a no less reasonable choice than the choice of 32 bits as the length of IP addresses some years earlier. And while we are still in the ALTQ world, the obvious workaround in settings where you have bandwidth that comes in increments of ten or forty gigabit is to do your filtering and traffic shaping at network locations where bandwidth is a bit more scarce. In the post-ALTQ world, things may become significantly different.

The early outline of the new world comes in the form of in-ruleset traffic classification, at the face of it not that different from ALTQ when seen from a user perspective. In the upcoming OpenBSD 5.0 release onwards, you will be able to skip the familiar ALTQ queue definitions, but still do traffic classification like this:
pass in proto tcp to port ssh prio 6
meaning that incoming ssh traffic will pass with a relatively high priority (the range is 0 through 7), hopefully making it through earlier than the rest.

The new scheme even lets you duplicate the old speedup trick of setting aside a separate subqueue for ACKs and other lowdelay packets, like so:
pass in proto tcp to port ssh prio (5, 6)
This has yet to reach downloadable binary snapshots, but if you have the required bits in place, you can get an early start by building your own OpenBSD-current. See the relevant pieces of The FAQ, supplemented by man release.

It is important to note that in the upcoming release, priority rules will coexist with the existing ALTQ infrastructure. Over time, however, the plan is to replace ALTQ with a largely rewritten system where the internal workings of the HFSC and CBQ queues, for example, are not all that different. We are also likely to see the total bandwidth definition for an interface move out ouf your pf.conf and re-emerge as an ifconfig option.

Once a traffic shaping infrastructure that's functionally equivalent to ALTQ (or it is hoped, superior in all ways including performance) is in place, ALTQ will eventually be removed. In the meantime, we have already seen commits that set default priority for certain traffic types, and reading source-changes@ along with testing snapshots will continue to be a fun and exciting experience in the weeks and months ahead.

Over time, it might even become necessary to do more book work, if so you will hear about it first here.

Whatever happens on the book front, I'll more than likely get back to those new features in more detail in upcoming PF tutorial sessions (See here, here and here for announcements about the ones scheduled so far), and I'm trying to figure out a way to improve the quality of the tutorial experiences.

The lecture plus demonstrations format of the tutorial sessions has so far allowed for very limited interactivity with little or no hands-on experience for attendees during the sessions themselves. I would be very happy to hear ideas for and input on practical implementation of changes that would improve the experience. One possibility is setting up a set of virtual labs, hosted somewhere reachable from the conference locations. Suggestions are welcome via email or the comments field.

Update 2011-07-10: The prio code is in snapshots dated July 10, 2011 or newer. If you're upgrading from a previous version, the installer will now also offer to run sysmerge(8) and it will even offer to upgrade any non-free firmware it finds installed on your system.

Update 2011-09-14: Chris Cappucio wrote in, adding some more useful detail to ALTQ's history. Chris writes:

I did the initial altq port to OpenBSD and Kenjiro Cho (author of ALTQ) was invited to the next hackathon (I believe it was c2k2) along with myself to integrate it into the tree. I declined to participate for personal reasons but Kenjiro did turn my patch into an in-tree implementation, one or two years later the PF guys replaced the ALTQ packet classifier and configuration utility with PF itself. Now they are replacing the ALTQ scheduler framework completely with a simpler framework that still uses the PF classifier and configurator :)

SEK 1995 for six months' worth of trademark protection?

2011-07-08T14:46:00.003+02:00

In a fit of rage, I went out and did something I wouldn't have even remotely considered doing just a few moments before. I'm now the proud owner of the bsdly.se domain.

Does somebody out there really want to register bsdly.se? If they did, it's probably too late by now.

My friends all know that I'm not too fond of talking on the phone, and trying to sell me anything by cold-calling is just never going to work. So when somebody calling themselves "Nordic Domain Hosting", calling from +46 406660225 (a Swedish unlisted number as far as I can tell) did just that, it had to end badly.

The lady at the other end claimed that somebody was trying to register the bsdly.se domain, and seeing that I was the owner of doubleU-doubleU-doubleU-dot-bee-ess-dee-ell-why-dot-enn-ee-tee, would I be interested in blocking the attempt? It would just be a matter of 'trademark protection', and it would cost me the mere trifle of SEK 1995 (roughly USD 310 or EUR 218 at today's rate).

I told them right off that I wasn't terribly interested in owning a Swedish domain, but the lady they tried to talk me through a series of yes/no answers I assume were designed to set up a legal-sounding agreement, all the better to bill me afterwards. More likely than not, she was reading it all off the whois info for bsdly.net.

They had the act pretty well rehearsed, except for one detail that irritated me immensely: they insisted on getting an oral confirmation that I was interested in their service. Only after an oral confirmation was in place would they be sending me anything in writing.

The conversation never progressed much beyond beyond the initial "Is your name Peter [...]", "Are you duly authorized to act on $company_name's behalf", which is where I broke it off. For one thing the connection was terrible, and for another I was beginning to smell a rat. (Yes, I'm dense at times, I know.) At the end of too-many minutes, I thought I had finally got them to agree to send me their papers for me to review and hung up. Only to have the same lady call once more, continuing the script.

The second conversation did not progress much either, and for the third call a male claiming to be a supervisor had taken over. He didn't actually clinch a sale either.

So after three phone conversations, I used my regular registrar's web interface to register the bsdly.se domain myself, setting me back a whopping NOK 140 (roughly EUR 18 or USD 26). While I was writing this note, the "Registration complete" message from my registrar arrived here in my inbox.

So now I have another domain. I've expanded into Sweden.

Or as they say in the trade, get a geek really riled up, he'll go right off and buy a domain. In Sweden.

My First IPv6 Spam

2011-06-08T22:41:00.005+02:00

On the day of The Great Experiment, an anecdote on how much the world stays the same even with IPv6, security-wise. No reason to stay all dotty, this is when the fun starts.

Happy IPv6 Day, everyone! As I write this column, the 24 hour worldwide Internet Protocol, version Six preparedness experiment is still in progress, with some hours to go before the summaries, no doubt penned by industry luminaries, will start appearing. In the meantime, I have a small IPv6 anecdote of my own to share.

Like most network oriented techies, I've had IPv6 somewhere in my field of vision for some time, with a footnoted TODO list to match. I started nagging local ISPs for their IPv6 plans some years ago, and as you've probably guessed, up until very recently the answer at essentially all European ISPs has been 'non-existent'. Among European nations, most of them pretty early in the queue when the original IPv4 address allocations were made, Norway was quite fortunate, and with a total population of less than five million and enough NAT to go around we have a good enough IPv4 address to total population ratio that makes for no panic of us running out of IPv4 locally.

So this left us early adopters to seek out the next generation connectivity via tunnel providers such as Hurricane Electric or Sixxs, with the dancing turtle at the Kame Project website as the early reward for venturing into 128-bit address territory.

For the longest while, that would be pretty much it. Once you'd kept your tunnel stable for a while, you could generally have a /48 subnet for the asking. But finding any good content on IPv6 or large amounts of IPv6 traffic of any kind was not easy. At parties we'd even bitch about the famous lines in one IPv6 textbook that claimed that IPv6 routing tables were generally much smaller and more compact than their IPv4 counterparts -- of course they're smaller, there are essentially no sites, and they produce next to no no traffic!

Then, famously, the time came when the last /8 range allocations of IPv4 address space were made, and the IPv4 internet was officially full (should we just start telling them to go away yet?), for some values of at least. Even if certain European organizations or nations are not in any danger of running out of IP addresses, sixteen years after the initial IPv6 RFCs entered the standards track (see RFC1883 and friends), we found ourselves in a situation where any new large-scale implementations would likely be built exclusively with IPv6 or very nearly so, and the historical backdrop for today's experiment was complete.

I expect the summaries to say mostly that modern operating systems, at least the free ones such as OpenBSD came very well prepared for the task. The two protocols are in fact not compatible, but running both will work, and for the vast majority of services, the sysadmin's worksheet looks like this:

1) Get hold of whatever number of IPv6 addresses your application requires

2) Configure your network interfaces with IPv6 addresses (some systems come preconfigured to do their own autoconfiguration)

3) Edit in the required IPv6 addresses into your services' configurations

4) Add any AAAA records required to your domains' externally-visible zones.

And you're done, at least for now. With quad-A entries in your externally-visible DNS, you will start seeing IPv6 traffic hitting your services from elsewhere, not just your own test traffic.

Your logs and other monitoring will show you how your rig behaves. Most likely it all just works, and your users won't notice anything different, except perhaps a dancing turtle (or missing ads on IPv6 for some web sites, as noted by some early IP version six day reports).

I did those things a little while back on my home network, and when the first few days did not show up anything I almost missed what could be an important event: my first spam email delivered over IPv6. I only read message headers when the message stands out somehow, and in this case it was clearly spam, so I took a peek at the headers in case I it would be worth blacklisting manually:

Received: from mo-p00-ob6.rzone.de ([2a01:238:20a:202:53f0::1])
by skapet.bsdly.net with esmtp (Exim 4.73)
(envelope-from )
id 1QFOdo-0001zq-E9
for bsdly@bsdly.net; Thu, 28 Apr 2011 12:40:25 +0200
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1303987210; l=2236144;
s=domk; d=edrenat.es;
h=Content-Type:Mime-Version:Subject:Reply-To:From:Date:X-RZG-CLASS-ID:
X-RZG-AUTH;
bh=CS67GJbTmE0wDhH/HieJqdndPSo=;
b=pNIf0dK3KnFhj/7mcP8KGm70V9/XxlPa/ALQSKRh0nBgWsT0mdoUxi2unepDaMlZMPW
cReoEgH/yMOF6T59Zf2X4fjrr358rU9hhxjQuV6lHwIpYoQWYwkxOqAB1iYL+SacIWspU
MYMrgju1GQLZps7hIeunhV0pfto8o4JeqqM=
X-RZG-AUTH: :KWgWcE6pb9/UNsdwkwZbgj6IM9/U3aYAugAbJE4rNBO+ejjApHAeOC4nD+Q=
X-RZG-CLASS-ID: mo00
Received: from PACO1 (cliente-98392.iberbanda.es [87.111.193.23])
by post.strato.de (jimi mo42) (RZmta 25.17)
with ESMTPA id J00252n3SABVBu ; Thu, 28 Apr 2011 12:29:29 +0200 (MEST)

The full message is preserved here with headers or as mainly message body as text if you're so inclined.

From the most recent Received: header we see that the delivery happened over IPv6, while the second tells us that the originator most likely did not have IPv6 connectivity, or at least did not see fit to use it if they did.

It's also worth noting that this message, like a surprising amount of spam in recent times, also comes with what appears to be a valid DKIM signature. So much for cryptograpic validation as an effective antispam measure.

Now of course my next step was to find out if this particular sender had tried to darken my mail spool before:

peter@skapet:~$ sudo grep edrenat@edrenat.es /var/spool/exim/logs/main.log
2010-11-07 14:21:44 1PF5Bm-0006RI-KB <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=251712 id=201011071353082974379@edrenat.es
2010-12-14 20:42:49 1PSaln-000255-PQ <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=759434 id=201012141954165325931@edrenat.es
2011-01-30 21:13:30 1PjdeA-0001Nn-If <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.161] P=esmtp S=1905781 id=201101302042161777948@edrenat.es
2011-02-22 21:59:05 1PrzIX-0006jv-7x <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=1227270 id=201102222041571040455@edrenat.es
2011-04-28 12:40:26 1QFOdo-0001zq-E9 <= edrenat@edrenat.es H=mo-p00-ob6.rzone.de [2a01:238:20a:202:53f0::1] P=esmtp S=2210478 id=201104281225569454004@edrenat.es

Apparently they had, on more or less a monthly basis, so let's take a peek at the log entries for all of those messages:

peter@skapet:~$ for foo in 1PF5Bm-0006RI-KB 1PSaln-000255-PQ 1PjdeA-0001Nn-If 1PrzIX-0006jv-7x 1QFOdo-0001zq-E9; do sudo grep $foo /var/spool/exim/logs/main.log ; done
2010-11-07 14:21:43 1PF5Bm-0006RI-KB DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1289136101 l=251099 [verification failed - signature did not verify (headers probably modified in transit)]
2010-11-07 14:21:44 1PF5Bm-0006RI-KB <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=251712 id=201011071353082974379@edrenat.es
2010-11-07 14:21:44 1PF5Bm-0006RI-KB => peter <bsdly@bsdly.net> R=localuser T=local_delivery
2010-11-07 14:21:44 1PF5Bm-0006RI-KB Completed
2010-12-14 20:42:46 1PSaln-000255-PQ DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1292355762 l=766689 [verification succeeded]
2010-12-14 20:42:49 1PSaln-000255-PQ <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=759434 id=201012141954165325931@edrenat.es
2010-12-14 20:42:49 1PSaln-000255-PQ => peter <bsdly@bsdly.net> R=localuser T=local_delivery
2010-12-14 20:42:49 1PSaln-000255-PQ Completed
2011-01-30 21:13:26 1PjdeA-0001Nn-If DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1296418396 l=1927465 [verification succeeded]
2011-01-30 21:13:30 1PjdeA-0001Nn-If <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.161] P=esmtp S=1905781 id=201101302042161777948@edrenat.es
2011-01-30 21:13:30 1PjdeA-0001Nn-If => peter <bsdly@bsdly.net> R=localuser T=local_delivery
2011-01-30 21:13:30 1PjdeA-0001Nn-If Completed
2011-02-22 21:59:03 1PrzIX-0006jv-7x DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1298408169 l=1240300 [verification succeeded]
2011-02-22 21:59:05 1PrzIX-0006jv-7x <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=1227270 id=201102222041571040455@edrenat.es
2011-02-22 21:59:06 1PrzIX-0006jv-7x => peter <bsdly@bsdly.net> R=localuser T=local_delivery
2011-02-22 21:59:06 1PrzIX-0006jv-7x Completed
2011-04-28 12:40:21 1QFOdo-0001zq-E9 DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1303987210 l=2236144 [verification succeeded]
2011-04-28 12:40:26 1QFOdo-0001zq-E9 <= edrenat@edrenat.es H=mo-p00-ob6.rzone.de [2a01:238:20a:202:53f0::1] P=esmtp S=2210478 id=201104281225569454004@edrenat.es
2011-04-28 12:40:26 1QFOdo-0001zq-E9 => peter <bsdly@bsdly.net> R=localuser T=local_delivery
2011-04-28 12:40:26 1QFOdo-0001zq-E9 Completed

This tells us that they've consistently sent to an address that features only on my website and has never been used as a sender or return address and has never been signed up for any mailing list of any kind. We also see that all the messages bar one pass the initial DKIM validity test.

There are several lessons to be learned here. One is that with a sensible choice of operating system and other software, the transition from classic IP version four only to a dual-stack configuration with both IP version four and IP version six running on your systems will likely be less dramatic than you have been lead to believe.

Another is that at least in some respect the world stays very much the same with IPv6 as it was with IPv4, and content is likely to cross the protocol border. Or to put it slightly differently, the same caveats about ill-intentioned traffic still apply; any security measures you had inplace before you went dual stack most likely need to be tuned to handle traffic from the brave new world of IPv6.

And of course, even if a number of implementation bugs have been found and fixed and a number of fundamental design flaws have been identified and worked around, only full scale testing like today's experiment (preferably sustained over longer periods) offer any hope of identifying and fixing the problems we haven't found yet. The looming IPv6 transition is likely to make and break careers, and some of us will have our share of both fun and nightmares while seeing to the confidentiality, integrity and general security of your data and your systems.

Good night and good luck, while we're slowly going from dots to colons.

Thanks to Sevan Janiyan for tweeting "Though the gmail website is reachable via IPv6, mail is still going via IPv4 :(" earlier today and reminding me of the incident that was the inspiration for this column.

How Do We Appropriately Celebrate The Arrival Of The 100,000th PF Tutorial Visitor?

2011-06-05T21:20:00.007+02:00

A nice surprise may be in line for a new visitor, and you (yes, you) can help me pick the surprise.

In late 2004, I started working on the text for a user group lecture for the BLUG meeting scheduled for the the following January.

The original manuscript was in Norwegian, but after a rather successful and surprisingly well attended user group meeting, I wrote up an English version and posted both online. With some encouragement from Greg Lehey (I'd participated in the group of volunteer reviewers for his third edition of The Complete FreeBSD), I submitted a proposal to give a half day tutorial for the 2005 AUUG conference in Sidney.

The proposal was accepted, as were several of the followup submissions to other conferences, and via a sequence of conferences and some private sessions, the document kept developing. In early 2007 I started working on turning the manuscript into a usable book. As regular readers will be aware, the much revised second edition was completed during the second half of 2010, and even that version has recently been subjected to its first update thanks to the ongoing development of the OpenBSD operating system.

The original tutorial has kept attracting a relatively steady stream of new visitors from all over the world, even though I have not added any new material to the document since I started working on the book version. New material will, rather, find its way into slides for the next session (such as the most recent one at BSDCan 2011), or will be put in the queue for possible upcoming book material.

During periods when I have had little visible output to offer, it has been interesting to see that the documents attract visitors and the occasional comment or suggestion for improvement. Then a little while back, I realized that in a not too distant future, the number of unique host names or IP addresses that have visited the tutorial tree will roll past a hundred thousand (100,000).

That particular number is possibly only significant to me, I keep the count of unique hosts accessing mainly to get an idea how many people have looked at my work. The raw number of page hits for the same location (we don't have any numbers for the early days when it was hosted at a now-defunct ISP) is fairly close to one and a half million, but I feel that number is a rather pointless statistic.

But when visitor number one hundred thousand arrives, how should we celebrate? I'm inclined to try to identify and contact the lucky visitor and offer a prize of sorts, but I have not quite made up my mind what and how. I'll welcome suggestions sent to via email (to peter@bsdly.net) with a recognizable subject.

It is worth mentioning that neither the tutorial nor this blog directly generates any revenue for me. I did for a short time have Google-supplied ads running on both sites, but for reasons that have never been quite clear to me, Google chose to terminate my AdSense account a few days before my second USD 100 transfer was due.

RFC1149: Ten Years of In-Flight Internet

2011-04-29T10:00:00.011+02:00

It's been ten years since a small group of Bergen hackers implemented RFC1149, the Carrier Pigeon Internet Protocol, making in-flight Internet a reality well ahead of any airline.

On April 28th, 2001, my laptop received four ping packets. That in itself, you might say, was not a particularly noteworthy event.

However, on this occasion, the message was in the medium. The traffic my laptop received that day was transmitted via avian carriers, more commonly known as carrier pigeons.

The story of the hows and whys was rather quickly documented in main implementer Vegard Engen's "preliminary writeup" (part of the CPIP Workgroup website (which was slashdotted and for a few days saw traffic in the million hits per day range).

Why did we do it? Because the RFC was implementable, and it was fun. Very few people outside tech circles have ever clued in on this, with a few notable exceptions such as Peter Meyers, who wrote the salon.com article The pigeon protocol that was published within a few days of the event.

If you're interested in this kind of thing (if you're still reading, I suppose you are), other writeups such as my own The kernel hacker speaks to polite people and assists in the flight of network packets - a conspirator's view of Alan Cox' April 2001 Bergen visit may be worth reading too.

After the event, Vegard made a presentation about the workgroup efforts at the next IETF conference, and was presented with a plaque (jpg, 76kB, jpg, 2.1MB) in return.

The CPIP WG activities have proceeded at a more leisurely pace in recent years. In 2005 I went to the AUUG 2005 conference to do an early version of the PF tutorial, and en route I made a presentation in Adelaide about the project (slides and accompanying notes are still avaliable).

We're still looking for independent, interoperable implementations, though. Preferably on other free operating systems besides Linux. If we can entice our old pigeon partners to participate, we're more than willing to arrange for interoperability tests.

The world needs this to be on the IETF Standards Track.

The Problem Isn't Email, It's Microsoft Exchange

2011-02-27T23:03:00.008+01:00

The takeaway: don't pretend your appointment book can handle your email. And don't blame the Internet for all the compatibility issues. The main problem is Microsoft Exchange.

I care about email. In fact, a large part of how I have made a living over the years has depended on a reliable email service. I get a lot of email, and I send my fair share of it too - some of it is correspondence directly related to whatever I'm working on at the moment, some of it is personal, quite a bit comes from topic-oriented mailing lists such as openbsd -misc, and a large chunk of my mail archive consists of automatically generated mail sent by systems in my care. I've also been known to treat email much the same as other correspondence, rarely if ever deleting messages. When the mailboxes became too unwieldy I would transfer some of the contents to archive storage.

I've become convinced that a large part of the reason I don't mind dealing with large volumes of email is that I started doing it before Microsoft became an actor in the Internet email market. Way back in the late eighties and early nineties, email of the Internet, TCP/IP, kind would be handled by some sort of Unix box (a BSD or, by the mid-nineties, a Linux variant, perhaps) that would frequently offer shell command line access, but more likely than not also email reading via POP or IMAP interfaces.

And it worked. Users who insisted on (or needed to be on) a Microsoft desktop could be persuaded to install a useful email client such as Eudora (now defunct but fortunately Qualcomm donated the code base to Mozilla for integration in Thunderbird), and for mailboxes that became too unwieldy, the advice would be to just move content to mailboxes that Eudora wouldn't load into memory by default, such as the ubiquitous Inbox. Over the years the volumes of and the nature of email changed gradually, so along the way we learned to deal with spam and mail-borne Microsoft worms by installing content filtering and setting up other tools. Still, everywhere I worked, apart from the unavoidable but infrequent freak incidents SMTP email was considered reliable, and your email archive was just that.

From other parts of the world we would hear every now and then stories about the death of email, and recently even a largish IT company announced that they were planning to get rid of all email in the near future. Email, the story goes, is just too time consuming and disruptive. I never quite understood what they were on about.

Then not too long ago I started working regularly in an environment where email is done the Microsoft way, via Exchange and Outlook. And it has struck me that they're right: If your email experience is via Exchange and Outlook, the net effect is both time consuming and disruptive.

Forced to work with an all-Microsoft desktop for the first time in years (where my most frequently used application by far is putty.exe, but that's beside the point here), I found Outlook's user interface clunky and with frankly insane default settings ("rich text" by default, newest messages on top and positively deranged quoting setups, more about that later) that were for the most part fortunately changeable, at least on a per mailbox basis.

The first revelation came when I heard a co-worker praise newer Microsoft Office releases "because 2007 and newer has discussions". I was forced to imagine how life must have been like without threading as we've tended to call it on the USENET and mailing lists since, well, the late 1980s. Outlook's predecessor Microsoft Mail of course did not support threading, but I suppose any plans to support threading via References: headers and suchlike received a major blow when the translators of MSMail decided not to leave the RFC-dictated "Re:" prefix alone, but rather translate it for local language versions and lead the way to the "Re: SV: Antw: VS:" and so on cascades we see in the Subject: fields for correspondence between users of Microsoft mail clients and others.

No big surprise then, that when Microsoft decided to "invent" threading for their messaging products, they again ignored the RFC-compliant References: header and chose to implement their very own version based on a set of X-something headers that appears to make the threading a local-to-this-Exchange-server (and Outlook clients only) thing. Messages that do not retain the X-something headers regularly show up as separate "discussions". All this is to a Unix-head much like the "Recall" functionality that always draws smiles on mailing lists.

Being robbed of any easy way to track the relationships between messages in your mailboxes is bad enough, but there's more. Even with a limited sort of threading in place (even one that would break at the slightest interference from outside software), the damage had already been done by software that introduced counterproductive, confusing and time consuming response practices.

For reasons that have never become entirely clear to me, the developers of Microsoft email client software decided that direct and limited quoting of text from previous messages was not a priority. So rather than build on earlier work where we would have exchanges like
From: First Correspondent <first.correspondent@onecompany.nx> To: Second Emailer <second.emailer@otherplace.nx> Subject: A most enlightening message Dear Second, Here I offer an important insight that I would like to share. Followed with random commentary that may or may not be important. I hope you agree this was worth sharing. Yours, First
where a typical response from Second would typically be something like this,
From: Second Emailer <second.emailer@otherplace.nx> To: First Correspondent <first.correspondent@onecompany.nx> Subject: Re: A most enlightening message First Correspondent <first.correspondent@onecompany.nx> writes: > Here I offer an important insight that I would like to share. Thanks for sharing that! The next bit was really about something else entirely, but is probably worth discussing over refreshments at an appropriate time. > I hope you agree this was worth sharing. Oh, definitely! We'll get plenty of good out of this as time goes by. Be seein' ya, Second, jr
they chose a different approach entirely.

Keep in mind that other parts of the world that were already used to email and related forms of communication such as Usenet news, where exchanges like these were commonplace and gave a reasonable certainty as to who said what, when.

What Microsoft did instead was to introduce a wholly new convention for email responses. The details vary over the various versions, but the main parts were to wrap any text information in pseudo-html formatting and place the entire previous message after the present correspondent's signature, with the cursor for the user to input text at the top.

Inline quoting like in the exchange I quoted earlier was tricky bordering on impossible, and adventurous users would resort to tricks like "my parts are the ones in magenta", only to discover that the carefully hand-painted text would fail to render correctly on any other software than their own version, down to the minutest patch level.

Thus was born the age of all-inclusive top-posting, where deciphering the true meaning of any of the paragraphs on top of the message could take more moments than you really have at your disposal, the time needed to decipher the cascade of earlier messages included. Not only would the ever-expanding, all-inclusive (but actually rather unreliable and far from tamper-proof) discussion-in-a-message convention confuse all readers involved, it also meant that the text and any file attachments would be stored multiple times, many times over for long discussions. It would take only a minimally uncharitable view of the average C?O's intellectual capacity to suggest that this was a prime mover behind the intense rush to "data deduplication" in storage marketing literature a few years ago.

Which takes us to the next item: Storage. Taken in semi-random order, the next hurdle for a Microsoft email user to overcome is storage. Outlook by default uses its own binary format for local message storage, know as PST files or Personal Storage Table files as the informative Wikipedia entry explains. In some configurations all mail is stored in a database of sorts on the Exchange server, and the user may or may not have the option to save messages to local PST files to work around space limitations on the server.

It is not uncommon for Exchange admins to turn off users' ability to save messages to PST files. One major reason is that more likely than not any saved PST file will end up on the end user's computer, with the consequence that potentially important messages may end up being backed up infrequently, if at all. Other reasons to avoid PSTs are size limits (originally 2GB but larger in newer releases), but the thing that tends to scare people the most are horror stories of data corruption to the point of absolute unrecoverability. As in gigabytes of your business or personal life gone, due to a scrambled PST file. There is anecdotal evidence that missing or scrambled PST files are a big headache for those who for various reasons want to look into the inner life of the Bush 43 administration.

So for records keeping involving your email, you're in a bind: Your mailbox size is likely to be limited -- every Exchange admin knows that large mailboxes will hurt performance, impacting all users of that server -- and the only way to save messages offline is a known-unsafe method. As far as I have been able to find out, there is no easy way (other than extracting messages to a separate system, say via IMAP) to export mail from the Microsoft product combo to any text or non-microsoft mailbox format.

Now weigh those practical considerations against legislation that dictates all business related correspondence be kept on file for a matter of several years. The exact number of years varies by location, but unless you've purchased one of the add-on solutions for archiving, you will be struggling to keep in line with requirements.

It all comes down to the shortsightedness or intellectual shallowness of Microsoft Exchange's designers, way back then. It does make sense that your appointment calendar application should be able to send and receive email, and it kind of makes sense that your appointments are within easy reach from your email client.

Those facts do not, however, dictate that the appointments calendar and your email archive should share a common storage backend. In fact, it's likely that the decision to merge the email storage and appointments storage into one is the direct cause of many of the inefficiencies of Microsoft Exchange.

In one recent incident involving a user mailbox of perhaps a couple of gigabytes, where the bulk of the data was made up of an estimated (since Outlook never managed to display totals before freezing) 1.5 million messages of about one kilobyte each, even deleting the messages using an Outlook filtering rule (the content was not of a nature that required long term storage) literally took weeks, typically proceeding at a rate of one message per second early in the process, speeding up to somewhere in the five to ten messages per second rate near the end. Fortunately the user in question was able to access email functionality via the Outlook web access interface while deletion proceeded, but anecdotal evidence suggests that the workload had measurable performance impact on other hosts attached to the same SAN.

Even if you tackle the storage hurdle, you more than likely will be tripped up by other inanities in the software design. There are bound to be other pitfalls, but here is my personal list of things that continue to irritate me (in addition to the default "rich text" formatting), coming as I do from the outside world:

Using Outlook it appears to be impossible to see what your From: address will be before you send the message. The effect is sometimes quite bizarre, in my case since the site has several domains, I of course ended up signing up to several mailing lists with a wrong address, banishing my posts there to moderator queues until I was able to study the real mail headers on a non-Microsoft system.

Also, Outlook is overly helpful in filling in adress fields such as To: and Cc: from common address books and Active Directory, leading in at least one case I know of to a supposed-to-be-private message to be sent to every mailbox in a largish corporation. That's when you learn that after the first reply, retracting the message won't actually work.

And no rant about Exchange would be complete without mention of the largely information-free bounce messages the system generates for non-delivery. A significant portion of the spamtrap addresses I use have been fished out of bounce messages, and the Exchange ones stand out as the ones practically guaranteed to exclude any information about where the triggering message came from, or when.

Summing up, if you're an executive who feels that your organization is saddled with inefficient email processing and dubious archiving, the likely culprit is not email as such, but rather the poorly constructed application some unscrupulous sales person inserted in your network for you.

Changing to a standards compliant, preferably open source, alternative is likely to save your organization costs at all levels, including hardware and software acquisition and maintenance costs as well as significant personell time. At the same time a move to a standards compliant, open source solution will likely leave you in a better position with respect to security, information consistency and verification. A full treatment of email as a business tool would have had at least one column of similar length as this one on each of these topics, and I may return to those in future columns. In the meantime, if inefficient emailing bothers you, you may need to realize that a large part of your problem is Microsoft Exchange.

St. Patrick's Day PF tutorial in Tokyo: Returning readers may already be aware that I will be giving a PF tutorial at AsiaBSDCon 2011. My session will be on March 17th, known in some parts of the world as St Patrick's Day. You can register for my session and others here, hope to see you there!

La oss prise fruktbarheten, for den lar de snakkesalige slippe til [.NO #DLD]

2011-01-31T22:24:00.005+01:00

Eller: Det hjertet er fullt av, renner munnen over med. Her: Trafikkdata og grådighet.

I dag har vi spesiell grunn til å være takknemlige for menneskelig fruktbarhet. Spesielt må vi takke justisminister Storbergets ektefelle for å ha medvirket til at mannen fikk anledning til å ta pappaperm akkurat nå. Uten en justisminister på pappaperm og med forsvarsministeren som vikar hadde vi nemlig ikke fått dagens oppslag i Aftenposten, med den betegnende tittelen "Faremo frykter hackere, men er samtidig ikke så bekymret".

Det interessante kommer først i tredje siterte svar på spørsmål, der vår vikarierende justisminister etter de obligatoriske floskelavleveringene med marginal sannhetsgehalt ('dette er ikke overvåking', 'vi styrker personvernet', osv) avslutter med

Det vil forbause meg mye om vi ikke også får en diskusjon om hvordan trafikkdata kan brukes kommersielt.

Vi skal huske på at Grete Faremo i sin forvisningstid fra politikken (hun forsvant fra justisministerposten i Torbjørn Jaglands regjering midt på nittitallet, etter bråket som oppstod da det viste seg at minst ett medlem av Lund-kommisjonen hadde vært under overvåking mens kommisjonens arbeid pågikk) var innom IT-bransjen, der hun antakelig må ha plukket opp tanken om adferdsdata som handelsvare. Det skulle ikke forundre om akkurat Faremos tidligere arbeidsgiver Microsoft ville være svært takknemlig mottaker av detaljerte oppplysninger om hvem som snakker sammen og skriver til hverandre, med noe nær komplette adferdsdata for hele befolkningen. Når kommersiell utnyttelse er bare en forsnakkelse unna, er det grunn til å spisse ørene.

Men for all del, i det perspektivet -- at motivene bak iveren for elektronisk brev- og besøkskontroll for hele nasjonen ikke er rent politiske, men også klart kommersielle -- avslører merkelig nok tilhengerne av det uspiselige direktivet mer menneskelighet enn noen gang før. Det handler kanskje helt enkelt om grådighet. Så er de kanskje ikke primært ute etter å ha kontroll på politiske motstandere likevel. De ønsker å selge detaljerte opplysninger om ditt og mitt privatliv til høystbydende, gjerne flere ganger til forskjellige aktører til hver sin pris.

Men før dataene kan selges eller på annen måte overlates til kommersielle interesser, er det avgjørende at dataene faktisk blir samlet inn. I dag er innsamling og oppbevaring av detaljerte trafikkdata i virkelig kommersielt interessant skala (les: Til: og Fra: med nøyaktig tidspunkt for all epost, samme for telefonsamtaler, men da supplert med posisjonsdata med brukbar oppløsning) ikke tillatt i Norge. Det er de enkelte operatørene som står for innsamlingen, og hvilke data som blir samlet inn, er styrt av hver enkelt operatørs faktureringsrutiner og driftstekniske hensyn.

Det lovforslaget som Arbeiderpartiet for tiden er alene om å gå inn for, forutsetter at større mengder adferdsdata enn noengang før blir samlet inn og lagret, og sannsynligvis vil man også ende opp med ett sentralt og ytterst attraktivt lagringssted. Lovforslaget legger riktignok opp til at utlevering av data fra det sentrale samlingsstedet skal være under domstolskontroll, men det er en smule påfallende hvor lite bekymret vår fungerende justisminister er for at brev- og besøkskontrolldataene skal komme på avveie.

Politiet har alltid hatt mulighet til å hente ut og reservere disse dataene når det er behov for det, sentral lagring vil gjøre det lettere å hente ut virkelig verdifulle data om deg og meg for de som ikke har skrupler med å gå bakveier til dataene. Samtidig vet vi fra de landene som har innarbeidet datalagringsdirektivet i nasjonal lov at direktivet helt klart ikke bidrar til å bedre oppklaringsgrad eller forekomst av kriminalitet, og at flere EU-land har kjent direktivet eller innføringslovene grunnlovsstridige. Men alt dette bekymrer tilsynelatende ikke vår fungerende justisminister. Kanskje neste datalagringslov vil gjøre det lettere å selge data om ditt og mitt privatliv over bordet.

Stortingsbehandlingen av datalagringsdirektivet innledes med høringer 7. og 9. februar. Se sakssidene på stortinget.no (med informativ samleside) for detaljer.

I will not mindlessly paste from HOWTOs

2011-01-30T15:40:00.004+01:00

Even with proper discouragement, mindless pasting is rampant, it seems

It had to happen sooner or later.

My incoming mail this morning had one item about what I thought was a fairly trivial misconfiguration, and I answered it like this

From: peter@bsdly.net (Peter N. M. Hansteen)
Subject: Re: interesting-traffic
To: Name Withheld <Name.Withheld@gmail.com>
Cc: peter@bsdly.net
Date: Sun, 30 Jan 2011 12:44:35 +0100

Name Withheld <Name.Withheld@gmail.com> writes:

> how should i handle the 'intersting-traffic macro not defined' error
> in pf.conf on obsd 4.8 reboot syntax error starting pf?

either define the macro (remove a # comment perhaps) or remove any
references to it.  Have you been pasting from a partial example
floating around the web perhaps?

- P

Then a few sips of coffee later, it dawned on me: the macro interstring-traffic is more than likely one I made up for the bridge example in the short (and now rarely updated) version of my PF tutorial document. (I added the strongly worded note there as a reaction to this incident).

So it's at least partly my fault. I put an incomplete example out there, hoping that whoever stumbled upon the material would grasp the context and fill in any needed details. The important bits are all there, but when pasted into a config without checking, the result will be just as Name Withheld experienced.

But then I can't really take the full blame: Had he bothered to read the rest of the document or even the book that's a further development, he would have seen this admonition which comes out even more clearly in the slides version. If for some reason the links are inoperative, here it is:


The Pledge of the Network Admin

This is my network. 

It is mine 
or technically my employer's, 
it is my responsibility 
and I care for it with all my heart

there are many other networks a lot like mine,

but none are just like it.

I solemnly swear 

that I will not mindlessly paste from HOWTOs.

I actually recite that at the very beginning of all my tutorial sessions, and while of course it's sometimes accompanied by giggles, the point remains: there is no substitute for actually understanding your configuration. Testing (if nothing else, a quick sudo pfctl -vnf /etc/pf.conf and reading the output before rebooting) would have helped enormously too.

For those hungry for fresh PF tutorials, I'll jump the gun and announce that there will be one by yours truly at AsiaBSDCon 2011, final schedule to appear on that URL shortly. A few other events are in the works too, more details here and at the PF tutorial page when details are settled.

Ikke styrket personvern, men brev- og besøkskontroll for hele folket [.NO]

2010-12-29T14:40:00.005+01:00

En ny dag og naturligvis enda et patetisk forsøk på nytale fra justisministeren. Pluss en interessant invitt.

[Another .no politics post, I'll be back with international-style geekery soonish]

Les gjerne Knut Storbergets magnum opus Vi styrker personvernet nå, og husk på å lagre det, slik at det bevares for ettertiden.

Når vi pålegger operatører som Telenor, Netcom og andre å lagre trafikkdata, så styrker dette personvernet, sies det.

Trafikkdata handler i praksis om hvem du snakker med. For de av oss som går rundt med mobiltelefon hele tiden, vil trafikkdataene også vise hvor vi har oppholdt oss på et gitt tidspunkt. Vanligvis ikke med samme nøyaktighet som GPS-sporingen i mobiltelefonen, men nøyaktig nok til at posisjonsdata fra telenettet allerede har vært brukt som bevis i straffesaker.

I dag logger og lagrer teleoperatører og internett-tilbydere de dataene som er interessante for faktureringsformål eller som gir saklig grunnlag for teknisk drift, vedlikehold og planlegging. Og i alle sammenhenger jeg er involvert, så kaller vi dette for nettopp overvåking. Datamaskiner er nemlig uhyre godt egnet til å ta unna arbeid som mennesker ville blitt anstrengt av å utføre. Og i sammenhenger som å sørge for at kunder blir fakturert for rett mengde trafikk, se utvikling over tid i for eksempel overført volum, teknisk diagnosearbeid på utstyr (for eksempel for å se hvordan det oppfører seg under stor last) og planlegging av endringer er slik overvåking faktisk både nødvendig og nyttig.

Men når fakturaene er betalt og analysene sluttført, er det ingen saklig grunn til å beholde dataene, og de blir slettet. Summer i regnskap og samleverdier for teknisk tilstand er alt vi trenger. Rådata overlates ikke til uvedkommende.

Men justisministeren liker altså ikke at vi kaller dette for overvåking, og jeg er glad vi får denne invitten til å være behjelpelige med å finne et mer presist uttrykk. Mitt forslag er, i all beskjedenhet:

brev- og besøkskontroll for hele folket

Dette er betydelig mer presist enn det noe ulne "overvåking", og vi oppnår samtidig å ta livet av forestillingen om at "Internett" er noe som eksisterer separat fra samfunnet. Det har nemlig aldri vært slik at datamaskiner og kommunikasjonen mellom datamaskinbrukere var noe som eksisterte utenfor samfunnet eller utenfor lovene vi må forholde oss til. I Norge har en helt dominerende del av befolkningen nettopp Internett som hovedkanal for kontakt med slekt, venner, kolleger og de fleste samfunnsfunksjoner.

Når alle våre trafikkdata blir registert og sammenholdt, blir vi i praksis utsatt for en grad av kontroll som ikke skiller seg vesentlig fra det vi ilegger vareteksfanger -- altså personer som holdes i forvaring på grunn av klar mistanke om straffbare forhold -- som skjerpende tiltak når det er fare for bevisforspillelse under etterforskning.

Jeg vil oppfordre justisministeren og andre debattanter til å ta fatt i dette terminologiskiftet. Først når vi diskuterer datalagringsdirektivet som innføring av elektronisk brev- og besøkskontroll har vi tatt steget over i en edruelig debatt med virkelighetsbasert terminologi.

Så får vi heller komme tilbake til de klart uttrykte ambisjonene om å lagre innholdet i kommunikasjonen senere, om historien faktisk gjentar seg som en tragedie.

Oppdatering 1: Petter Reinholdtsen minte meg på et interessant datapunkt: I Danmark, der EUs datalagringsdirektiv allerede er innbakt i lovverket, blir det lagret gjennomsnittlig mer enn åtti tusen dataelementer per danske per år som følge av datalagringsdirektivet. En opplysende artikkel finnes i norske Digi.no: Lagrer 82 000 DLD-poster per danske i år. Vel verd å lese for interesserte.

Du er en forbryter eller et virus, så Arbeiderpartiet vil overvåke alle, hele tiden

2010-12-23T21:08:00.006+01:00

Er du ikke forbryter, kommer du til å bli det. Og vi vet at du kommer til å begå overgrep mot barn, mener Arbeiderpartiet.

[This post is in Norwegian only. I'll be back with the regular more-or-less-ordinary geekery for my international friends later.]

Hovedoppslaget på NRK1 Dagsrevyen 23. desember 2010 dreide seg i følge annonseringen om 'manglende tiltak mot spredning av overgrepsbilder på Internett'.

"Både politiet og internettselskapene kjenner til metoden for å begrense flyten av overgrepsbilder på nett. Men de tar den ikke i bruk."¹⁾

Metoden de henviser til, er om vi tolker innslaget rett, å bruke samme metoder som man bruker i antivirus-systemer til å identifisere og stoppe overgrepsbilder. Som politisk propaganda for totalovervåking og som reklame for Norman Antivirus fungerte dette reklameinnslaget kanskje bra, men det er verd å påpeke en del åpenbare tekniske og samfunnsmessige svakheter.

Den teknologien som blir omtalt i innslaget, går enkelt sagt ut på å generere en 'signatur' for hvert virus. Dette gjør man typisk ved å utføre en serie beregninger på innholdet og dermed få en slags avansert tverrsum. Denne tverrsummen eller signaturen kan sin tur med stor grad av sikkerhet (men aldri absolutt) kan brukes til å identifisere nye forekomster av de samme dataene. Dersom en ny datamengde eller fil som blir utsatt for beregningene gir samme resultat som en tidligere kjent fil, så er det med overveiende sannynlighet snakk om identiske data, eller samme fil.

Dette er teknikken som antivirusprodukter har brukt i mer enn 20 år, og et typisk antivirusprodukt i dag gjør stadige oppslag mot lister på flere hundre tusen signaturer.

Så langt alt vel, dette er kjent teknologi. Det reportasjen nokså beleilig unnlot å nevne, er at signatur-teknikken fungerer bare når det er tale om identiske data. Virusfirmaer som Norman vet smertelig vel at det trengs bare en triviell endring, for eksempel å endre et enkelt tegn i en del av viruset som ikke har direkte innvirkning på hvordan det kjører, så slipper det nå 'muterte' viruset forbi filtreringen, siden innholdet skiller seg fra det som allerede er kjent.

Resultatet er velkjent: Jo flere påfunn de slemme kommer med og jo flere nye varianter som dukker opp, jo flere signaturer må vi sjekke mot. Et typisk antivirus-sysmem (Clam Antivirus) har til sammen i sine databaser vel 1,25 millioner signaturer.

Om man bruker den samme tilnærmingen på bilder, vil det være tilstrekkelig å justere fargepalett eller gjøre en enkel beskjæring slik at formen på bildeflaten endres, så vil bildet passere. For video vil helt ubetydelige redigeringer som å fjerne eller flytte rundt på noen utvalgte øyeblikk i en sekvens være tilstrekkelig. Og siden den nødvendige manipuleringen av et bilde er betydelig enklere å gjøre enn programmeringsgrepene som må til for å gjøre endringer i et virus, må vi anta at antallet signaturer vil vokse betydelig fortere enn det tilsvarende antallet signaturer for skadelig programvare.

Så lenge dette dreier seg om filer man finner under beslag hos mistenkte, er antakelig dette likevel et håndterlig problem. Filer med ukjent innhold kan man sjekke manuelt.

Noe mer interessant blir det når Inger-Marie Sunde argumenterer for at "politiet og internetttilbyderne med loven i hånden kan filtrere folks datatrafikk". I klartekst betyr dette at Inger-Marie Sunde endelig sier offentlig og i klartekst det vi lenge har ant hun mente, nemlig at totalovervåking av oss alle, med fullstendig innholdsfiltrering av all datatrafikk er både ønskelig og nødvendig. I innslaget brukte hun til overmål formuleringer som at 'staten har plikt til' kontrollere innholdet i internett-trafikk.

Det bør heller ikke være noen stor overraskelse at vår justisminister Storberget avslutningsvis så ut til å omfavne dette påståtte behovet for innholdsfiltrering av all kommunikasjon i befolkningen som argument for sin aktuelle hjertesak: innføring av datalagringsdirektivet.

Så om det noen gang har vært grunn til å tvile på det, er det nå helt klart:

Arbeiderpartiet mener du er forbryter eller kommer til å bli det.
Derfor skal du overvåkes, hele tiden.

At det også kan oppleves som plagsomt for et parti som regner seg som statsbærende å ikke ha fullt innsyn i hva opposisjonen holder på med, kan vi levende tenke oss. For de av oss som har blitt satt under overvåking kun med bakgrunn i politiske standpunkter²⁾ er det blitt stadig mer interessant å vite om det fortsatt finnes politikere som har et snev av demokratisk sinnelag. Arbeiderpartiet er med dette varig diskvalifisert.

Noter:
1) Tatt fra tekstutgaven av innslaget, http://www.nrk.no/nyheter/norge/1.7438572

2) Første halvår 1985 var jeg som nyvalgt tillitsmann for sivilarbeiderne i mitt fylke litt overrasket over at det som regel tok flere sekunder med spolelyder før summetonen dukket opp når jeg skulle ringe. Og til de som kom med de litt pussige sidekommentarene: vi hørte dere.

The Book of PF, 2nd ed: It's Here!

2010-11-09T17:11:00.007+01:00

Yes, it's that time of the year again -- we missed both Halloween and the OpenBSD 4.8 release, but hot on the heels of both, here it is:

The Book of PF, 2nd Edition is here, a box of author's copies turned up here just after lunchtime, and were taken well care of by Nora and Birthe.

This means, of course, that those of you who preordered will be receiving your copies shortly (mod the usual factors eloquently described by Michael Lucas here, the printer in my case is in Louiseville, Quebec), those who have reason to expect copies from my hoard here can rest assured that I'm taking them to the post office right after this. There's an illegible scrawl on some early pages, sorry 'bout that.

Better bookstores online and elsewhere will have it, or you could make it part of a bundle by ordering from the OpenBSD orders page. You will be going there for your six monthly fix anyway, won't you?

Upcoming events: The plans are not fixed yet, but you should expect me to turn up at BSD-themed events over the next few months. Look for announcements here, tweeted, or via the usual mailing lists.

If It Runs OpenBSD, It Has To Be Important

2010-10-24T22:58:00.003+02:00

If we are starting to see targeted attacks on OpenBSD systems, the world has become more interesting.

A lazy Sunday turned a tad more interesting when Dragos Rui, a person usually involved in high-quality security work (or somebody with access to his twitter account) tweeted this:

Analysis of recently deleted data on compromised server has turned up an OpenBSD boot sector trojan worm? Volunteer deconstuctors?

There have been no further updates as of this writing, and without access to the actual files in question, we can not offer any real meat on the actual payload, so any preliminary conclusions will be uncertain at best. But it is possible to come up with a threat assessment based on general knowledge of what it would take for a boot sector based OpenBSD exploit to make it into the wild.

OpenBSD is used in a wide range of environments (see the OpenBSD at work page for some examples, the list is by no means exhaustive), and since it is commonly perceived to be one of the most secure general-purpose operating systems available, it should be no surprise to find OpenBSD systems in mission critical roles.

So the motive for trying to forge a way in, past a system that is generally considered trustworthy, is obvious. If you hit the right targets, the potential payoff could be huge.

Then it becomes interesting to consider how you would go about to compromise a system running OpenBSD. If we leave aside social engineering such as bribing the sysadmin or stealing the passwords database for the moment, the traditional, and obvious, way to compromise a system is to find an exploitable, unpatched bug and use that to take control of the system.

The OpenBSD source tree available to the world (in fact, OpenBSD was the first project to make its CVS repository available via anonymous read-only CVS), but writable only by small group of developers. The developers tend to concentrate on the in-development -current but they do produce a small number of patches to the released version that become available via the patches page. If you want to study the OpenBSD development process more closely, there are a number of good articles and presentationas available via this page, but the important message is that insisting on code correctness and continuous code audits have yielded quite solid results. Exploiting any bugs you would be lucky enough to find in OpenBSD is harder than elsewhere.

But apart from patches to -stable, OpenBSD users usually do not update their systems by recompiling source. Neither do we usually run automatic system upgrades via an upgrade service. For packages, $ sudo pkg_add -u (assuming your PKG_PATH is set to something sensible) provides binary upgrades. For base system upgrades (the only time the boot sector code is likely to be updated), we tend to run the installer in upgrade mode and point it to a set of known good file sets. If the installer finds that a file set does not match the expected SHA256 checksum, it will warn you in no uncertain terms. The same messages will turn up if you, like me, tend to install snapshots on at least some systems as they become available, but sometimes forget to copy the correct bsd.rd into the right place.

The lesson here is that to taint an OpenBSD system, your most likely way in would be to replace a full set of installation files on your likely target's usual mirror, complete with checksums that makes the installer report your modified file sets as genuine. This has two obvious implications: one, you should never install anything on a mission critical system unless you find identical files on several different mirrors, and you should make sure that checksums from one mirror match the files on another. Watching the errata page for the release you are running is a good idea, too, of course.

This gives the proper context to the suspected exploit Dragos reported. The suspect still has some interesting properties. The boot sector code is very small and any changes would be relatively easy to spot once the system is running, but the code there runs early in the boot process and is there mainly to fetch other code that makes sure the system boots. The only useful modification here for an attacker would be to modify the boot loader code to load something that replaces the OpenBSD kernel and performs actions whatever the attacker wants.

The result of modifying the early stage boot code would more than likely be a trashed system unable to boot, but an appropriately skilled attacker who managed to insert code in the right places might be able to pass off a subtly modified system as a genuine one and keep it running for long enough to matter. That's why it will be very interesting to hear whatever real information becomes available about this suspected OpenBSD-targeting attempt.

If we are starting to see attack attempts specifically targeting OpenBSD systems, it could be an indication that at least some criminals have achieved a level of skill, or at least reached a new level of ambition. I have a strong feeling that the OpenBSD developers' efforts have paid off and creating a workable exploit will be very hard, but in the meantime, now is not the time to be slacking, even if all you critical system run OpenBSD. But you have a head start.

If you are curious about the status of the The Book of PF, second edition, preorders have started, and a PDF version is available right away. Physical copies will start shipping as soon as they exist, likely around November 10th.

EuroBSDCon 2010: The Finest Software Tool Is Alive And Well

2010-10-10T20:52:00.006+02:00

From mainframe replacements to firewall appliances, the BSD family of systems is a toolbox flexible enough to baffle insiders and newbies alike. EuroBSDCon 2010 was good fun.

I arrived in Karlsruhe on Thursday night, and ran into Erwin Lansing, Mark Linimon and a few other FreeBSD devsummit attendees at the perfect moment to tag along to dinner at a sort-of-greek place. Forgettable food, but fortunately the dark beer was quite drinkable and the various FreeBSDers made for good company.

Then up relatively early on the Friday. My own path through the conference started with the by now fairly familiar PF tutorial, which as you may be aware, is a close relative of The Book of PF, really soon now out in its second edition. Off the top of my head I'm not sure how many times I've given some version of the PF tutorial, but BSD-DK members will find that this edition of the slides borrows heavily from the somewhat swifter paced introduction they saw in Copenhagen this August.

Among my seven attendees were several who had hoped to be able to catch early copies of The Book of PF, second edition, which I had strongly hinted in the tutorial description would be available by now.

That did not happen, unfortunately, but it's getting very close -- last week entering final-really-final corrections to the index and the laid out book itself took up a good part of my non-office time, and the last word from my contact at No Starch was that the complete and final PDF would land in my inbox for final approval before going to the printers. Amazon.com now lists a likely delivery date of November 15th, which I for one think is a rather realistic guesstimate.

The tutorial went roughly as expected, unfortunately without live demos (demo equipment has a tendency to break badly during air transport or soon afterwards just in time muck up your presentation), and produced just enough good questions that it's likely useful to keep up the effort to maintain the tutorial. Slides are available from roughly where you would expect them.

After the session I ran into Thordur Bjornson (thib@) in the bar-cum-waiting area downstairs at the hotel, waiting for various other OpenBSD developers to arrive. This year's conference had a pleasantly larger than usual number of OpenBSD topics on the program, with some rather interesting talks scheduled for the first day. After a few beers we had reached critical mass with the arrival of among others Theo (deraadt@), Henning Brauer (henning@) and Felix Kronlage (fkr@) we found food and beer at a conveniently local eatery. Then back to the hotel bar for (slightly better) beer and mingling with arriving conference attendees and organizers.

The conference proper started on the Saturday with a "Software tools" themed keynote by Poul-Henning Kamp. PHK is always witty and fun to listen to, and he took us through a number of fresh perspectives on how, even though the world has changed dramatically in several ways and the BSDs still manage to kick ass, we need to keep up the effort to stay relevant.

FreeBSD jails have been a major attraction for quite a while, and I took in the two back to back jails talks by Bjoern A. Zeeb and james Britton. Both talks were fun refreshers on jails, with each presenting a preview of what may turn up in FreeBSD 9 jails code, plus of course tidbits like Bjoern's anectdote about setting up a million jails on the same physical server.

I was intending to attend thib@ and oga@'s OpenBSD on large memory systems talk, but was unfortunately diverted into a meeting that lead to me becoming slightly more involved in future EuroBSDCons. More details on that at a later time, the meeting ran for long enough that the next talk I did catch was reyk@'s iked(8) talk.

If you're on OpenBSD 4.8 or newer, man iked will give you the full story. If you're not that fortunate, it's nice to know that OpenBSD 4.8 gives you a new key exchange daemon for IPsec, up to date with the latest versions of all relevant protocols and able to handle all the nasty little details for IPsec communication with operating systems this column would rather not mention. A good talk about a very useful program, and during the questions part, Theo de Raadt pulled out OpenBSD 4.8 CD sets for attendees who had not already preordered to buy. It's almost a month until the official release date, but the CDs do exist and are likely on their way to early preorderers.

Henning's talk about the state of the OpenBSD networking code was good fun and to the point as always. No shocking new revelations for those who have followed the subject closely like yours truly, but do look out for this talk's slides when it hits the openbsd.org papers section along with the other EuroBSDCon 2010 presentations, hopefully soon.

The social event was conveniently placed in the hotel restaurant and bar area, where something called "Phönix Disco" had set up their equipment, including earth mover size speakers and a mirror ball hanging from the ceiling. Inbetween the dining noises and music, techie talk (at my table, OpenBSD internals, with a helping of ACPI insanities) could be heard. At some point they turned on their laser strobes, which made me queasy enough that I retired to my room soon enough to be in reasonable shape for the early Sunday morning sessions.

The 09:15 sessions on the Sunday offered a choice between Dru Lavigne's BSD Certification talk (highly recommended for your next event if you haven't taken it in already, she not only writes well, she's a brilliant presenter as well), and "Hacking NanoBSD for fun and profit" by Patrick Hausen, who has twisted the NanoBSD setup (originally intended for tiny machines) to serve as a basis for maintaining a hosting environment consisting mainly of regular-sized and capable servers. I'm already quite familiar with the bsdcertification.org efforts (and I recommend getting involved if you aren't already), so I decided to try Patrick's talk which turned out to be very enjoyable and presented some good ideas that could very well be carried over to other BSDs.

One other interesting talk that morning was Hans-Martin Rasch's "From Mainframe to FreeBSD", chronicling the gradual and successful migration by a subscriptions and mass mailings company from their legacy mainframe based system to an all-FreeBSD setup, of course shedding costs in the pretty serious range along the way.

The next time slots had a "Quo vadis ZFS" talk by Martin Matuska, that fortunately contained not the usual "see how great ZFS is" but rather focused on the challenges involved in using ZFS code, technical and legal as things stand today. The FreeBSD project seems to have concluded that the way ZFS is included in their code base does not pose legal problems in itself, but there could be other submarine issues. Apparently the NetApp vs Sun patent suit had ended with one patent invalidated and a settlement whose terms were not disclosed plus the remaining two patents under reexamination. Two unresolved patent issues would be enough to scare me off, but then again the ZFS feature set is perhaps too tempting to not take a few risks for.

Next up were espie@'s back to back talks about the *amazing* work he's been doing with OpenBSD packages. Before lunch, "the long road to pkg_add -u ... and beyond" which took us through the background and the design choices and evolution that took us to the point where upgrading your packages with pkg_add -u can be reliably expected to work. All I can say is that it was an excellent presentation about top-notch work that has made the life of OpenBSD users everywhere a lot better. The after lunch part, "efficient distributed package builds in OpenBSD" took us to the slightly more esoteric part of the world that contains the magic that makes sure the binary packages you expect to have at the other end of your pkg_add command actually exist. Another very enlightening talk, and this set of talks was certainly my favorite at this conference.

For further information on the topics mentioned in this column, the EuroBSDCon web site is the natural place to start. The OpenBSD developers' presentations will appear in the papers section of the OpenBSD web site, while my slides, as usual are available from the NUUG site.

The Goodness of Men and Machinery

2010-01-03T20:30:00.019+01:00

If you keep them to their promises, what will corporations and individuals do? Plus, the general goodness of OpenBSD and its installer.

Happy new year, everyone!

As I write this, 2010 is still quite new, and I'm writing this on a new laptop, running, of course, OpenBSD. I've been mostly quite happy with the ThinkPad R60 that has served me since late 2006, but a while back the fan started giving off ugly noises and the organization nominally in charge of doing Lenovo repairs in my area seemed quite uninterested in actually doing repairs on the unit.

So after the usual browsing of web shops I ended up going for a simpler ThinkPad, the SL500 model. From the looks of it, the machine would be a tad faster and support more physical memory than the one I had already, and most likely the newer Intel processor would support running in true 64-bit (amd64) mode.

Click the buttons, whip out the credit card and away we go. Of course, the next day the repair shop turned called in to say that they could in fact get that fan repair done after all. So now I had a once-more-silent and oldish but working machine and a new one on the way.

The new one would come with Microsoft Windows XP preinstalled, Vista Business recovery media and an option to upgrade to Windows 7. I certainly did not want any of those. I make my living in free software consulting, with a bit of promoting and training thrown in and could credibly be denoted part of the competition. Microsoft's End User License Agreement in most of its incarnations (there are several) promises a full refund if you find their licensing terms unacceptable.

So with a minimal delay I fired off an email (sorry, Norwegian only) to what I thought was the right addresses at the retailer, saying essentially that I would want to arrange for the return of the software, in accordance with Microsoft's standard contract terms.

After all, we had heard of successes such as MacSlow in Germany and Poul-Henning Kamp's (of FreeBSD fame) slow but steady progress, also involving Lenovo hardware.

I expected that the response would be either that they have a process set up to make a symbolic refund, or they would refuse, saying that the software and the hardware are inseparable parts of the product, no matter what the clickwrap screen says.

Their actual response was mostly the latter, but with the twist that 'we do not get refunds for this from the manufacturer'. Distinctly odd, if you ask me, but I suppose we will find out more once people with actual decision making power are back after the holidays.

So no other alternative than start establishing the facts of the matter, that is, wait for the package to turn up and see what the clickwrap screen actually says.

Windows Refund Norway: The Fact Finding Mission
The package arrived this Saturday, and after the outer wrapping came off, sure enough there was a warning on the outside of the box:

That was pretty much as expected, so I continued unpacking.

Plugging in and turning on gave me this screen:

That eventually segued into

('please wait while Windows is preparing for startup') and proceeded through a few more largely information-free screenfuls up to the the point where you choose your location.

That choice likely serves several purposes. It's useful for choosing user interface languages as well as the legal terms for using the software or not, so I proceeded.

Finally, the Microsoft End User Agreement, Norwegian version, started to appear, in a miniature text box that needed several scrolling motions to reveal much of anything:

It would take only a minimal dose of negativity to suspect that these screens were in fact designed to make the user just click agreement without ever reading the full text. But taken together, these screenfuls gave me all the information I needed.

Microsoft promises a refund if you find their license terms unacceptable, but farms out the responsibility for the refund to the manufacturer, not the retailer.

So I had been barking up the wrong tree after all. Fortunately, in the meantime my friends at NUUG offered some good advice on how to proceed, and I will likely be offering updates on "Windows Refund, Norwegian version" as the tale progresses.

The promise of a refund has clearly been made. I suspect that with a sufficiently staffed legal team, playing a symbolic blame game over who should honor that promise is enough of a deterrent to the average end user, if they do not just go away after the initial rejection.

The fact-finding mission complete, it was time to try installing OpenBSD. If you are not at all interested in OpenBSD, you can scroll to the end of the article for more on the Windows refund and legalities part.

The Further Adventures of the OpenBSD Installer
The OpenBSD installer has a somewhat ugly reputation, as witnessed by this recent article over at TechRepublic. In all fairness, some years back one of the developers was quoted as saying about the installer, "it was never released, it just kind of escaped".

But things have change since then, to the point that I was wondering just which version of OpenBSD Mr. Wallen had tried to get running on his hardware, nevermind that he seems totally unaware of the automatization features you can exploit by simply adding hostnameVv.tgz or siteVv.tgz (Vv being Major.minor version numbers) file sets to your install media.

The following sequence shows you what it really looks like, with my most sincere apologies for my near-total lack of photography skills.

Booting from the CD image from the latest amd64 snapshot, my first screenful was

This proceeds through what we 'greybacks' recognize as kernel messages that scroll of the screen rather rapidly (dmesg output of the installed system is available here), ending with a menu of possible actions:

Choosing (I)nstall produces an encouraging message and a prompt to choose your preferred keyboard layout:

Next up, you get to choose a hostname

followed by a prompt which network interfaces you want to configure.

This is where I hit my first and only snag. I had tried to do the install with only wireless networking available, and for reasons known only to Intel, they have not allowed the OpenBSD project redistribution rights to the firmware files that turn the Intel WiFi Link 5100 circuitry into a working wireless networking component.

Here's what it looks like when a manufacturer chooses not to play nice with free software:

Fortunately, the OpenBSD man page for the iwn driver shows you just how to fetch and install the firware via the OpenBSD package tools. Other manufacturers have granted the OpenBSD project redistribution rights to similar firmware files needed by their products, with the result that you can perform a network install of OpenBSD directly from a typical bsd.rd over a Ralink wireless network card (using the ral driver), but not over most of the wireless network cards from Intel and some other manufacturers.

I was in fact prepared that this exact thing would happen, and it was time to move up to the attic where the wired Ethernets live. (Our house is an 18th century wooden building, and my sweetheart wanted no more holes drilled, certainly not to accommodate Ethernet cabling for her and others to trip over).

Restarting the install with a wired Ethernet got me through the entire sequence so far inside one screenful.

Once I told the installer I was done with network configuration, it naturally used all the configuration information my DHCP server had supplied:

You can go in with manual network configuration if you want, but in my case that was not needed.

Next up, your root account will need a password, you choose to run or not run some basic services such as sshd and ntpd, and you can choose to add a regular user too.

Notice the second part of the dialogue here, where the installer notices that I added a regular user and offers to disable root logins over ssh, adding the regular user to the wheel group so it's easier to add the regular user to sudoers.

The installer does not edit sudoers for you, but it's the little things like these that warms the hearts of greybacks like me.

And naturally, the installer correctly guesses my time zone but offers me a prompt to change it if the guess was not the correct one.

Network setup done, the installer presents the choice of disks available and asks you to specify which device will contain the root file system. In this case my new laptop had only one usable disk, so I proceeded directly to the dreaded partitioning part.

The first part here shows the disk partitioning before the OpenBSD installer touched it, with an option to either use the whole disk or edit the Master Boot Record (MBR). I did not plan to run several operating systems on this machine, at least not natively, so I opted for using the whole disk for OpenBSD instead.

That lead to the second part, where the installer suggests a disk partitioning scheme based on typical use, with the utterly sane choice of leaving the largest part (the k partition in this case) for user files as the /home partition.

You can edit the setup or choose your own entirely, the main point here is that the defaults are sane and unless you know a good reason to choose differently, the default choice is the safe and comfortable choice.

The installer goes on to create and mount file systems,

and the informed reader will recognize that the default mount options make a lot of sense, too.

With file systems mounted, the actual installation can proceed once you have indicated where you want to fetch the file sets from. Here I chose the http install method, and the installer made a reasonable guess at what was my closest mirror. I chose ftp.eu.openbsd.org (based in Sweden) instead, and chose not to exclude any file sets.

The next part is when you can go fetch your cup of coffee, get some other refreshment or simply a breath of fresh air. Even on a very good link, the transfers will take a few minutes.

When you get back, the installer prompts for further install sets, and if you do not have any to offer, it will finish up the install, choose the right kernel for your hardware and offer some advice for how to proceed from here on.

That is, enter reboot, let the system boot, log in as your new user or root if you did not create a user as prompted, read the (almost) personalized message from Theo de Raadt, and go on using the system as you please. Actually, you can skip reading the message if you like, but the time is likely well spent with the tips (or reminders if you like) it contains.

Here is a photograph of the disk utilization of my newly installed OpenBSD laptop a few moments after I had started transferring my files over. Notice the /home file system is not quite empty, since I almost forgot to try to document the pristine state of the system, but I am reasonably sure this picture was taken before I added any packages.

So please, Mr. Lenovo, may I have that Windows money refunded?

As the numbers add up, there is no space for your precious proprietary software on my system, and I would gladly offer to send those restore CDs and the license label back.

It was indeed Microsoft that made that promise on your behalf, but then you must have been aware of these legal issues.

I am not about to abuse your trust or your software. However I do like the hardware that I have bought, am sure we will come to a reasonable agreement.

It's all about honesty in your business life.

Now back to the tech. After installing a few packages, my desktop looks roughly like this,

and disk utilization is
peter@deeperthought:~$ df -h Filesystem Size Used Avail Capacity Mounted on /dev/sd0a 1005M 49.8M 905M 5% / /dev/sd0k 263G 44.5G 205G 18% /home /dev/sd0d 3.9G 40.0K 3.7G 0% /tmp /dev/sd0f 2.0G 1.3G 542M 72% /usr /dev/sd0g 1005M 178M 776M 19% /usr/X11R6 /dev/sd0h 5.9G 2.1G 3.5G 38% /usr/local /dev/sd0j 2.0G 2.0K 1.9G 0% /usr/obj /dev/sd0i 2.0G 732M 1.2G 38% /usr/src /dev/sd0e 9.8G 51.5M 9.2G 1% /var
Yes, there's probably a lot of old data there I don'really need. But then, the time it takes to identify and remove old cruft is hard to come by.

Good night and good luck.

Raw size versions of the illustrations are available here, other updates after the footer.

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact FreeCode (via email or other means) to make arrangements.

Other updates: The password guessing Hail Mary Cloud is still with us, and I occasionally update the data referenced in that article. As to the antics of the various spammers, they haven't changed much either. They're still fun to watch from time to time, though.

As should be fairly obvious from the above, this article was produced using only free software: OpenBSD and various software available through the OpenBSD packages system.

Rickrolled? Get Ready for the Hail Mary Cloud!

2009-11-15T14:21:00.005+01:00

If you publish your user name and password, somebody who is not you will use it, sooner or later.

It's been a fun few weeks. Over in Microsoft land, it must have been a big issue that according to malware hunters Sophos, the newly released Windows 7 with no extras is roughly as vulnerable as its older siblings. No great surprises there, I suppose.

For those of us with a more Unixish leaning, the more interesting piece of news involved Apple iPhones. These phones apparently run a version of MacOS that has enough Unix in it that with a bit of tinkering, it is possible to install a variety of Unix software, such as the ubiquitous secure shell daemon sshd. To some, there is a certain attraction in knowing that you have an SSH server in your pocket or handbag. Too bad, then that enough of those adventurous iPhone owners never read up on the instructions and chose to run their toy with the default password for the root account and were vulnerable to a wonderful prank perpetrated by a programmer down under.

The prank (described in the inimitable The Register style here) demonstrated just how bad an idea it is to publish your user name and password. If you followed the news around last weekend you would notice that a large segment of the Microsoft-attached instapunditry got their facts wrong as usual with the this proves that Apple (and by extension any Unix and of course Linux) is just as vulnerable as Microsoft mantra repeated over and over.

In fact, there are two historical incidents that point to Unix being no silver bullet: the 2002 Linux Slapper Worm and the original network-enabled worm, the 1988 Morris Worm. Those two prove mainly that yes, some bugs are exploitable, and the way forward is to fix bugs and make them harder to exploit in the first place (alternates here and here). Now these two famous exploits is possibly to be joined by a third, the rickrolling prank.

I beg to differ. The rickroller is about bad passwords, no more, no less. I've spent considerable time ranting about passwords in earlier columns, and this incident only underscores what we've been repeating until your eardrums wear thin an my vocal cords swell from exhaustion:

Publishing your username and password is a really bad idea.
It's almost as bad as picking a guessable password.

Add to this that the fact, as we've noted here earlier, there is a whole cloud of hijacked machines out there beavering away at guessing passwords right now, and they have been at it for quite a while.

The most remarkable of these botnets is the one that tries to avoid detection by distributing the password guessing for any target across a large number of hosts, so each guesser never shows high enough rates of activity to trigger any of the traditional bruteforce detection mechanism. The attempts look something like this in your authentication log:
Nov 13 21:10:14 rosalita sshd[50401]: error: PAM: authentication error for illegal user mars from 125.40.69.208 Nov 13 21:10:14 rosalita sshd[50401]: Failed keyboard-interactive/pam for invalid user mars from 125.40.69.208 port 38052 ssh2 Nov 13 21:11:20 rosalita sshd[50419]: reverse mapping checking getaddrinfo for 115-186-131-90.nayatel.pk [115.186.131.90] failed - POSSIBLE BREAK-IN ATTEMPT! Nov 13 21:11:20 rosalita sshd[50419]: Invalid user mars from 115.186.131.90 Nov 13 21:11:21 rosalita sshd[50419]: error: PAM: authentication error for illegal user mars from 115.186.131.90 Nov 13 21:11:21 rosalita sshd[50419]: Failed keyboard-interactive/pam for invalid user mars from 115.186.131.90 port 42235 ssh2 Nov 13 21:13:43 rosalita sshd[50428]: Invalid user mars from 58.247.222.163 Nov 13 21:13:43 rosalita sshd[50428]: error: PAM: authentication error for illegal user mars from 58.247.222.163 Nov 13 21:13:43 rosalita sshd[50428]: Failed keyboard-interactive/pam for invalid user mars from 58.247.222.163 port 35134 ssh2 Nov 13 21:15:59 rosalita sshd[50440]: Invalid user mars from 89.76.186.99 Nov 13 21:16:00 rosalita sshd[50440]: error: PAM: authentication error for illegal user mars from chello089076186099.chello.pl Nov 13 21:16:00 rosalita sshd[50440]: Failed keyboard-interactive/pam for invalid user mars from 89.76.186.99 port 52156 ssh2 Nov 13 21:17:16 rosalita sshd[50448]: Invalid user mars2 from 88.134.166.31 Nov 13 21:17:16 rosalita sshd[50448]: error: PAM: authentication error for illegal user mars2 from 88-134-166-31-dynip.superkabel.de Nov 13 21:17:16 rosalita sshd[50448]: Failed keyboard-interactive/pam for invalid user mars2 from 88.134.166.31 port 39254 ssh2 Nov 13 21:18:14 rosalita sshd[50452]: Invalid user room from 62.198.66.107 Nov 13 21:18:14 rosalita sshd[50452]: error: PAM: authentication error for illegal user room from 62.198.66.107 Nov 13 21:18:14 rosalita sshd[50452]: Failed keyboard-interactive/pam for invalid user room from 62.198.66.107 port 47557 ssh2 Nov 13 21:19:27 rosalita sshd[50458]: Invalid user room from 61.74.75.43 Nov 13 21:19:27 rosalita sshd[50458]: error: PAM: authentication error for illegal user room from 61.74.75.43 Nov 13 21:19:27 rosalita sshd[50458]: Failed keyboard-interactive/pam for invalid user room from 61.74.75.43 port 57794 ssh2 Nov 13 21:21:41 rosalita sshd[50472]: Invalid user room from 212.243.41.9 Nov 13 21:21:41 rosalita sshd[50472]: error: PAM: authentication error for illegal user room from 212.243.41.9 Nov 13 21:21:41 rosalita sshd[50472]: Failed keyboard-interactive/pam for invalid user room from 212.243.41.9 port 38396 ssh2 Nov 13 21:22:58 rosalita sshd[50491]: reverse mapping checking getaddrinfo for static-ip-cr1901468058.cable.net.co [190.146.80.58] failed - POSSIBLE BREAK-IN ATTEMPT! Nov 13 21:22:58 rosalita sshd[50491]: Invalid user room from 190.146.80.58 Nov 13 21:22:58 rosalita sshd[50491]: error: PAM: authentication error for illegal user room from 190.146.80.58 Nov 13 21:22:58 rosalita sshd[50491]: Failed keyboard-interactive/pam for invalid user room from 190.146.80.58 port 4420 ssh2 Nov 13 21:24:01 rosalita sshd[50509]: Invalid user room from 62.23.130.173 Nov 13 21:24:01 rosalita sshd[50509]: error: PAM: authentication error for illegal user room from host.173.130.23.62.rev.coltfrance.com Nov 13 21:24:01 rosalita sshd[50509]: Failed keyboard-interactive/pam for invalid user room from 62.23.130.173 port 3904 ssh2 Nov 13 21:25:21 rosalita sshd[50517]: reverse mapping checking getaddrinfo for hn.kd.ny.adsl [125.40.69.208] failed - POSSIBLE BREAK-IN ATTEMPT! Nov 13 21:25:21 rosalita sshd[50517]: Invalid user room from 125.40.69.208 Nov 13 21:25:21 rosalita sshd[50517]: error: PAM: authentication error for illegal user room from 125.40.69.208 Nov 13 21:25:21 rosalita sshd[50517]: Failed keyboard-interactive/pam for invalid user room from 125.40.69.208 port 3294 ssh2
and so on.

I put it to you: What you see here is the cybercrime equivalent of the Hail Mary Pass.

Each attempt in theory has monumental odds against succeeding, but occasionally the guess will be right and they have scored a login. As far as we know, this is at least the third round of password guessing from the Hail Mary Cloud (see the archives for earlier postings about slow bruteforcers), but there could have been earlier rounds that escaped our attention.

The fact that we see the Hail Mary Cloud keeping up the guessing is a strong indicator that there are a lot of guessable passwords and possibly badly maintained systems out there, and that even against the very long odds they are succeeding often enough in their attempts to gain a foothold somewhere that it is worth keeping up the efforts. For one thing, the cost of using other people's equipment is likely to be quite low.

There are a lot of things about the Hail Mary Cloud and its overseers that we do not know. People who responded to the earlier articles with reports of similar activity also reported pretty consistently something like a sixty to seventy percent match in hosts making the attempts.

With 1767 hosts in the current sample it is likely that we have a cloud of at least several thousand, and most likely no single guessing host in the cloud ever gets around to contacting every host in the target list. The busier your SSH deamon is with normal traffic, the harder it will be to detect the footprint of Hail Mary activity, and likely a lot of this goes undetected.

The data, as I am sure you have been waiting for it, is available in these forms: Raw log data, with 3-4 lines per attempt (as illustrated above and requested by some correspondents), one line per attempt (shows the pattern a little more clearly), a list of the hosts participating in the Hail Mary Cloud sorted by number of attempts, and the user names attempted, sorted by number of attempts.

The pattern is fairly familiar by now, but this time the alphabetic cycles are shorter and at times the coordination seems to have broken down. My guess is that the apparent breakdowns are due to silly factors like the guessing machines running without time synchronization or other signs of incompetence.

And finally, some words of advice for those of you who want to avoid both rickrolling and getting cracked by other password guessing.

You should at least consider setting a password policy and enforcing it with something like John the ripper, which more than likely is available at the cost of a few keystrokes from your package system. And of course there is the fine art of sshd configuration. Some of the things you could do are, in no particular order:

disable root logins over the network
use packet filtering or other means to restrict where users can log in from
disable password logins entirely allowing only key-based logins
set up your sshd to listen on a non-standard port

whatever your users can bear to live with.

If you see traces of the Hail Mary Cloud's activity in your logs and you want to share and study, I would very much like to hear from you. I will most likely be updating the log data and extracts at intervals.

A Third Time, Uncharmed

2009-10-04T22:10:00.013+02:00

Spamwashers hijacked, a wake-up call for lazy sysadmins everywhere. The slow bruteforcers are back for another round.

A new round of slow, distributed bruteforce attacks is in progress. Just like the other times we know about (see references later), the initial target is root. This time around I see only one of my ssh-contactable machines targeted, and the dribble started on September 30th.

I've put the raw data so far up for study here (a total of 6067 attempts), and a list of hosts sorted by number of attempts (the first column) can be found here (770 hosts, with up to 32 attempts each). Quite likely I'll be collecting more data and publishing updates when I have a few free moments.

A number of people were kind enough to contact me in the followup of the earlier articles, and from one of my correspendents (who asked not to be named) I learned that the likely culprit is a piece of Linux malware known as dt_ssh5. If you type dt_ssh5 into your favorite search engine, it will turn up a few hits, but significantly fewer than the number of hosts in my sample. A couple of those documents have some analysis of how a badly secured web application let the miscreants in.

We should not be surprised that whoever is behind this has more than one takeover method in their arsenal. Unfortunately it should not surprise anybody either that we're now seeing a third round of those attacks. Most likely the perpetrators keep going because they occasionally succeed, and when they do, it's because every now and then they luck on a Linux machine with either

a maintenance regime that's disorganized enough that software with known and exploitable bugs is left in place for long enough to open the doors to undesirables, or

at least one user (whoever is manning root or any of the other user IDs we know they will be sniffing out later) with a guessable password and a system administration regime that lets weak passwords exist in the first place.

You see where this is heading? Over the last few years we have seen Linux and other free systems take over ever more niches and even mission critical application areas in businesses and organizations all over the world. I for one think this move to free systems with source code accessible is a good thing.

But there is a downside: In our efforts to entice the suits into the wonderful new world of free software, we likely oversold the security part.

Yes, bugs are easier to find and fix when the source code is available, yes, the Unix security model or some variant thereof is vastly preferable to that disorienting hellhole system marketed from somewhere up Seattle way, and yes, there are a few hundred more good and valid reasons (technical and otherwise) why basing your business on free license software is generally a good idea. Security-wise you have a better starting point, and with a little effort you will gain control and stay in control. But it does take some degree of effort by one or more qualified persons who are willing and able to take charge and make sensible decisions.

Looking at the data earlier I can see that one of the hosts now trying to gain illegitimate access to one of my systems was originally set up as a "spam washer" (grep for spamvask in the data). Setting up a Linux box to filter spam is likely a good idea in many contexts, but please keep this in mind: The fact that your rig runs Linux does not mean you're home free. You need to keep paying attention. When your spam washer has been hijacked and tries to break into other people's systems, you urgently need to get your act together, right now.

We do not have self-propagating worms on Linux just yet (and in all fairness the likelihood of that ever happening is rather slim), but we certainly do not have a system that will stay secure if you ignore the basics of useful system administration. This recurring episode of dt_ssh5 and the slow bruteforcers should serve as a wakeup call for system administrators everywhere: Your system stays secure only as long as you pay proper attention to maintenance. Out there in the real world there are at least 770 machines where the maintenance regime slipped, either through incompetence or work overload or a combination of the two with a dash of bad planning thrown in.

The only way to make a fourth round of slow bruteforcers not happen is to make the people responsible for the 770 hosts listed here do the right thing. I suppose we will find out soon enough how many of those domains actually have the RFC2142-mandated mailboxes in place.

References: Previous posts on this topic include A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009).

The reason for my rather long silence in the blogosphere can be summed up in two words: Client Confidentiality. Some of the projects I've been working on might have been material for interesting articles, but separating the generally useful bits from what the customer could reasonably expect to be kept confidential has proved difficult, at least in the short run.

Update 2009-10-05 just before 10:00 CEST: Slightly newer log extracts uploaded, total attempts now 6590, from 775 hosts. I will likely upload fresher versions at intervals.

Update 2009-10-14 19:30ish: After a couple of days with literally none to three or four attempts per hour, my afternoon log check turned up activity almost at start of run levels, so total numbers are now 9405 attempts from a total of 946 unique hosts. I never bothered to change the file names, but the data have been updated a couple of times each day since I started publishing them.

Update 2009-10-29 00:10: Because a file transfer took a little longer than expected, I was awake to see what may actually be the start of the alphabetic phase -

Oct 28 23:58:35 rosalita sshd[61664]: Invalid user anthony from 62.225.63.99 Oct 28 23:58:35 rosalita sshd[61664]: error: PAM: authentication error for illegal user anthony from 62.225.63.99 Oct 28 23:58:35 rosalita sshd[61664]: Failed keyboard-interactive/pam for invalid user anthony from 62.225.63.99 port 58683 ssh2 Oct 28 23:59:43 rosalita sshd[61668]: Invalid user anthony from 217.136.253.239 Oct 28 23:59:43 rosalita sshd[61668]: error: PAM: authentication error for illegal user anthony from 239.253-136-217.adsl-static.isp.belgacom.be Oct 28 23:59:43 rosalita sshd[61668]: Failed keyboard-interactive/pam for invalid user anthony from 217.136.253.239 port 54728 ssh2 Oct 29 00:00:26 rosalita sshd[61695]: Invalid user barbara from 112.78.124.159 Oct 29 00:00:27 rosalita sshd[61695]: error: PAM: authentication error for illegal user barbara from 112.78.124.159 Oct 29 00:00:27 rosalita sshd[61695]: Failed keyboard-interactive/pam for invalid user barbara from 112.78.124.159 port 56990 ssh2 Oct 29 00:03:42 rosalita sshd[61722]: Invalid user brian from 122.224.128.197 Oct 29 00:03:42 rosalita sshd[61722]: error: PAM: authentication error for illegal user brian from 122.224.128.197 Oct 29 00:03:42 rosalita sshd[61722]: Failed keyboard-interactive/pam for invalid user brian from 122.224.128.197 port 47058 ssh2 Oct 29 00:04:21 rosalita sshd[61725]: Invalid user brian from 218.69.27.138 Oct 29 00:04:21 rosalita sshd[61725]: error: PAM: authentication error for illegal user brian from 218.69.27.138 Oct 29 00:04:21 rosalita sshd[61725]: Failed keyboard-interactive/pam for invalid user brian from 218.69.27.138 port 41622 ssh2 Oct 29 00:05:00 rosalita sshd[61738]: Invalid user carol from 58.60.106.121 Oct 29 00:05:01 rosalita sshd[61738]: error: PAM: authentication error for illegal user carol from 58.60.106.121 Oct 29 00:05:01 rosalita sshd[61738]: Failed keyboard-interactive/pam for invalid user carol from 58.60.106.121 port 55916 ssh2 Oct 29 00:05:51 rosalita sshd[61744]: Invalid user carol from 200.248.242.218 Oct 29 00:05:52 rosalita sshd[61744]: error: PAM: authentication error for illegal user carol from 200.248.242.218 Oct 29 00:05:52 rosalita sshd[61744]: Failed keyboard-interactive/pam for invalid user carol from 200.248.242.218 port 54547 ssh2 Oct 29 00:07:24 rosalita sshd[61748]: Invalid user charles from 222.128.36.60 Oct 29 00:07:24 rosalita sshd[61748]: error: PAM: authentication error for illegal user charles from 222.128.36.60 Oct 29 00:07:24 rosalita sshd[61748]: Failed keyboard-interactive/pam for invalid user charles from 222.128.36.60 port 42322 ssh2 Oct 29 00:08:09 rosalita sshd[61755]: Invalid user christopher from 72.148.242.188 Oct 29 00:08:10 rosalita sshd[61755]: error: PAM: authentication error for illegal user christopher from adsl-072-148-242-188.sip.asm.bellsouth.net Oct 29 00:08:10 rosalita sshd[61755]: Failed keyboard-interactive/pam for invalid user christopher from 72.148.242.188 port 37157 ssh2 Oct 29 00:09:06 rosalita sshd[61758]: Invalid user christopher from 58.223.251.232 Oct 29 00:09:06 rosalita sshd[61758]: error: PAM: authentication error for illegal user christopher from 58.223.251.232 Oct 29 00:09:06 rosalita sshd[61758]: Failed keyboard-interactive/pam for invalid user christopher from 58.223.251.232 port 46350 ssh2 Oct 29 00:09:48 rosalita sshd[61762]: Invalid user daniel from 64.69.114.115 Oct 29 00:09:49 rosalita sshd[61762]: error: PAM: authentication error for illegal user daniel from mulligan.softspike.org Oct 29 00:09:49 rosalita sshd[61762]: Failed keyboard-interactive/pam for invalid user daniel from 64.69.114.115 port 39960 ssh2 Oct 29 00:11:29 rosalita sshd[61781]: Invalid user david from 80.255.179.150 Oct 29 00:11:29 rosalita sshd[61781]: error: PAM: authentication error for illegal user david from ppp-vpdn-80.255.179.150.yarnet.ru Oct 29 00:11:29 rosalita sshd[61781]: Failed keyboard-interactive/pam for invalid user david from 80.255.179.150 port 49013 ssh2
More material later, for now I'd be happy to hear confirmation (reports of similar activity) if you see it in your logs.

The slow brute zombies are back

2009-04-12T22:17:00.005+02:00

Real-life zombies feed off weak passwords.

Regular readers will remember that late last year we saw a peculiar form of distributed bruteforce attack on certain ssh servers. What made this particular batch of failed login attempts stand out was that unlike the traditional rapid-fire bruteforce attempts we were quite adept at heading off with state tracking tricks (such as the OpenBSD PF method described here or a slightly different Linux-specific method), the technique seemed to be specifically designed to slip past such defenses.

I described the method and the evidence at some length in a series of colums, A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008) and finally The slow brutes, a final roundup (January 22, 2009).

If you do not want to read through all of that, I'll recap: The traditional bruteforce attack is characterized by somebody trying again and again to find a combination of logon credentials (typically sets of user names and passwords) that will let them gain access to the system. The basic idea is that given enough tries, you will sooner or later find a user who has set a guessable password, and you have access.

One of the important weaknesses of the method is that the typical attacker would have gained access to only one or a few systems, and the attacker would then typically try to connect a lot more often than a typical user, and sysadmins learned to adapt their firewalls and other systems to lock out activity that fit the bruteforcer profile.

If you run a ssh service anywhere Internet-facing, you will be used to seeing a steady stream of failed logons for both existing and non-existing users. There's nothing new in seeing failed logons in your log files. However, what happened late last year was that we started seeing large numbers of failed ssh logon attempts, with the new twist that the same user would be trying to log on a large number of times, but never from the same place twice in rapid succession. This log data sample will give you an idea. The data will show you the pattern, as will the summary article.

Back then in January, my conclusion was that we had seen the conclusion of a largely failed experiment. After all, proceeding at a truly glacial pace and decreasing the number of attempts per user name as they went along hardly seemed like a recipe for success even if they could sneak past packet filtering based defenses.

It turns out I was wrong. Returning to my log summaries after taking a few days off from log data (yes, easter is big here, even moves geeks sometimes to take time off) I find data with almost exactly the same profile as last December's series.

This can only mean that there were enough successful attempts at guessing people's weak passwords in the last round that our unknown perpetrators found it worthwhile to start another round. For all I know they may have been at it all along, probing other parts of the Internet far from my unfasionable backwaters.

Anyway, they are most certainly back. A summary of the data so far follows in the concluding section. Please note that I would be very interested in communicating with other parties who see similar activity in their logs, and if you are responsible for one or more of the systems identified or you know who is, I urge you to take appropriate action.

The Data So Far

First, We Take root
The full log data reveal that they started rubbing up against my listening posts during the early hours of April 7th 2009 (as measured in CEST), concentrating first on the user root (attempts sorted by hostname here, hostnames here. Not terribly surprising that they tried to take on this one, as most Unix(ish) systems tend to at least have a root account.

Then We Take admin
After about twenty-four hours, the zombies in the cloud tired of root and turned their collective attention to admin (attempts sorted by hostname here, hostnames here. This is a user name I would not expect to be on a Unix syste by default. In my limited experience Microsoft systems tend to use this one more often, but then there may be enough of those systems out there with a SSH service and weak passwords out there to try.

And james Will Make Us Famous
Well into civilized lunchtime (CEST) on April 9th, our would-be guests turned their attention to james (attempts sorted by hostname here, hostnames here. Why this user name received so much attention is a complete mystery to me. They did tire in the end, though.

Before We Turn To 123456, aadi And The Rest Of The Rabble
As if to confirm that we are indeed seeing a rerun of last November and December's sequence of events, the robots started on their regular alphabetic attempts on April 11th:
Apr 11 17:44:15 rosalita sshd[51241]: error: PAM: authentication error for illegal user james from 221.130.177.154 Apr 11 17:50:47 rosalita sshd[51258]: error: PAM: authentication error for illegal user 123456 from 220.194.54.41 Apr 11 17:53:24 rosalita sshd[51262]: error: PAM: authentication error for illegal user 123456 from webmail.sistemafieg.org.br Apr 11 17:58:25 rosalita sshd[51285]: error: PAM: authentication error for illegal user 123456 from 202.82.25.161 Apr 11 18:00:59 rosalita sshd[51307]: error: PAM: authentication error for illegal user aadi from mail.cgconsultoriacontabil.com.br
The various comments to my earlier columns tended to point out the near-futility of low-intensity bruteforcing, and I took the fact that the last series of attempts did not run to the end of the alphabet as an indication that the perpetrators themselves had reached the same conclusion.

Now we know that they just moved on, went elsewhere for a while and now they're back. In the meantime, they must have hit on enough weak minds and associated weak passwords that they were able to take over and zombiefy enough systems to be worth their while.

The data so far (collected up to April 12, 2009, roughly 21:00 CEST) indicates that the total number of hosts involved in the attempts at my listening posts is just over a thousand (a list of hosts available here).

It is probably worth repeating that in real life, zombies feed on both weak minds and weak passwords.

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available for free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact FreeCode to make arrangements.

Also worth noting, I will be doing a PF tutorial at BSDCan 2009 in May, and I will be staying around for the rest of the conference. I look forward to seeing you there.

Oh yes, you signed up for this. You did. Honest.

2009-03-21T19:41:00.004+01:00

Honesty in marketing. You may have heard of it.

It may come as a surprise to some, but I generally do not spend much time on spam related matters. Occasionally I need to do some manual labor to keep spamd and spamassasin in trim, but at most times my little robot helpers just keep running, leaving my desktop essentially spam free.

That changed slightly late last month. Messages hawking the oddest wares started arriving, with a largish number of messages claiming that I had actually signed up to receive them:
You are receiving this message because on 2/26/2009 at 3:57 PM peter@bsdly.net 64.12.116.10 registered to receive messages from e-researchcouncil.com and its partners. To change your preferences with e-researchcouncil.com, go to the website and select "Contact Us" to review your options.
To give you an idea how likely that statement is to be true, consider this: The 64.12.116.10 address resolves back to somewhere in America Online's network, pretty much an ocean and then some away from where I'm usually located.

I assume entering my address into a few web forms is somebody's idea of a joke, and the net effect was that a number of spammy messages started appearing in my mailbox, starting on February 27th. Only about third of the messages contained that particular claim, and a typical message would contain headers like these:
X-From-Line: eHarmonyDating@BranchSprint.com Fri Feb 27 16:30:36 2009 Return-path: <3f5.4.73479158-21937306@BranchSprint.com> Envelope-to: peter@bsdly.net Delivery-date: Fri, 27 Feb 2009 19:15:13 +0100 Received: from [99.198.152.161] (helo=dns7-cronomagic-biz.BranchSprint.com) by skapet.bsdly.net with esmtp (Exim 4.69) (envelope-from <3f5.4.73479158-21937306@BranchSprint.com>) id 1Ld7Eu-00074N-NF for peter@bsdly.net; Fri, 27 Feb 2009 19:15:13 +0100 X-Gnus-Mail-Source: pop:peter@bsdly.net Message-Id: <KKcbjdhdagmcfbVN@BranchSprint.com> Reply-To: <eHarmonyDating@BranchSprint.com< From: eHarmonyDating <eHarmonyDating@BranchSprint.com> Subject: eHarmony could match you with the right singles Date: Fri, 27 Feb 2009 16:30:36 GMT X-Information: 73479158_21937306 ListZA251 X-Complaints-To: <complaints@BranchSprint.com> To: <peter@bsdly.net>
My first impulse was, in case this is an honest mistake somewhere, let's try and play nice at first. That meant sending messages to the X-Complaints-To: addresses and waiting to see what would happen.

You should not be terribly surprised to hear that those addresses all turned out to be invalid, the messages undeliverable.

In the meantime, I went on collecting messages, and the amount of data I had accumulated was large enough that I could reach some preliminary conclusions.

It's obvious that in order to reach me, the messages would need to clear greylisting and avoid triggering too many of my spamassassin rules. That meant in turn that the messages were sent using real mail servers. So I started collecting the messages with that claim for further study. The messages were almost all sent from a few distinct subnets, all of them apparently fairly well stocked with real mailservers.

Based on data from the spam messages and whois lookups and the larger groupings of messages, the professional spammers are, for your convenience in case you want to visit them:

NN, LLC
4001 Kennett Pike
Suite 134-910
Greenville, DE 19807
US

Spiesigma PLC
P.O. BOX 243, 2221 S Webster Ave
Green Bay, WI 54301
US

GreenButtonMedia.com
5580 La Jolla Blvd # 73
La Jolla, CA 92037
US

AdSelectMedia.com
5482 Wilshire Blvd. #302
Los Angeles, CA 90036
US

BestOnlineGreetings.com
5482 Wilshire Blvd. #302
Los Angeles, CA 90036
US

MyPromotionNetwork.com
970 West Valley Parkway
Suite 604
Escondido, CA 92025

GreatTechsOnline.com
5580 La Jolla Blvd # 73
La Jolla, CA 92037
US

CrownVenturesMedia.com
7127 Hollister Ave., Suite 25A, #145
Goleta, CA 93117

Top Notch Media, Inc.
1735 Market Street · Suite A · PMB 429
Philadelphia, PA 19103-7588

In addition, some of the domain names used in the spam messages were registered via an anonymizing service whose whois data comes out as:

Dynamic Dolphin Privacy Protect
5023 W 120th Ave #233
Broomfield
null,80020

The spam volume from all of them swelled at roughly the same time, so it is likely that they cooperate on keeping their lists up to date.

So we see spammers evolving: They buy or rent real mail servers now and they have even started coordinating. Using greylisting has actually increased the cost of becoming a successful spammer.

At our end of the game, we stay ahead of their game thanks to tools like spamd, and several of us dump and share our greytrap lists. It is even possible to collect IP addresses and feed a large number at the time to spamdb, but after a little while I grew tired of the increased manual work decided it was time for a counterprank. Cleaning up after spammers is no fun, unless you can have little robot helpers do the heavy lifting.

The Counterprank: A Feedback Loop

Regular readers will remember that I have a collection of known bad addresses in my domains that I use for my greytrapping, all generated elsewhere, that has come in handy at times. Run of the mill spam operators tend to just suck in anything that looks like email addresses, and keeping the list available on the web has served us extremely well here.

The professional spammers are apparently not quite that stupid, so the problem became a little different. They were able to sneak past greylisting and conventional content filtering. Also, they were apparently oblivious to email communication and as far as I can tell their Unsubscribe pages are not entirely believeable.

So it was a relief to find that places such as http://e-researchcouncil.com/ are very happy to accept any email addresss you can come up with. Time to enlist a few of our imaginary friends, drawn from the obvious source.

I did ponder the ethics for a few moments. After all, the forms included sentences such as "I certify that I am a US citizen", which was about as true as the assertion that I had signed up via an AOL proxy. But I did not ponder that matter for long. Moments later, most of the spam operators found themselves with new neighbors with odd names and foreign email addresses.

The net result is that the hosts start appearing automagically in the hourly dump of my list of greytrapped addresses and in the daily spamd activity report. With a little luck, we have succeeded in increasing the cost of spamming one tiny increment.

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive or non-trivial assistance, please contact FreeCode to make arrangements.

Note that the list of greytrapped addresses is updated ten past every full hour, fetching it every minute like some Americans have started doing is not a good use of your resources.

What have the black boxes wrought

2009-03-14T21:00:00.002+01:00

How compartmentalization turned into a security disaster. Greed, incompetence and dishonesty was involved.

IT security or the lack of any such thing has grabbed headlines lately here in Norway. A series of high profile public institutions have seen large scale worm infections on their Microsoft based networks. Late last year the regional government agency responsible for essentially all health care in the western part of the country had a worm infection so bad they essentially shut down their network as a preventive measure. During the last few weeks, the national police force, of all thinkable organizations, has seen not one, but two large-scale incidents.

Use of Microsoft products and sloppy system maintenance are both pervasive enough that similar incidents are likely happening right now elsewhere, somewhere near you too. The news reports about the Norwegian police force's IT problems contained one item that was particularly shocking to IT types like me:

Apparently large parts of the bureaucracy that is responsible for the confidential and correct processing of criminal matters and all sorts of sensitive personal information associated with the crimes runs essential services on Microsoft Windows NT 4.0.

That version of the Microsoft product is so old it is officially abandonware, and early reports of the police network problems included the oldish news that even the antiware vendors have stopped supporting the system. Later reports had police IT department officials claim that the worm infections were not that much of a security problem, since at this point all the worm actually did was spread.

Break out the popcorn, boys and girls: In an upcoming episode, we will see how the worm infected Windows machines the Royal Norwegian Police did not find or couldn't clean well enough are used in the perpetration of some cybercrime or other.

It's all pretty sickening, and at this point it would be rather tempting to spend the rest of the column ranting about the general stupidity of Windows users. But a smarter approach is to see if there is a lesson to be learned. To do that, we need to backtrack quite a bit and look at the cult of the little black boxes.

The cult of the little black boxes, and Microsoft the 1980s corporation

We need to go back and take in what the world was like in the nineteen-eighties. This was back when the world was divided into real computers (from the likes of IBM, Digital, and regional quasi-monopolies like our own Norsk Data) and those annoying toys called 'personal' microcomputers, where the 'IBM PC compatibles' had emerged as the surprise leader of the pack. Computer networks were usually private, corporate things and rarely interconnected much, with the rare exception of those institutions that were part of the US Department of Defense science experiment that was becoming known variously as ARPANET or 'the Internet'.

If you took your programming classes back in the nineteen-eighties, you likely know that we were taught that black boxes were good. Compartmentalization was the order of the day, meaning that as a developer you were supposed to create code with a well defined inteface, so anybody interacting with your code would be presented with a cleanly predictable result to any given input. Your code should be a black box and for any particular specificiation there could be written several interchangeable modules to fit the bill.

So far so good, predictability is nice and with compartmentalization comes, we hope at least, clear chains of responsibility. But several factors combined to take the cult of the black boxes and turn it into a corporate culture at a corporation that was growing from a handful of furry hackers into a corporation at the time, namely Microsoft. Microsoft' early successes all came from writing software for single-user systems that were so limited that working around the limitations of the hardware became much of a lifestyle. At the start of the decade, networking on microcomputers in Microsoft's range was pretty much out of the question, and by the end of the eighties any sort of connectivity (even dial-up) was still an expensive optional extra that was still too hard to configure for most people. On the storage side, we progressed from 128 kilobyte floppies to hard drives of just over a hundred megabytes for personal systems, with the 32 megabyte partition size still a very present limiting factor.

Amazing developments all, but both the applications and the organization grew faster than the hardware could keep up with. The organization now had several levels of management, and each one demanded maximum productivity from their underlings. Keeping in mind that each programmer or team would be writing little black boxes anyway, it made perfect sense to set up the software production line so each developer only had access to those parts of the system he or she was supposed to be working on. That way developers would be concentrating on their main task and minimize time spent waiting for compiles to finish. At predetermined times the developers would then upload the source code for their little black boxes to a central build system. The only people who had all the parts of the projects were in fact the custodians of the build system. Source code version control systems were made part of the process, but there is anecdotal evidence that the transition from standalone hacking to a version control regime was a rough one for many early Microsoft developers.

Only a few days ago I offered pretty much the content of the last paragraph to a table of youngish free software developers over beer. The reaction was quick an unanimous: "That way, nobody really knows what's going on in the software". That is a very valid point, and it proves how far we've come with free software. At the same time there is every reason to believe that the extreme compartmentalization that Microsoft established for its product development in the 1980s was the way things were done there until very recently, if indeed it has changed at all.

By the mid-1990s when Microsoft had been dragged kicking and screaming into modern-day network environments, and the ongoing saga of internet-enabled malware started in earnest (I've written a summary in a malware paper), with the company moving from early denial of any bugs whatsoever through a near-constant barrage of emergency hotfixes to today's monthly megapatch regime. With the source code still a closely guarded (if occasionally leaked) secret, there is really no way for us to know if they've learned any lessons at all. One indication that they still have some way to go is this Infoworld article about the state of their protocol documentation (summary: it's not to be trusted at all). As for the state of the source code, all we can do is to study the flow of urgent patches.

Much better then to learn how it should be done - Say from Theo de Raadt's AsiaBSDCon 2009 presentation about how OpenBSD's release process works, and if you want more of the gory details, do check his classic exploit mitigation presentation. Also, most likely you could do worse than read Damien Miller's OpenSSH security presentation (full text here). It's all those little things we do, at FreeCode and in free software in general.

If you found this column useful, entertaining or irritating, please drop me a line.

The slow brutes, a final roundup

2009-01-22T17:57:00.007+01:00

The slow brutes stopped their churning. Their last call was for sophia.

Over the last few columns, we have followed the progress of what appears to be a botnet cloud's attempt at gaining access to a couple of FreeBSD machines I have in my care. One of my predictions about the distributed, slow ssh bruteforce attempts we started seeing in November of 2008 was that at the rate they were going at the time, it would be well into the new year before we would see the end of their alphabetic progression. As it turns out, they stopped just before year end, before even reaching the 'T's. The last attempt recorded was this:
Dec 30 11:09:03 filehut sshd[54981]: error: PAM: authentication error for illegal user sophia from static-98-119-110-139.lsanca.dsl-w.verizon.net
The full collection of raw data is available here, with a .csv summarising number of attempts, user names and hosts per day here.

With the incident apparently over, we can sit back and study the data and see what patterns emerge.

The anatomy of the attack

There are a number of ways to slice and dice the data. One useful way to view the collection is to do day to day statistics, such as the ones in this .csv file, numbers extracted by some simple greping and awkery. Based on the day to day data, I made this graph to illustrate the progression.

Then for your data overload cravings, I turned the same data to a log scale for enhancement and added number of attempts -

hopefully adding some insight into just what happened when and maybe supporting some guessing about what they were indeed trying to achieve.

It is possible that we missed the actual start of those coordinated attempts, but the data we do have show a few interesting points.

The earliest preserved data from November 19th shows the most attempts per user name (average 13.29), with 7 unique user names tried and a relatively low number of hosts (76).

On November 20th, the cloud turned its attention to one user, root, trying only that user name a total of 1697 times from 566 different hosts. It would be well into November 21st before the cloud moved on to admin (128 attempts, 107 different hosts) and an apparently coordinated alphabetic progression.

The absolute number of attempts and hosts involved per day fell quickly, with average number of attempts per user name stabilizing at a fairly low number after a few days. The exception is the peak on December 27th, which could perhaps be explained by owners of compromised computers returning from holiday celebrations and turning their computers back on. The sharp decline in all numbers over the next few days before the attempts stop seems consistent with what we assumed: That the botnet masters were allocating resources according to likelihood of success.

So why is this incident important, or even interesting? After all, attempts to gain access to services by brute force or dictionary based attacks are nothing new. I was rather intrigued to see clear evidence that miscreants were trying to find the way under the radar of intrusion detectors by distributing the intrusion task over a large number of hosts. If their success rate at my sites is anything to go by, this may be just a weird anomaly and an idea that did not lead anywhere. I haven't heard from anybody who was actually compromised by this particular set of clouds, but then again anybody who got bitten would likely be rather shy about telling the world at large or even fairly obscure researchers about the fact.

Always looking for patterns I even went to the trouble of extracting some data from the logs about the individual hosts that participated in the attack. After some basic shell gymnastics I ended up with a .csv of hostnames, number of attempts from the host as well as the date of first and last contact (available here). Next I tried (and failed - gnuplot gurus, here's your chance) to graph the data usefully in OpenOffice, but ended up with a sorted version (sorted by attempts, start and end date) that at least shows us that a surprising number of hosts actually hung on for most of the time the coordinated attepts went on.

The lessons learned: security the old-fashioned way

The general lesson of this incident is rather predictably that miscreants will occasionally try new and original ways to try to crack their way into your system. The slow method was a refreshing variation, and for all we know they may have succeeded in places where the people in charge remain blissfully unaware. Trying to catalogue and detect all kinds of variations on the theme of "attempts at unauthorized access" is the kind of activity that has kept "antivirus" people in beer money for quite a while, and if there is a lesson to be learned here, it is that trying to enumerate badness (Yes, do look it up using your favorite search) is a losing game. Make sure whatever system you run is sanely constructed, any bugs that do turn up are fixable within a reasonable time frame, and so on. I suppose I will come back later with a rant about how much damage the "black boxes" school of thinking about software has done, especially after it got elevated to practically religious dogma by certain major players. And yes, you can usefully look that up as well.

For those of you who are interested in the data, here are the now complete extracts for your perusal:

The full set of log data
The per day .csv file - and the same in an .ods sheet with some graphing attempts
Per host data in the "Host,Attempts,StartDate,EndDate" format and sorted by attempts, start and end

For those of you interested in learning about OpenBSD and related delights like PF, FreeCode is set to start offering courses featuring among others yours truly as well as the usual support and consulting offerings. Contact the good front end people at FreeCode for further details.

International readers are at liberty to ignore the following, but Norwegian online IT magazine digi.no are apparently in the process of setting up a sort of census of Norwegian bloggers. The following is there to make this blog show up in their listing. You can read their article about the initiative (Norwegian only, unfortunately).

Into a new year, slowly pounding the gates

2008-12-21T23:29:00.003+01:00

The distributed but clearly coordinated bruteforcers are still at it. How long until they reach the end of the alphabet? And why are they staying away from my OpenBSD machines? Are we seeing the contours of a controlling intelligence?

As large parts of the Western world prepares for the holidays, the swarm of little robots that started trying to pry open the doors to my machines some weeks back are still at it. As far as we can tell, the coordinated attempts started some time in early November or perhaps late October (we don't keep logs around for long enough to be sure), with an alphabetic progression that has now progressed to somewere into the os. The complete listing from the time I started noticing up to the time I started writing this column can be found here.

I've written about this before, and in fact one of those columns was slashdotted, a pleasant surprise to me and a cause of some excitement among my colleagues at FreeCode.

After writing that article, I did some further research and found out that a precursor to what we are seeing now was observed as early as May 2008, as described in an Ars Technica article published at the time. That article also reveals, via Linuxtoday, that yours truly was among the many who failed to understand the problem, at least for a while. Then again, maybe actual log excerpts would have helped.

The problem, such as it is, seems to be that a somebody who herds a botnet has decided that the laws of big numbers favors those who keep trying for long enough. User names and passwords are far generally far enough from random that if you are allowed to go on for long enough, you will sooner or later manage to guess a correct combination of username and password and get access to a machine somewhere.

Sysadmins have been seeing bruteforce attacks for years. The traditional brute force attack would be a rapid succession of login attempts from one host, and usable countermeasures were devised in short order. My favorite of course involves PF, and the description of how to thwart traditional bruteforcers is one of the more popular pages in my PF tutorial.

The distributed, slow bruteforcers are different. For one, the login attempts from each host out in the cloud are spaced far enough apart in time that intrusion attmpt detectors will not trigger. Next, it takes a keen eye to spot the common thread in the attempts spaced up to a number of minutes apart: a monotonously alphabetic progression of user names, with attempts coming in from different hosts. Some number of attemtps at a specific user name, before the cloud moves on the next one, in alphabetic order.

During the period we have been observing the slow brute activity, a total of 695 hosts have been involved. A total of 665 hosts made unsuccessful attempts at authenticating at the hosts we are observing during November, while the number for December so far is 346. The typical number of attempts per user name has decreased, too, from a typical ten do fifteen during the early days down to between one and four during the last couple of weeks.

I thought at first that the decrease in activity was just an indicator that compromised hosts were getting cleaned up, but my colleague Egil Möller was the first to suggest that since we know the attempts are coordinated, it is not too far fetched to assume that the controlling system measures the rates of success for each of the chosen targets and allocates resources accordingly.

If Egil's assumption is right, we are seeing the bad guys adapting. My systems do not run any services they do not need to, and apparently all attempts at gaining access have been futile so far. So, the controlling system shifts resources to elsewhere, even if the access attempts do not stop entirely. Come to think of it, I'm not seeing any attempts at all on my OpenBSD systems, so it is possible to speculate that whoever is behind this phenomenon has decided that OpenBSD systems are hardened enough to begin with and usually run by compentent paranoids as to be useless as targets. That would be a comforting thought at the end of a long and sometimes trying year.

Speaking of the new year, look for exciting announcements coming from FreeCode. We're working on some cool things. And with a bit of luck, I might run into you at one conference or the other during the coming year.

Happy holidays to everyone.

A Small Update About The Slow Brutes

2008-12-06T22:41:00.003+01:00

Slow and steady might actually do it, eventually.

The reactions to my December 2nd column hit me with a bit of surprise. The column was taken on by slashdot and Linux Today both, producing a largish number of page views, but only two clicks on my featured ads. But while my clickthrough rate is not particularly interesting to others, the comments to the columns sometimes are.

If you look at the comments at slashdot and elsewhere, most of the commenters most likely did not actually read the column in full or did not take the time to digest what it actually said, with some notable exceptions. And yes, there were others, some also wrote in via email with informed comment - thanks!

For the benefit of those who did not get the point the first time around, I'll try once more to explain what the observations are and what they may in fact mean.

A number of commenters offered well meant advice to use packages like fail2ban, denyhosts or a few others.

The common denominator for all of them is that they track single hosts that make a larger than usual number of connections or are the source of a number of failed logins higher than a certain threshold value over a set time period. I appreciate your concerns, but the subject of the column did not fit well with the way those the packages work.

In fact, a similar scheme was already in place at the site that provided the data. The machines that provided the ssh logs are FreeBSD ones (as the sharper ones have observed already, and the reasons may possibly be revealed over beer sometime), but any gateway under my control will run OpenBSD, and by extension, PF (and yes, there is a book you might want to order from one (North America) or the other (Europe and elsewhere) of the OpenBSD project's sites). For a quick fix of background, the online PF tutorial may be worth a look.

Anyway, the /etc/pf.conf at that site's gateway contains the lines
table <abusive_hosts> persist block log quick from <abusive_hosts>
and
pass log (all) quick proto { tcp, udp } from any to any port ssh flags S/SA keep state \ (max-src-conn 15, max-src-conn-rate 7/3, overload <abusive_hosts> flush global)
Those lines provide a variation on the logic that those posters recommended. Essentially, any host that tries 15 or more simultaneous ssh connections, or come in at a rate of more than seven over the span of three seconds, will be added to the table <abusive_hosts>, and the block quick rule blocks any further access from those hosts. Yes, flush global means what you think it does.

This works at the network level. For a gateway with a potentially large number of hosts on either side, the success or otherwise of eventual authentication may not be relevant and may be better dealt with elsewhere. Anyway, at the time I started working on this column, the table <abusive_hosts> on the gateway contained only two hosts:
:~$ sudo pfctl -t abusive_hosts -v show 194.204.37.93 201.57.187.114
I keep offenders in that table for 24 hours only, I do not believe in the permanent bans that some commenters advocate. After all, there is such a thing as DHCP, and entire netblocks are reallocated with amazingly short intervals.

Anyway, looking at the authentication log on the gateway reveals how those hosts got added to the the table in the first place:
:~$ grep 194.204.37.93 /var/log/authlog Dec 5 22:50:30 delilah sshd[15266]: Did not receive identification string from 194.204.37.93 Dec 5 22:50:37 delilah sshd[6106]: Did not receive identification string from 194.204.37.93 Dec 6 01:29:58 delilah sshd[30359]: Failed password for root from 194.204.37.93 port 47071 ssh2 Dec 6 01:29:58 delilah sshd[7293]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:29:59 delilah sshd[4395]: Failed password for root from 194.204.37.93 port 47296 ssh2 Dec 6 01:29:59 delilah sshd[27615]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:00 delilah sshd[12248]: Failed password for root from 194.204.37.93 port 47330 ssh2 Dec 6 01:30:00 delilah sshd[24579]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:01 delilah sshd[3434]: Failed password for root from 194.204.37.93 port 47380 ssh2 Dec 6 01:30:01 delilah sshd[32737]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:02 delilah sshd[11984]: Failed password for root from 194.204.37.93 port 47425 ssh2 Dec 6 01:30:02 delilah sshd[27059]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:03 delilah sshd[13345]: Failed password for root from 194.204.37.93 port 47459 ssh2 Dec 6 01:30:03 delilah sshd[1858]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:04 delilah sshd[12739]: Failed password for root from 194.204.37.93 port 47516 ssh2 Dec 6 01:30:04 delilah sshd[16843]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:05 delilah sshd[13796]: Failed password for root from 194.204.37.93 port 47564 ssh2 Dec 6 01:30:05 delilah sshd[16789]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:06 delilah sshd[628]: Failed password for root from 194.204.37.93 port 47602 ssh2 Dec 6 01:30:06 delilah sshd[6162]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:07 delilah sshd[2579]: Failed password for root from 194.204.37.93 port 47646 ssh2 Dec 6 01:30:07 delilah sshd[12461]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:08 delilah sshd[12725]: Failed password for root from 194.204.37.93 port 47685 ssh2 Dec 6 01:30:08 delilah sshd[29909]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:09 delilah sshd[16560]: Failed password for root from 194.204.37.93 port 47724 ssh2 Dec 6 01:30:09 delilah sshd[1690]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:09 delilah sshd[1600]: Failed password for root from 194.204.37.93 port 47771 ssh2 Dec 6 01:30:09 delilah sshd[28882]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:10 delilah sshd[29953]: Failed password for root from 194.204.37.93 port 47807 ssh2 Dec 6 01:30:10 delilah sshd[15349]: Received disconnect from 194.204.37.93: 11: Bye Bye Dec 6 01:30:11 delilah sshd[2962]: Failed password for root from 194.204.37.93 port 47845 ssh2 Dec 6 01:30:11 delilah sshd[27557]: Received disconnect from 194.204.37.93: 11: Bye Bye
and
:~$ grep 201.57.187.114 /var/log/authlog Dec 5 23:55:30 delilah sshd[24338]: Did not receive identification string from 201.57.187.114 Dec 5 23:55:35 delilah sshd[23570]: Did not receive identification string from 201.57.187.114 Dec 5 23:59:58 delilah sshd[10216]: Invalid user raimundo from 201.57.187.114 Dec 5 23:59:58 delilah sshd[10216]: Failed password for invalid user raimundo from 201.57.187.114 port 35776 ssh2 Dec 5 23:59:58 delilah sshd[18515]: Received disconnect from 201.57.187.114: 11: Bye Bye Dec 6 00:00:01 delilah sshd[17353]: Invalid user joan from 201.57.187.114 Dec 6 00:00:01 delilah sshd[17353]: Failed password for invalid user joan from 201.57.187.114 port 37570 ssh2 Dec 6 00:00:02 delilah sshd[30314]: Received disconnect from 201.57.187.114: 11: Bye Bye
which again shows that these were the old-fashioned, rapid-fire kind of bots. Looking a bit closer even reveals that they kept trying after they were put in the doghouse:
:~$ sudo pfctl -t abusive_hosts -vT show 194.204.37.93 Cleared: Sat Dec 6 01:30:11 2008 In/Block: [ Packets: 18985 Bytes: 835336 ] In/Pass: [ Packets: 0 Bytes: 0 ] Out/Block: [ Packets: 0 Bytes: 0 ] Out/Pass: [ Packets: 0 Bytes: 0 ] 201.57.187.114 Cleared: Sat Dec 6 00:00:02 2008 In/Block: [ Packets: 800 Bytes: 48268 ] In/Pass: [ Packets: 0 Bytes: 0 ] Out/Block: [ Packets: 0 Bytes: 0 ] Out/Pass: [ Packets: 0 Bytes: 0 ]
And of course there were no traces of those IP addresses or corresponding host names in the authentication logs in the machines where I have collected the data about slow bots. My best guess at why is that the gateway's IP address is at the low end of the routable range for that site. Note to bruteforcers: try working the Internet in reverse next time. (Not that it would help much here, but that's another story.)

The reason why I don't see much activity for other services is simply that those machines do not run all that many services, and only the services they actually run for the world's benefit are in fact available to the outside.

My log data shows a definite pattern, and the alphabetic progression points to a degree of coordination. The slow bots are, I theorize, operated by a botnet herder who has a large pool of compromised hosts available and who also believes that given enough time, sooner or later you will find a correct combination of user names and passwords for a given host. Statisticians tell me that assumption is in fact valid, at least to some extent.

By setting up the necessarily large number of attempts to come from a sizable number of hosts in either round-robin or pseudo-random order and intervals for each individual host's attempts in the several minutes to hours range, there is a very real possibility that the slow but determined campaign for control of any single system will drown in the noise.

It is useful to keep in mind that malware moved out of the hands of pranksters and vandals years ago. Mass destruction of systems or data might still make the occasional headline, but staying out of the limelight is likely to be a lot more profitable. Modern malware masters want their creations and charges to stay undetected. What we may be seeing right at this moment is that they have realized the herd may only be sustainable if they grow it slowly.

If you are interested in researching the phenomena I've blogged about, you're welcome to contact me directly for more information or raw data.

A low intensity, distributed bruteforce attempt

2008-12-02T19:50:00.010+01:00

We have seen the future of botnets, and it is a distributed, low-key affair. Are sites running free software finally becoming malware targets?

Phase 1: “That's odd …”
During the last few weeks, I noticed an anomaly in the authentication logs on one of my listening posts. There were a larger than usual number of ssh login attempts overall, a higher than usual number of attempts for non-existent user names as well as some failures for a few that actually exist as well.

Looking at the log directly a typical progression would look like this:
Nov 19 15:04:22 rosalita sshd[40232]: error: PAM: authentication error for illegal user alias from s514.nxs.nl Nov 19 15:07:32 rosalita sshd[40239]: error: PAM: authentication error for illegal user alias from c90678d3.static.spo.virtua.com.br Nov 19 15:10:20 rosalita sshd[40247]: error: PAM: authentication error for illegal user alias from 207-47-162-126.prna.static.sasknet.sk.ca Nov 19 15:13:46 rosalita sshd[40268]: error: PAM: authentication error for illegal user alias from 125-236-218-109.adsl.xtra.co.nz Nov 19 15:16:29 rosalita sshd[40275]: error: PAM: authentication error for illegal user alias from 200.93.147.114 Nov 19 15:19:12 rosalita sshd[40279]: error: PAM: authentication error for illegal user alias from 62.225.15.82 Nov 19 15:22:29 rosalita sshd[40298]: error: PAM: authentication error for illegal user alias from 121.33.199.39 Nov 19 15:25:14 rosalita sshd[40305]: error: PAM: authentication error for illegal user alias from 130.red-80-37-213.staticip.rima-tde.net Nov 19 15:28:23 rosalita sshd[40309]: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com Nov 19 15:31:17 rosalita sshd[40316]: error: PAM: authentication error for illegal user alias from gate-dialog-simet.jgora.dialog.net.pl Nov 19 15:34:18 rosalita sshd[40334]: error: PAM: authentication error for illegal user alias from 80.51.31.84 Nov 19 15:37:23 rosalita sshd[40342]: error: PAM: authentication error for illegal user alias from 82.207.104.34 Nov 19 15:40:20 rosalita sshd[40350]: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com Nov 19 15:43:39 rosalita sshd[40354]: error: PAM: authentication error for illegal user alias from 200.20.187.222 Nov 19 15:46:41 rosalita sshd[40374]: error: PAM: authentication error for illegal user amanda from 58.196.4.2 Nov 19 15:49:31 rosalita sshd[40378]: error: PAM: authentication error for illegal user amanda from host116-164.dissent.birch.net Nov 19 15:55:47 rosalita sshd[40408]: error: PAM: authentication error for illegal user amanda from robert71.lnk.telstra.net Nov 19 15:59:08 rosalita sshd[40412]: error: PAM: authentication error for illegal user amanda from static-71-166-159-177.washdc.east.verizon.net Nov 19 16:02:06 rosalita sshd[40455]: error: PAM: authentication error for illegal user amanda from host87-163-static.30-87-b.business.telecomitalia.it Nov 19 16:05:08 rosalita sshd[40459]: error: PAM: authentication error for illegal user amanda from 213.150.184.70 Nov 19 16:08:16 rosalita sshd[40465]: error: PAM: authentication error for illegal user amanda from mail.pddsl.de Nov 19 16:11:24 rosalita sshd[40486]: error: PAM: authentication error for illegal user amanda from abu66.internetdsl.tpnet.pl Nov 19 16:15:00 rosalita sshd[40491]: error: PAM: authentication error for illegal user amanda from 125.77.106.246 Nov 19 16:17:43 rosalita sshd[40497]: error: PAM: authentication error for illegal user amanda from 217.76.34.230 Nov 19 16:20:54 rosalita sshd[40506]: error: PAM: authentication error for illegal user amanda from robert71.lnk.telstra.net Nov 19 16:24:09 rosalita sshd[40529]: error: PAM: authentication error for illegal user amanda from p578b4f0b.dip0.t-ipconnect.de Nov 19 16:28:11 rosalita sshd[40538]: error: PAM: authentication error for illegal user amanda from mail.carena-ci.com Nov 19 16:30:15 rosalita sshd[40551]: error: PAM: authentication error for illegal user amavis from 87.229.3.89 Nov 19 16:34:31 rosalita sshd[40567]: error: PAM: authentication error for illegal user amavis from 218.248.79.251 Nov 19 16:36:40 rosalita sshd[40574]: error: PAM: authentication error for illegal user amavis from 83-103-70-170.ip.fastwebnet.it Nov 19 16:40:05 rosalita sshd[40596]: error: PAM: authentication error for illegal user amavis from 75-49-251-71.lightspeed.snjsca.sbcglobal.net
- and so on, with a striking regularity. See for example the attempts to log on as the alias user, 14 attempts are made from 13 different hosts, with only 70-46-140-187.orl.fdn.com trying more than once. Then thirteen attempts are made for the amanda user, from 13 other hosts. The pattern repeats again for users amavis, apache, at, and goes on with others, apparently trying users in an alphabetic sequence.

Phase 2: Not your run of the mill screwup, the data say
Repeated login attempts for non-existing users are nothing new (in fact the bruteforce avoidance section is one of the more popular parts of the PF tutorial), but I was a bit surprised to see the attempts actually reaching this machine, which is on a local network behind a PF gateway with a configuration that is in fact closely related to the one in the tutorial (and the book for that matter). Then looking at the log entries, I noticed a few more things: The attempts are never less than a minute apart, and the attempts from a single host are separated by much longer intervals. The full data set I extracted from the point I started noticing those anomalies sum up to these figures can be found here, in case you want to look at it and draw you own conclusions

Some one-liners give us illustrative numbers:
peter@thingy:~$ wc -l slowbrutes.txt 16727 slowbrutes.txt
That is, over this period there were 16727 failed ssh login attempts at this host. A large number for this particular machine, but not enough to raise eyebrows by itself at larger or busier sites.

More than sixteen thousand attempts, but for how many invalid user names?
peter@thingy:~$ grep illegal slowbrutes.txt | awk '{print $13}' | sort -u | wc -l 2962 peter@thingy:~$ grep illegal slowbrutes.txt | awk '{print $15}' | sort -u | wc -l 671
That is, approaching three thousand unlucky guesses, coming from 671 different hosts.

How many valid user names did they stumble upon?
peter@thingy:~$ grep -v illegal slowbrutes.txt | awk '{print $11}' | sort -u | wc -l 2
A grand total of two, one of them the rather obvious root, for a total of
peter@thingy:~$ grep -vc illegal slowbrutes.txt 1698
1698 attempts, coming from
peter@thingy:~$ grep -v illegal slowbrutes.txt | awk '{print $13}' | sort -u | wc -l 566
566 different hosts.

The patterns that emerge from the data, with the alphabetical ordering and apparent coordination, point to a botnet herder trying out new methods. Intrusion detection systems and adaptive firewalls are generally tuned to detecting things like large numbers of simultaneous connecions or a high rate of new connections from a host. Distributing the task of bruteforcing passwords to several hosts could seem like an inspired way to come in under the radar wherever relatively smart systems are in place. Setting the herd to attempt at a low frequency would likely mean that those failed attempts simply drown in the noise at higher volume sites, and will not be noticed.

Phase 3: Are you one of their guinea pigs, too?
There are indications that the method has not been quite perfected yet. At the start of this run, the bots would make at least ten attempts before moving on down the alphabet. Now it seems enough bots have been taken out of circulation that the typical number of attempts per user name is closer to three, with some tried only once:
Dec 2 11:45:59 rosalita sshd[55775]: error: PAM: authentication error for illegal user heaven from cpe001217e403b3-cm000f9fa6157c.cpe.net.cable.rogers.com Dec 2 11:48:16 rosalita sshd[55778]: error: PAM: authentication error for illegal user heaven from 90.190.96.46 Dec 2 11:50:39 rosalita sshd[55791]: error: PAM: authentication error for illegal user heaven from static-71-117-126-102.snloca.dsl-w.verizon.net Dec 2 11:55:26 rosalita sshd[55811]: error: PAM: authentication error for illegal user heavynne from dsl-217-155-184-54.zen.co.uk Dec 2 11:57:57 rosalita sshd[55814]: error: PAM: authentication error for illegal user heavynne from pd907ee1e.dip0.t-ipconnect.de Dec 2 12:00:20 rosalita sshd[55836]: error: PAM: authentication error for illegal user heba from 201-26-172-213.dial-up.telesp.net.br Dec 2 12:07:37 rosalita sshd[55879]: error: PAM: authentication error for illegal user hector from 75.145.16.83 Dec 2 12:09:58 rosalita sshd[55882]: error: PAM: authentication error for illegal user hector from ppp-69-217-30-214.dsl.applwi.ameritech.net Dec 2 12:12:33 rosalita sshd[55901]: error: PAM: authentication error for illegal user hector from 75-49-251-71.lightspeed.snjsca.sbcglobal.net Dec 2 12:14:51 rosalita sshd[55905]: error: PAM: authentication error for illegal user hedda from 201.218.231.142 Dec 2 12:17:21 rosalita sshd[55911]: error: PAM: authentication error for illegal user hedda from 75.147.27.85 Dec 2 12:19:48 rosalita sshd[55914]: error: PAM: authentication error for illegal user hedda from 203.70.179.113
From where I'm sitting it's hard to tell whether the lower number of attempts means that the machines have cleaned up by their legal owners or whether they have simply taken out of rotation by their herders. Even with the initial 14 attempts per user name the chance of actually finding a valid combination of user names and passwords would be slim but not non-existent, but decreasing the number of attempts per time unit will necessarily make the chance of eventually finding a valid pair even smaller.

Apparently I'm not the only one seeing the slow brutes, as this post to openbsd-misc indicates. The sensible countermeasure could be to disallow shh password logins and allow only key logins, probably easier to set up and enforce than network-level measures. With the slow rate of attempts and the relatively large number of hosts involved, the undesirable traffic here is relatively hard to distinguish automatically from innocent errors unless you make have any attempt to log in with an invalid user name a sufficent reason for blocking traffic from that host.

Phase N: The shape of things to come
In the longer term view, this may very well be the shape of botnets to come. With a large enough pool of compromised hosts under their control, future botnet herders can afford to organize their activity so any one host only participates in undesirable activity at intervals long enough that malware detectors do not trigger (and thinking further ahead, if the world ever does go IPv6 wholesale and you can expect any one network interface to have dozens of IP addresses, think again how much more interesting detecting botnets becomes).

Antiware vendors will likely put their spin on this too when their marketing departments start noticing columns (Hey! It's Linux they're targeting!), but then as regular readers know, the more productive approach is always to reduce malware masters' target area by using systems that are less vulnerable because they have been extensively audited and whose makers are unafraid to make source code available for public inspection and experimentation.

As people who ran into me at the recent PF tutorial in London or at OpenCON will already know, have I joined FreeCode, the Norwegian free software consultancy. We expect to keep doing fun and useful things with free software for customers and friends, and some of it may be interesting enough to be the topics of future columns.

IETF failed to account for greylisting

2008-10-20T15:17:00.002+02:00

The potential for conflict between greylisting and sites with large pools of outgoing SMTP senders is well known and in need of resolution. Why does the SMTP RFC moving along the standards track fail to address this?

Standardization efforts rarely grab headlines. Except in rather exceptional circumstances (think Microsoft's recent ISO buyout), standards are formulated in response to specific technical needs, or as frequently happens, standards documents are written to codify existing and well known best practices.

As regular readers of this column will be aware, I tend to argue that in the email domain, greylisting is one such 'best practice' that would deserve to be included in a standards or best practice document. In a way it already is, but it was never actually referred to in any standard or standards draft, and its main claim to standards compliance relied on extrapolating some crucial points in RFC2821.

The technical alibi for claiming that greylisting is a valid, RFC-compliant technique comes from reading section 4.5.4.1, Sending Strategy. It's really quite straightforward: An SMTP sender that receives a temporary error (451) when it tries to deliver is required to try again later, after a reasonable time. The RFC gives a few more details such as recommended retry intervals and time to give up trying, but does not explicitly say where those repeated attempts should come from. At the time RFC2821 was written in early 2001, it was almost certainly implicit in the formulation used that the retries would come from the same hosts, and specifying that as an explicit requirement most likely seemed more than a little redundant.

The MUST requirement in RFC2821 is what cleared the way for greylisting (see greylisting.org and the relevant parts of my PF tutorial (or the book)), and the earliest implementations started appearing from 2003 onwards.

Fast-forward a few years, and you have a situation where several large operators have set up their networks with large pools of outgoing SMTP servers and no guarantee which machine in the pool will be the next to handle a message queued for a delivery retry. Some greylisting implementations use a modified algorithm that stores or at least acts upon the subnet a greylistes message came from instead of the specific IP address, while others such as OpenBSD's spamd operate strictly on individual IP addresses.

It does not exactly take a rocket scientist to figure out that here is a potential problem, at least when it comes to the stricter implementations. Sites that do not retry from the same IP address can claim to be RFC2821 compliant, since the RFC does not contain a specific requirement that the delivery retries have to be from the same IP address. Again quoting my tutorial, the solution so far has been to whitelist those sites, extracting their SPF info for whitelisting purposes if necessary.

The large number of outgoing SMTP hosts per site problem has been widely discussed, and anybody even marginally interested in mail and spam avoidance should be aware if this. Then here's a surprise for you: Apparently the writers of RFCs are actually unaware of this, or chose not to care. On October 1, 2008, I found in my IETF mailbox RFC5321, which obsoletes RFC2821. The new RFC contains a number of things, but section 4.5.4.1, "Sending Strategy" (yes, they even kept the numbering) is unchanged.

That means that the working group were either unaware of the problem or chose not to resolve the conflict at this time. It would have been a very sensible thing to explicitly state that retries MUST come from the same IP address. The only halfway sane reason not to resolve the conflict that way would be that this would have made greylisting powerful enough to possibly go a long way towards eliminating the need for SPF and other more convoluted schemes.

I can not bring myself to believe that the working group does not have at least one member who is aware of what greylisting is and knows about that sole remaining problem that needs to be addressed. RFC5321 has almost everything else covered, so I hope the working group will listen to reason and move resolve this problem before the standard is finalized.

In other news, this year's EuroBSDCon in Strasbourg went smoothly with a lot of good content if a somewhat smaller number of participants than the previous one. The next chance to catch my PF tutorial live will be in London on November 26th, 2008. Contact the UKUUG for details and booking.

“Name and Shame”, or socially responsible use of your log data

2008-09-22T19:32:00.007+02:00

Your logs contain an ever-growing mass of data on spammers. How about making an effort to make that data useful to others?

Those of us who run email services know, from sometimes painful experience, what it takes to ensure that the minimum possible amount of unwanted advertising and scams that may turn out to be security hazards reaches our users' inboxes.

Email: This should have been very simple
Handling email should really be quite simple: The server is configured to know what domains it receives mail for and what users actually exist in those domains. When a machine makes contact and indicates that it intends to deliver email, the server check if the recipient is a valid user. If the recipient is valid, the message is received and put in the relevant user's mailbox. Otherwise, a message about a failed delivery and optionally the reason for the failure is sent to the user specified as the sender.

If they were all honest people
In each part of the process, the underlying premise is that the communicating parners offer each other correct information. Frequently that is the case, and we have legitimate communications between partners with a valid reason for contacting each other. Unfortunately there are other cases where the implicit trust is abused, such as when email messages are sent with a sender address other than the real one, quite likely a made-up one in a domain that belongs to other people. Some of us occasionally receive delivery failure messages for messages we verfiably did not send[1]. If we take the time to study the contents of those messages, in almost all cases we will find that the messages are spam, sometimes the scamming kind and perhaps part of an attempt to take control of the recipient's computer or steal sensitive data.

What do the ones in charge do, then?
If you ask a typical system administrator what measures are in effect to thwart attempts at delivering unwanted or malicious messages to their users, you will most likely get a description that says, essentially, the messages are filtered through systems that inspect message contents. If the message does not contain anything known to be bad (known spam or malware) or something sufficiently similar to a known bad, the message is delivered to the user's mailbox. If the system determines that the message contents indicates it should not be delivered, the messages is thrown away undelivered, and some system administrators will tell you that the system also sends a message about the decision not to deliver the message to the stated sender address.

Large parts or this is likely part of moderately educated users' passive knowledge, and most of us are likely to accept that content filtering is all we can do to keep dubious or downright criminal elements out of our working environment. For the individual end user, only minor adjustments to this are likely to be possible.

Measures based on observed behavior
But those of us who actually run the service also have the opportunity to study the automatically generated log data from our systems and use spammers' (that is, senders of all types of unwanted mail, including malware) behavior patterns to remove most of the unwanted traffic before actual message content is known. In order to do that, it is necessary to go to a more basic level of network traffic and study sender behavior on the network level.

One of the simpler forms of behavior based measures emerged in the form of a technique called greylisting in 2003. The technique is based on a slightly pedantic and rather creative interpretation of established standards. The Internet protocol for email transfer, SMTP (the Simple Mail Transfer Protocol) allows servers that experience temporary problems that make it impossible to receive mail to report a specific 'temporary local problem' status code to correspondents trying to deliver mail. Correctly configured senders will interpret and act on the status code and delay delivery for a short time. In most circumstances, the delivery will succeed within a short time. It is worth noting that this part of the standard was formulated to help the mail service's reliability. At most times, the retries happen without alerting the person who wrote and sent the message. The messages generally reach their destination eventually.

Lists of grey and black, little white lies
Greylisting works like this: the server reports a temporary local problem to all attempts at delivery from machines the server has not exchanged mail with earlier. Experience shows that the pre-experiment hypothesis was mainly correct: Essentially all machines that try to deliver valid email are configured to check return codes and act on them, while almost all spam senders dump as many messages as possible, and never check any return codes. This means that somewhere in the eighty to high nineties percentage of all spam volume is discarded at the first delivery attempt (before any content filtering), while legitimate email reaches its intended recipients, occasionally with delayed delivery of the initial message from a new correspondent.

One other behavior based technique that predates greylisting is the use of 'blacklists' - lists of machines that have been classified as spam senders - and rejecting mail from machines on such lists. Some groups eventually started experimenting with 'tarpits', a technique that essentially means your end of the communication moves along very slowly. A much cited example is the spamd program, released as a part of the free operating system OpenBSD in May of 2003. The program's main purpose at the time was to answer email traffic from blacklisted hosts one byte per second, never leaving a blacklisted host any real chance of delivering messages.

The combination of blacklists and greylisting proved to work very well, but the quest for even more effective measures continued. Yet again, the next logical step grew out of observing spammer behavior. We saw earlier that spammers do not bother to check whether individual messages are in fact delivered.

Laying traps and bait
By early 2005, these observations lead to a theory that was soon proved useful: If we have one or more addresses in our own domains the are certain to never receive any valid mail, we can be almost a hundred percent certain that any mail addressed to those addresses is spam. The addresses are spamtraps. Any machines that try to deliver spam to those addresses are placed in a local blacklist, and we keep them busy by answering their traffic at a rate of one byte per second. The machines stay on the blacklist for 24 hours unless otherwise specified.

The new technique, dubbed greytrapping was launched as part of the improved spamd in OpenBSD 3.8, released May 2005. In early 2006, Bob Beck, one of the main spamd developers announced that his greytrapping hosts at the University of Alberta generates a downloadable blacklist based on the greyptrap data, updated once per hour, ready for inclusion in spamd setups elsewhere. This is obviously useful. Machines that try to deliver mail to addresses that were never deliverable most likely do not have any valid mail to deliver, and it we are doing society at large a favor by delaying their deliveries and wasting their time to the maximum extent possible.

It is worth mentioning that during the period we have used the University of Alberta blacklist at our site, it has contained a minimum of twenty-some thousand IP addresses, and during some busy periods have reached almost two hundred thousand.

You can help, too
Fortunately you do not need to be a core developer to be able to contribute. The exact same tools Bob Beck uses to generate his blacklist is available to everybody else as part of OpenBSD, and they are actually not very hard to use productively.

Here at BSDdly.net and associated domains we saw during the (Northern hemisphere) summer of 2007 a marked increase in email sent to addresses that have never actually existed in our domains. This was clearly a case of somebody, one or more groups, making up or generating sender addresses to avoid seeing any reactions to the spam they were sending. This in turn lead to us starting an experiment that is still ongoing. We record invalid addresses in our own domains as they turn up in our logs. From these addresses we pick the really improbable ones, put them in our local spamtrap list and publish the list on a specific web page on our server[2].

Experience shows that it it takes a very short time for the addresses we put on the web page to turn up as target addresses for spam. This means that we have succeeded in feeding the spammers data that makes it easier for us to stop their attempts, and frequently we make spam senders use significant amounts of time communicating with our machines with no chance of actually achieving anything. The number of spamtrap addresses has reached fifteen thousand, and we have at times observed groups of machines that spend weeks working through the whole list, with average time spent per unsuccessful delivery attempt clocked at roughly seven minutes.

As a byproduct of the active spammer trapping we started exporting our own list of machines that had been trapped via the spamtrap addresses during the last 24 hours and making the list available for download. This list's existence has only been announced via the spamtrap addresses web page and a few blog posts, but we see that it's retrieved, most likely automatically, at intervals and is apparently used by other sites in their systems.

At this point we have established that it is possible to create a system that makes it very unlikely that spam actually makes it through to users, while at the same time it is quite unlikely that legitimate mail is adversely affected. In other words, we have the cyberspace equivalent of good fences around our property, but spammers are still out there and may create serious probles for those who are without adequate protection.

Collecting evidence, or at least seek clarity
We would have loved to see law enforcement take the spammer problem seriously. This is not just because the spam that reaches its targets is irritating, but rather because almost all spam is sent via equipment that spammers use without the legal owners' consent. We would have liked to see resources allocated in proportion to the criminal activity the spam represents. We would have liked to help, but it might seem that we would not have usable evidence available due to the fact that we do not actually receive the messages the spammers try to deliver. On the other hand, we have at all times a list of machines that have tried to deliver spam, identified with an almost hundred percent certainty based on the spammer trapping addresses. In addition, our systems routinely produce logs of all activity, with the level of detail we set ourselves. This means that it is possible to search our logs for the IP addresses that have tried to deliver spam to our systems during the last 24 hours, and get a summary of what those machines have done.

A search of this kind typically yields a result like this:
Aug 10 02:34:29 skapet spamd[13548]: 190.20.132.16: connected (4/3) Aug 10 02:34:41 skapet spamd[13548]: (GREY) 190.20.132.16: <kristie@iland.net> -> <asasaskosmicki@bsdly.net> Aug 10 02:34:41 skapet spamd[13548]: 190.20.132.16: disconnected after 12 seconds. Aug 10 03:41:42 skapet spamd[13548]: 190.20.132.16: connected (14/13), lists: spamd-greytrap Aug 10 03:42:23 skapet spamd[13548]: 190.20.132.16: disconnected after 41 seconds. lists: spamd-greytrap Aug 10 06:30:35 skapet spamd[13548]: 190.20.132.16: connected (23/22), lists: spamd-greytrap becks Aug 10 06:31:16 skapet spamd[13548]: 190.20.132.16: disconnected after 41 seconds. lists: spamd-greytrap becks

The first line here states that 190.20.132.16 contacts our system at 02:34:29 AM on August tenth, as the fourth active SMTP connection, three blacklisted. A few seconds later it appears that this is an attempt at delivering a message to the address asasaskosmicki@bsdly.net. That address was already one of our spamtraps, most likely one that was harvested from our logs and was originally made up somewhere else. After 12 seconds, the machine disconnects. The attempted delivery to a spamtrap address means that the machine is added to our local spamd-greytrap blacklist, as indicated in the entry for the next attempt about one hour later. This second attempt lasts for 41 seconds. The third try in our log material happens just after 06:30, and the addition of the list name becks indicates that in the meantime has tried to deliver to one of Bob Beck's spammer trap addresses and has entered that blacklist, too.

Unfortunately, it is unlikely that logs of this kind are sufficient as evidence for criminal prosecution purposes, but the data may be of some use to those who have an interest in keeping machines in their care from sending spam.

“Name And Shame“, or just being neighborly?
After some discussions with colleagues I decided in early August 2008 to generate daily reports of the activities of machines that had made it into the local blacklist on bsdly.net and publish the results. If all we have is the fact that a machine has entered a blacklist as an IP address (such as 24.165.4.190), and there is no supporting material, it is fairly easy for whoever is in charge of that address range to just ignore the entry as an unsupported allegation. We hope that when whoever is responsible for the network containing 24.165.4.190 sees a sequence like this,
Host 24.165.4.190: Aug 10 02:57:40 skapet spamd[13548]: 24.165.4.190: connected (9/8) Aug 10 02:57:54 skapet spamd[13548]: (GREY) 24.165.4.190: <hand@itnmiami.com> -> <kimberlee.ledet@ehtrib.org> Aug 10 02:57:55 skapet spamd[13548]: (GREY) 24.165.4.190: <hand@itnmiami.com> -> <kimberliereffett@ehtrib.org> Aug 10 02:57:56 skapet spamd[13548]: 24.165.4.190: disconnected after 16 seconds. Aug 10 02:58:16 skapet spamd[13548]: 24.165.4.190: connected (8/6) Aug 10 02:58:30 skapet spamd[13548]: (GREY) 24.165.4.190: <brunson@jebconet.com> -> <kimberlee.ledet@ehtrib.org> Aug 10 02:58:31 skapet spamd[13548]: (GREY) 24.165.4.190: <brunson@jebconet.com> -> <kimberliereffett@ehtrib.org> Aug 10 02:58:32 skapet spamd[13548]: 24.165.4.190: disconnected after 16 seconds. Aug 10 02:58:39 skapet spamd[13548]: 24.165.4.190: connected (7/6), lists: spamd-greytrap Aug 10 03:02:24 skapet spamd[13548]: (BLACK) 24.165.4.190: <aarnq@abtinc.com> -> <kimberlee.ledet@ehtrib.org> Aug 10 03:03:17 skapet spamd[13548]: (BLACK) 24.165.4.190: <aarnq@abtinc.com> -> <kimberliereffett@ehtrib.org> Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: From: "Preston Amos" <aarnq@abtinc.com> Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: To: kimberlee.ledet@ehtrib.org Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: Subject: Wonderful enhancing effect on your manhood. Aug 10 03:06:04 skapet spamd[13548]: 24.165.4.190: disconnected after 445 seconds. lists: spamd-greytrap
they will find that to be a sufficient for action of some kind. The material we generate is available via the “The Name And Shame Robot” web page. The latest complete report of log excerpts is available via links at that page. Previous versions are archived offline, but will be made available on request to parties with valid reasons to request the data.

“The Name And Shame Robot”" is rather new, and it is too early to say what effect, if any, the publication has had. We hope that others will do similar things based on their local log data or even synchronize their data with ours. If you are interested in participating, please make contact.

Regardless of other factors, we hope that the data can be useful as indicators of potential for improvement in the networks that appear regularly in the reports as well as material for studies that will produce even better techniques for spam avoidance.

A shorter version of this article in Norwegian was published in Computerworld's Norwegian edition on August 22, 2008; the longer Norwegian version is available as an earlier blog post.

[1] A collection of such failure messages collected earlier this year is available at http://www.bsdly.net/~peter/joejob-archive.2008-07-28.txt.

[2] See http://www.bsdly.net/~peter/traplist.shtml, references at that page lead to my blog, which consists of public field notes, as well as other relevant material.

About the author
Peter N. M. Hansteen (peter@bsdly.net) is a consultant, system administrator and writer, based in Bergen, Norway. In October 2008, he joined FreeCode, the Norwegian free software consultancy. He has written various articles as well as "The Book of PF", published by No Starch Press in 2007, and lectures on Unix- and network-related topics. He is a main organizer of BLUG (Bergen (BSD and) Linux User Group), vice president of NUUG (Norwegian Unix User Group) and an occasional activist for EFF's Norwegian sister organization EFN (Elektronisk Forpost Norge).

[.NO] “Name and Shame” eller samfunnsnyttig bruk av loggdata om spammere

2008-08-31T17:51:00.007+02:00

Today's post is in Norwegian - I'll be back in English later

Vi sitter med stadig voksende mengder med data om spammere. Kan vi bruke dette på en måte som er nyttig for andre?

Vi som selv står for driften av eposttjenester vet av tidvis smertelig erfaring hva som skal til for at minimalt med uønsket reklame og lureri som kan være direkte trusler mot sikkerheten faktisk havner i innboksene til brukerne våre.

Epost: Dette burde vært enkelt
I utgangspunktet burde det være en enkel sak å håndtere epost: Serveren er satt opp slik at den vet hvilke domener den skal ta imot post for, og hvilke brukere som eksisterer i domenene. Når en maskin tar kontakt og signaliserer at den ønsker å levere epost, sjekker serveren om meldingen er adressert til en gyldig bruker. Er det snakk om en gyldig bruker, blir meldingen mottatt og lagt i innboksen til den aktuelle brukeren, i motsatt fall får den som er oppgitt som avsender beskjed om at meldingen ikke lot seg levere, og hvorfor.

Hvis bare alle var ærlige
I hvert enkelt ledd av denne prosessen er det en underliggende forutsetning at kommunikasjonspartnerne oppgir korrekt informasjon. I mange tilfeller er det slik, og det dreier seg om legitim kommunikasjon mellom parter som har grunn til å ønske kontakt. Dessverre finnes det også tilfeller der denne grunnleggende tilliten blir brutt eller misbrukt, for eksempel når epostmeldinger blir sendt med annen avsenderadresse enn den reelle, gjerne noe oppdiktet i et domene som tilhører andre. En del av oss har også opplevd å få returmeldinger om at levering ikke var mulig på grunnlag av meldinger som vi vitterlig ikke har sendt[1]. Når vi ser nøyere på innholdet i disse meldingene, vil vi i nesten alle tilfeller se at dette er søppelpost, tidvis med innslag av svindel og kanskje ledd i førsøk på å overta mottakerens maskin eller stjele sensitive data.

Hva gjør de ansvarlige?
Hvis du spør en typisk systemadministrator om hvilke tiltak som er satt i verk for å hindre at uønskede eller skadelige meldinger faktisk kommer frem til brukerne, vil du antakelig få høre en beskrivelse som stort sett kan kokes ned til at posten blir filtrert gjennom systemer som studerer innholdet i meldingene. Hvis meldingen ikke inneholder noe som er kjent som uønsket (kjent spam eller skadelig programvare) eller noe som likner på noe som kunne være det, blir meldingen levert til brukeren den er adressert til. Hvis systemet avgjør at meldingen har innhold som gjør at den ikke skal leveres til mottaker, blir meldingen i mange tilfeller kastet uten å bli levert, og noen systemadministratorer vil nok også fortelle deg at systemet sender melding om avgjørelsen tilbake til adressen som er oppgitt som avsender.

Mye av dette hører antakelig til den passive kunnskapen hos mange, og de fleste slår seg til ro med at innholdsfiltrering er det eneste vi kan gjøre for å holde tvilsomme eller direkte kriminelle elementer unna arbeidsmiljøet. For en enkelt sluttbruker er det sannsynligvis bare mindre justeringer ut fra dette som er mulig.

Tiltak basert på observert adferd
Men for oss som håndterer driften av selve tjenesten er det mulig å studere dataene som registreres automatisk i loggene våre og bruke spammernes (her brukt om avsendere av alle typer uønsket epost, inkludert skadevare) adferd til å fjerne mesteparten av den uønskede trafikken før innholdet i meldingene er kjent. Det er nødvendig å gå ned på et noe mer grunnleggende nivå i nettverkstrafikken og studere adferden på nettverksnivå.

En av de enkleste formene av slike adferdsbaserte tiltak dukket opp i form av en teknikk som fikk navnet grålisting (greylisting) i 2003. Teknikken bygger på en litt pedantisk og ganske kreativ tolkning av allerede vedtatte standarder. Protokollen som brukes for epost-overføring på Internett, SMTP (Simple Mail Transfer Protocol) inneholder en mulighet for at en server som har antatt forbigående problemer med å ta imot epost kan rapportere om tilstanden ved å svare med en spesiell feilkode når andre maskiner prøver å levere. Hvis avsendermaskinen er korrekt konfigurert, vil den vente en viss tid før den gjør et nytt forsøk på levering, og etter all sannsynlighet vil den lykkes etter kort tid. Det er verd å merke seg at dette er en del av standarden som først og fremst skulle sørge for at eposttjenesten skulle være så pålitelig som mulig, og i de fleste tilfeller skjer alt dette uten at personen som skrev og sendte meldingen merker noe til det. Meldingene kommer frem, og alle er fornøyde.

Grå og svarte lister, små hvite løgner
Grålisting går ut på at serveren gir melding om midlertidig feil til alle forsøk på epostlevering fra maskiner den ikke har hatt kontakt med før. Erfaringene viste at antakelsene fra før eksperimentet i hovedsak var korrekte: De aller fleste maskiner som sender legitim epost er satt opp til å sjekke returkoder og handle etter dem, mens de aller fleste avsendere av spam bare pøser på så mange meldinger som mulig, uten å sjekke returkoder. Resultatet er at et sted mellom åtti og noenognitti prosent av spam-mengden blir stoppet på første forsøk (før eventuell innholdsfiltrering), mens legitim post kommer frem, i noen tilfeller med en viss forsinkelse ved første kontakt.

En annen adferdsbasert teknikk, som faktisk var i bruk før grålisting ble utbredt, er å bruke såkalte svartlister - lister over maskiner som er blitt klassifisert som spam-avsendere - og avvise post fra maskiner som var med i listen. Noen grupper begynte etterhvert å eksperimentere med såkalte tjærehull (tarpit), der teknikken går ut på å forsinke trafikk fra maskiner som er med i en svartliste ved å la sin del av kommunikasjonen gå svært sakte. Et eksempel som ofte nevnes er programmet spamd, som det frie operativsystemet OpenBSD lanserte i mai 2003. Programmet hadde da som sin hovedoppgave å svare på eposttrafikk fra svartlistede maskiner med ett tegn i sekundet, uten at avsendere som var med i en svartliste hadde noen reell mulighet til å få levert meldingene.

Kombinasjonen av svartlister og grålisting har vist seg å fungere utmerket. Likevel fortsetter praktikere og utviklere forsøkene på å få til enda mer effektive teknikker. Enda en gang kom neste logiske steg som resultat av observert adferd. Vi så tidligere at spammere ikke bryr seg om å sjekke om hver enkelt melding faktisk kommer frem.

Legge ut snarer og agn
Tidlig i 2005 førte dette til at det ble det formulert en teori som det viste seg å være hold i: Hvis vi så lager en eller flere adresser i vårt eget domene som vi vet aldri vil ha noen grunn til å få legitim post, så kan vi med nesten hundre prosent sikkerhet vite at post som blir forsøkt levert til disse adressene er spam. Adressene er spammerfeller. Maskiner som prøver å levere spam til disse adressene, kan vi legge i en lokal svartliste som vi så oppholder med ett tegn i sekundet. Maskinene blir liggende i svartlisten i 24 timer dersom ikke annet tilsier det.

Den nye teknikken fikk navnet greytrapping, og ble lansert som del av en forbedret spamd i OpenBSD 3.8 i mai 2005. Bob Beck, som var sentral deltaker i utviklingen av spamd, annonserte tidlig i 2006 at han brukte greytrapping på sentralt plasserte maskiner ved University of Alberta, og gjorde svartlistene som er resultat av fangsten, med oppdateringer hver time, tilgjengelig for nedlasting slik at andre kan bruke listene i sine oppsett. Dette er åpenbart nyttig, maskiner som sender til adresser som aldri har vært leverbare, har etter all sannsynlighet ikke legitim epost å levere, og vi gjør samfunnet en tjeneste ved å hindre dem i leveringen og å få dem til å kaste bort så mye tid som mulig.

Det er kanskje verd å nevne at svartlisten fra University of Alberta så lenge undertegnede har fulgt den har inneholdt minst noenogtyvetusen IP-adresser, og har i travle perioder vært oppe i nesten tohundretusen maskiner.

Også du kan bidra
Likevel er det ikke nødvendig å være sentralt plassert programvareutvikler for å kunne bidra positivt. De samme verktøyene som Beck bruker til å generere sin svartliste er tilgjengelige for alle som del av OpenBSD, og er ikke så vanskelig å bruke som man kunne frykte.

Her på bsdly.net og beslektede domener observerte vi sommeren 2007 en markert økning i meldinger om ikke leverbar epost til epostadresser som aldri har eksistert i noen av våre domener. Her var det helt klart snakk om at noen, en eller flere grupper, genererte eller fant på avsenderadresser for å unngå å få reaksjoner på spammen sin tilbake til seg. Dette førte i sin tur til et eksperiment som vi fortsatt har gående. Vi registrerer ugyldige adresser i våre egne domener som dukker opp i loggene våre. Av disse adressene velger vi ut de helt usannsynlige, legger dem inn i vår lokale spammerfelle-liste og legger ut listen på en egen side på webserveren vår[2].

Erfaringene viser at det tar svært kort tid før adressene vi fører opp på denne siden dukker opp som mottakeradresser. Kort og godt har vi klart å fore spammere med data som gjør det enklere for oss å stoppe dem, og i mange tilfeller får vi i tillegg spamsenderne til å bruke betydelige mengder tid på å kommunisere med våre maskiner uten å oppnå noe som helst. Antallet spammerfelle-adresser i vår liste er nå oppe i rundt femtentusen, og vi har tidvis observert grupper av maskiner som bruker noen uker på å arbeide seg gjennom hele listen, med gjennomsnittlig noe under syv minutter brukt per mislykkede leveringsforsøk.

Som et biprodukt av denne aktive spammerfangingen begynte vi å eksportere vår egen liste over maskiner som har blitt fanget via spammerfelle-adressene de siste 24 timer og legge den ut tilgjengelig for nedlasting. At denne listen finnes, er kun annonsert via siden med spammerfelle-adressene og noen bloggposter, men vi ser at den blir hentet regelmessig og antakelig automatisk av andre som bruker den i sine systemer.

Så langt har vi etablert at det er mulig å lage et system som gjør at sannsynligheten for at spam kommer gjennom til brukere er svært liten, samtidig som det er nesten helt usannsynlig at legitim post blir merkbart hindret. Dermed har vi det som tilsvarer gode gjerder rundt egen eiendom, men spammerne finnes fortsatt der ute og er et potensielt alvorlig problem for de som ikke har tilstrekkelig beskyttelse mot dem.

Samle bevis, eller i det minste skape klarhet
Aller helst hadde vi ønsket at politi og påtalemyndigheter hadde tatt spammerproblemet alvorlig. Dette ikke bare fordi den spammen som kommer frem er irriterende å se, men fordi nesten all spam sendes via utstyr som spammerne bruker uten eiernes samtykke. Kort og godt ønsker vi at det blir satt inn ressurser som står i forhold til den kriminelle virksomheten spammen representerer. Vi ville gjerne hjelpe til, men i utgangspunktet kan det virke som vi kan ha problem med å skaffe til veie brukbart bevismateriale siden vi ikke mottar meldingene som spammerne forsøker å levere. På den annen side har vi til enhver tid en liste over maskiner som har prøvd å levere spam, noe nær hundre prosent sikkert identifisert på grunnlag av spammerfelle-adressene. I tillegg produserer systemene våre rutinemessig logger over all aktivitet, med det detaljnivået vi selv velger. Dermed går det an å søke i loggene etter IP-adressene som har forsøkt å levere spam til oss siste 24 timer, og få oversikt over hva maskinene har foretatt seg.

Resultatet av et typisk søk av denne typen ser slik ut:
Aug 10 02:34:29 skapet spamd[13548]: 190.20.132.16: connected (4/3) Aug 10 02:34:41 skapet spamd[13548]: (GREY) 190.20.132.16: <kristie@iland.net> -> <asasaskosmicki@bsdly.net> Aug 10 02:34:41 skapet spamd[13548]: 190.20.132.16: disconnected after 12 seconds. Aug 10 03:41:42 skapet spamd[13548]: 190.20.132.16: connected (14/13), lists: spamd-greytrap Aug 10 03:42:23 skapet spamd[13548]: 190.20.132.16: disconnected after 41 seconds. lists: spamd-greytrap Aug 10 06:30:35 skapet spamd[13548]: 190.20.132.16: connected (23/22), lists: spamd-greytrap becks Aug 10 06:31:16 skapet spamd[13548]: 190.20.132.16: disconnected after 41 seconds. lists: spamd-greytrap becks
Den første linjen angir at 190.20.132.16 tar kontakt med vårt system klokken 02.34.29 om morgenen tiende august, som fjerde aktive forbindelse, derav tre svartlistede. Noen sekunder senere blir det klart at dette er et forsøk på å levere en melding til adressen asasaskosmicki@bsdly.net, som er blant de vi har som spammerfelle, sannsynligvis høstet fra logger og diktet opp et annet sted i verden. Etter 12 sekunder kobler denne maskinen fra. Forsøket på å levere til en spammerfelle gjør at maskinen blir oppført i vår lokale svartliste, spamd-greytrap, noe som vises klart når maskinen prøver igjen litt mer enn en time senere. På dette forsøket blir den oppholdt i 41 sekunder. Det tredje forsøket i vårt loggmateriale skjer like etter 06.30, og at listenavnet becks har kommet til, viser at maskinen i mellomtiden har forsøkt å levere til en av Bob Becks spammerfelle-adresser og nå er med i også den svartlisten.

Det er dessverre lite sannsynlig at slike logger er tilstrekkelige som bevismateriale i straffesaker, men for de som har interesse av enten å sørge for at maskinene de administrerer i så liten grad som mulig brukes til spamutsendelse eller de som har interesse av spammeradferd er dette nyttige data.

“Name And Shame”, eller kanskje bare godt naboskap?
Etter noen diskusjoner med kolleger bestemte jeg meg tidlig i august 2008 for å generere daglige oversikter over aktivitetene til maskiner som har kommet inn i vår lokale svartliste på bsdly.net og legge dem ut offentlig. Når en maskin er oppført i en svartliste med bare IP-adresse (for eksempel 24.165.4.190), uten annet materiale om oppføringen, er oppføringen mest en påstand som de ansvarlige godt kan velge å ikke tro på. Vårt håp er at om den som er ansvarlig for nettverket der 24.165.4.190 hører hjemme ser en sekvens som denne,
Host 24.165.4.190: Aug 10 02:57:40 skapet spamd[13548]: 24.165.4.190: connected (9/8) Aug 10 02:57:54 skapet spamd[13548]: (GREY) 24.165.4.190: <hand@itnmiami.com> -> <kimberlee.ledet@ehtrib.org> Aug 10 02:57:55 skapet spamd[13548]: (GREY) 24.165.4.190: <hand@itnmiami.com> -> <kimberliereffett@ehtrib.org> Aug 10 02:57:56 skapet spamd[13548]: 24.165.4.190: disconnected after 16 seconds. Aug 10 02:58:16 skapet spamd[13548]: 24.165.4.190: connected (8/6) Aug 10 02:58:30 skapet spamd[13548]: (GREY) 24.165.4.190: <brunson@jebconet.com> -> <kimberlee.ledet@ehtrib.org> Aug 10 02:58:31 skapet spamd[13548]: (GREY) 24.165.4.190: <brunson@jebconet.com> -> <kimberliereffett@ehtrib.org> Aug 10 02:58:32 skapet spamd[13548]: 24.165.4.190: disconnected after 16 seconds. Aug 10 02:58:39 skapet spamd[13548]: 24.165.4.190: connected (7/6), lists: spamd-greytrap Aug 10 03:02:24 skapet spamd[13548]: (BLACK) 24.165.4.190: <aarnq@abtinc.com> -> <kimberlee.ledet@ehtrib.org> Aug 10 03:03:17 skapet spamd[13548]: (BLACK) 24.165.4.190: <aarnq@abtinc.com> -> <kimberliereffett@ehtrib.org> Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: From: "Preston Amos" <aarnq@abtinc.com> Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: To: kimberlee.ledet@ehtrib.org Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: Subject: Wonderful enhancing effect on your manhood. Aug 10 03:06:04 skapet spamd[13548]: 24.165.4.190: disconnected after 445 seconds. lists: spamd-greytrap
så er det tilstrekkelig grunnlag for å gjøre noe aktivt. Materialet er tilgjengelig via The Name And Shame Robot-siden på http://www.bsdly.net/~peter/nameandshame.html. Siste genererte loggoversikt er tilgjengelig via referanser på den siden, tidligere utgaver blir arkivert, men vil være tilgjengelige ved godt begrunnet forespørsel.

The Name and Shame Robot er såpass ny at vi ikke kan si spesielt mye om effekten av offentliggjøringen. Det er lov å håpe på at andre vil gjøre noe tilsvarende ut fra sine lokale loggdata eller kanskje til og med synkronisere sine data med våre. Ta gjerne kontakt hvis du er interessert i dette arbeidet.

Uavhengig av alt annet håper vi at dataene kan være nyttige, både som påpeking av forbedringspotensiale for de nettverkene som opptrer jevnlig i oversiktene og som materiale for studier som kan gi oss enda bedre spambekjempelse.

Noter

[1] En samling av slike returmeldinger fra tidligere i år kan beskues på http://www.bsdly.net/~peter/joejob-archive.2008-07-28.txt

[2] http://www.bsdly.net/~peter/traplist.shtml, referanser på den siden fører videre til bloggen min som jeg bruker til offentlige notater, og annet relevant materiale.

En forkortet utgave av denne artikkelen ble trykt i Computerworld Norge 22. august 2008.

Logfiles in the buff

2008-08-27T14:14:00.005+02:00

Search engine optimization, deflowered.

Logs are important. Depending on the specific kind of log, the data may shape lives and generate fortunes (how many times were those ads displayed, your clickthrough rate), reveal suspicious behavior and trigger actions (such as shutting the door to that bruteforcer) or provide sysadmins such as yours truly a general idea of what works and not or anything inbetween.

If you're a sysadmin, log data or log data derivatives such as a monitoring tool's graphical status display is more likely than not an important underlying factor to determine how you spend your day.

Then of course most of the material for these columns comes from log files, too. Depending on the specific log file, I tend to either just peek at the data my monitoring scripts offer me or do some manual greping for any patterns that interest me.

One such pattern matches the filename for my resume. I put that online for job hunting purposes, and now that I'm basically a gun for hire, it's slightly interesting to see any activity involving that file.

So at semi-random intervals, I check the apache log for references to my resume. Today, the grepery turned up this nugget
92.48.107.33 - - [27/Aug/2008:04:41:12 +0200] "GET /%7Epeter/PNMH-cv.html HTTP/1.0" 200 12318 "http://afmfokuv.fcpages.com/hot-anime-lesbians.html" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
and I count myself lucky that I had thoroughly swallowed my last mouthful of coffee before reading that.

In the Era of PageRank, the Age Search Engine Optimization Consultant, Season of the Clickthrough rage, I suppose we should not be entirely surprised to see such things. Just what the two documents have in common, perhaps other than targeting a very specific market, is left as an excercise for the terminally curious. I would advise some caution in choice of browser and operating system if your research takes you to the referring URL. One of the lessons of the day is, it doesn't always take a spamd log to crack you up.

PF tutorial in London, November 26
In other news, the UKUUG are hosting a full day PF tutorial featuring yours truly in London on November 26th, 2008. See the UKUUG web site for details. OpenCON is the following weekend in Venice, and I hope to make it there too.

The Name and Shame Robot
Last week the Norwegian edition of Computerworld published an article about the Name and Shame Robot, unfortunately in the paper edition only (yes, I've got an English article in process too). The article did spur some nameandshame.html traffic from unexpected places, but no offers of cooperation or spamd synchronization so far. In the meantime, I'm running into odd cron behavior differences when trying to run the generator script I wrote on my OpenBSD machines on a few FreeBSD hosts. More than likely there is a lesson to be learned there too.

Is one of your machines secretly a spambot?

2008-08-09T13:27:00.007+02:00

Some times we just need facts on the table, automated.

In my previous blog post, I wondered aloud about publishing data about the machines that verifiably tried to spam us. The response was other than overwhelming, and with the script running once per day anyway, I now publish the results via the Name And Shame Robot page.

The annoucement below is very close to the text there, so by way of explanation, here is a gift to all my fellow spamd junkies out there:

We started actively greytrapping and publishing our list of greytrap addresses (almost exclusively addresses generated or made up elsewhere and harvested from our logs) during July 2007. The list of greytrap addresses is published on the Traplist page along with some commentary. You can find related comments in this blog post and its followups.

One byproduct of the greytrapping is a list of IP addresses that has tried to deliver mail to one or more of our greytrap addresses during the last 24 hours. The reasoning is, none of these addresses are valid, and any attempts at delivering to those addresses is more likely than not spam. You can download that list here as a raw list of IP addresses, or as a DNS zone file intended as a DNS blacklist here.

In early August 2008, I wrote a small script that copies (rsyncs, actually) the current list of trapped IP addresses as well as the spamd log off the firewall and for each IP address collects all log entries from the spamd log. The resulting file is rsynced back to the webserver, and you can view the latest version here.

The material here is useful mainly to the system administrators responsible for the machines that appear in it, or people who are interested in studying spammer or spambot behavior. Times are given according to the Europe/Oslo time zone (CET or CEST according to season), and if a date appears several times for an IP address entry, the reason is simply that the log data spans several years. The default syslog settings do not record the year.

In the data you will find several kinds of entries, most of them are pretty obvious and straightforward, others less so. The likely FAQ is, "what are the entries with no log data?". The answer is, the spamd here synchronizes with a spamds at other sites. The entries without log data entered our traplist through a sync operation, but the host did not attempt direct contact here.

The other likely question is, "what is that becks list?". It's what the rest of the world refers to as uatraps. I copied the data for that list into my config from Bob Beck's message on OpenBSD-misc and didn't notice that the list had an official name until much later.

Please note that this is not an up-to-the minute list. Depending on the number of hosts currently in the list of trapped addresses, the script's run time could be anything up to several hours. For that reason, the script starts at the time stated at the beginning of the report file and runs until it finishes generating. The last thing the script does is to rsync the report file to the webserver. For the time being, I archive older versions off-line.

This is now a totally hands-off, automated operation. The report is currently generated on a Pentium IV-class computer with few and only occasional other duties. If you have any comments or concerns, the address in the next sentence is the one I use for day to day email. If you find this data useful, donations of faster hardware or money (paypal to peter@bsdly.net or contact me for bank information) is of course welcome.

Now that we have their addresses, do we name and shame?

2008-08-07T14:57:00.003+02:00

The legal owners of botnet-controlled spam senders are quite likely unaware what their machines are doing. Do they deserve to be outed, named and shamed?

Earlier this week a friendly Australian who I think had been reading my blog sent me a few questions about spam, spammers and what to do with them. Would it for example be useful to forward the IP addresses in the local traplist to law enforcement? After all, I publish a dump of IP addresses from my local-greytrap once per hour, and apparently at least some people are fetching and using that as a valid blacklist on a regular basis.

(On a side note: if you do fetch that list regularly, keep in mind that the data is dumped ten past every hour, that's when the data is fresh. If you fetch at every full hour, the data is already fifty minutes old).

Anyway, my initial reaction to the question about forwarding the list of IP addresses to law enforcement was, along the lines of "Well, a raw list of IP addresses doesn't really add up to a lot of evidence, but if you can extract the log entries for each one, you may have something". My actual answer was phrased a little differently, but even while I was writing my reply I started fiddling with a script to read my list of trapped IP addresses and grep the spamd log for all entries for each IP address.

My complete collection of spamd logs goes back a few years, so searching for a complete history does take a while. (For techies: for each IP address, a grep of the entire log takes at least a few seconds (s) , total time is s is times number of entries (N), typically a few thousand, and grepping in parallel is difficult, because you want the output per IP address, not interlaced like in the raw log data).

After a while, you can see output roughly like this:

Host 81.183.80.187: Aug 7 07:24:16 skapet spamd[13548]: 81.183.80.187: connected (12/9) Aug 7 07:24:30 skapet spamd[13548]: (GREY) 81.183.80.187: <akstcabrushwithafricamnsdgs@abrushwithafrica.com> -> <bennett-gauvin@ehtrib.org> Aug 7 07:24:31 skapet spamd[13548]: 81.183.80.187: disconnected after 15 seconds. Aug 7 07:24:44 skapet spamd[13548]: 81.183.80.187: connected (9/7) Aug 7 07:25:06 skapet spamd[13548]: (GREY) 81.183.80.187: <akstcamplepleasuremnsdgs@amplepleasure.net> -> <bennett-gauvin@ehtrib.org> Aug 7 07:25:07 skapet spamd[13548]: 81.183.80.187: disconnected after 23 seconds. Aug 7 07:25:08 skapet spamd[13548]: 81.183.80.187: connected (11/9) Aug 7 07:25:23 skapet spamd[13548]: (GREY) 81.183.80.187: <akstcaesamnsdgs@aesa.ch> -> <bennett-gauvin@ehtrib.org> Aug 7 07:25:24 skapet spamd[13548]: 81.183.80.187: disconnected after 16 seconds. Aug 7 07:26:16 skapet spamd[13548]: 81.183.80.187: connected (11/9), lists: spamd-greytrap Aug 7 07:30:00 skapet spamd[13548]: (BLACK) 81.183.80.187: -> <bennett-gauvin@ehtrib.org> -> <bennett-gauvin@ehtrib.org> Aug 7 07:31:43 skapet spamd[13548]: 81.183.80.187: From: "Frances Ballard" -> <bennett-gauvin@ehtrib.org> Aug 7 07:31:43 skapet spamd[13548]: 81.183.80.187: To: <bennett-gauvin@ehtrib.org> Aug 7 07:31:43 skapet spamd[13548]: 81.183.80.187: Subject: Extraordinary Narcotic Deals Aug 7 07:32:47 skapet spamd[13548]: 81.183.80.187: disconnected after 391 seconds. lists: spamd-greytrap

That's rougly what I would have expected to see: A host tries to send obvious spam to one of the trap addresses (one I harvested from incoming noise earlier), is added to spamd-greytrap and on the next attempts gets stuck for a few minutes. (Notice that this spammer has another version of grepable From: addresses - prepend akstc and append mnsdgs to the basename so abrushwithafrica.com becomes the junk address akstcabrushwithafricamnsdgs@abrushwithafrica.com. Content and header filterers, please take note.) I thought that this would be the typical behavior, but browsing the output from my script, entries of this kind seems to be more of the norm:
Host 81.192.185.9: Aug 6 12:47:15 skapet spamd[13548]: 81.192.185.9: connected (12/8) Aug 6 12:47:27 skapet spamd[13548]: (GREY) 81.192.185.9: <jacowen@teaneckschools.org> -> <hevadcouture@bsdly.net> Aug 6 12:47:27 skapet spamd[13548]: 81.192.185.9: disconnected after 12 seconds.
Here, the spambot tries exactly once, never to return. It's possible they detect the stuttering (our side answers one byte per second for the first ten seconds) and give up for that reason, but it could equally well be that it's classic fire-and-forget, the reason why greylisting still works. Or both, for that matter.

But back to the real question: Now that we have the data, what do we do with it?

With the script I have now, extracting the history for each of several thousand IP addresses takes some hours. The output is enlightening, but by the time the run is complete, it could be significantly more than twenty-four hours since the machines listed tried to send spam.

Should we name and shame anyway? If we forward the data to law enforcement, would they care?

For the time being, I'll try to think of a quicker way to extract the data. Any input on how to make the process more efficient is welcome, as is considered (learned or otherwise) opinion on the ethical up- or downside of publishing spamd log data.

Is there really a market for an open source router?

2008-07-02T13:08:00.005+02:00

Open source goodness. Coming soon to a router near you (if it isn't there already).

I have a confession to make. Today's headline isn't mine. I snatched it from Dana Blankenhorn's June 30th piece over at ZDNet. It almost made me utter a Simpsonian grunt and start ranting about my more than 40,000 visitors again. Maybe my readers don't constitute a market, and in a consumerland context, a mere forty thousand (they've been coming in at a rate of about five hundre new uniques a week for a little while now) is possibly small potatoes indeed.

On the other hand, there are good indications that significant parts of the Internet actually runs on open source in some form, regardless of sundry punditry or for that matter how many people have found my online or printed work. And then again, if even a small subset of those who have downloaded my work actually do some of the things I write about, there is reason to believe that they have achieved a degree of insulation between any local stupidity and the Internet at large.

But back to the ZDNet piece. The interesting news there is that Netgear are apparently coming around to support open source via the MyOpenRouter web site and at least one wireless router appliance with firmware source code available. It will be kind of interesting to see if they've actually made their code and specifications open enough that we have a reasonable chance of seeing non-Linux open source systems such as OpenBSD run on the platform.

If you read the OpenBSD source-changes list you probably know this already, but even if you don't, OpenBSD just turned -current into 4.4-beta (see Theo's commit message). I take that to mean that the various significant changes such as the overhaul of the PF code will be thoroughly tested in time for the 4.4 release. That change hasn't made it into snapshots just yet (but likely will witin the next few hours), but you can take a peek at the OpenBSD change log for a preview of the goodies that will be officially released on November 1st. I for one am looking forward to that date.

Yes, we can! Make a difference, that is

2008-06-25T16:23:00.004+02:00

Good netizenship sometimes comes with a green tinge.

Taking in my daily Linuxtoday dose this morning, there was one item grabbed that my attention, with the headline "Botnets and You: Save the World--Install Linux", and the Linuxtoday entry in turn points to Ross Brunson's blog post with the same title. Do click the link to Ross' blog, it's well worth reading.

What I particularly like about the piece is that he makes the point that you can actually make a difference. More specifically, if you run Linux (being a Novellian, he naturally recommends SLES or SLED) and eliminate Microsoft from your system, you are not only gaining for yourself a safer and more reliable platform, you are also helping everybody else by making the probability of your machine ever joining a botnet a lot smaller.

As regular readers here will recognize, I rate being a good netizen (aka net citizen) as extremely important. Let others get on with their business while we tend to our own tasks, not interfering unless we really have to. If you opt to run your day to day business on the same software your machine most likely came with, the likelihood that somebody else will be taking control of your machine and using it for less than desirable purposes is in fact anything but negligible. I could have used stronger words ("reckless endangerment" comes to mind), but then Redmondians would have just shut off all remnants of rationality. I have argued earlier (article in Norwegian only, sorry) that a computer owner's responsibility should be roughly on par with a dog owner's, but it's possible I should return to that in a future column. And besides, any Linux I've touched for the last ten years is easier to install and operate than the Microsoft offering.

If you followed the Linuxtoday link earlier, you know that I could not resist making the suggestion there that it is in fact possible to be an even better netizen. As outlined in an earlier column (and its followups), if you do your greytrapping properly, you can keep the bad guys occupied and have fun at the same time, consuming next to no resources locally. How's that for green computing.

For example, the crew who started sending messages with headers like

From: "Mrs Maria Jose" <cordinator@euros-ukolympics.com>
Subject: Immediate Response Required.(Euro Award)

to various spamtrap addresses on May 15th are still patiently trying to deliver. A very superficial log analysis shows that there were originally four hosts sending those messages from the 217.70.178.0/24 network. There appears to be only one left now, but collectively these machines have so far made 476,787 attempts at delivery to my data collection points. Judging from a sample of some 21,000 connections from one of the hosts, the average connection time was 389.68 seconds, which in turn means that we've had those spam senders waste approximately 185792441 seconds, or time equal to 5.89 years.

Not bad in a little more than a month. On the downside, the predictions that spambots would sooner or later learn to do things in parallel have been proved true. My logs indicate that the current crop is able to handle at least sixty simultaneous delivery attempts. Even bogged down by a suboptimal operating system at the sending end, modern desktop computers are in fact powerful beasts. In my book it's just good netizenry to set up a machine to keep the garbage they send off your own network, and by extension off others since they don't get around to try delivering to others. By the way, that list is now almost 15,000 addresses long, all non-deliverable garbage. You could be excused for thinking it a twisted art project.

BSD Unix? That's purely historical

2008-06-17T15:36:00.009+02:00

If you've ever bought something that ended up disappointing you to the point where you wanted to yell at somebody, you will recognize the frame of mind I was in after reading the book I ended up reviewing. With perfect hindsight I should of course have smelled the rat - anything written in this century about "BSD UNIX" is either a retrospective or ill informed. If you don't fancy a book review, tune in next time for something completely different instead.

Book Review: BSD Unix Toolbox

I came across this title while browsing an online bookstore for possible supporting literature for a course I'm planning. The teaser text boasts "1000+ Commands for FreeBSD, OpenBSD and NetBSD", and with a 2008 publication date I thought this one was definitely worth checking out.

At roughly 300 pages, covering 3-4 commands per page usefully would be a tall order for anyone, so I was a little surprised to find that the book sets aside two chapters to preliminaries, first "Starting with BSD systems" with a brief and not very complete overview of BSDish systems and some pointers to online resources, before moving on to an entire chapter on installing FreeBSD.

In a book that's supposedly about more than a thousand useful commands on FreeBSD, OpenBSD and NetBSD, setting aside an entire chapter to a rather superficial description of how to install one of the systems seems to me a very odd choice. Odder still, how to install either OpenBSD or NetBSD is not covered at all. Now installing any of those systems is not in fact too difficult, and I for one do not think the world needs yet another walkthrough of FreeBSD's or the other systems' install process. In my view, it would have been better if the authors had concentrated on getting around to describing those 1000+ commands, the sooner the better.

In the installation chapter, which also covers the ports and packages system, the authors seem to be unaware that each system has its own variety, and that on NetBSD the system goes under a different name. As the chapter title implies, this chapter is clearly FreeBSD specific, and would have been a lot more useful if the authors had noted at least some of the significant differences that exist between the systems the book sets out to cover.

Probably the most useful chapters in the book are chapters 3 through 6, where essentially all information is likely to be portable to all covered systems. However, Chapter 3, "Using the shell", is not without its oddities: The authors seem to assume that the user has installed Gnome as the preferred desktop environment and covers mainly using Bash as the shell. That is a slightly odd choice since to my knowledge Bash is not the default shell on any of the BSDs, but available as an optional extra through the package system.

Chapter 4, "Working with Files", walks through the basics of file types, file permissions, file system operations and commands such as cp, file, mount and a few others. After reading the chapter, you will be aware that these commands exist, if not much else.

Chapter 5, "Manipulating Text", offers the briefest treatment I've ever seen of regular expressions, mentions in passing vi and emacs as possible tools and then moves on to describing what appears to be the authors' favorite text editors (joe, nano and pico, neither of those are in the base system anywhere) and a brief mention of some graphical tools. The chapter then offers samples of using cat, head, tail, more, less, pr, grep, wc, sort, strings, sed, tr, diff, sdiff, awk, cut, od and finally unix2dos. Again, after reading the chapter you will be aware that the commands exist, but you will be looking elsewhere for detailed information.

Large chunks of Chapter 6, "Playing with Multimedia" would be useful on most unixlike systems (oggenc and convert likely perform much the same anywhere), but once again what little is offered as tips for getting sound or other functionality to work on your system is strictly FreeBSD-specific.

In Chapter 7, "Administering File Systems" the perspective is again distinctively FreeBSD-centric with little or no note of even potential differences across systems. For a FreeBSD user it may offer a useful if very brief and shallow walkthrough, though.

Chapter 8, "Backups and Removable Media", shows some examples of tar, gzip and rsync use, sometimes in combination, supplemented with brief mention of some common and less common file compression tools. The removable media section covers CD burning with cdrtools in more detail than most other software mentioned in this book, but fail to mention useful tidbits such as how to use OpenBSD's cdio command (which is in base) for similar tasks. In a book that claims to be up to date as of 2008, I find that a very curious omission.

Chapter 9, "Checking and Managing Running Processes" does little more than mention the names of some process management relevant commands such as nice, renice, fg, bg, kill and killall in passing before delving into a surprisingly detailed walkthrough of ps. It proceeds to describing top, pgrep, fuser (which the user is instructed to install via pkg_add), then returns to nice and renice and offering a couple of examples of fg and bg use before really picking up speed with kill and killall (fortunately with a list of signals), background processes started via either nohup or by appending an & character, and spending a couple of sentences each on at, gatch, atq, atrm and crontab.

Chapter 10, "Managing the System", weighs in at a little more than 20 pages, and characteristically slips back into FreeBSD-centric mode where it offers much detail at all. For some reason this is where the instructions about setting up your system for booting several operating systems turns up, along with a description of using GRUB as your boot loader.

Chapter 11, "Managing Network Connections" is again quite FreeBSD centric even if it does rattle off the names of the others at apparently random intervals. The information is as far as I can tell mostly correct for FreeBSD and some, if not all, commands will work elsewhere, but superficial enough that a user will have to turn elsewhere for help in resolving any problems that turn up.

Chapter 12, "Accessing Network Resources" covers anything from browsing the web (the authors state confidently that lynx 'has been supplanted [...] by the links browser, which was later replaced by elinks', apparently unaware that in OpenBSD at least, lynx is in fact part of the base system), fetching files with wget, curl, lftp (curiously recommending lftp even though in at least OpenBSD the base system's ftp client offers essentially all the required functionality), before spending four pages doing some handwaving about how to set up samba. IRC and mail clients are also mentioned, but I was a bit surprised that 'managing mail' apparently does not even touch on running a mail service.

Chapter 13, "Doing Remote System Administration", covers ssh, screen, tsclient (Gnome's windows remote desktop client), xhost, vnc and vino (another Gnome applet, this one for sharing your Gnome desktop) in that order, none of them in any great detail, bringing to mind the mantra 'now at least you know the commands exist'.

Chapter 14, "Locking Down Security", sprints through the basics of user and groups administration on FreeBSD, moves on to some tips about running services in general and via inetd, and also mentions firewalls.

The section "Configuring the Built-In Firewall" has me really baffled. The authors claim not only that ipfw has been ported to NetBSD and OpenBSD (where PF, mentioned here only as PacketFilter, has been the only packet filter since 2001), but the online reference they give (http://www.phildev.net/ipf) actually points to information about Darren Reed's IPFilter, also known as IPF.

It is unclear which firewall the authors think they have configured, but the actual rule sets they offer are ipfw scripts. It is likely that a user trying to run with the rc.conf snippet supplied and the rule set would in fact end up with both ipf and ipfw enabled, but likely with no working packet filtering (and depending on how the kernel with ipf and ipfw was compiled, the configuration would be either completely open or completely shut, another point apparently unknown or considered irrelevant by the authors). After a few examples of ipfw operations, the chapter then moves on to mention that yes, you can actually input your own information into the system logs, before recommending that you set up a centralized syslog server and instructing you to look into the third-party tools tripwire and chkrootkit.

The three appendixes have reference-style information (finally!) about using vi or vim, shell special characters and variables, and 'personal configuration files', aka dotfiles. All very brief, of course.

Unfortunately, this is a book I can not recommend. Large chunks of it or something very similar is available elsewhere, some of it for free and reasonably well written. If you're a FreeBSD user you will find yourself looking up the topics in the Handbook anyway, if you are a NetBSD or OpenBSD user, the relatively platform independent parts are addressed equally well in several 'Linux' books and other online resources.

There may in fact be more than a thousand monospace type 'command' examples in the book (I never counted), but other than that this title fails to live up to the expectations set up by the cover and other marketing. The book goes to some length to give the impression that it's current, with all dates I could see in examples set somewhere in the second half of 2008, but the authors appear to have been working with FreeBSD 6.3 as their current version.

The relatively frequent mention of 'BSD distributions' - a term that never entered the BSD vocabulary, mainly because the BSDs are maintained as separate systems - and various odd details such as the plainly wrong firewalls examples makes me suspect strongly that much of the material is in fact warmed over from a Linux book, slightly edited where the authors thought it was necessary. Unfortunately, they missed more than a few spots and for whatever reason the fact checking and testing did not get all the attention it should have. The result is a book that is very superficial where it's right and has enough spots that are anything from misleading to downright wrong that it gets rather irritating to an experienced reader. I can only imagine how frustrating it would be to use this as a resource for learning.

For a long time *BSD user (yes, all three), it seemed like a nice surprise that Wiley had discovered that there is such a thing as a BSD market. After a few hours looking at this entry, I hope they take the task a little more seriously the next time they try offering to sell us BSD literature.

Book info:

Title: BSD UNIX Toolbox: 1000+ Commands for FreeBSD, OpenBSD and NetBSD
Authors: Christopher Negus and Francois Caen
Publisher: Wiley Publishing, Inc. (Indianapolis)
Copyright: 2008
ISBN: 978-0-470-37603-4

More than 40,000 served

2008-06-04T15:16:00.004+02:00

Today's blog comes to you from sunny Aalborg in northern Denmark, where our Danish friends had the good sense to put together a one-day conference. Go to the web site at http://www.foss-aalborg.dk/ for details of the programme, I certainly hope the organizers will start a tradition and put on another conference soon.

For my own part, the PF tutorial (the free predecessor to the book), saw its confirmed unique visitor number forty thousand today, apparently a user somewhere in Ukraine:
$ ./mystats.sh Wed Jun 4 14:57:26 CEST 2008 Total PF tutorial visitors : 40000
I promise I won't bother you with updates of the number of visitors again until we reach fifty thousand. I'll do my very best to have produced some other interesting material before then.

I challenge your response, backscatterer

2008-05-25T21:02:00.004+02:00

A few weeks before I left Datadok, some real user accounts there, including mine, were joejobbed. That is, somebody, somewhere started sending spam with the From: or return address set to a real, live mail account in one of our domains. There were several incidents, the backscatter output of one of the more recent ones is preserved here.

Of course there were a handful of the "please prove to me that you are a human" messages from challenge-response systems too, but more about that later.

I normally keep the composed, sceptical attitude you will have come to expect from system administrators, but at fairly random intervals I turn into a regular Goody Two-Shoes. So faced with a collection of bounces for messages I certainly did not send, I decided to take some time to contact the people responsible for the machines that sent the spam. Owners of spam sending machines (as distinct from their Pwners) generally are not aware what the machines are doing, and at times I turn into that soft-hearted guy who just wants to help.

As you will see from the collection, not all bounces contain enough of the original message to track down where the spam was actually sent from. Then there were others where it was possible to dechipher where the message came from, and I made a canned message and sent it off to whatever addresses seemed likely from the whois output.

You should not be surprised to hear that quite a handful of those produced regular bounces (yes, postmaster@ and several other RFC2142 mandated mailboxes are non-existent at several sites), but what struck me was that of those that did not bounce there were several that instead produced "Hi, I'm the challenge/response robot at site.nx, please follow my instructions to prove you are human" responses.

Seriously, folks, does anybody in this day and age actually believe that these systems work? It's possible that challenge-response adherents' claims that their system stops all incoming spam are true, but seen from the outside, the only guaranteed effect is that your site will produce backscatter, and lots of it. It would perhaps have been interesting to know just how many of the undeliverables I've squirted into my spammer bait list were actually produced by challenge-response systems, but unfortunately, we have no way of knowing.

There are a few other things that challenge-response systems do not protect you against, such as plain old stupidity and malicious clicking by those who receive your challenge-response backscatter. I take the spam bounces in my collection as proof of that.

So challenge-response people, yes, I agree that you had no way of knowing whether Hampe Ivancevic, Hamada Kirkegaard, Caixia Kluge, Bitte Kutschinski, Anant Johaneson, Sumol Przygocki or a few others actually worked for me at the time you sent your challenge. I will however promise those fine people that if they exist, if they have the right qualifications and I am in a position to offer them employment, I will consider their applications in all seriousness.

The rant about certain blacklists and certain unwise ways to use them will have to wait until later. There are other deadlines to keep, the house still needs the odd splash of paint, but I'll catch up with you all later.

Fake Address Round Trip Time: 13 days

2008-05-21T21:08:00.003+02:00

The results are in. Our adversaries really are mindless automata.

Regular readers will have noticed that I've been running a small scale experiment over the last few months, feeding one spammer byproduct back to them via a reasonably accessible web page. The hope was that I would learn a few things about spammer behavior in the process.

After collecting fake addresses in my domains generated elsewhere for a while, I started noticing that a short time after I'd put addresses on the traplist page, they would start appearing as To: addresses on what appeared to be spam message entries in my logs. After a while more, I was certain that the round trip time was down to a few days, but my notes did not include exact dates for when each individual harvested address made it into my spamtrap list. Meaning, of course, I had no way of telling just how fast the process is.

Time to cheat slightly, or, as they say in the trade, perform a controlled experiment. Instead of sticking to the original plan of strictly collecting addresses generated elsewhere and feed them back to the harvesters via the web page, I decided to deviate slightly and plant an address at a specified date, and maybe add another few for data points later. I did the obvious thing and on October 11th, 2007, I slipped put-here-2007-10-11@datadok.no into one of that day's batches of collected addresses.

Needless to say, my attention wandered from the project in the meantime. After all, in October there was still that book to finish, and after the book was done, other developments had my attention or at least drained my energy for quite a while. So it was only last Sunday it struck me that by now I should have at least some data on what actually happened. One other reason it was suddenly quite appropriate to sum up the data was that the datadok.no domain had been turned over to new owners, and it will likely move out of my sphere of responsibility in the near future.

So here's the result, the fake address round trip time:
$ grep put-here-2007-10-11@datadok.no /var/log/spamd Oct 24 03:40:40 skapet spamd[20795]: (BLACK) 60.50.174.129: <pepgyoygq@boisdelan.com> -> <put-here-2007-10-11@datadok.no>
That is, the first time my artificially inserted address was used as a spam target was thirteen days after I put it on the traplist page. Since then, something, somewhere, has tried
$ grep -c put-here-2007-10-11@datadok.no /var/log/spamd 300
to deliver email to our imaginary friend a total of 300 times. Data taken from the spamd in front of that domain's secondary mail exchanger, of course. As always, I would love to hear from you about any related experiences.

In upcoming columns we will see, er, actually I find myself with such a selection of tempting topics to choose from, it is really hard to decide what to cover next. But the next one will appear here shortly.

Network devices that lie

2008-05-14T10:55:00.003+02:00

Do some network devices lie about their config, or are they just talking past each other?

One of my tasks recently has been to start integrating the networks of two companies, because one of the companies bought the other.

At my end you would have guessed already that anything exposed to the Internet would be running OpenBSD, while at the other end they were strangely attached to products from a certain software marketer based in the north-western part of the USA. But for the IPSEC part, they had decided that a certain proprietary "Internet Security Appliance" would fill their needs, and at each of their locations they had identical appliances running for "VPN" purposes.

Therein lies a story. After a few working days of debugging my way to an almost working configuration, my main conclusion is, that it is quite likely that proprietary security appliance lies. That is, it will let its administrator choose a wide range of very specific configuation options, but will actually override them, at least if the device at the other end does not do the "I'm one of your kind" secret handshake.

Here's what it looks like in the logs on the OpenBSD end, the part that is supposed to be a negotiation but actually isn't and the options on the OpenBSD side matched the options given in the screendumps of the other side's GUI that I had to work with:


May 13 14:00:28 delilah isakmpd[13547]: attribute_unacceptable: 
ENCRYPTION_ALGORITHM: got DES_CBC, expected 3DES_CBC
May 13 14:00:28 delilah isakmpd[13547]: attribute_unacceptable: 
HASH_ALGORITHM: got MD5, expected SHA

and so on, through the various options, each time expecting what matched the screendumps, getting something else entirely, up until the inevitable


May 13 14:00:28 delilah isakmpd[13547]: 
message_negotiate_sa: no compatible proposal found

or as they say elsewhere, Do not pass GO.

Moving on from there to actual tunnel setup reveals troublesome vendor IDs and other strangeness. This could very well be an attempt at vendor lock-in, but then the conventional wisdom anyway seems to be that the only way to set up a VPN is to have identical devices at every endpoint. There is no good reason why it should be like that, but I may be discovering the less good ones.

The main reason I'm not naming names or stating flat-out that "this device lies about its configuration" is that my analysis is not complete yet. If you have data you want to share with me on this or similar experiences, I would love to hear from you, at peter@bsdly.net. It's likely I'll write a followup based on what I hear and what happens.

During my silent period blogging wise, some notable events did occur, and I may return to them in later posts.

UKUUG Spring 2008
In late March I went to Birmingham to attend the UKUUG Sping 2008 conference - with a PF tutorial much like the one in Riga in February and a refreshed malware talk. A number of very good talks, and a number of Perl Mongers there, notably Barbie, who told me about his "Understanding Malware" talk that actually overlaps quite a bit with my paper. The social parts were good too, and yes, in Birmingham it is possible to find Real Ale.

BSD Magazine #1 is out
Not too long after the UKUUG Spring Conference, the first issue of BSD Magazine was published. Readers of this blog may find one article to be slightly familiar, but there's a nice selection of articles and reviews, covering all the BSD systems and a wide range of topics. And I should mention, they are looking for articles for upcoming issues.

OpenBSD 4.3 is out
OpenBSD 4.3 was released on schedule on May 1st, and as usual those who pre-ordered got their copies early. As always, exciting new stuff as well as a number of incremental improvements. There's interesting stuff happening post-4.3 as well, watch undeadly for updates.

Next up: FOSS Aalborg, June 4th
Our Danish friends are kind enough to give us yet another excuse to come visit. A group of BSD and Linux people have been hosting good conferences in Copenhagen for years, last year's EuroBSDCon being one.

This time it's Aalborg's turn, with a one day conference on June 4th. Yours truly will be on the speaker list. I hope to see you there, I will be bringing a small stack of books.

Does anybody here remember Artie Eff?

2008-04-13T21:11:00.009+02:00

That's what you thought you heard, right? Actually, this is not Uncle BSDly's nostalgia for a backstreet hoodlum he might have known way back when. It's Uncle BSDly trying to point out what happened the last time a standard was entrusted to that little Seattle company called Microsoft. R-T-F. The Rich Text format.

So the question is,

Does anybody here remember RTF?

It's been a long, long time, so you probably do not remember. But take my word for it, way back when OS/2 was to be the Next Operating System, the Rich Text Format was generally hailed as a Very Good Thing. It was a document and data format which came with a published specification, and the plan was that this was to be the vendor neutral data exchange format for all applications which would talk RTF compatibly and henceforth data destruction due to inane incompatibilities and fuzzy interpretations would be a thing of the past.

The original RTF 1.0 specification has been preserved at sourceforge as http://latex2rtf.sourceforge.net/RTF-Spec-1.0.txt, and our friends at WikiPedia have a short but nice article at http://en.wikipedia.org/wiki/Rich_Text_Format.

I originally started writing that piece at the end of July 2007, before the ISO voting process over Microsofts "Office Open XML" specification had started. I was working on a book at the time, and never got very far with my planned overview with a very personal slant on that file format. The RTF format was actually useful in its time, and how it is constructed and how it was maintained can show us a lot of useful things. Now that it's clear that Microsoft succeeded in getting ISO approval for its specification, it's time to return to that subject.

What a process it has been. Microsoft has succeeded in buying itself an ISO standard. Piece by piece, in a process that included ballot-stuffing, bribes, and the administrative overruling of the relevant technical committee's decision by a national standards body (right here in Norway), Microsoft bought itself or is very close to getting its wholly-owned ISO standard. Nevermind that there is at least one formal investigation by the EU into the process and that the chairman of one national body's technical committee (once again, in Norway) has lodged a formal protest over procedure with ISO.

Microsoft won, and we will all have to learn to live with the consequences.

One of the very effective tactics was to have Microsoft partners sign up as voting members of national standards bodies, with clear instructions on how to vote. This lead to an influx of new voting members in various parts of the standards organization so large that other standardization efforts where Microsoft has no interest at the moment have reportedly ground to a halt. The reason is that what remains of formal procedure in ISO dictates that if enough members of a national body fail to vote on a proposed standard, the standard is not approved and fails by default. So the first victim of Microsoft's ISO takeover is no new standards. Who would want more ISO standards anyway?

And oh yes, Microsoft's main OOXML propagandist in Norway has offered up this document as proof that OOXML has been implemented and is in fact useable in OpenOffice too. It did not open at all in my OpenOffice 2.3 running on OpenBSD 4.3-current, (see here and here for the two stages of the results), and informed sources tell me that the much-ballyhooed formula is not editable when the document is imported into other applications such as Microsoft's own Office 2003 plus the compatibility pack.

And as if to emphasize the fact that all you can guarantee when you call something "XML" is that the data will start with a < and end with a >, this XML document in fact handles the equation part as an embedded binary object, intimately tied to Microsoft's equation editor. I suppose this means that even at 9,000 pages and counting, the spanking new ISO standard does not cover handling equations in a satisfactory manner. All in all, just how much more weirdness this will lead to is hard to predict.

But I digress. As I said earlier, RTF was thought to be a very good thing. The specification was a lot less ambitious and a lot slimmer than the XML based one, and since RTF was the source format for the Microsoft Windows Help file format, recommended for all apps to run on Windows, there were even usable examples documented in one of the several manuals you would get with the Microsoft programming tools.

As you would expect, word processors and other applications were rewritten to be able RTF. I worked in documentation and localization at the time, and over the years I had a fair amount of exposure to the format and the tools. Basically, any Windows application worth documenting would require an online help system and the source format would be RTF.

Just like an XML file has to start with a < and end with >, RTF files are made up of elements that are delimited by matching { and } curly braces. In principle, you could write valid RTF using any text editor, and the Microsoft documentation would show you how. That fact came in handy a few times when Word managed to mangle some other document with its 'fast save' feature or as punishment for sending the document to be edited at one machine too many during a technical review. The solution to essentially any problem, as long as the .DOC file was possible to open at all, would be to save the document as .RTF, count the number of {s and }s and see if the numbers were equal. If they were not, you would add one of the missing kind at either the top or the bottom, using any straight text editor. The magic would work, or at least put you in a position where you could extract useful data.

Over time you would upgrade to newer tools, and Microsoft's development discipline (amply documentend in the OOXML tomes) would show up at every Word upgrade if you had not been paying attention.

It would go like this: You get the fresh Word version with all the new bells and whistles, and do another revision of the help system for an app. The help compile would break horribly, and after some hair-pulling and tabletop-thumping you would eventually remember that this had in fact happened before.

The definition of RTF had changed.

After a while you would discover that hidden somewhere deep in the new Word's online help or README you would find Microsoft noting that application developers would need to install a newer version of the help compiler to get their RTF to compile. Or to put it slightly differently, the operative definition of what "RTF" is would always be "whatever Word produces when you save as RTF". To their credit, the RTF people at Microsoft would usually publish a new version of the specification some time after a new Word release.

So, we've been there before. "Whatever Microsoft {Word,Excel,PowerPoint} produces" is likely to be the operative definition of what OOXML is, even if the 9,000 pages and counting specification says something subtly different.

With a specification that lets dates be represented in radically different ways depending on context, say if you're saving from a word processor, a spreadsheet or a presentation program, we're in for a lot of entertainment, I'm sure.

The other things I'm sure about are that there will never be an implementation of OOXML as currently written (even if the OpenOffice.org people have made noises that they might try), and that a full implementation of anything vaguely similar will have to come from Microsoft.

Real gluttons for punishment could try the new hobby (or profession, if you can get a sponsor) of tracking the the day to day development in the set of differences between OOXML-as-published and
OOXML-as-implemented.

Stuff I'll be getting back to: OpenBSD 4.3 is about to hit a mirror near you and hopefully your mailbox very soon, and more spam follies.

Riga, here we come, OpenBSD 4.3 on the horizon

2008-02-16T10:30:00.005+01:00

On Wednesday, Riga is the place to be, if attending a PF tutorial session for free is how you want to spend a day. You may have missed the announcement (and the update), but no matter - on Wednesday, February 20th I will be giving a full day tutorial in Riga, Latvia. bsdly.net has a Riga event page which is the space to watch for updates about the event.

A few things about the event is settled, though: The content will have quite a bit in common with the book, and I will be bringing a some copies of in case you need to pick up one (and promo postcards if you just want a souvenir). And time allowing, we will be previewing some of the things we can expect to see in the upcoming OpenBSD 4.3 release.

OpenBSD 4.3-RELEASE is likely to be cut in a few weeks' time (with a likely release date May 1st), so if you want to get an idea what it is going to look like or even want to help track down and squash any remaining bugs, now is the time to grab a snapshot and start putting it through the paces. If you're a little bit like me and can spare some capacity for testing, you will enjoy the process.

The next chance to catch a PF tutorial by me will be at the UKUUG Spring 2008 conference (not a free as in beer event), unless you arrange something yourself. This year's events look like they'll be a lot more ad hoc than earlier, so there may be other events announced later. Keep an eye on the OpenBSD Events page and any updates here.

Also keep an eye out for the upcoming premiere issue of BSD magazine. It promises to have about 60 pages of BSD related material in the first issue. I have an article in there that I hope you will enjoy.

See you in Riga or some other time,

A year ends; what to do next?

2007-12-27T07:47:00.000+01:00

It's the end of a year already. The end of the year is among other things the traditional time for tallying up totals to see what the year brought and looking forward to the fresh year ahead. Now at this point the inner geek in me and probably you too rebels with Why should any arbitrarily chosen point in time be assigned that much significance, huh?, but let's face it, it's one convention we will just have to live with.

The past year included a number of events, some entirely expected like the formal beginning of the end of the corporation once known as Caldera (yes, I know, it's not quite over yet, and I've written about that earlier), some rather surprising like the recent EU brokered patents and specifications deal which apparently means that the Samba team and other interested parties will not only be given access to usable protocol specs, they will even be furnished with a list of what Microsoft believes to be their relevant patents. That at least puts a serious dent in that corporation's patent FUD capability.

Any pundit in the Microsoft/Linux/FOSS "watcher" crowd who left that one out of their year end summary pieces should consider themselves cautioned: You were not paying attention to what could be this year's most important single piece of news in our field.

The general picture of the IT field is rather one of vast crowds of users who simply want to get on with their lives. The typical user is weary of the seventeen and a half times a week ritual Microsoft malware scare, and doesn't really see any benefit in getting a new computer with Vista to slow it down, now that they've finally weeded out or gotten used to all the annoyances of Windows XP and the background noise of unwanted popups and spam.

Rather more depressingly for us in the FOSS field, the typical user wants to just get on with his or her life and is weary, too, of the constantly overhyped "solutions" IT types are peddling. Faced with < insert your favorite product selling point here > , the stock answer now is, "gimme a break, that's what the last one said too".

This goes for almost any selling point, including vastly improved security along any measurable axis, efficient spam killing (including avoidance techniques like greylisting), the lightweight while useable desktop, and during the past year Microsoft even made a credible attempt at taking "open standards" prisoner. There is clearly a lot of work to be done, and we need to find ways to do that work better and present it in ways that actually add to FOSS people's credibility.

That includes, in my view, finding better ways to handle the periodic squabbles over licenses such as the GPL vs BSD shouting matches. It is likely that I will return to that topic in a future column, if and when I find the time to write it properly.

In my own little corner of the world, the publication of The Book of PF, marked here by the arrival of the author copies, marked the end of a long process that consumed rather more time and resources than I had anticipated. Before those copies arrived I had some copies made for OpenCon which were auctioned off for amazing sums that were subsequently donated to the OpenBSD project (see undeadly.org for details). Even though Amazon.com now lists the book as due for release January 11th, I have confirmation that No Starch shipped all preorders before they closed for the holidays, and I know at least one correspondent who got a message from the UK arm of Amazon that his copy was on its way. I'm interested in hearing from you about the book, of course, even reports that it has arrived safely in your mailbox.

Now other opportunities beckon, and I promise that in the coming year I will be writing about developments, confidentiality agreements allowing. If there is anything specific you want me to write about, please let me know.

I give you all my best wishes for the new year.

PS I almost neglected to mention that the PF tutorial (the forerunner of the Book of PF) saw its visitor (unique IP address or host name) number 27,000 for the period we have log data for on December 24th.

I Must Be Living in a Parallel Universe, Then

2007-11-25T10:23:00.000+01:00

It's Sunday morning, and I'm having my morning coffee while getting ready for a long session of editing my OpenCON presentation. By working on adapting the presentation tailored to the tutorial I've been rediscovering just how much work went into making the book, so a long Sunday session is needed, if not more.

Then courtesy of Groklaw's news picks comes the USA today piece called Despite filters, tidal wave of spam bears down on e-mailers.

A tidal wave of spam, no less. Well, we're seeing a lot of attempts at sending, like the sequence here (text link, formatting it would take too long) that I captured from the xterm running a tail -f on my spamd log a little while back. That sequence tells me, for one thing, that the naive spambot thinks my spamd looks like an open relay.

The other interesting thing about the sequence there is the pattern you can see in the From: addresses. It may have dawned on some of the spammers that generating random addresses in other people's domains might end up poisoning their own well, so they started introducing patterns to be able to weed out their own made up addresses from their lists. I take that as a confirmation that our harvesting and republishing efforts here and elsewhere have been working rather well.

Here the method seems to be that they take the victim domain name, prepend "dw" and append "m" to make up the local part and then append the domain, so starting from sia.com we get dwsiam@sia.com.

There is one other common variation on that theme, where the prepend string is "lin" and the append string is "met", producing addresses like linhrimet@hri.de, used just a few minutes ago to try to spam malseeinvmk@bsdly.net from the apparently Polish adress 89.228.40.80. This is of course very interesting, as is the fact that right now about two and a half thousand machines are in my spamd-greytrap list . That's where they end up, making no waves at all.

On the subject of patterns, earlier this month the address capitalgain02@gmail.com started appearing frequently enough that it caught my attention in my greylist dumps and log files.

The earliest contact as far as I can see was at Nov 10 14:30:57, trying to spam wkzp0jq0n6.fsf@datadok.no from 193.252.22.241 (apparently a France Telecom customer). The last attempt seems to have been ten days later, at Nov 20 15:20:31, from the Swedish machine 217.10.96.36.

My logs show me that during that period 6531 attempts had been made to deliver mail from capitalgain02@gmail.com via bsdly.net, from 35 different IP addresses, to 131 different recipients in our domains. Those recipients included three deliverable addresses, mine or aliases I receive mail for. None of those attempts actually succeeded, of course. With a little more time on my hands I'm sure I could have made a good regular expression to calculate to the second how much time those spam senders wasted here, too.

So where's the tidal wave? Back when PDF spam was the new horror, it actually took three weeks for one to reach me, and then only via an alias on a machine I really don't have much control over anymore. The number of spam sending machines does seem to be increasing, though.

Bob Beck's uatraps list is a good indicator, and the tendency is clear from the graph in my malware paper. The number did dip just below 100,000 addresses earlier this month, and it now seems to have stabilized in the 110,000 to 120,000 range.

From my perspective, it looks like a reasonably configured spamd is really all we need to observe the tidal wave at a safe distance and have fun all the while.

It's almost like living in a parallel universe.

Of Course, It Had To Be A Webshield

2007-10-28T19:55:00.000+01:00

In an earlier blog post, I mentioned that I would buy a round of drinks the first time I saw an attempt to deliver a message with both the From: and To: addresses already on my spammer baiting list.

In fact it happened very soon afterwards, and as luck, misfortune or just plain old incompetence would have it, that message apparently came from a WebShield appliance not too far from here:
Oct 17 23:03:52 skapet spamd[20795]: 194.54.96.18: connected (6/4) Oct 17 23:04:03 skapet spamd[20795]: (GREY) 194.54.96.18: <capitulations7@datadok.no> -> <capitulations7@datadok.no> Oct 17 23:04:03 skapet spamd[20795]: 194.54.96.18: disconnected after 11 seconds. Oct 17 23:19:21 skapet spamd[20795]: 194.54.96.18: connected (4/3) Oct 17 23:19:32 skapet spamd[20795]: (GREY) 194.54.96.18: <capitulations7@datadok.no> -> <capitulations7@datadok.no> Oct 17 23:19:32 skapet spamd[20795]: 194.54.96.18: disconnected after 11 seconds. Oct 17 23:30:30 skapet spamd[20795]: 194.54.96.18: connected (4/4), lists: spamd-greytrap Oct 17 23:34:14 skapet spamd[20795]: (BLACK) 194.54.96.18: <capitulations7@datadok.no> -> <capitulations7@datadok.no> Oct 17 23:35:58 skapet spamd[20795]: 194.54.96.18: From: Webshield.SMTP.V4.5.MR1a.Mail.Service@vs4.bgnett.no Oct 17 23:35:58 skapet spamd[20795]: 194.54.96.18: To: <capitulations7@datadok.no> Oct 17 23:35:58 skapet spamd[20795]: 194.54.96.18: Subject: Returned Mail: Error During Delivery Oct 17 23:37:00 skapet spamd[20795]: 194.54.96.18: disconnected after 390 seconds. lists: spamd-greytrap Oct 17 23:57:18 skapet spamd[20795]: 194.54.96.18: connected (6/6), lists: spamd-greytrap

I sent the operators at that site a polite message right away, pointing out the misconfiguration. Two weeks later I have seen no response other than the automatic acknowledgement, but it looks like the machine has managed to get itself automatically whitelisted in the meantime. So perhaps they found the button that actually does something.

Since my last blog post I have completed the book, and I expect the last bit of proofing to be done during the coming week. Then a few other necessary processes, and physical copies available for mid December if all goes well. With the cover in place, it looks like it's become attractive and popular over at amazon.com in its various categories. The BSD category there looks pretty No Starch dominated at the moment.

That can not be a bad thing. It's been a real pleasure working with the people at No Starch Press. If you think you want write a tech book, they should be on the list of publishers to contact with your proposal.

While all this was happening, the spammer baiting operation seems to have reached a critical mass of sorts. With roughly 7,200 addresses in the spamtrap list there are several hundred bait addresses for each real one in those domains taken together, so it's extremely unlikely that the spammers will ever get a chance to try delivery to a real address before they hit the tar pit. Over the last couple of weeks, my gateways have had anywhere between 2,500 and 4,000 hosts in the local spamd-greytrap, and anywhere from 0 to about 300 spambots pushing bytes into the tar pits at any time. It's fun to watch (some of the bots labor through the bait list from top to bottom), and the net effect is, well, we're not seeing much spam.

I think I've mentioned it before, but it bears repeating: To naive spammers and the tools they use, spamd looks like an open relay. Spamd never actually delivers any messages, but this

GREY|201.250.57.147|sofia|<vdaegkoxgk@bonana.com>| <brad.james.anderson@jhg.com.au>|1193105605|1193127205|1193127205|1|0

says that whoever operates 201.250.57.147 (according to whois, likely located in or near Buenos Aires, Argentina), is unable to tell the difference between an open relay and spamd's 451 and subsequent "this is going to hurt you more than it hurts me" messages.

Another variation on that theme is what I think is some sort of amateurish relay testing, which typically produces anywhere from five hundred to a thousand greylist entries of the type

GREY|59.35.4.51|UATIM-F7E7949C7|<adgjnq@194.54.103.104>| <ariel5268@yahoo.com.tw>|1193084672|1193113472|1193113472|2|0 GREY|59.35.4.51|UATIM-F7E7949C7|<xaehkn@rosalita.datadok.no>| <ariel5268@yahoo.com.tw>|1193084675|1193113475|1193113475|2|0 GREY|59.35.4.51|UATIM-F7E7949C7|<qswyd@brutha.datadok.no>| <ariel5268@yahoo.com.tw>|1193084691|1193113491|1193113491|2|0 GREY|59.35.4.51|UATIM-F7E7949C7|<nqtw@monalisa.datadok.no>| <ariel5268@yahoo.com.tw>|1193084733|1193113533|1193113533|2|0

where the From parts are made up of host names and IP addresses in our local net, including in this case, the host name for one of our laser printers. Those floods have tended to swell the bait list a bit, even if I strip out the invalid @<IP address> ones.

Spamd makes the naive relay testers think we have a whole network of open relays, and we harvest the noise they generate to lead the spambots to the tarpit. That's pretty close to a hands-off spammer repellent for us, and a serious auto-LART for the spammers.

OpenCON is sneaking up on us in a month's time, and we're heading for Venice with a refreshed tutorial session. See you there!

PS - [non-IT PS coming up] Bergen's football (soccer) team SK Brann has just won the national league for the first time in 44 years. With one game to go before end of season they are so far ahead in points there is no way any other team will be able to catch up. The town is predictably going gaga over the event, and we joined the thousands at the central Festplassen square for the city sponsored celebration tonight. I'm surprised how many songs have been written about that team and how everybody around me seened to know every last word of the lyrics. Good fun, ending with fireworks.

Always a pleasure to be wasting your time, guv

2007-09-29T19:09:00.000+02:00

This week has been a little unusual around the BSDly household. So far I've generally been doing my regular job in the daytime (with longish office hours), only working on the book evenings and weekends. That the arrangement would lead to "Exhaustion is my middle name" status was obvious to everyone except me, but I finally saw where it could be going. So for a little more than the past week I've been working on the book full time.

The state of perpetual exhaustion has had some not too happy consequences. Of course the general progress on the book suffered, but it also lead to me missing the monthly BLUG meeting in August. Of course much of that particular day I had spent persuading somebody not too bright that it indeed had to be a reconfiguration they said had never happend at their end which ended up breaking things at our end, and I was just too tired and missed what I assume was a well executed lecture on networking basics by Vegard Engen (of RFC1149 implementation fame).

This week with only one job I needed to tackle, I was there for an enjoyable one and a half hours of Bacula, well presented by Bård Aase (aka elzapp). Off to Henrik (the regular BLUG pub) for a few beers afterwards, and with Johan Riise volunteering to put together a 'Unix and time' lecture for next month, the BLUG calender seems to be in order after all, with Jill Walker doing the end of semester talk in November, on whatever interesting stuff she has been up to lately. Unfortunately it looks like the last Thursday of November is close enough to OpenCON that I'll likely miss Jill's session.

In the meantime, there are signs that the greytrapping and my bait list is working. Looking over the spamd logs today I found quite a few entries like these:


Sep 29 15:29:23 skapet spamd[20795]: (BLACK) 84.76.177.159: 
<royaleuromillion2007@yahoo.es> -> <211hgsreliart7@datadok.no>
Sep 29 15:29:32 skapet spamd[20795]: (BLACK) 84.76.177.159: 
<royaleuromillion2007@yahoo.es> -> <00b27f18@datadok.no>

which looks strikingly like the Spanish lottery scam spammers patiently and methodically working their way through my list of bait addresses, all the way from top to bottom, at roughly 3000 addresses it's going be a while. All I can say is, we are extremely pleased to be wasting your time, senor.

Also while the girls were off to the Raptus comics festival (an annual event, and one of the big things here in Bergen), I found enough trash backscatter to non-existent bsdly.net addresses that it's likely that the same weekend spambot operators who spewed their spam with @ehtrib.org and @skapet.datadok.no addresses earlier (both times at weekends) have now discovered bsdly.net and are doing their damnedest.

Why they prefer to generate a few hundred fake addresses and use them all in one go is beyond me. The other groups seem to generate only a handful of new addresses each every day, and for good measure at least one of them sort of reuse the generated addresses by using a forward and a reverse (such as in this morning's preserved greylist dumps, there was a potterv76@datadok.no as well as the reverse 67VRETTOP3@datadok.no). This lot just dumps all they have in one go, mainly contributing to swelling that file in my home directory with the totally unprintable file name which is the temporary storage before they go to into the traplist and on to the bait page.

Distractions of that kind from my main task is never entirely welcome, but with a larger influx of new addresses to be added to the bait list I made some small changes to make the maintenance of that page a bit more sane, rediscovering server-side includes and redirects along the
way. And the data I keep collecting may become the basis for other projects later.

Anyway, it is increasingly clear that the spammers are including the generated fake addresses in their "known good" lists. Consider the spambot at 210.111.190.216 (apparently in Korea), which insists on delivering to an address somebody generated in early July:


peter@skapet:~/www_sider$ grep  210.111.190.216 /var/log/spamd
Sep 29 15:58:07 skapet spamd[20795]: 210.111.190.216: 
connected (5/4)
Sep 29 15:58:21 skapet spamd[20795]: (GREY) 210.111.190.216: 
<jim.vance@presentsmadeeasy.com> -> 
<careersogt2083@datadok.no>
Sep 29 15:58:22 skapet spamd[20795]: 210.111.190.216: 
disconnected after 15 seconds.
Sep 29 15:58:35 skapet spamd[20795]: 210.111.190.216: 
onnected (4/3)
Sep 29 15:58:49 skapet spamd[20795]: (GREY) 210.111.190.216: 
<tbaker@groupecdb.com> -> 
lt;careersogt2083@datadok.no>
Sep 29 15:58:50 skapet spamd[20795]: 210.111.190.216: 
disconnected after 15 seconds.
Sep 29 15:59:03 skapet spamd[20795]: 210.111.190.216: 
connected (5/3)
Sep 29 15:59:17 skapet spamd[20795]: (GREY) 210.111.190.216: 
<wotan@4vsi.com> -> <careersogt2083@datadok.no>
Sep 29 15:59:18 skapet spamd[20795]: 210.111.190.216: 
disconnected after 15 seconds.
Sep 29 15:59:30 skapet spamd[20795]: 210.111.190.216: 
connected (6/5), lists: spamd-greytrap
Sep 29 16:03:14 skapet spamd[20795]: (BLACK) 210.111.190.216: 
<sylviacastleman@alltypecalligraphy.com> -> 
<careersogt2083@datadok.no>
Sep 29 16:04:59 skapet spamd[20795]: 210.111.190.216: 
From: "Marguerite Casey" <sylviacastleman@alltypecalligraphy.com>
Sep 29 16:04:59 skapet spamd[20795]: 210.111.190.216: 
To: <careersogt2083@datadok.no>
Sep 29 16:04:59 skapet spamd[20795]: 210.111.190.216: 
Subject: 100mg x 60 pills US $ 129.95 buy now
Sep 29 16:06:04 skapet spamd[20795]: 210.111.190.216: 
disconnected after 394 seconds. lists: spamd-greytrap

I have no real opinion on the validity of the From: addresses, but the address they are trying their best to deliver spam to here never actually existed, of course. The first record of it at datadok.no was this bounce from a Russian site:


Jul 12 23:38:52 delilah spamd[29851]: (GREY) 81.177.34.190: 
<> -> <careersogt2083@datadok.no>

Dumping their trash back at them is good for a laugh, and I am quite amazed how shortsighted the spambot operators appear to be. They get yelled at for spamming, so to avoid detection, they start using fake addresses. This in turn means they have no feedback whatsoever on the quality of their address lists, and with well pissers like me in action, they are getting less effectitive each day, reducing themselves to background noise in the network.

Now with this blog post done I will go back and finish the edits on the logs chapter. With the early parts of the book about to enter the layout phase while the last bits get written over the next few days, there is a chance that there will be a physical copies of the book to pass around at OpenCON. Not quite there yet, but the fulltime push is certainly helping. The preface with a list of thanks is part of what is entering layout; I think a few people who did not expect to be in there will soon have a pleasant surprise.

Also this week, the PF tutorial saw its unique visitor number 19,000 since EuroBSDCon 2006 on Thursday morning (September 27th). We certainly hope at least some of them will come back for the book.

The Great SCO Swindle Winding Down, But Will They All Get Away With It?

2007-09-21T12:03:00.000+02:00

Poor Dan Lyons. He thought like a bookmaker and wrote what he thought was right.

You see, a few years back, when Caldera was still Caldera, that company had successfully sued a large corporation and won. Then Caldera changed its name to SCO and sued another huge corporation. Dan the bookie thought it was a sure bet, and started cheering them on. Four years on, the sure bet went south on a technicality. They did not actually own the code they had accused others of stealing. At least that's the way I read his Snowed by SCO article over at Forbes.

My take on this is, Dan, you only had to look at the facts. Knowing a bit of IT history is also a plus. When the SCOX matter came up, I like most people thought that you can never rule out the possibility that some code might have been copied. After all, Unix source code was never particularly hard to get you hands on and was widely used as classroom examples all over the world.

Then if that code was just identified, it would be ripped out and replaced. It's happened before. In the free software world, whole subsystems get replaced when there's a good reason to, and if the reason is copyright violation it gets somewhat urgent. The problem is, in the SCO matter, no code was ever identified.

Some journalists went through an elaborate procedure involving non-disclosure agreements and were, we are told, showed code from Linux and somewhere else which showed remarkable similarity. When Darl McBride used the SCOForum 2003 conference to show something he passed off as ripped off code, it took only hours to identify the exact chunks through the obfuscation (yep, formatting comments in the Symbol font) and the code proved to be irrelevant.

None of these events helped convince techies of their claims, but for me the tipping point was when they claimed to have a reason to sue the BSDs as well. Anyone who had been paying any attention at all to Unix history knew that the ATT vs BSD lawsuit was finally settled in 1994, with most of the terms sealed, but one of the few things that was made public was that the parties had forfeited any right to sue each other over the Unix code base. To me and quite a few others, this was proof positive that they were 'misguided or dishonest', as a commentator put it at the time.

One of my favorite summaries of the facts of the case was written by Greg Lehey (of The Complete FreeBSD fame), who looked at the various announcements from the technical side. He stopped maintaining it after a while, but it's still there at his website, with as far as I tested with all links intact.

Most people seem to be relieved that the matter seems to be over. I beg to differ.

For one thing, the main characteristic of this matter has been the amazing ability of the SCO crowd to drag out the proceedings over irrelevant, mainly procedural matters. They will have more tricks up their sleeves, for certain.

The other thing is, with Dan's friends out on the technicality that they did in fact not have the legal standing to sue, we will never get that detailed walkthrough of the code where Darl and his covert experts are supposed to point out the infringing code. I, for one would have looked forward to that. Then we would have had a chance of getting to know their real motivation too, and possibly some solid leads on the planning and funding. Now that will just not happen.

Then of course there's the stockholder lawsuits and possibly the FTC. If you were one of those chumps who bought SCOX stock at roughly twenty dollars a share based on Dan Lyons' recommendations, wouldn't you feel a little sore now that your investment is about a cent to your original dollar? That is, if you can unload it before SCOX are finally kicked out of NASDAQ for good?

So poor Dan Lyons for not seeing this coming. And damn the technicalities for cancelling the main event.

For those of you eager for news of the book, we're working hard to get it out there.

Update 2007-09-25: Another non-apology, this one from Rob Enderle.

According to linuxtoday, Rob Enderle claims he was tricked by (wait for it) both SCOX and those ever-bullying Linux people.

Actually, there's not much to see there, You can read it as just another non-apologizing apology, with some tall tales about death threats and DOS attacks thrown in (yes, really).

As I've said a few times earlier, enough facts were on the table right from the start of this timewasting story to show that more than likely the SCOX crowd did in fact not have a case.

Now I wonder what, if anything we will be hearing from
John Parkinson, who wrote in CIO:

"a lot of the intellectual property in Linux is actually owned by companies that never officially agreed to make it available under an open-source license."

Interestingly enough, that came without any qualification at all.

That irritated me enough at the time that I wrote to them (pasted into some inane feedback form):

Alleged intellectual property theft

In the article called "The End of Idealism" (http://www.cio.com/archive/070103/et_pundit.html), John Parkinson writes, "a lot of the intellectual property in Linux is actually owned by companies that never officially agreed to make it available under an open-source license."

Please take a moment to consider the seriousness of this allegation. What Parkinson actually says here is, "large parts of Linux consist of stolen property".

Reading such allegations in an article written by a senior executive of Cap Gemini Ernst & Young is quite shocking in itself.

It is only reasonable that Mr Parkinson or Cap Gemini Ernst & Young specify which parts of the Linux kernel they consider to consist of stolen property.

All versions of the Linux kernel, along with detailed change logs and archives of the developer mailing lists are available to the public. Using these resources, all parts of the code base can be traced to the individual who submitted them for inclusion.

In other words, it is quite easy to pinpoint who did what, and Mr Parkinson and Cap Gemini Ernst & Young would be doing the public a great disservice by refusing to help point out code which was illegally included in the open source operating system.

Quite a few articles, well informed and otherwise, have been written about the SCO vs IBM lawsuit and SCO's allegations. I suggest interested readers browse FreeBSD Core member Greg Lehey's overview at http://www.lemis.com/grog/SCO/index.html while we wait for more details from Mr Parkinson or Cap Gemini Ernst & Young.

Slashdot It!

EuroBSDCon was great, disks dying and some scary Windows stuff

2007-09-17T22:36:00.000+02:00

This Monday finds me safely back from EuroBSDCon and trying to do useful things while the file server gets restored.

Of course it had to be that way. With me off to EuroBSDCon to do the tutorial and other refreshing geekiness, in the first batch of mail I retrieved after arriving in Copenhagen was a log summary from the machine which holds pretty much everything datadok is working on at any time, with these nuggets:
> > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=410884031 > > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=410912703 > > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=410884575 > > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=410905887 > > ad6: FAILURE - READ_DMA status=51 error=40 LBA=410857151 > > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=446104667 > > ad6: TIMEOUT - READ_DMA retrying (1 retry left) LBA=446104667 > > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=522840603
ouch.

This is what disks say when they've run out of space to map bad sectors into. The disk wasn't quite dead yet, but definitely time to plan a replacement. Not much to be done about that right away except alert the colleagues that there would be file server downtime on the Monday afternoon. Disks will die, and sysadmins end up with the task of replacing them.

My brief summary of EuroBSDCon is that it was an excellent conference, lots of good talks, interesting people to see and in a good, clean location with a network connectivity which worked, most of the time. update: finally my eurobsdcon pictures are on flickr

For my own part the PF tutorial went reasonably well, with 24 people signed up and I think one or two sit-ins. People were paying attention and there were a few good questions which made the session more interesting with a little more improv than the last few times I did this tutorial. Answers were had, though, and I believe a good time with useful info for the people who had signed up for the session. Not too many hours after we were done, the number of unique visitors (aka host names or addresses) to the tutorial tree since last EuroBSDCon rolled past 18,000.

After lunch Marco Zec's session about virtualizing the FreeBSD network stack was really interesting. Unfortunately none of the Thinkpads present were able to boot from the FreeBSD-current image Marco had prepared and supplied on USB thumbdrives, actually producing pretty much the same crash (illustrated here). But a very interesting topic and session. I'm glad I stuck around for it.

The Wednesday I had the choice of sightseeing, sitting in on Kirk's session and holing up in the hostel basement's hacker room to get some writing done, and I ended up going for the latter option, getting significant parts of the logging chapter done. There is of course a limit to how long you will avoid interruption in a semi-public area, but that session was certainly useful.

The EuroBSDCon hacker area with both wired and wireless networks was available to conference attendees all conference and tutorial days. Naturally it took on a social function in addition to being a convenient way to surf and fetch your email.

For the conference itself, it was sometimes hard to choose which talks to go to. I still think Ike's jails talk (pix here, here, here) was my favorite (similar but not identical to the one he gave at AsiaBSDCon in Tokyo), but there were a lot of good ones. I ended up managing to miss Pierre-Yves Ritschard's Load Balancing talk since they'd switched the schedule around. I hope there's a chance to pick up the essentials at some later date.

Fortunately Wim and Machtelt turned up to organize the OpenBSD booth (convenient for restocking your clothes cupboard) and some news about OpenCON - there will be an OpenCon 2007, but there's still some organizing to do. I hope to be seeing you there, Venice November 30th through December 2nd.

From the Windows Is Scary department, one episode from a few weeks back which I suddenly remembered when I realized the guy quietly hacking to the left of me was FreeBSD USB guru Hans Petter Selasky:

When I saw 4GB USB thumb drives priced at just under NOK 300 (USD 55), I decided I needed one. The drive mounted with no trouble at all in in OpenBSD (mount /dev/sd1i at the location of your choice), and I thought good, I'll just delete those .exe files to make room. A few days later I needed to retrieve som files which turned out were most easily accessible from my Windows machine at work. So I plugged in the new 4GB thumb drive.

Windows machines always do strange things and take a while to recognize new hardware, but this time it claimed to have found a new CD drive. A few confusing minutes later, with various message boxes flashing across the screen, the machine begged for a reboot. I let it have that, slightly puzzled but not entirely surprised that Windows wanted the user to jump through a few extra hoops to make something work.

I was able to retrieve the files eventually, while trying to avoid yet another quirky Windows application which wanted to handle my files. As it turns out, the device actually emulates a CD drive as well as USB mass storage. Here's what it looks like in /var/log/messages on my OpenBSD laptop:
Sep 17 22:23:23 thingy /bsd: umass0: SanDisk Corporation U3 Cruzer Micro, rev 2.00/0.10, addr 2 Sep 17 22:23:23 thingy /bsd: umass0: using SCSI over Bulk-Only Sep 17 22:23:23 thingy /bsd: scsibus2 at umass0: 2 targets Sep 17 22:23:23 thingy /bsd: sd1 at scsibus2 targ 1 lun 0: SCSI2 0/direct removable Sep 17 22:23:23 thingy /bsd: sd1: 3913MB, 498 cyl, 255 head, 63 sec, 512 bytes/sec, 8015502 sec total Sep 17 22:23:23 thingy /bsd: cd1 at scsibus2 targ 1 lun 1: SCSI2 5/cdrom removable
The reason all the strange and scary things happened with the Windows machine is that the emulated CD contains Windows Autorun files, which it seems there is no easy way to turn off or is at least enabled by default in that operating system. What I find slightly disturbing is that, as Hans Petter explained, this behavior is part of the device's firmware and you can't get rid of that five or six megabytes of useless software in these devices. The best you can do is use a system which ignores such silliness.

Returning to the file server, the box is a few years old and has by now probably had most of the original components replaced. The last time we replaced the motherboard, we were still thinking that SCSI was the only way to go for storage, disks and tape both. Not too long after that, we decided that actually SATA was OK for that little office of ours, but when the time came to replace that disk, I discovered that actually the motherboard had only two SATA ports on it, one for the system disk and one for the dying data disk. So copying across from one SATA disk to another had to be done via Ethernet instead. Fortunately installing a useful operating system takes only about twenty minutes, and the some tens of gigabytes transferred while I was writing this article. Far faster than restoring the same data via rsync from our offline backup, though.

Among the things announced in Copenhagen were that there will be an AsiaBSDCon in March 2008, NYCBSDCon will maybe be next year in the fall, and the next EuroBSDCon will be in Strasbourg. I hope to be at several of those, time and money allowing. But now on to finish that book.

Wanna help science? Study your greylists' innards!

2007-09-08T15:13:00.000+02:00

If somebody, say five years ago, had told me that I would be spending a little time, every day, studying data about what invalid addresses some unknown miscreants are making up in my domains, I would have thought them to be slighly off their rockers.

Yet here I am, actually maintaining a publicly available list of addresses which do not stand a chance of becoming valid, ever. It all started with a log data anomaly - I noticed an increase in the number of failed delivery messages to non-existent addresses in our domains. I had expected that the bounces to invalid addresses would appear for a short period only, but for one reason or the other it looks like it's here to stay, with some dips and peaks like the ehtrib.org flood.

The list is apparently working as intended too. These addresses are on my local greytrap list, and I have started seeing addresses I put in there as all uppercase turn up in my logs in all lowercase variants. Fun to watch, sort of.

Anyway, the supply of new bogus addresses proved to be larger than I had expected. So to get a handle on just what is happening I ended up doing periodic dumps of the live greylist data. This is really easy to do if you're using spamd as your greylister, your basic command is

$ sudo spamdb | grep GREY

and you redirect to a file, pipe to mail, or whatever you like.

Now if you're a bit like me, looking for patterns in the noise like this makes you feel a little weirder than usual and possibly lead you to think of a Clive Barker novel (specifically the bits about the dead letter file in The Great and Secret Show) and you wonder why this is worth doing at all. After all there is precious little spam that actuall reaches my users, so like I said earlier, for us spamd users it really looks like spam is a solved problem. I guess I'm just a bit fascinated by the pure irrationality of the spammers' behavior.

From the data I collect here in my tiny corner of the world to browse when time allows there may be useful information lurking somewhere.

Typical entries show things like the host 202.152.33.43 tried to send with a From: address jcejft@charter.com to dkqvujfn@datadok.no and sdenuuu@datadok.no. Using a few common networking commands we see that there is no reason why charter.com email should come from the IP range belonging to idola.net.id, and as the admin of datadok.no I know these two addresses have never been deliverable. Most likely the admin at charter.com can tell you if that from address is deliverable, but I keep wondering how much of the spam out there is stuffed into the pipe with bogus From: and To: addresses both. Or in other words, purely useless noise, never to be delivered anywhere.

On a side note, with one or more of the spammer operations trying to sneak through using sender and recipient addresses in the target domain, I assume it is just a matter of time before I see a tuple with both sender and recipient addresses already in my spamtraps list. When that happens, I think I will feel inclined to let my friends have a round of refreshments on my tab.

It's obvious that there are a handful of spammer operations that have decided to use datadok.no (and to a lesser extent, dataped.no and ehtrib.org) From: addresses on the spam they send, apparently in an attempt to cover their tracks. I will probably never know why they decided to do that, but I wonder why they keep it up and for that matter how many other domains are seeing this, with bounces from strange places, directed at non-existent, fairly obviously generated bogus addresses.

So if you are seeing similar stupidity in your logs and if you are running a sensible greylister such as spamd, I would be interested in hearing from you so we can compare notes.

Out there in meatspace, EuroBSDCon 2007 is coming up. I'll be there with the PF tutorial on Wednesday. This Friday's deadline for an updated manuscript had totally slipped my mind (I blame the book and a few other, less rational, factors), but hopefully the 24 who signed up for the session will find it useful anyhow - there will be new bits and as much interesting stuff as I can manage. I'll be around for the rest of the conference too, but unfortunately I'll have to give the Legonland trip a miss.

Be seeing you in Copenhagen! The book is getting closer to finished, I promise!

A Lady in Distress; or Then Again, Maybe Not

2007-08-19T14:42:00.000+02:00

A two user domain gets bounces for seven hundred, grep and sed to the rescue, spamd saves the day

The past week moved along with only minor disturbances on the keep-systems-running front. The time consuming frustrations were generated elsewhere, and (un?)fortunately I am not at liberty to discuss the details. Incompetence was involved, next week it's somebody else's problem.

All the while, the spammer trapping experiment has been moving along at a leisurely pace.

Generally keeping the lists (both the web version and the live one) updated would cost me a few minutes' browsing of greylist dumps two or three times a day or whenever I felt like it, with a typical catch of maybe fifteen new bogus addresses to feed to the trap list each day.

For the last three or four days the haul has been smaller, with essentially no new captures yesterday, for example. Now I've found out why. They have moved on, alpabetically.

Done with bsdly.net, the dominant group of spammers moved on to generating addresses in the D domains including datadok.no and dataped.no. I'm bound to have missed a few, since the grand total by this morning had yet to reach a full thousand. By now, they seem to have reached the Es. This morning I noticed the overnight greylist dumps were bigger than usual.

The reason: ehtrib.org, the domain we set up mainly for my wife's use (read: her email), appears to be the current home of made up From: addresses, with roughly seven hundred accumulated by the time I was done with morning routines of breakfast with coffee and browsing the overnight incoming mail.

That is by far the largest addition to the flypaper list ever.

Fortunately, with only two active addresses in the domain (I'm not telling what either other one is) it's fairly trivial to extract the bogus ones.

Up to now I've been integrating the noise into the traplist page manually, for now I've put this batch up at http://www.bsdly.net/~peter/ehtrib-1stbatch.

They're all in the active traplist at the gateways, of course. It's the editing into the page the spammers will slurp via unattended robot I'm putting off for a little more while I'm doing some other writing. [not any more. all there now, but the original list is preserved too]

Just why this time we are seeing this number of addresses over a short period of time, and not a handful each day over several months is an open question. One likely explanation is that one of the chickenboners fell asleep at the wheel and let the junk generator run longer than they actually intended. Time will show if this means they move on more quickly.

When I have more time, I will probably analyse the data I am accumulating at the moment and tell the tales of the silly lamer tricks the spammers try to pull.

In the meantime, following up on earlier posts, there are still a few people who Just Don't Get It:


Aug 19 13:28:03 delilah spamd[3712]: 217.159.231.230: connected 
(9/9), lists: spamd-greytrap
Aug 19 13:31:49 delilah spamd[3712]: (BLACK) 217.159.231.230:
<> -> <armrest10@datadok.no>
Aug 19 13:33:32 delilah spamd[3712]: 217.159.231.230: Subject:
Considered UNSOLICITED BULK EMAIL, apparently from you
Aug 19 13:33:32 delilah spamd[3712]: 217.159.231.230: From:
"Content-filter at linux.byroomaailm.ee" 
<postmaster@linux.byroomaailm.ee>
Aug 19 13:33:32 delilah spamd[3712]: 217.159.231.230: To: 
<armrest10@datadok.no>
Aug 19 13:34:38 delilah spamd[3712]: 217.159.231.230: disconnected 
after 395 seconds. lists: spamd-greytrap

And it looks like the published list is having the effect I was hoping for. I keep seeing quite a few of the addresses in ALLCAPS (with numbers tacked on) I put on the web page a few weeks back beginning to appear in lowercase but otherwise identical in my greylist dumps.

In other news, the PF tutorial session at EuroBSDCon is now a definite.

See you in Copenhagen, if not before!

Now for that other bit of writing. The Book of PF page now refers to the tutorial page at bsdly.net. Now let's get that baby done.

The lady is, in fact, not too distressed.

And in case you were wondering - Yes, you can use my auto-generated list of trapped hosts for your own blacklisting purposes if you like. Here it's just a supplement to Bob Beck's traplist, and most likely you're better off using the Beck/UofA list along with your own greytrapping, but if you really want to use mine, be my guest. It gets updated ten past every hour.