The numbers are in. The slow dunces still don't get it.
After five days of activity and no wins on my machines, the Hail Mary Cloud moved on. That means we have yet another complete set of data to summarize and analyze. The numbers are:
A total of 4773 attempts, none of them successful, involving 338 distinctive source addresses, the most active host (109.237.210.147, according to whois located somewhere in the Netherlands, made 109, while at the other end of the scale 30 hosts made only a single attempt). The wannabe attackers attempted to access 944 different user names, the most frequently attempted user name by far was root, with several blocks of root-only accesses even during the otherwise purely alphabetical stage.
The current sample is too small to support any far reaching conclusions, but it is tempting to speculate that with only 338 hosts participating we are seeing an indication that their success rate is sinking (previous attempts counted a cople of thousand hosts), even though they may be at least partially succeeding in their secondary goal: avoiding detection. That success is partial at best, this blog post and the earlier ones pluss varied commentary at Slashdot are indications that at least some of us are paying attention to our logs.
Another few observations worth making: 1) I have still not seen any of these sequences aimed at my Internet-facing OpenBSD systems, only Linux and FreeBSD ones. 2) It's likely that the miscreants are directing their attempts at several targets at the same time, so this sample is only a tiny fraction of the whole.
Reports of similar activity are surfacing from elsewhere, but very few people appear to be willing to share their data. It is of course even possible that the earlier episodes generated enough noise that better password policies (or preferably key logins only policies) are now in place, frustrating the random password guessers' attempts.
Whether or not you have been seeing these sequences in you authentication logs, please do yourself a favor and study your logs every now and then. It might even be worth the trouble to set up some kind of log collection and analysis infrastructure. Europeans may have to consider the legal implications of storing logs in light of the Data Retention Directive, denizens of the great elsewhere would do well to check if any similar legislation applies.
Good night and good luck.
Broken link fixed, sorry. Also, of course this has been discussed earlier, most recently in this post, also in this one as well as A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009). Read those for further info.
Update 2011-11-06: Another round of attempts has started, see the data aggregation page for the November 2011 entries. Of particular interest, perhaps is the List of participating hosts, sorted by number of attempts.
Update 2011-11-06 part 2: A note over at the ISC, "New, odd SSH brute force behavior" linked here, generating some additional traffic. Commenting over there requires a login and the confirmation email appears to be delayed by greylisting, so I'll comment here instead: I would not call this a particularly new approach. We've been seeing these attempts on and off since we started noticing them sometime in 2008, and it's entirely possible that there have been earlier attempts that did slip in under our radars. Analyses based on data from other sites beside mine would be very welcome indeed.
Update 2011-11-20: They keep coming back, now again after taking a 9 day breather (or possibly poking elsewhere in the meantime). Data accumulating again at the Hail Mary Cloud Data Page, with notes on the most recent activity at the very end. Please do play with the data, there's hope yet that some useful insights are to be found.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Saturday, October 29, 2011
Sunday, October 23, 2011
You're Doing It Wrong, Or, The Return Of The Son Of The Hail Mary Cloud
Do Linux system administrators still in this day and age run with PermitRootLogins yes in their sshd configurations? Do they also allow password logins? Do they ever attempt to keep their systems up to date and reasonably secure?
Apparently the answers are yes, yes, and no, at least for some. The evidence is slowly accumulating in the authentication logs on one of my servers, published via the The Hail Mary Cloud Data Page. There are several reasons why these attempts stand out, but it kind of helps that the number of users with sensible or indeed legitimate reasons for shell access to this particular server is quite limited.
I've ranted about this before, famously but not exclusively in a series of slashdotted and much-syndicated blog posts such as this one. For the TL;DR crowd, here's the summary:
If you're allowing root logins from the great elsewhere, you're doing it wrong.
If you've been allowing root logins from the great elsewhere, I wouldn't be surprised it's one or more of your boxes doing the distributed password guessing.
If you can't remember the last time you checked that your system is up to date and properly configured, you're doing it wrong.
So nothing really new to see here, it's only yours truly seeing his hope of never seeing this silliness repeated dashed, again.
If you're interested in background information about the Hail Mary Cloud phenomenon, please do read the previous posts (A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009) as well as the one referenced earlier.
Good night and good luck.
Update 2011-10-27: The alphabetic stage has started, see refreshed data for details.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Apparently the answers are yes, yes, and no, at least for some. The evidence is slowly accumulating in the authentication logs on one of my servers, published via the The Hail Mary Cloud Data Page. There are several reasons why these attempts stand out, but it kind of helps that the number of users with sensible or indeed legitimate reasons for shell access to this particular server is quite limited.
I've ranted about this before, famously but not exclusively in a series of slashdotted and much-syndicated blog posts such as this one. For the TL;DR crowd, here's the summary:
If you're allowing root logins from the great elsewhere, you're doing it wrong.
If you've been allowing root logins from the great elsewhere, I wouldn't be surprised it's one or more of your boxes doing the distributed password guessing.
If you can't remember the last time you checked that your system is up to date and properly configured, you're doing it wrong.
So nothing really new to see here, it's only yours truly seeing his hope of never seeing this silliness repeated dashed, again.
If you're interested in background information about the Hail Mary Cloud phenomenon, please do read the previous posts (A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009) as well as the one referenced earlier.
Good night and good luck.
Update 2011-10-27: The alphabetic stage has started, see refreshed data for details.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Wednesday, August 3, 2011
Practical Packet Analysis Is Good Fun [Book Review]
If you've wanted to take a peek at what network traffic really looks like, but put it off because it sounded all too complicated, a new book may be just what you need to get started. Practical Packet Analysis is an excellent network analysis introduction.
When Chris Sanders' Practical Packet Analysis (with the subtitle Using Wireshark to Solve Real-World Network Problems) was originally published in mid-2007, to generally very favorable reviews and quite respectable sales for a network book, the memory of Wireshark's predecessor Ethereal's eviction from the OpenBSD package system for too many unfixed security bugs in a short time was fresh enough that most of us in the OpenBSD contingent generally shrugged and moved on. The criticism of Ethereal centered on the fact that there was next to no separation between the packet capture part of the system (which needs to run with elevated privilege) and the analysis-related parts (that do not in fact need to). At the time I was also rather busy working on a book of my own (full disclosure: also a No Starch title), so I mentally filed Wireshark under 'things to get back to', possibly to look into reviving the OpenBSD port.
Nevertheless I was quite pleasantly surprised when No Starch Press contacted me a little while back and asked if I would like to have a review copy of Practical Packet Analysis, second edition. The book is a pleasant read and good fun, and in fact the program compiles from source fine on OpenBSD, but more about that later.
After a brief introduction that contains summaries of the numbered chapters and practical information such as where to get the sample packet capture files, Practical Packet Analysis leads in with a chapter called Packet Analysis and Network Basics, which introduces the reader through an introduction to network protocols in general and how packet sniffers fit in the general picture.
When Chris Sanders' Practical Packet Analysis (with the subtitle Using Wireshark to Solve Real-World Network Problems) was originally published in mid-2007, to generally very favorable reviews and quite respectable sales for a network book, the memory of Wireshark's predecessor Ethereal's eviction from the OpenBSD package system for too many unfixed security bugs in a short time was fresh enough that most of us in the OpenBSD contingent generally shrugged and moved on. The criticism of Ethereal centered on the fact that there was next to no separation between the packet capture part of the system (which needs to run with elevated privilege) and the analysis-related parts (that do not in fact need to). At the time I was also rather busy working on a book of my own (full disclosure: also a No Starch title), so I mentally filed Wireshark under 'things to get back to', possibly to look into reviving the OpenBSD port.
Nevertheless I was quite pleasantly surprised when No Starch Press contacted me a little while back and asked if I would like to have a review copy of Practical Packet Analysis, second edition. The book is a pleasant read and good fun, and in fact the program compiles from source fine on OpenBSD, but more about that later.
After a brief introduction that contains summaries of the numbered chapters and practical information such as where to get the sample packet capture files, Practical Packet Analysis leads in with a chapter called Packet Analysis and Network Basics, which introduces the reader through an introduction to network protocols in general and how packet sniffers fit in the general picture.
The chapter then goes on to present the classical ISO seven layer model and makes a credible attempt at mapping those to the slightly fewer layers of the TCP/IP stack before going through a discussion of common variants of networking hardware and touches briefly on classes of network traffic (multicast, broadcast and unicast).
The second chapter, Tapping into the Wire focuses mainly on how and where to position your network tap, introduces a couple of different hardware devices for the purpose and even touches on ARP spoofing as a tap technique before giving a first glimpse of how to display and interpret captured traffic using the Windows utility Cain & Abel, and ends with a summary that includes a flowchart to help decide on the most useful tapping technique for the task at hand.
The third chapter, Introduction to Wireshark, introduces the book's main tool, with installation instructions for the more common operating systems and an initial walkthrough of the graphical user interface. While the install instructions are not correct in all details (when installing from source, you do not actually need elevated privileges until you get to the concluding make install step), they should be sufficient to get most readers started with minimal fuss.
The second chapter, Tapping into the Wire focuses mainly on how and where to position your network tap, introduces a couple of different hardware devices for the purpose and even touches on ARP spoofing as a tap technique before giving a first glimpse of how to display and interpret captured traffic using the Windows utility Cain & Abel, and ends with a summary that includes a flowchart to help decide on the most useful tapping technique for the task at hand.
The third chapter, Introduction to Wireshark, introduces the book's main tool, with installation instructions for the more common operating systems and an initial walkthrough of the graphical user interface. While the install instructions are not correct in all details (when installing from source, you do not actually need elevated privileges until you get to the concluding make install step), they should be sufficient to get most readers started with minimal fuss.
Finishing up the chapter with a section that moves quickly to Your First Packet Capture is a nice touch that emphasizes the practical approach that's typical of this book.
Chapter 4, Working with Captured Packets, walks the user through some highlights of the filtering, analysis and presentation features that makes working with the likes of Wireshark fun. Much of this functionality would be available or at least fairly familiar to a seasoned tcpdump user, but this walkthrough does illustrate that sometimes a graphical interface can be fun too.
Chapter 4, Working with Captured Packets, walks the user through some highlights of the filtering, analysis and presentation features that makes working with the likes of Wireshark fun. Much of this functionality would be available or at least fairly familiar to a seasoned tcpdump user, but this walkthrough does illustrate that sometimes a graphical interface can be fun too.
The chapter also leads off with fairly weakly worded advice that packet capture and analysis are likely to be separate activities. Considering that Chapter 3 introduced Wireshark's ability to choose interfaces by point and click in a dialog box (indicating that the program runs with elevated privileges), I for one would have found a stronger admonition very helpful.
An inexperienced reader will likely want to view the packet capture of her own network traffic animated and in full color, so at this point or earlier it would have been useful to include a note that on Unixish operating systems, either of these three commands
$ sudo tshark -w - | wireshark -k -i -
$ sudo dumpcap -w - | wireshark -k -i -
$ sudo tcpdump -e -s 65535 -i <interfacename> -w - | wireshark -k -i -
will at least buy token security in that only the packet capture runs with elevated privileges, the analysis tools run as your regular user (and yes, <interfacename> is where the actual interface name goes).
Fortunately, for the rest of the book, the activities are firmly centered around the packet capture files collected by the author himself. Chapters 5 through 11 present a variety of specific traffic scenarios that showcase the analysis and presentation features (including various graphing options) that make Wireshark a useful tool.
$ sudo tshark -w - | wireshark -k -i -
$ sudo dumpcap -w - | wireshark -k -i -
$ sudo tcpdump -e -s 65535 -i <interfacename> -w - | wireshark -k -i -
will at least buy token security in that only the packet capture runs with elevated privileges, the analysis tools run as your regular user (and yes, <interfacename> is where the actual interface name goes).
Fortunately, for the rest of the book, the activities are firmly centered around the packet capture files collected by the author himself. Chapters 5 through 11 present a variety of specific traffic scenarios that showcase the analysis and presentation features (including various graphing options) that make Wireshark a useful tool.
There are several memorable moments to be found here, including a packet capture that demonstrates how a successful compromise of a Microsoft Windows system (getting a privileged command shell) could look like at the network level. There is a wide variety of examples, and most of them are clearly designed to nudge the reader into exploring further. A good selection of protocols and protocol features are explained with a learning by doing approach, but there is enough that's only hinted at to let an interested reader look up the Further Reading appendix to dive into the rest.
All in all Practical Packet Analysis, second edition stands out as a book that's a very useful learning resource, and one that makes the learning process a lot of fun. Seasoned network professionals will most likely not find much new material here, but the book is a good read for anyone with a networking interest and I'm pretty sure you'll enjoy the hours you spend leafing through it before you hand it over to your junior network admin or your students.
Title: Practical Packet Analysis, 2nd Edition - Using Wireshark to Solve Real-World Network Problems
Author: Chris Sanders
Publisher: No Starch Press, San Francisco
Published: July 2011
Pages: 280
ISBN: 978-1-59327-266-1
Price: USD 49.95 for print + ebook, USD 39.95 for ebook (PDF, Mobi, and ePub formats)
All in all Practical Packet Analysis, second edition stands out as a book that's a very useful learning resource, and one that makes the learning process a lot of fun. Seasoned network professionals will most likely not find much new material here, but the book is a good read for anyone with a networking interest and I'm pretty sure you'll enjoy the hours you spend leafing through it before you hand it over to your junior network admin or your students.
Title: Practical Packet Analysis, 2nd Edition - Using Wireshark to Solve Real-World Network Problems
Author: Chris Sanders
Publisher: No Starch Press, San Francisco
Published: July 2011
Pages: 280
ISBN: 978-1-59327-266-1
Price: USD 49.95 for print + ebook, USD 39.95 for ebook (PDF, Mobi, and ePub formats)
Tuesday, July 19, 2011
What to expect in OpenBSD 5.0 onwards
With OpenBSD-current tagged as 5.0-beta it's time to take a closer look at the upcoming release and the processes that make the OpenBSD project work.
Before you start getting all worked up about an upcoming dot-zero release, I'll tell you right away: Don't. Or maybe just a little bit. Release numbering in OpenBSD simply does not work the way most people expect. For this upcoming release, slated for general availability via FTP and other download methods on November 1, 2011 (and CD in pre-orderers' hands up to about two weeks earlier), 5.0 was simply the next available version number increment. OpenBSD releases every six months like clockwork, with the version number incremented by exactly 0.1 each time.
That does not mean that there is nothing to be excited about this time around, only that the OpenBSD approach is about guided and well planned evolution rather than revolutionary changes where large chunks of code are thrown away and replaced with new, untested code with bugs to be explored and exploited until a future dot-something-else release is finally considered stable.
The snapshots, as always available from your friendly local mirror, turned 5.0-beta during the early hours of today (July 19th 2011) CEST, and upgraders will notice two significant improvements before they're done running the installer (which in most other respects is fairly identical to what I described in an earlier article).
The most visible set of changes are near the end of the process:
As the illustration shows, once the install sets are in place, the upgrade program will offer to run sysmerge(8), the program that's specifically designed to help you merge any required changes into your configuration files with minimum fuss and disruption. Once you have chosen to either run the merge or skip it, the upgrade program will offer to fetch updated versions of any non-free firmware detected on your system at first boot.
When that first boot is finished, you will find a number of improvements are in place. You can find an overview of the changes at on the Daily changelog page over at the OpenBSD web site. We'll get into some of the more significant ones in this article.
But before we go into details of those changes, let's take a look at how the OpenBSD development process works. The Goals page on the project web site lays out the main project goals, while the Security page goes into somewhat more detail about the OpenBSD approach to security (including code audits and general all-tree bug sqashing), with a very worthwhile Further reading section at the bottom of the page. If you want to go into even more detail (short of reading source code and commit logs), the Papers page has a large selection of presentations and papers by various OpenBSD developers.
There's a lot of good material linked from that page; among the more recent favorites are Damien Miller's Recent developments in OpenSSH, Henning Brauer and Sven Dehmlow's Puffy At Work - getting code right and secure, the OpenBSD way and Theo de Raadt's The OpenBSD release process: A success story.
The last one is Theo's own first-hand account of how the project has been able to consistently deliver high quality releases every six months since the project started more than a decade ago. It's well worth your time flipping through the entire presentation, but for an outsider the possibly most enlightening slides are the ones contrasting the traditional release process with the OpenBSD one.
In particular, pay attention to the two slides describing the OpenBSD release calendar, slide 16 and slide 17. If you do the math, you see every six-month cycle is divided into roughly four months of intense development followed by roughly two months of stabilization before the release is cut.
Anything that will not fit within four months of hacking without breaking the tree (leaving the system in an unusable state) will not make it in. Any large changes need to be carefully planned and introduced via a number of substeps, some purely preparatory, others introducing user-visible changes. One visible effect of this is that truly incompatible changes such as the NAT and redirection syntax change that occured in OpenBSD 4.7 (prompting among other things an extensive rewrite of a certain book) are exceedingly rare, and if you're capable of reading the commit logs, you will see that it took several years of preparation before the switch happened.
It's all about clear thinking and proper planning, and changes will be introduced gradually and as they fit into the overall system. With all this as the background, it makes sense that 5.0 is just another number. Now let's take a peek at what changed since OpenBSD 4.9 and why it makes sense for both enthusiasts and others to start testing snapshots.
The changes listed here are my particular favorites, what you yourself will consider important may be different, depending on your specific use case:
BIGMEM on by default on amd64 - this is literally a big one. The amd64 architecture has true 64-bit capability, but the ghosts of hardware designs past keep hauting us, with legacy devices that continue to require a lot of dark magic to be handled correctly and safely. Excpect Ariane van der Steldt (ariane@) to be presenting at EuroBSDCon with the gory details. This commit happened fairly early in the OpenBSD 5.0 cycle.
Disk UID (DUID) support in all storage related parts such as mount, and by extension fstab (and DUID-style fstab enabled by default for new installs), so disk device renumbering will not be such a headache the next time you add or remove disks.
The proxies (ftp-proxy(8), tftp-proxy(8)) now use divert(4) sockets for performance, meaning that your rdr-to rules for those proxies need to be rewritten to divert-to rules.
Rewrite of the old /etc/security (a shell script) to the new security(8) by Andrew Fresh and Ingo Schwartze, a much needed refresh of a crucial component.
Various IPv6-related improvements in PF and other parts of the networking code, meaning that traffic using the newer generation protocol is now a bit more manageable. Just like everywhere else, the IPv6-related work is still in the early stages of seeing full scale use, and more likely than not future releases will have more news in this area.
The beginning of the end for ALTQ signalled by the introduction of always-on priority queues available as PF per-rule options, as noted in a previous article. This is an example of the longer term, incremental and well planned change like the ones I mentioned earlier. The internals of queueing and traffic shaping in OpenBSD is about to change in the long run, an it's even conceivable that the current ALTQ grammar will at one point use the newer code, but existing ALTQ setups will continue to work in OpenBSD 5.0.
Further package system improvements Marc Espie (espie@) continues his refinement of the package handling with an overhauled pkg_delete(8) that has several important improvements, including the -a option (click the link to read the man page online).
The Daily changelog page contains a lot more changes than the ones I've highlighted here. Among the things I've mostly skipped are added support for new hardware and a host of platform-specific fixes and enhancements. And of course it's likely that the list of changes will grow visibly over the next few weeks via commits caused by you testing and user feedback.
The exact point in time when a release is cut and shipped off to production is never pre-announced. The best indicator is to look for the commit message that changes the -current version string back to N.m.-current again after a brief period as N.m. A little later, the OpenBSD Orders page will allow preorders, and if you get your order in soon enough, you'll have your CDs and other swag before the official release date.
In about six months' time, you will see blog posts and other news items announcing the change to OpenBSD 5.1-beta, and we will be gearing up for yet another OpenBSD release. In any case, the best way to support the project (that produces, among other widely used software, OpenSSH, more likely than not by a wide margin, your remote login system) in addition to contributing code, testing and direct donations is to go to the OpenBSD Orders page and order one or more items.
Update: The sysmerge run from upgrade feature was backed out in a last minute commit by deraadt@, but it's possible it will return in time for the 5.0 to 5.1 upgrade cycle.
Update 2011-09-09: Pre-orders have started, the 5.0 release now has both a release page and a detailed list of changes since previous release. Expect more details to emerge over the coming days and weeks, and please do go order some items to help fund further OpenBSD development!
Before you start getting all worked up about an upcoming dot-zero release, I'll tell you right away: Don't. Or maybe just a little bit. Release numbering in OpenBSD simply does not work the way most people expect. For this upcoming release, slated for general availability via FTP and other download methods on November 1, 2011 (and CD in pre-orderers' hands up to about two weeks earlier), 5.0 was simply the next available version number increment. OpenBSD releases every six months like clockwork, with the version number incremented by exactly 0.1 each time.
That does not mean that there is nothing to be excited about this time around, only that the OpenBSD approach is about guided and well planned evolution rather than revolutionary changes where large chunks of code are thrown away and replaced with new, untested code with bugs to be explored and exploited until a future dot-something-else release is finally considered stable.
The snapshots, as always available from your friendly local mirror, turned 5.0-beta during the early hours of today (July 19th 2011) CEST, and upgraders will notice two significant improvements before they're done running the installer (which in most other respects is fairly identical to what I described in an earlier article).
The most visible set of changes are near the end of the process:
As the illustration shows, once the install sets are in place, the upgrade program will offer to run sysmerge(8), the program that's specifically designed to help you merge any required changes into your configuration files with minimum fuss and disruption. Once you have chosen to either run the merge or skip it, the upgrade program will offer to fetch updated versions of any non-free firmware detected on your system at first boot.
When that first boot is finished, you will find a number of improvements are in place. You can find an overview of the changes at on the Daily changelog page over at the OpenBSD web site. We'll get into some of the more significant ones in this article.
But before we go into details of those changes, let's take a look at how the OpenBSD development process works. The Goals page on the project web site lays out the main project goals, while the Security page goes into somewhat more detail about the OpenBSD approach to security (including code audits and general all-tree bug sqashing), with a very worthwhile Further reading section at the bottom of the page. If you want to go into even more detail (short of reading source code and commit logs), the Papers page has a large selection of presentations and papers by various OpenBSD developers.
There's a lot of good material linked from that page; among the more recent favorites are Damien Miller's Recent developments in OpenSSH, Henning Brauer and Sven Dehmlow's Puffy At Work - getting code right and secure, the OpenBSD way and Theo de Raadt's The OpenBSD release process: A success story.
The last one is Theo's own first-hand account of how the project has been able to consistently deliver high quality releases every six months since the project started more than a decade ago. It's well worth your time flipping through the entire presentation, but for an outsider the possibly most enlightening slides are the ones contrasting the traditional release process with the OpenBSD one.
In particular, pay attention to the two slides describing the OpenBSD release calendar, slide 16 and slide 17. If you do the math, you see every six-month cycle is divided into roughly four months of intense development followed by roughly two months of stabilization before the release is cut.
Anything that will not fit within four months of hacking without breaking the tree (leaving the system in an unusable state) will not make it in. Any large changes need to be carefully planned and introduced via a number of substeps, some purely preparatory, others introducing user-visible changes. One visible effect of this is that truly incompatible changes such as the NAT and redirection syntax change that occured in OpenBSD 4.7 (prompting among other things an extensive rewrite of a certain book) are exceedingly rare, and if you're capable of reading the commit logs, you will see that it took several years of preparation before the switch happened.
It's all about clear thinking and proper planning, and changes will be introduced gradually and as they fit into the overall system. With all this as the background, it makes sense that 5.0 is just another number. Now let's take a peek at what changed since OpenBSD 4.9 and why it makes sense for both enthusiasts and others to start testing snapshots.
The changes listed here are my particular favorites, what you yourself will consider important may be different, depending on your specific use case:
BIGMEM on by default on amd64 - this is literally a big one. The amd64 architecture has true 64-bit capability, but the ghosts of hardware designs past keep hauting us, with legacy devices that continue to require a lot of dark magic to be handled correctly and safely. Excpect Ariane van der Steldt (ariane@) to be presenting at EuroBSDCon with the gory details. This commit happened fairly early in the OpenBSD 5.0 cycle.
Disk UID (DUID) support in all storage related parts such as mount, and by extension fstab (and DUID-style fstab enabled by default for new installs), so disk device renumbering will not be such a headache the next time you add or remove disks.
The proxies (ftp-proxy(8), tftp-proxy(8)) now use divert(4) sockets for performance, meaning that your rdr-to rules for those proxies need to be rewritten to divert-to rules.
Rewrite of the old /etc/security (a shell script) to the new security(8) by Andrew Fresh and Ingo Schwartze, a much needed refresh of a crucial component.
Various IPv6-related improvements in PF and other parts of the networking code, meaning that traffic using the newer generation protocol is now a bit more manageable. Just like everywhere else, the IPv6-related work is still in the early stages of seeing full scale use, and more likely than not future releases will have more news in this area.
The beginning of the end for ALTQ signalled by the introduction of always-on priority queues available as PF per-rule options, as noted in a previous article. This is an example of the longer term, incremental and well planned change like the ones I mentioned earlier. The internals of queueing and traffic shaping in OpenBSD is about to change in the long run, an it's even conceivable that the current ALTQ grammar will at one point use the newer code, but existing ALTQ setups will continue to work in OpenBSD 5.0.
Further package system improvements Marc Espie (espie@) continues his refinement of the package handling with an overhauled pkg_delete(8) that has several important improvements, including the -a option (click the link to read the man page online).
The Daily changelog page contains a lot more changes than the ones I've highlighted here. Among the things I've mostly skipped are added support for new hardware and a host of platform-specific fixes and enhancements. And of course it's likely that the list of changes will grow visibly over the next few weeks via commits caused by you testing and user feedback.
The exact point in time when a release is cut and shipped off to production is never pre-announced. The best indicator is to look for the commit message that changes the -current version string back to N.m.-current again after a brief period as N.m. A little later, the OpenBSD Orders page will allow preorders, and if you get your order in soon enough, you'll have your CDs and other swag before the official release date.
In about six months' time, you will see blog posts and other news items announcing the change to OpenBSD 5.1-beta, and we will be gearing up for yet another OpenBSD release. In any case, the best way to support the project (that produces, among other widely used software, OpenSSH, more likely than not by a wide margin, your remote login system) in addition to contributing code, testing and direct donations is to go to the OpenBSD Orders page and order one or more items.
Update: The sysmerge run from upgrade feature was backed out in a last minute commit by deraadt@, but it's possible it will return in time for the 5.0 to 5.1 upgrade cycle.
Update 2011-09-09: Pre-orders have started, the 5.0 release now has both a release page and a detailed list of changes since previous release. Expect more details to emerge over the coming days and weeks, and please do go order some items to help fund further OpenBSD development!
Saturday, July 9, 2011
Anticipating the Post-ALTQ World
A peek into exciting new features in the upcoming OpenBSD 5.0 release and beyond, plus how to teach our favorite operating system better?
One of the more exciting pieces of news to come out of the Edmonton OpenBSD hackathon this week was that the face of OpenBSD traffic shaping is about to change.
This change is by no means the only news to come out of Edmonton (read source-changes@, either by subscribing or via public archives such as marc.info and, if you're at all BSD geekish, feel your excitement levels rise), but the new prio keyword that was added to PF grammar and announced via a message to the tech@ mailing list titled new small, fast, always on priority queueing is both a major new feature and a prime example of how the OpenBSD project works, avoiding sudden "revolutionary" steps in favor of well planned evolutionary steps. The stepwise approach ensures, among other things, that changes are properly tested, and is visible in the fact that the OpenBSD source tree is as close as doesn't matter to always being in a buildable state, even during hackathons.
ALTQ, the ALTernate Queueing subsystem, was integrated into the PF codebase for OpenBSD version 3.3 (mainly by the efforts of the very same Henning Brauer who posted the message linked in the previous paragraph), and from that release onward, managing your filtering and your traffic shaping via your pf.conf file has been a highly appreciated feature of OpenBSD and other systems that have adopted the PF networking toolset.
Although much appreciated by its users everywhere, the integration was never quite an easy fit and the developers have been privately discussing how the queuing should be rewritten to avoid the separateness of ALTQ compared to the rest of the PF features.
There have been other concerns too. Not all of the features outlined in Cho's original USENIX paper were ever fully implemented, and the code had begun showing its age in several respects. By amazing coincidence another mailing list thread appeared in the same week as the new queueing system was announced, pointing out that ALTQ's total bandwidth parameter is a 32-bit value, meaning in practical terms that the maximum bandwidth ALTQ will be able to manage is in the 4Gbit range. Just changing the relevant variables to a wider type apparently breaks the code in other interesting ways (if you're adventurous, you could try playing with FreeBSD developer Ermal Luci's patch (available among other places here) and see what happens). The results could be hackishly interesting, but with the new queueing system set for gradual introduction, this may not be the optimal time to submit ALTQ patches.
Once again, the 32 bit value is a sign of the code's age. At the time ALTQ was originally written, a 32-bit value for bandwidth, and by extension an absolute cap at four gigabits per second, was a no less reasonable choice than the choice of 32 bits as the length of IP addresses some years earlier. And while we are still in the ALTQ world, the obvious workaround in settings where you have bandwidth that comes in increments of ten or forty gigabit is to do your filtering and traffic shaping at network locations where bandwidth is a bit more scarce. In the post-ALTQ world, things may become significantly different.
The early outline of the new world comes in the form of in-ruleset traffic classification, at the face of it not that different from ALTQ when seen from a user perspective. In the upcoming OpenBSD 5.0 release onwards, you will be able to skip the familiar ALTQ queue definitions, but still do traffic classification like this:
pass in proto tcp to port ssh prio 6
meaning that incoming ssh traffic will pass with a relatively high priority (the range is 0 through 7), hopefully making it through earlier than the rest.
The new scheme even lets you duplicate the old speedup trick of setting aside a separate subqueue for ACKs and other lowdelay packets, like so:
pass in proto tcp to port ssh prio (5, 6)
This has yet to reach downloadable binary snapshots, but if you have the required bits in place, you can get an early start by building your own OpenBSD-current. See the relevant pieces of The FAQ, supplemented by man release.
It is important to note that in the upcoming release, priority rules will coexist with the existing ALTQ infrastructure. Over time, however, the plan is to replace ALTQ with a largely rewritten system where the internal workings of the HFSC and CBQ queues, for example, are not all that different. We are also likely to see the total bandwidth definition for an interface move out ouf your pf.conf and re-emerge as an ifconfig option.
Update 2011-07-10: The prio code is in snapshots dated July 10, 2011 or newer. If you're upgrading from a previous version, the installer will now also offer to run sysmerge(8) and it will even offer to upgrade any non-free firmware it finds installed on your system.
Update 2011-09-14: Chris Cappucio wrote in, adding some more useful detail to ALTQ's history. Chris writes:
One of the more exciting pieces of news to come out of the Edmonton OpenBSD hackathon this week was that the face of OpenBSD traffic shaping is about to change.
This change is by no means the only news to come out of Edmonton (read source-changes@, either by subscribing or via public archives such as marc.info and, if you're at all BSD geekish, feel your excitement levels rise), but the new prio keyword that was added to PF grammar and announced via a message to the tech@ mailing list titled new small, fast, always on priority queueing is both a major new feature and a prime example of how the OpenBSD project works, avoiding sudden "revolutionary" steps in favor of well planned evolutionary steps. The stepwise approach ensures, among other things, that changes are properly tested, and is visible in the fact that the OpenBSD source tree is as close as doesn't matter to always being in a buildable state, even during hackathons.
ALTQ, the ALTernate Queueing subsystem, was integrated into the PF codebase for OpenBSD version 3.3 (mainly by the efforts of the very same Henning Brauer who posted the message linked in the previous paragraph), and from that release onward, managing your filtering and your traffic shaping via your pf.conf file has been a highly appreciated feature of OpenBSD and other systems that have adopted the PF networking toolset.
Although much appreciated by its users everywhere, the integration was never quite an easy fit and the developers have been privately discussing how the queuing should be rewritten to avoid the separateness of ALTQ compared to the rest of the PF features.
There have been other concerns too. Not all of the features outlined in Cho's original USENIX paper were ever fully implemented, and the code had begun showing its age in several respects. By amazing coincidence another mailing list thread appeared in the same week as the new queueing system was announced, pointing out that ALTQ's total bandwidth parameter is a 32-bit value, meaning in practical terms that the maximum bandwidth ALTQ will be able to manage is in the 4Gbit range. Just changing the relevant variables to a wider type apparently breaks the code in other interesting ways (if you're adventurous, you could try playing with FreeBSD developer Ermal Luci's patch (available among other places here) and see what happens). The results could be hackishly interesting, but with the new queueing system set for gradual introduction, this may not be the optimal time to submit ALTQ patches.
Once again, the 32 bit value is a sign of the code's age. At the time ALTQ was originally written, a 32-bit value for bandwidth, and by extension an absolute cap at four gigabits per second, was a no less reasonable choice than the choice of 32 bits as the length of IP addresses some years earlier. And while we are still in the ALTQ world, the obvious workaround in settings where you have bandwidth that comes in increments of ten or forty gigabit is to do your filtering and traffic shaping at network locations where bandwidth is a bit more scarce. In the post-ALTQ world, things may become significantly different.
The early outline of the new world comes in the form of in-ruleset traffic classification, at the face of it not that different from ALTQ when seen from a user perspective. In the upcoming OpenBSD 5.0 release onwards, you will be able to skip the familiar ALTQ queue definitions, but still do traffic classification like this:
pass in proto tcp to port ssh prio 6
meaning that incoming ssh traffic will pass with a relatively high priority (the range is 0 through 7), hopefully making it through earlier than the rest.
The new scheme even lets you duplicate the old speedup trick of setting aside a separate subqueue for ACKs and other lowdelay packets, like so:
pass in proto tcp to port ssh prio (5, 6)
This has yet to reach downloadable binary snapshots, but if you have the required bits in place, you can get an early start by building your own OpenBSD-current. See the relevant pieces of The FAQ, supplemented by man release.
It is important to note that in the upcoming release, priority rules will coexist with the existing ALTQ infrastructure. Over time, however, the plan is to replace ALTQ with a largely rewritten system where the internal workings of the HFSC and CBQ queues, for example, are not all that different. We are also likely to see the total bandwidth definition for an interface move out ouf your pf.conf and re-emerge as an ifconfig option.
Once a traffic shaping infrastructure that's functionally equivalent to ALTQ (or it is hoped, superior in all ways including performance) is in place, ALTQ will eventually be removed. In the meantime, we have already seen commits that set default priority for certain traffic types, and reading source-changes@ along with testing snapshots will continue to be a fun and exciting experience in the weeks and months ahead.
Over time, it might even become necessary to do more book work, if so you will hear about it first here.
Whatever happens on the book front, I'll more than likely get back to those new features in more detail in upcoming PF tutorial sessions (See here, here and here for announcements about the ones scheduled so far), and I'm trying to figure out a way to improve the quality of the tutorial experiences.
Over time, it might even become necessary to do more book work, if so you will hear about it first here.
Whatever happens on the book front, I'll more than likely get back to those new features in more detail in upcoming PF tutorial sessions (See here, here and here for announcements about the ones scheduled so far), and I'm trying to figure out a way to improve the quality of the tutorial experiences.
The lecture plus demonstrations format of the tutorial sessions has so far allowed for very limited interactivity with little or no hands-on experience for attendees during the sessions themselves. I would be very happy to hear ideas for and input on practical implementation of changes that would improve the experience. One possibility is setting up a set of virtual labs, hosted somewhere reachable from the conference locations. Suggestions are welcome via email or the comments field.
Update 2011-07-10: The prio code is in snapshots dated July 10, 2011 or newer. If you're upgrading from a previous version, the installer will now also offer to run sysmerge(8) and it will even offer to upgrade any non-free firmware it finds installed on your system.
Update 2011-09-14: Chris Cappucio wrote in, adding some more useful detail to ALTQ's history. Chris writes:
I did the initial altq port to OpenBSD and Kenjiro Cho (author of ALTQ) was invited to the next hackathon (I believe it was c2k2) along with myself to integrate it into the tree. I declined to participate for personal reasons but Kenjiro did turn my patch into an in-tree implementation, one or two years later the PF guys replaced the ALTQ packet classifier and configuration utility with PF itself. Now they are replacing the ALTQ scheduler framework completely with a simpler framework that still uses the PF classifier and configurator :)Update 2015-04-02: The new queueing subsystem finally appeared in OpenBSD 5.5, and was the main motivation for revising The Book of PF into its 2014 third edition. If you follow the book links in this article, you will find the edition that describes both the new queueing system and the legacy ALTQ system.
Friday, July 8, 2011
SEK 1995 for six months' worth of trademark protection?
In a fit of rage, I went out and did something I wouldn't have even remotely considered doing just a few moments before. I'm now the proud owner of the bsdly.se domain.
Does somebody out there really want to register bsdly.se? If they did, it's probably too late by now.
My friends all know that I'm not too fond of talking on the phone, and trying to sell me anything by cold-calling is just never going to work. So when somebody calling themselves "Nordic Domain Hosting", calling from +46 406660225 (a Swedish unlisted number as far as I can tell) did just that, it had to end badly.
The lady at the other end claimed that somebody was trying to register the bsdly.se domain, and seeing that I was the owner of doubleU-doubleU-doubleU-dot-bee-ess-dee-ell-why-dot-enn-ee-tee, would I be interested in blocking the attempt? It would just be a matter of 'trademark protection', and it would cost me the mere trifle of SEK 1995 (roughly USD 310 or EUR 218 at today's rate).
I told them right off that I wasn't terribly interested in owning a Swedish domain, but the lady they tried to talk me through a series of yes/no answers I assume were designed to set up a legal-sounding agreement, all the better to bill me afterwards. More likely than not, she was reading it all off the whois info for bsdly.net.
They had the act pretty well rehearsed, except for one detail that irritated me immensely: they insisted on getting an oral confirmation that I was interested in their service. Only after an oral confirmation was in place would they be sending me anything in writing.
The conversation never progressed much beyond beyond the initial "Is your name Peter [...]", "Are you duly authorized to act on $company_name's behalf", which is where I broke it off. For one thing the connection was terrible, and for another I was beginning to smell a rat. (Yes, I'm dense at times, I know.) At the end of too-many minutes, I thought I had finally got them to agree to send me their papers for me to review and hung up. Only to have the same lady call once more, continuing the script.
The second conversation did not progress much either, and for the third call a male claiming to be a supervisor had taken over. He didn't actually clinch a sale either.
So after three phone conversations, I used my regular registrar's web interface to register the bsdly.se domain myself, setting me back a whopping NOK 140 (roughly EUR 18 or USD 26). While I was writing this note, the "Registration complete" message from my registrar arrived here in my inbox.
So now I have another domain. I've expanded into Sweden.
Or as they say in the trade, get a geek really riled up, he'll go right off and buy a domain. In Sweden.
Does somebody out there really want to register bsdly.se? If they did, it's probably too late by now.
My friends all know that I'm not too fond of talking on the phone, and trying to sell me anything by cold-calling is just never going to work. So when somebody calling themselves "Nordic Domain Hosting", calling from +46 406660225 (a Swedish unlisted number as far as I can tell) did just that, it had to end badly.
The lady at the other end claimed that somebody was trying to register the bsdly.se domain, and seeing that I was the owner of doubleU-doubleU-doubleU-dot-bee-ess-dee-ell-why-dot-enn-ee-tee, would I be interested in blocking the attempt? It would just be a matter of 'trademark protection', and it would cost me the mere trifle of SEK 1995 (roughly USD 310 or EUR 218 at today's rate).
I told them right off that I wasn't terribly interested in owning a Swedish domain, but the lady they tried to talk me through a series of yes/no answers I assume were designed to set up a legal-sounding agreement, all the better to bill me afterwards. More likely than not, she was reading it all off the whois info for bsdly.net.
They had the act pretty well rehearsed, except for one detail that irritated me immensely: they insisted on getting an oral confirmation that I was interested in their service. Only after an oral confirmation was in place would they be sending me anything in writing.
The conversation never progressed much beyond beyond the initial "Is your name Peter [...]", "Are you duly authorized to act on $company_name's behalf", which is where I broke it off. For one thing the connection was terrible, and for another I was beginning to smell a rat. (Yes, I'm dense at times, I know.) At the end of too-many minutes, I thought I had finally got them to agree to send me their papers for me to review and hung up. Only to have the same lady call once more, continuing the script.
The second conversation did not progress much either, and for the third call a male claiming to be a supervisor had taken over. He didn't actually clinch a sale either.
So after three phone conversations, I used my regular registrar's web interface to register the bsdly.se domain myself, setting me back a whopping NOK 140 (roughly EUR 18 or USD 26). While I was writing this note, the "Registration complete" message from my registrar arrived here in my inbox.
So now I have another domain. I've expanded into Sweden.
Or as they say in the trade, get a geek really riled up, he'll go right off and buy a domain. In Sweden.
Wednesday, June 8, 2011
My First IPv6 Spam
On the day of The Great Experiment, an anecdote on how much the world stays the same even with IPv6, security-wise. No reason to stay all dotty, this is when the fun starts.
Happy IPv6 Day, everyone! As I write this column, the 24 hour worldwide Internet Protocol, version Six preparedness experiment is still in progress, with some hours to go before the summaries, no doubt penned by industry luminaries, will start appearing. In the meantime, I have a small IPv6 anecdote of my own to share.
Note: This piece is also available without trackers but classic formatting only here.
The full message is preserved here with headers or as mainly message body as text if you're so inclined.
From the most recent Received: header we see that the delivery happened over IPv6, while the second tells us that the originator most likely did not have IPv6 connectivity, or at least did not see fit to use it if they did.
It's also worth noting that this message, like a surprising amount of spam in recent times, also comes with what appears to be a valid DKIM signature. So much for cryptograpic validation as an effective antispam measure.
Now of course my next step was to find out if this particular sender had tried to darken my mail spool before:
peter@skapet:~$ sudo grep edrenat@edrenat.es /var/spool/exim/logs/main.log 2010-11-07 14:21:44 1PF5Bm-0006RI-KB <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=251712 id=201011071353082974379@edrenat.es 2010-12-14 20:42:49 1PSaln-000255-PQ <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=759434 id=201012141954165325931@edrenat.es 2011-01-30 21:13:30 1PjdeA-0001Nn-If <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.161] P=esmtp S=1905781 id=201101302042161777948@edrenat.es
2011-02-22 21:59:05 1PrzIX-0006jv-7x <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=1227270 id=201102222041571040455@edrenat.es
2011-04-28 12:40:26 1QFOdo-0001zq-E9 <= edrenat@edrenat.es H=mo-p00-ob6.rzone.de [2a01:238:20a:202:53f0::1] P=esmtp S=2210478 id=201104281225569454004@edrenat.es
Apparently they had, on more or less a monthly basis, so let's take a peek at the log entries for all of those messages:
peter@skapet:~$ for foo in 1PF5Bm-0006RI-KB 1PSaln-000255-PQ 1PjdeA-0001Nn-If 1PrzIX-0006jv-7x 1QFOdo-0001zq-E9; do sudo grep $foo /var/spool/exim/logs/main.log ; done
2010-11-07 14:21:43 1PF5Bm-0006RI-KB DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1289136101 l=251099 [verification failed - signature did not verify (headers probably modified in transit)]
2010-11-07 14:21:44 1PF5Bm-0006RI-KB <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=251712 id=201011071353082974379@edrenat.es
2010-11-07 14:21:44 1PF5Bm-0006RI-KB => peter <bsdly@bsdly.net> R=localuser T=local_delivery 2010-11-07 14:21:44 1PF5Bm-0006RI-KB Completed 2010-12-14 20:42:46 1PSaln-000255-PQ DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1292355762 l=766689 [verification succeeded]
2010-12-14 20:42:49 1PSaln-000255-PQ <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=759434 id=201012141954165325931@edrenat.es
2010-12-14 20:42:49 1PSaln-000255-PQ => peter <bsdly@bsdly.net> R=localuser T=local_delivery 2010-12-14 20:42:49 1PSaln-000255-PQ Completed
2011-01-30 21:13:26 1PjdeA-0001Nn-If DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1296418396 l=1927465 [verification succeeded]
2011-01-30 21:13:30 1PjdeA-0001Nn-If <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.161] P=esmtp S=1905781 id=201101302042161777948@edrenat.es
2011-01-30 21:13:30 1PjdeA-0001Nn-If => peter <bsdly@bsdly.net> R=localuser T=local_delivery 2011-01-30 21:13:30 1PjdeA-0001Nn-If Completed
2011-02-22 21:59:03 1PrzIX-0006jv-7x DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1298408169 l=1240300 [verification succeeded]
2011-02-22 21:59:05 1PrzIX-0006jv-7x <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=1227270 id=201102222041571040455@edrenat.es
2011-02-22 21:59:06 1PrzIX-0006jv-7x => peter <bsdly@bsdly.net> R=localuser T=local_delivery 2011-02-22 21:59:06 1PrzIX-0006jv-7x Completed
2011-04-28 12:40:21 1QFOdo-0001zq-E9 DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1303987210 l=2236144 [verification succeeded]
2011-04-28 12:40:26 1QFOdo-0001zq-E9 <= edrenat@edrenat.es H=mo-p00-ob6.rzone.de [2a01:238:20a:202:53f0::1] P=esmtp S=2210478 id=201104281225569454004@edrenat.es
2011-04-28 12:40:26 1QFOdo-0001zq-E9 => peter <bsdly@bsdly.net> R=localuser T=local_delivery
2011-04-28 12:40:26 1QFOdo-0001zq-E9 Completed
This tells us that they've consistently sent to an address that features only on my website and has never been used as a sender or return address and has never been signed up for any mailing list of any kind. We also see that all the messages bar one pass the initial DKIM validity test.
There are several lessons to be learned here. One is that with a sensible choice of operating system and other software, the transition from classic IP version four only to a dual-stack configuration with both IP version four and IP version six running on your systems will likely be less dramatic than you have been lead to believe.
Another is that at least in some respect the world stays very much the same with IPv6 as it was with IPv4, and content is likely to cross the protocol border. Or to put it slightly differently, the same caveats about ill-intentioned traffic still apply; any security measures you had inplace before you went dual stack most likely need to be tuned to handle traffic from the brave new world of IPv6.
And of course, even if a number of implementation bugs have been found and fixed and a number of fundamental design flaws have been identified and worked around, only full scale testing like today's experiment (preferably sustained over longer periods) offer any hope of identifying and fixing the problems we haven't found yet. The looming IPv6 transition is likely to make and break careers, and some of us will have our share of both fun and nightmares while seeing to the confidentiality, integrity and general security of your data and your systems.
Good night and good luck, while we're slowly going from dots to colons.
Thanks to Sevan Janiyan for tweeting "Though the gmail website is reachable via IPv6, mail is still going via IPv4 :(" earlier today and reminding me of the incident that was the inspiration for this column.
Happy IPv6 Day, everyone! As I write this column, the 24 hour worldwide Internet Protocol, version Six preparedness experiment is still in progress, with some hours to go before the summaries, no doubt penned by industry luminaries, will start appearing. In the meantime, I have a small IPv6 anecdote of my own to share.
Note: This piece is also available without trackers but classic formatting only here.
Like most network oriented techies, I've had IPv6 somewhere in my field of vision for some time, with a footnoted TODO list to match. I started nagging local ISPs for their IPv6 plans some years ago, and as you've probably guessed, up until very recently the answer at essentially all European ISPs has been 'non-existent'. Among European nations, most of them pretty early in the queue when the original IPv4 address allocations were made, Norway was quite fortunate, and with a total population of less than five million and enough NAT to go around we have a good enough IPv4 address to total population ratio that makes for no panic of us running out of IPv4 locally.
So this left us early adopters to seek out the next generation connectivity via tunnel providers such as Hurricane Electric or Sixxs, with the dancing turtle at the Kame Project website as the early reward for venturing into 128-bit address territory.
For the longest while, that would be pretty much it. Once you'd kept your tunnel stable for a while, you could generally have a /48 subnet for the asking. But finding any good content on IPv6 or large amounts of IPv6 traffic of any kind was not easy. At parties we'd even bitch about the famous lines in one IPv6 textbook that claimed that IPv6 routing tables were generally much smaller and more compact than their IPv4 counterparts -- of course they're smaller, there are essentially no sites, and they produce next to no traffic!
Then, famously, the time came when the last /8 range allocations of IPv4 address space were made, and the IPv4 internet was officially full (should we just start telling them to go away yet?), for some values of at least. Even if certain European organizations or nations are not in any danger of running out of IP addresses, sixteen years after the initial IPv6 RFCs entered the standards track (see RFC1883 and friends), we found ourselves in a situation where any new large-scale implementations would likely be built exclusively with IPv6 or very nearly so, and the historical backdrop for today's experiment was complete.
I expect the summaries to say mostly that modern operating systems, at least the free ones such as OpenBSD came very well prepared for the task. The two protocols are in fact not compatible, but running both will work, and for the vast majority of services, the sysadmin's worksheet looks like this:
1) Get hold of whatever number of IPv6 addresses your application requires
2) Configure your network interfaces with IPv6 addresses (some systems come preconfigured to do their own autoconfiguration)
3) Edit in the required IPv6 addresses into your services' configurations
4) Add any AAAA records required to your domains' externally-visible zones.
And you're done, at least for now. With quad-A entries in your externally-visible DNS, you will start seeing IPv6 traffic hitting your services from elsewhere, not just your own test traffic.
Your logs and other monitoring will show you how your rig behaves. Most likely it all just works, and your users won't notice anything different, except perhaps a dancing turtle (or missing ads on IPv6 for some web sites, as noted by some early IP version six day reports).
I did those things a little while back on my home network, and when the first few days did not show up anything I almost missed what could be an important event: my first spam email delivered over IPv6. I only read message headers when the message stands out somehow, and in this case it was clearly spam, so I took a peek at the headers in case I it would be worth blacklisting manually:
Received: from mo-p00-ob6.rzone.de ([2a01:238:20a:202:53f0::1]) by skapet.bsdly.net with esmtp (Exim 4.73) (envelope-from) id 1QFOdo-0001zq-E9 for bsdly@bsdly.net; Thu, 28 Apr 2011 12:40:25 +0200 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1303987210; l=2236144; s=domk; d=edrenat.es; h=Content-Type:Mime-Version:Subject:Reply-To:From:Date:X-RZG-CLASS-ID: X-RZG-AUTH; bh=CS67GJbTmE0wDhH/HieJqdndPSo=; b=pNIf0dK3KnFhj/7mcP8KGm70V9/XxlPa/ALQSKRh0nBgWsT0mdoUxi2unepDaMlZMPW cReoEgH/yMOF6T59Zf2X4fjrr358rU9hhxjQuV6lHwIpYoQWYwkxOqAB1iYL+SacIWspU MYMrgju1GQLZps7hIeunhV0pfto8o4JeqqM= X-RZG-AUTH: :KWgWcE6pb9/UNsdwkwZbgj6IM9/U3aYAugAbJE4rNBO+ejjApHAeOC4nD+Q= X-RZG-CLASS-ID: mo00 Received: from PACO1 (cliente-98392.iberbanda.es [87.111.193.23]) by post.strato.de (jimi mo42) (RZmta 25.17) with ESMTPA id J00252n3SABVBu ; Thu, 28 Apr 2011 12:29:29 +0200 (MEST)
The full message is preserved here with headers or as mainly message body as text if you're so inclined.
From the most recent Received: header we see that the delivery happened over IPv6, while the second tells us that the originator most likely did not have IPv6 connectivity, or at least did not see fit to use it if they did.
It's also worth noting that this message, like a surprising amount of spam in recent times, also comes with what appears to be a valid DKIM signature. So much for cryptograpic validation as an effective antispam measure.
Now of course my next step was to find out if this particular sender had tried to darken my mail spool before:
peter@skapet:~$ sudo grep edrenat@edrenat.es /var/spool/exim/logs/main.log 2010-11-07 14:21:44 1PF5Bm-0006RI-KB <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=251712 id=201011071353082974379@edrenat.es 2010-12-14 20:42:49 1PSaln-000255-PQ <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=759434 id=201012141954165325931@edrenat.es 2011-01-30 21:13:30 1PjdeA-0001Nn-If <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.161] P=esmtp S=1905781 id=201101302042161777948@edrenat.es
2011-02-22 21:59:05 1PrzIX-0006jv-7x <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=1227270 id=201102222041571040455@edrenat.es
2011-04-28 12:40:26 1QFOdo-0001zq-E9 <= edrenat@edrenat.es H=mo-p00-ob6.rzone.de [2a01:238:20a:202:53f0::1] P=esmtp S=2210478 id=201104281225569454004@edrenat.es
Apparently they had, on more or less a monthly basis, so let's take a peek at the log entries for all of those messages:
peter@skapet:~$ for foo in 1PF5Bm-0006RI-KB 1PSaln-000255-PQ 1PjdeA-0001Nn-If 1PrzIX-0006jv-7x 1QFOdo-0001zq-E9; do sudo grep $foo /var/spool/exim/logs/main.log ; done
2010-11-07 14:21:43 1PF5Bm-0006RI-KB DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1289136101 l=251099 [verification failed - signature did not verify (headers probably modified in transit)]
2010-11-07 14:21:44 1PF5Bm-0006RI-KB <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=251712 id=201011071353082974379@edrenat.es
2010-11-07 14:21:44 1PF5Bm-0006RI-KB => peter <bsdly@bsdly.net> R=localuser T=local_delivery 2010-11-07 14:21:44 1PF5Bm-0006RI-KB Completed 2010-12-14 20:42:46 1PSaln-000255-PQ DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1292355762 l=766689 [verification succeeded]
2010-12-14 20:42:49 1PSaln-000255-PQ <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=759434 id=201012141954165325931@edrenat.es
2010-12-14 20:42:49 1PSaln-000255-PQ => peter <bsdly@bsdly.net> R=localuser T=local_delivery 2010-12-14 20:42:49 1PSaln-000255-PQ Completed
2011-01-30 21:13:26 1PjdeA-0001Nn-If DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1296418396 l=1927465 [verification succeeded]
2011-01-30 21:13:30 1PjdeA-0001Nn-If <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.161] P=esmtp S=1905781 id=201101302042161777948@edrenat.es
2011-01-30 21:13:30 1PjdeA-0001Nn-If => peter <bsdly@bsdly.net> R=localuser T=local_delivery 2011-01-30 21:13:30 1PjdeA-0001Nn-If Completed
2011-02-22 21:59:03 1PrzIX-0006jv-7x DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1298408169 l=1240300 [verification succeeded]
2011-02-22 21:59:05 1PrzIX-0006jv-7x <= edrenat@edrenat.es H=mo-p00-ob.rzone.de [81.169.146.162] P=esmtp S=1227270 id=201102222041571040455@edrenat.es
2011-02-22 21:59:06 1PrzIX-0006jv-7x => peter <bsdly@bsdly.net> R=localuser T=local_delivery 2011-02-22 21:59:06 1PrzIX-0006jv-7x Completed
2011-04-28 12:40:21 1QFOdo-0001zq-E9 DKIM: d=edrenat.es s=domk c=relaxed/relaxed a=rsa-sha1 t=1303987210 l=2236144 [verification succeeded]
2011-04-28 12:40:26 1QFOdo-0001zq-E9 <= edrenat@edrenat.es H=mo-p00-ob6.rzone.de [2a01:238:20a:202:53f0::1] P=esmtp S=2210478 id=201104281225569454004@edrenat.es
2011-04-28 12:40:26 1QFOdo-0001zq-E9 => peter <bsdly@bsdly.net> R=localuser T=local_delivery
2011-04-28 12:40:26 1QFOdo-0001zq-E9 Completed
This tells us that they've consistently sent to an address that features only on my website and has never been used as a sender or return address and has never been signed up for any mailing list of any kind. We also see that all the messages bar one pass the initial DKIM validity test.
There are several lessons to be learned here. One is that with a sensible choice of operating system and other software, the transition from classic IP version four only to a dual-stack configuration with both IP version four and IP version six running on your systems will likely be less dramatic than you have been lead to believe.
Another is that at least in some respect the world stays very much the same with IPv6 as it was with IPv4, and content is likely to cross the protocol border. Or to put it slightly differently, the same caveats about ill-intentioned traffic still apply; any security measures you had inplace before you went dual stack most likely need to be tuned to handle traffic from the brave new world of IPv6.
And of course, even if a number of implementation bugs have been found and fixed and a number of fundamental design flaws have been identified and worked around, only full scale testing like today's experiment (preferably sustained over longer periods) offer any hope of identifying and fixing the problems we haven't found yet. The looming IPv6 transition is likely to make and break careers, and some of us will have our share of both fun and nightmares while seeing to the confidentiality, integrity and general security of your data and your systems.
Good night and good luck, while we're slowly going from dots to colons.
Thanks to Sevan Janiyan for tweeting "Though the gmail website is reachable via IPv6, mail is still going via IPv4 :(" earlier today and reminding me of the incident that was the inspiration for this column.
Subscribe to:
Posts (Atom)
Posts
