From mainframe replacements to firewall appliances, the BSD family of systems is a toolbox flexible enough to baffle insiders and newbies alike. EuroBSDCon 2010 was good fun.
I arrived in Karlsruhe on Thursday night, and ran into Erwin Lansing, Mark Linimon and a few other FreeBSD devsummit attendees at the perfect moment to tag along to dinner at a sort-of-greek place. Forgettable food, but fortunately the dark beer was quite drinkable and the various FreeBSDers made for good company.
Then up relatively early on the Friday. My own path through the conference started with the by now fairly familiar PF tutorial, which as you may be aware, is a close relative of The Book of PF, really soon now out in its second edition. Off the top of my head I'm not sure how many times I've given some version of the PF tutorial, but BSD-DK members will find that this edition of the slides borrows heavily from the somewhat swifter paced introduction they saw in Copenhagen this August.
Among my seven attendees were several who had hoped to be able to catch early copies of The Book of PF, second edition, which I had strongly hinted in the tutorial description would be available by now.
That did not happen, unfortunately, but it's getting very close -- last week entering final-really-final corrections to the index and the laid out book itself took up a good part of my non-office time, and the last word from my contact at No Starch was that the complete and final PDF would land in my inbox for final approval before going to the printers. Amazon.com now lists a likely delivery date of November 15th, which I for one think is a rather realistic guesstimate.
The tutorial went roughly as expected, unfortunately without live demos (demo equipment has a tendency to break badly during air transport or soon afterwards just in time muck up your presentation), and produced just enough good questions that it's likely useful to keep up the effort to maintain the tutorial. Slides are available from roughly where you would expect them.
After the session I ran into Thordur Bjornson (thib@) in the bar-cum-waiting area downstairs at the hotel, waiting for various other OpenBSD developers to arrive. This year's conference had a pleasantly larger than usual number of OpenBSD topics on the program, with some rather interesting talks scheduled for the first day. After a few beers we had reached critical mass with the arrival of among others Theo (deraadt@), Henning Brauer (henning@) and Felix Kronlage (fkr@) we found food and beer at a conveniently local eatery. Then back to the hotel bar for (slightly better) beer and mingling with arriving conference attendees and organizers.
The conference proper started on the Saturday with a "Software tools" themed keynote by Poul-Henning Kamp. PHK is always witty and fun to listen to, and he took us through a number of fresh perspectives on how, even though the world has changed dramatically in several ways and the BSDs still manage to kick ass, we need to keep up the effort to stay relevant.
FreeBSD jails have been a major attraction for quite a while, and I took in the two back to back jails talks by Bjoern A. Zeeb and james Britton. Both talks were fun refreshers on jails, with each presenting a preview of what may turn up in FreeBSD 9 jails code, plus of course tidbits like Bjoern's anectdote about setting up a million jails on the same physical server.
I was intending to attend thib@ and oga@'s OpenBSD on large memory systems talk, but was unfortunately diverted into a meeting that lead to me becoming slightly more involved in future EuroBSDCons. More details on that at a later time, the meeting ran for long enough that the next talk I did catch was reyk@'s iked(8) talk.
If you're on OpenBSD 4.8 or newer, man iked will give you the full story. If you're not that fortunate, it's nice to know that OpenBSD 4.8 gives you a new key exchange daemon for IPsec, up to date with the latest versions of all relevant protocols and able to handle all the nasty little details for IPsec communication with operating systems this column would rather not mention. A good talk about a very useful program, and during the questions part, Theo de Raadt pulled out OpenBSD 4.8 CD sets for attendees who had not already preordered to buy. It's almost a month until the official release date, but the CDs do exist and are likely on their way to early preorderers.
Henning's talk about the state of the OpenBSD networking code was good fun and to the point as always. No shocking new revelations for those who have followed the subject closely like yours truly, but do look out for this talk's slides when it hits the openbsd.org papers section along with the other EuroBSDCon 2010 presentations, hopefully soon.
The social event was conveniently placed in the hotel restaurant and bar area, where something called "Phönix Disco" had set up their equipment, including earth mover size speakers and a mirror ball hanging from the ceiling. Inbetween the dining noises and music, techie talk (at my table, OpenBSD internals, with a helping of ACPI insanities) could be heard. At some point they turned on their laser strobes, which made me queasy enough that I retired to my room soon enough to be in reasonable shape for the early Sunday morning sessions.
The 09:15 sessions on the Sunday offered a choice between Dru Lavigne's BSD Certification talk (highly recommended for your next event if you haven't taken it in already, she not only writes well, she's a brilliant presenter as well), and "Hacking NanoBSD for fun and profit" by Patrick Hausen, who has twisted the NanoBSD setup (originally intended for tiny machines) to serve as a basis for maintaining a hosting environment consisting mainly of regular-sized and capable servers. I'm already quite familiar with the bsdcertification.org efforts (and I recommend getting involved if you aren't already), so I decided to try Patrick's talk which turned out to be very enjoyable and presented some good ideas that could very well be carried over to other BSDs.
One other interesting talk that morning was Hans-Martin Rasch's "From Mainframe to FreeBSD", chronicling the gradual and successful migration by a subscriptions and mass mailings company from their legacy mainframe based system to an all-FreeBSD setup, of course shedding costs in the pretty serious range along the way.
The next time slots had a "Quo vadis ZFS" talk by Martin Matuska, that fortunately contained not the usual "see how great ZFS is" but rather focused on the challenges involved in using ZFS code, technical and legal as things stand today. The FreeBSD project seems to have concluded that the way ZFS is included in their code base does not pose legal problems in itself, but there could be other submarine issues. Apparently the NetApp vs Sun patent suit had ended with one patent invalidated and a settlement whose terms were not disclosed plus the remaining two patents under reexamination. Two unresolved patent issues would be enough to scare me off, but then again the ZFS feature set is perhaps too tempting to not take a few risks for.
Next up were espie@'s back to back talks about the *amazing* work he's been doing with OpenBSD packages. Before lunch, "the long road to pkg_add -u ... and beyond" which took us through the background and the design choices and evolution that took us to the point where upgrading your packages with pkg_add -u can be reliably expected to work. All I can say is that it was an excellent presentation about top-notch work that has made the life of OpenBSD users everywhere a lot better. The after lunch part, "efficient distributed package builds in OpenBSD" took us to the slightly more esoteric part of the world that contains the magic that makes sure the binary packages you expect to have at the other end of your pkg_add command actually exist. Another very enlightening talk, and this set of talks was certainly my favorite at this conference.
For further information on the topics mentioned in this column, the EuroBSDCon web site is the natural place to start. The OpenBSD developers' presentations will appear in the papers section of the OpenBSD web site, while my slides, as usual are available from the NUUG site.
Update 2015-04-02: This article refers to the second edition of The Book of PF. The third edition of that title (linked in the previous sentence) became available, with significant updates, in late 2014 and is overall a better resource for learning about networking and PF on all PF-capable systems.
Sunday, October 10, 2010
Sunday, January 3, 2010
The Goodness of Men and Machinery
If you keep them to their promises, what will corporations and individuals do? Plus, the general goodness of OpenBSD and its installer.
Happy new year, everyone!
As I write this, 2010 is still quite new, and I'm writing this on a new laptop, running, of course, OpenBSD. I've been mostly quite happy with the ThinkPad R60 that has served me since late 2006, but a while back the fan started giving off ugly noises and the organization nominally in charge of doing Lenovo repairs in my area seemed quite uninterested in actually doing repairs on the unit.
So after the usual browsing of web shops I ended up going for a simpler ThinkPad, the SL500 model. From the looks of it, the machine would be a tad faster and support more physical memory than the one I had already, and most likely the newer Intel processor would support running in true 64-bit (amd64) mode.
Click the buttons, whip out the credit card and away we go. Of course, the next day the repair shop turned called in to say that they could in fact get that fan repair done after all. So now I had a once-more-silent and oldish but working machine and a new one on the way.
The new one would come with Microsoft Windows XP preinstalled, Vista Business recovery media and an option to upgrade to Windows 7. I certainly did not want any of those. I make my living in free software consulting, with a bit of promoting and training thrown in and could credibly be denoted part of the competition. Microsoft's End User License Agreement in most of its incarnations (there are several) promises a full refund if you find their licensing terms unacceptable.
So with a minimal delay I fired off an email (sorry, Norwegian only) to what I thought was the right addresses at the retailer, saying essentially that I would want to arrange for the return of the software, in accordance with Microsoft's standard contract terms.
After all, we had heard of successes such as MacSlow in Germany and Poul-Henning Kamp's (of FreeBSD fame) slow but steady progress, also involving Lenovo hardware.
I expected that the response would be either that they have a process set up to make a symbolic refund, or they would refuse, saying that the software and the hardware are inseparable parts of the product, no matter what the clickwrap screen says.
Their actual response was mostly the latter, but with the twist that 'we do not get refunds for this from the manufacturer'. Distinctly odd, if you ask me, but I suppose we will find out more once people with actual decision making power are back after the holidays.
So no other alternative than start establishing the facts of the matter, that is, wait for the package to turn up and see what the clickwrap screen actually says.
Windows Refund Norway: The Fact Finding Mission
The package arrived this Saturday, and after the outer wrapping came off, sure enough there was a warning on the outside of the box:
That was pretty much as expected, so I continued unpacking.
Plugging in and turning on gave me this screen:
That eventually segued into
('please wait while Windows is preparing for startup') and proceeded through a few more largely information-free screenfuls up to the the point where you choose your location.
That choice likely serves several purposes. It's useful for choosing user interface languages as well as the legal terms for using the software or not, so I proceeded.
Finally, the Microsoft End User Agreement, Norwegian version, started to appear, in a miniature text box that needed several scrolling motions to reveal much of anything:
It would take only a minimal dose of negativity to suspect that these screens were in fact designed to make the user just click agreement without ever reading the full text. But taken together, these screenfuls gave me all the information I needed.
Microsoft promises a refund if you find their license terms unacceptable, but farms out the responsibility for the refund to the manufacturer, not the retailer.
So I had been barking up the wrong tree after all. Fortunately, in the meantime my friends at NUUG offered some good advice on how to proceed, and I will likely be offering updates on "Windows Refund, Norwegian version" as the tale progresses.
The promise of a refund has clearly been made. I suspect that with a sufficiently staffed legal team, playing a symbolic blame game over who should honor that promise is enough of a deterrent to the average end user, if they do not just go away after the initial rejection.
The fact-finding mission complete, it was time to try installing OpenBSD. If you are not at all interested in OpenBSD, you can scroll to the end of the article for more on the Windows refund and legalities part.
The Further Adventures of the OpenBSD Installer
The OpenBSD installer has a somewhat ugly reputation, as witnessed by this recent article over at TechRepublic. In all fairness, some years back one of the developers was quoted as saying about the installer, "it was never released, it just kind of escaped".
But things have change since then, to the point that I was wondering just which version of OpenBSD Mr. Wallen had tried to get running on his hardware, nevermind that he seems totally unaware of the automatization features you can exploit by simply adding hostnameVv.tgz or siteVv.tgz (Vv being Major.minor version numbers) file sets to your install media.
The following sequence shows you what it really looks like, with my most sincere apologies for my near-total lack of photography skills.
Booting from the CD image from the latest amd64 snapshot, my first screenful was
This proceeds through what we 'greybacks' recognize as kernel messages that scroll of the screen rather rapidly (dmesg output of the installed system is available here), ending with a menu of possible actions:
Choosing (I)nstall produces an encouraging message and a prompt to choose your preferred keyboard layout:
Next up, you get to choose a hostname
followed by a prompt which network interfaces you want to configure.
This is where I hit my first and only snag. I had tried to do the install with only wireless networking available, and for reasons known only to Intel, they have not allowed the OpenBSD project redistribution rights to the firmware files that turn the Intel WiFi Link 5100 circuitry into a working wireless networking component.
Here's what it looks like when a manufacturer chooses not to play nice with free software:
Fortunately, the OpenBSD man page for the iwn driver shows you just how to fetch and install the firware via the OpenBSD package tools. Other manufacturers have granted the OpenBSD project redistribution rights to similar firmware files needed by their products, with the result that you can perform a network install of OpenBSD directly from a typical bsd.rd over a Ralink wireless network card (using the ral driver), but not over most of the wireless network cards from Intel and some other manufacturers.
I was in fact prepared that this exact thing would happen, and it was time to move up to the attic where the wired Ethernets live. (Our house is an 18th century wooden building, and my sweetheart wanted no more holes drilled, certainly not to accommodate Ethernet cabling for her and others to trip over).
Restarting the install with a wired Ethernet got me through the entire sequence so far inside one screenful.
Once I told the installer I was done with network configuration, it naturally used all the configuration information my DHCP server had supplied:
You can go in with manual network configuration if you want, but in my case that was not needed.
Next up, your root account will need a password, you choose to run or not run some basic services such as sshd and ntpd, and you can choose to add a regular user too.
Notice the second part of the dialogue here, where the installer notices that I added a regular user and offers to disable root logins over ssh, adding the regular user to the wheel group so it's easier to add the regular user to sudoers.
The installer does not edit sudoers for you, but it's the little things like these that warms the hearts of greybacks like me.
And naturally, the installer correctly guesses my time zone but offers me a prompt to change it if the guess was not the correct one.
Network setup done, the installer presents the choice of disks available and asks you to specify which device will contain the root file system. In this case my new laptop had only one usable disk, so I proceeded directly to the dreaded partitioning part.
The first part here shows the disk partitioning before the OpenBSD installer touched it, with an option to either use the whole disk or edit the Master Boot Record (MBR). I did not plan to run several operating systems on this machine, at least not natively, so I opted for using the whole disk for OpenBSD instead.
That lead to the second part, where the installer suggests a disk partitioning scheme based on typical use, with the utterly sane choice of leaving the largest part (the k partition in this case) for user files as the /home partition.
You can edit the setup or choose your own entirely, the main point here is that the defaults are sane and unless you know a good reason to choose differently, the default choice is the safe and comfortable choice.
The installer goes on to create and mount file systems,
and the informed reader will recognize that the default mount options make a lot of sense, too.
With file systems mounted, the actual installation can proceed once you have indicated where you want to fetch the file sets from. Here I chose the http install method, and the installer made a reasonable guess at what was my closest mirror. I chose ftp.eu.openbsd.org (based in Sweden) instead, and chose not to exclude any file sets.
The next part is when you can go fetch your cup of coffee, get some other refreshment or simply a breath of fresh air. Even on a very good link, the transfers will take a few minutes.
When you get back, the installer prompts for further install sets, and if you do not have any to offer, it will finish up the install, choose the right kernel for your hardware and offer some advice for how to proceed from here on.
That is, enter reboot, let the system boot, log in as your new user or root if you did not create a user as prompted, read the (almost) personalized message from Theo de Raadt, and go on using the system as you please. Actually, you can skip reading the message if you like, but the time is likely well spent with the tips (or reminders if you like) it contains.
Here is a photograph of the disk utilization of my newly installed OpenBSD laptop a few moments after I had started transferring my files over. Notice the /home file system is not quite empty, since I almost forgot to try to document the pristine state of the system, but I am reasonably sure this picture was taken before I added any packages.
So please, Mr. Lenovo, may I have that Windows money refunded?
As the numbers add up, there is no space for your precious proprietary software on my system, and I would gladly offer to send those restore CDs and the license label back.
It was indeed Microsoft that made that promise on your behalf, but then you must have been aware of these legal issues.
I am not about to abuse your trust or your software. However I do like the hardware that I have bought, am sure we will come to a reasonable agreement.
Now back to the tech. After installing a few packages, my desktop looks roughly like this,
and disk utilization is
peter@deeperthought:~$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/sd0a 1005M 49.8M 905M 5% /
/dev/sd0k 263G 44.5G 205G 18% /home
/dev/sd0d 3.9G 40.0K 3.7G 0% /tmp
/dev/sd0f 2.0G 1.3G 542M 72% /usr
/dev/sd0g 1005M 178M 776M 19% /usr/X11R6
/dev/sd0h 5.9G 2.1G 3.5G 38% /usr/local
/dev/sd0j 2.0G 2.0K 1.9G 0% /usr/obj
/dev/sd0i 2.0G 732M 1.2G 38% /usr/src
/dev/sd0e 9.8G 51.5M 9.2G 1% /var
Yes, there's probably a lot of old data there I don'really need. But then, the time it takes to identify and remove old cruft is hard to come by.
Good night and good luck.
Raw size versions of the illustrations are available here, other updates after the footer.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
Other updates: The password guessing Hail Mary Cloud is still with us, and I occasionally update the data referenced in that article. As to the antics of the various spammers, they haven't changed much either. They're still fun to watch from time to time, though.
As should be fairly obvious from the above, this article was produced using only free software: OpenBSD and various software available through the OpenBSD packages system.
Happy new year, everyone!
As I write this, 2010 is still quite new, and I'm writing this on a new laptop, running, of course, OpenBSD. I've been mostly quite happy with the ThinkPad R60 that has served me since late 2006, but a while back the fan started giving off ugly noises and the organization nominally in charge of doing Lenovo repairs in my area seemed quite uninterested in actually doing repairs on the unit.
So after the usual browsing of web shops I ended up going for a simpler ThinkPad, the SL500 model. From the looks of it, the machine would be a tad faster and support more physical memory than the one I had already, and most likely the newer Intel processor would support running in true 64-bit (amd64) mode.
Click the buttons, whip out the credit card and away we go. Of course, the next day the repair shop turned called in to say that they could in fact get that fan repair done after all. So now I had a once-more-silent and oldish but working machine and a new one on the way.
The new one would come with Microsoft Windows XP preinstalled, Vista Business recovery media and an option to upgrade to Windows 7. I certainly did not want any of those. I make my living in free software consulting, with a bit of promoting and training thrown in and could credibly be denoted part of the competition. Microsoft's End User License Agreement in most of its incarnations (there are several) promises a full refund if you find their licensing terms unacceptable.
So with a minimal delay I fired off an email (sorry, Norwegian only) to what I thought was the right addresses at the retailer, saying essentially that I would want to arrange for the return of the software, in accordance with Microsoft's standard contract terms.
After all, we had heard of successes such as MacSlow in Germany and Poul-Henning Kamp's (of FreeBSD fame) slow but steady progress, also involving Lenovo hardware.
I expected that the response would be either that they have a process set up to make a symbolic refund, or they would refuse, saying that the software and the hardware are inseparable parts of the product, no matter what the clickwrap screen says.
Their actual response was mostly the latter, but with the twist that 'we do not get refunds for this from the manufacturer'. Distinctly odd, if you ask me, but I suppose we will find out more once people with actual decision making power are back after the holidays.
So no other alternative than start establishing the facts of the matter, that is, wait for the package to turn up and see what the clickwrap screen actually says.
Windows Refund Norway: The Fact Finding Mission
The package arrived this Saturday, and after the outer wrapping came off, sure enough there was a warning on the outside of the box:
That was pretty much as expected, so I continued unpacking.
Plugging in and turning on gave me this screen:
That eventually segued into
('please wait while Windows is preparing for startup') and proceeded through a few more largely information-free screenfuls up to the the point where you choose your location.
That choice likely serves several purposes. It's useful for choosing user interface languages as well as the legal terms for using the software or not, so I proceeded.
Finally, the Microsoft End User Agreement, Norwegian version, started to appear, in a miniature text box that needed several scrolling motions to reveal much of anything:
It would take only a minimal dose of negativity to suspect that these screens were in fact designed to make the user just click agreement without ever reading the full text. But taken together, these screenfuls gave me all the information I needed.
Microsoft promises a refund if you find their license terms unacceptable, but farms out the responsibility for the refund to the manufacturer, not the retailer.
So I had been barking up the wrong tree after all. Fortunately, in the meantime my friends at NUUG offered some good advice on how to proceed, and I will likely be offering updates on "Windows Refund, Norwegian version" as the tale progresses.
The promise of a refund has clearly been made. I suspect that with a sufficiently staffed legal team, playing a symbolic blame game over who should honor that promise is enough of a deterrent to the average end user, if they do not just go away after the initial rejection.
The fact-finding mission complete, it was time to try installing OpenBSD. If you are not at all interested in OpenBSD, you can scroll to the end of the article for more on the Windows refund and legalities part.
The Further Adventures of the OpenBSD Installer
The OpenBSD installer has a somewhat ugly reputation, as witnessed by this recent article over at TechRepublic. In all fairness, some years back one of the developers was quoted as saying about the installer, "it was never released, it just kind of escaped".
But things have change since then, to the point that I was wondering just which version of OpenBSD Mr. Wallen had tried to get running on his hardware, nevermind that he seems totally unaware of the automatization features you can exploit by simply adding hostnameVv.tgz or siteVv.tgz (Vv being Major.minor version numbers) file sets to your install media.
The following sequence shows you what it really looks like, with my most sincere apologies for my near-total lack of photography skills.
Booting from the CD image from the latest amd64 snapshot, my first screenful was
This proceeds through what we 'greybacks' recognize as kernel messages that scroll of the screen rather rapidly (dmesg output of the installed system is available here), ending with a menu of possible actions:
Choosing (I)nstall produces an encouraging message and a prompt to choose your preferred keyboard layout:
Next up, you get to choose a hostname
followed by a prompt which network interfaces you want to configure.
This is where I hit my first and only snag. I had tried to do the install with only wireless networking available, and for reasons known only to Intel, they have not allowed the OpenBSD project redistribution rights to the firmware files that turn the Intel WiFi Link 5100 circuitry into a working wireless networking component.
Here's what it looks like when a manufacturer chooses not to play nice with free software:
Fortunately, the OpenBSD man page for the iwn driver shows you just how to fetch and install the firware via the OpenBSD package tools. Other manufacturers have granted the OpenBSD project redistribution rights to similar firmware files needed by their products, with the result that you can perform a network install of OpenBSD directly from a typical bsd.rd over a Ralink wireless network card (using the ral driver), but not over most of the wireless network cards from Intel and some other manufacturers.
I was in fact prepared that this exact thing would happen, and it was time to move up to the attic where the wired Ethernets live. (Our house is an 18th century wooden building, and my sweetheart wanted no more holes drilled, certainly not to accommodate Ethernet cabling for her and others to trip over).
Restarting the install with a wired Ethernet got me through the entire sequence so far inside one screenful.
Once I told the installer I was done with network configuration, it naturally used all the configuration information my DHCP server had supplied:
You can go in with manual network configuration if you want, but in my case that was not needed.Next up, your root account will need a password, you choose to run or not run some basic services such as sshd and ntpd, and you can choose to add a regular user too.
Notice the second part of the dialogue here, where the installer notices that I added a regular user and offers to disable root logins over ssh, adding the regular user to the wheel group so it's easier to add the regular user to sudoers.
The installer does not edit sudoers for you, but it's the little things like these that warms the hearts of greybacks like me.
And naturally, the installer correctly guesses my time zone but offers me a prompt to change it if the guess was not the correct one.
Network setup done, the installer presents the choice of disks available and asks you to specify which device will contain the root file system. In this case my new laptop had only one usable disk, so I proceeded directly to the dreaded partitioning part.
The first part here shows the disk partitioning before the OpenBSD installer touched it, with an option to either use the whole disk or edit the Master Boot Record (MBR). I did not plan to run several operating systems on this machine, at least not natively, so I opted for using the whole disk for OpenBSD instead.
That lead to the second part, where the installer suggests a disk partitioning scheme based on typical use, with the utterly sane choice of leaving the largest part (the k partition in this case) for user files as the /home partition.
You can edit the setup or choose your own entirely, the main point here is that the defaults are sane and unless you know a good reason to choose differently, the default choice is the safe and comfortable choice.
The installer goes on to create and mount file systems,
and the informed reader will recognize that the default mount options make a lot of sense, too.
With file systems mounted, the actual installation can proceed once you have indicated where you want to fetch the file sets from. Here I chose the http install method, and the installer made a reasonable guess at what was my closest mirror. I chose ftp.eu.openbsd.org (based in Sweden) instead, and chose not to exclude any file sets.
The next part is when you can go fetch your cup of coffee, get some other refreshment or simply a breath of fresh air. Even on a very good link, the transfers will take a few minutes.
When you get back, the installer prompts for further install sets, and if you do not have any to offer, it will finish up the install, choose the right kernel for your hardware and offer some advice for how to proceed from here on.
That is, enter reboot, let the system boot, log in as your new user or root if you did not create a user as prompted, read the (almost) personalized message from Theo de Raadt, and go on using the system as you please. Actually, you can skip reading the message if you like, but the time is likely well spent with the tips (or reminders if you like) it contains.
Here is a photograph of the disk utilization of my newly installed OpenBSD laptop a few moments after I had started transferring my files over. Notice the /home file system is not quite empty, since I almost forgot to try to document the pristine state of the system, but I am reasonably sure this picture was taken before I added any packages.
So please, Mr. Lenovo, may I have that Windows money refunded?
As the numbers add up, there is no space for your precious proprietary software on my system, and I would gladly offer to send those restore CDs and the license label back.
It was indeed Microsoft that made that promise on your behalf, but then you must have been aware of these legal issues.
I am not about to abuse your trust or your software. However I do like the hardware that I have bought, am sure we will come to a reasonable agreement.
It's all about honesty in your business life.
Now back to the tech. After installing a few packages, my desktop looks roughly like this,
and disk utilization is
peter@deeperthought:~$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/sd0a 1005M 49.8M 905M 5% /
/dev/sd0k 263G 44.5G 205G 18% /home
/dev/sd0d 3.9G 40.0K 3.7G 0% /tmp
/dev/sd0f 2.0G 1.3G 542M 72% /usr
/dev/sd0g 1005M 178M 776M 19% /usr/X11R6
/dev/sd0h 5.9G 2.1G 3.5G 38% /usr/local
/dev/sd0j 2.0G 2.0K 1.9G 0% /usr/obj
/dev/sd0i 2.0G 732M 1.2G 38% /usr/src
/dev/sd0e 9.8G 51.5M 9.2G 1% /var
Yes, there's probably a lot of old data there I don'really need. But then, the time it takes to identify and remove old cruft is hard to come by.
Good night and good luck.
Raw size versions of the illustrations are available here, other updates after the footer.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
Other updates: The password guessing Hail Mary Cloud is still with us, and I occasionally update the data referenced in that article. As to the antics of the various spammers, they haven't changed much either. They're still fun to watch from time to time, though.
As should be fairly obvious from the above, this article was produced using only free software: OpenBSD and various software available through the OpenBSD packages system.
Sunday, November 15, 2009
Rickrolled? Get Ready for the Hail Mary Cloud!
If you publish your user name and password, somebody who is not you will use it, sooner or later.
It's been a fun few weeks. Over in Microsoft land, it must have been a big issue that according to malware hunters Sophos, the newly released Windows 7 with no extras is roughly as vulnerable as its older siblings. No great surprises there, I suppose.
For those of us with a more Unixish leaning, the more interesting piece of news involved Apple iPhones. These phones apparently run a version of MacOS that has enough Unix in it that with a bit of tinkering, it is possible to install a variety of Unix software, such as the ubiquitous secure shell daemon sshd. To some, there is a certain attraction in knowing that you have an SSH server in your pocket or handbag. Too bad, then that enough of those adventurous iPhone owners never read up on the instructions and chose to run their toy with the default password for the root account and were vulnerable to a wonderful prank perpetrated by a programmer down under.
The prank (described in the inimitable The Register style here) demonstrated just how bad an idea it is to publish your user name and password. If you followed the news around last weekend you would notice that a large segment of the Microsoft-attached instapunditry got their facts wrong as usual with the this proves that Apple (and by extension any Unix and of course Linux) is just as vulnerable as Microsoft mantra repeated over and over.
In fact, there are two historical incidents that point to Unix being no silver bullet: the 2002 Linux Slapper Worm and the original network-enabled worm, the 1988 Morris Worm. Those two prove mainly that yes, some bugs are exploitable, and the way forward is to fix bugs and make them harder to exploit in the first place. Now these two famous exploits is possibly to be joined by a third, the rickrolling prank.
I beg to differ. The rickroller is about bad passwords, no more, no less. I've spent considerable time ranting about passwords in earlier columns, and this incident only underscores what we've been repeating until your eardrums wear thin an my vocal cords swell from exhaustion:
Add to this that the fact, as we've noted here earlier, there is a whole cloud of hijacked machines out there beavering away at guessing passwords right now, and they have been at it for quite a while.
The most remarkable of these botnets is the one that tries to avoid detection by distributing the password guessing for any target across a large number of hosts, so each guesser never shows high enough rates of activity to trigger any of the traditional bruteforce detection mechanism. The attempts look something like this in your authentication log:
and so on.
I put it to you: What you see here is the cybercrime equivalent of the Hail Mary Pass".
Each attempt in theory has monumental odds against succeeding, but occasionally the guess will be right and they have scored a login. As far as we know, this is at least the third round of password guessing from the Hail Mary Cloud (see the archives for earlier postings about slow bruteforcers), but there could have been earlier rounds that escaped our attention.
The fact that we see the Hail Mary Cloud keeping up the guessing is a strong indicator that there are a lot of guessable passwords and possibly badly maintained systems out there, and that even against the very long odds they are succeeding often enough in their attempts to gain a foothold somewhere that it is worth keeping up the efforts. For one thing, the cost of using other people's equipment is likely to be quite low.
There are a lot of things about the Hail Mary Cloud and its overseers that we do not know. People who responded to the earlier articles with reports of similar activity also reported pretty consistently something like a sixty to seventy percent match in hosts making the attempts.
With 1767 hosts in the current sample it is likely that we have a cloud of at least several thousand, and most likely no single guessing host in the cloud ever gets around to contacting every host in the target list. The busier your SSH deamon is with normal traffic, the harder it will be to detect the footprint of Hail Mary activity, and likely a lot of this goes undetected.
The data, as I am sure you have been waiting for it, is available in these forms: Raw log data, with 3-4 lines per attempt (as illustrated above and requested by some correspondents), one line per attempt (shows the pattern a little more clearly), a list of the hosts participating in the Hail Mary Cloud sorted by number of attempts, and the user names attempted, sorted by number of attempts.
The pattern is fairly familiar by now, but this time the alphabetic cycles are shorter and at times the coordination seems to have broken down. My guess is that the apparent breakdowns are due to silly factors like the guessing machines running without time synchronization or other signs of incompetence.
And finally, some words of advice for those of you who want to avoid both rickrolling and getting cracked by other password guessing.
You should at least consider setting a password policy and enforcing it with something like John the ripper, which more than likely is available at the cost of a few keystrokes from your package system. And of course there is the fine art of sshd configuration. Some of the things you could do are, in no particular order:
If you see traces of the Hail Mary Cloud's activity in your logs and you want to share and study, I would very much like to hear from you. I will most likely be updating the log data and extracts at intervals.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
It's been a fun few weeks. Over in Microsoft land, it must have been a big issue that according to malware hunters Sophos, the newly released Windows 7 with no extras is roughly as vulnerable as its older siblings. No great surprises there, I suppose.
For those of us with a more Unixish leaning, the more interesting piece of news involved Apple iPhones. These phones apparently run a version of MacOS that has enough Unix in it that with a bit of tinkering, it is possible to install a variety of Unix software, such as the ubiquitous secure shell daemon sshd. To some, there is a certain attraction in knowing that you have an SSH server in your pocket or handbag. Too bad, then that enough of those adventurous iPhone owners never read up on the instructions and chose to run their toy with the default password for the root account and were vulnerable to a wonderful prank perpetrated by a programmer down under.
The prank (described in the inimitable The Register style here) demonstrated just how bad an idea it is to publish your user name and password. If you followed the news around last weekend you would notice that a large segment of the Microsoft-attached instapunditry got their facts wrong as usual with the this proves that Apple (and by extension any Unix and of course Linux) is just as vulnerable as Microsoft mantra repeated over and over.
In fact, there are two historical incidents that point to Unix being no silver bullet: the 2002 Linux Slapper Worm and the original network-enabled worm, the 1988 Morris Worm. Those two prove mainly that yes, some bugs are exploitable, and the way forward is to fix bugs and make them harder to exploit in the first place. Now these two famous exploits is possibly to be joined by a third, the rickrolling prank.
I beg to differ. The rickroller is about bad passwords, no more, no less. I've spent considerable time ranting about passwords in earlier columns, and this incident only underscores what we've been repeating until your eardrums wear thin an my vocal cords swell from exhaustion:
Publishing your username and password is a really bad idea.
It's almost as bad as picking a guessable password.
Add to this that the fact, as we've noted here earlier, there is a whole cloud of hijacked machines out there beavering away at guessing passwords right now, and they have been at it for quite a while.
The most remarkable of these botnets is the one that tries to avoid detection by distributing the password guessing for any target across a large number of hosts, so each guesser never shows high enough rates of activity to trigger any of the traditional bruteforce detection mechanism. The attempts look something like this in your authentication log:
Nov 13 21:10:14 rosalita sshd[50401]: error: PAM: authentication error for illegal user mars from 125.40.69.208
Nov 13 21:10:14 rosalita sshd[50401]: Failed keyboard-interactive/pam for invalid user mars from 125.40.69.208 port 38052 ssh2
Nov 13 21:11:20 rosalita sshd[50419]: reverse mapping checking getaddrinfo for 115-186-131-90.nayatel.pk [115.186.131.90] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:11:20 rosalita sshd[50419]: Invalid user mars from 115.186.131.90
Nov 13 21:11:21 rosalita sshd[50419]: error: PAM: authentication error for illegal user mars from 115.186.131.90
Nov 13 21:11:21 rosalita sshd[50419]: Failed keyboard-interactive/pam for invalid user mars from 115.186.131.90 port 42235 ssh2
Nov 13 21:13:43 rosalita sshd[50428]: Invalid user mars from 58.247.222.163
Nov 13 21:13:43 rosalita sshd[50428]: error: PAM: authentication error for illegal user mars from 58.247.222.163
Nov 13 21:13:43 rosalita sshd[50428]: Failed keyboard-interactive/pam for invalid user mars from 58.247.222.163 port 35134 ssh2
Nov 13 21:15:59 rosalita sshd[50440]: Invalid user mars from 89.76.186.99
Nov 13 21:16:00 rosalita sshd[50440]: error: PAM: authentication error for illegal user mars from chello089076186099.chello.pl
Nov 13 21:16:00 rosalita sshd[50440]: Failed keyboard-interactive/pam for invalid user mars from 89.76.186.99 port 52156 ssh2
Nov 13 21:17:16 rosalita sshd[50448]: Invalid user mars2 from 88.134.166.31
Nov 13 21:17:16 rosalita sshd[50448]: error: PAM: authentication error for illegal user mars2 from 88-134-166-31-dynip.superkabel.de
Nov 13 21:17:16 rosalita sshd[50448]: Failed keyboard-interactive/pam for invalid user mars2 from 88.134.166.31 port 39254 ssh2
Nov 13 21:18:14 rosalita sshd[50452]: Invalid user room from 62.198.66.107
Nov 13 21:18:14 rosalita sshd[50452]: error: PAM: authentication error for illegal user room from 62.198.66.107
Nov 13 21:18:14 rosalita sshd[50452]: Failed keyboard-interactive/pam for invalid user room from 62.198.66.107 port 47557 ssh2
Nov 13 21:19:27 rosalita sshd[50458]: Invalid user room from 61.74.75.43
Nov 13 21:19:27 rosalita sshd[50458]: error: PAM: authentication error for illegal user room from 61.74.75.43
Nov 13 21:19:27 rosalita sshd[50458]: Failed keyboard-interactive/pam for invalid user room from 61.74.75.43 port 57794 ssh2
Nov 13 21:21:41 rosalita sshd[50472]: Invalid user room from 212.243.41.9
Nov 13 21:21:41 rosalita sshd[50472]: error: PAM: authentication error for illegal user room from 212.243.41.9
Nov 13 21:21:41 rosalita sshd[50472]: Failed keyboard-interactive/pam for invalid user room from 212.243.41.9 port 38396 ssh2
Nov 13 21:22:58 rosalita sshd[50491]: reverse mapping checking getaddrinfo for static-ip-cr1901468058.cable.net.co [190.146.80.58] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:22:58 rosalita sshd[50491]: Invalid user room from 190.146.80.58
Nov 13 21:22:58 rosalita sshd[50491]: error: PAM: authentication error for illegal user room from 190.146.80.58
Nov 13 21:22:58 rosalita sshd[50491]: Failed keyboard-interactive/pam for invalid user room from 190.146.80.58 port 4420 ssh2
Nov 13 21:24:01 rosalita sshd[50509]: Invalid user room from 62.23.130.173
Nov 13 21:24:01 rosalita sshd[50509]: error: PAM: authentication error for illegal user room from host.173.130.23.62.rev.coltfrance.com
Nov 13 21:24:01 rosalita sshd[50509]: Failed keyboard-interactive/pam for invalid user room from 62.23.130.173 port 3904 ssh2
Nov 13 21:25:21 rosalita sshd[50517]: reverse mapping checking getaddrinfo for hn.kd.ny.adsl [125.40.69.208] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:25:21 rosalita sshd[50517]: Invalid user room from 125.40.69.208
Nov 13 21:25:21 rosalita sshd[50517]: error: PAM: authentication error for illegal user room from 125.40.69.208
Nov 13 21:25:21 rosalita sshd[50517]: Failed keyboard-interactive/pam for invalid user room from 125.40.69.208 port 3294 ssh2and so on.
I put it to you: What you see here is the cybercrime equivalent of the Hail Mary Pass".
Each attempt in theory has monumental odds against succeeding, but occasionally the guess will be right and they have scored a login. As far as we know, this is at least the third round of password guessing from the Hail Mary Cloud (see the archives for earlier postings about slow bruteforcers), but there could have been earlier rounds that escaped our attention.
The fact that we see the Hail Mary Cloud keeping up the guessing is a strong indicator that there are a lot of guessable passwords and possibly badly maintained systems out there, and that even against the very long odds they are succeeding often enough in their attempts to gain a foothold somewhere that it is worth keeping up the efforts. For one thing, the cost of using other people's equipment is likely to be quite low.
There are a lot of things about the Hail Mary Cloud and its overseers that we do not know. People who responded to the earlier articles with reports of similar activity also reported pretty consistently something like a sixty to seventy percent match in hosts making the attempts.
With 1767 hosts in the current sample it is likely that we have a cloud of at least several thousand, and most likely no single guessing host in the cloud ever gets around to contacting every host in the target list. The busier your SSH deamon is with normal traffic, the harder it will be to detect the footprint of Hail Mary activity, and likely a lot of this goes undetected.
The data, as I am sure you have been waiting for it, is available in these forms: Raw log data, with 3-4 lines per attempt (as illustrated above and requested by some correspondents), one line per attempt (shows the pattern a little more clearly), a list of the hosts participating in the Hail Mary Cloud sorted by number of attempts, and the user names attempted, sorted by number of attempts.
The pattern is fairly familiar by now, but this time the alphabetic cycles are shorter and at times the coordination seems to have broken down. My guess is that the apparent breakdowns are due to silly factors like the guessing machines running without time synchronization or other signs of incompetence.
And finally, some words of advice for those of you who want to avoid both rickrolling and getting cracked by other password guessing.
You should at least consider setting a password policy and enforcing it with something like John the ripper, which more than likely is available at the cost of a few keystrokes from your package system. And of course there is the fine art of sshd configuration. Some of the things you could do are, in no particular order:
- disable root logins over the network
- use packet filtering or other means to restrict where users can log in from
- disable password logins entirely allowing only key-based logins
- set up your sshd to listen on a non-standard port
If you see traces of the Hail Mary Cloud's activity in your logs and you want to share and study, I would very much like to hear from you. I will most likely be updating the log data and extracts at intervals.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Sunday, October 4, 2009
A Third Time, Uncharmed
Spamwashers hijacked, a wake-up call for lazy sysadmins everywhere. The slow bruteforcers are back for another round.
A new round of slow, distributed bruteforce attacks is in progress. Just like the other times we know about (see references later), the initial target is root. This time around I see only one of my ssh-contactable machines targeted, and the dribble started on September 30th.
I've put the raw data so far up for study here (a total of 6067 attempts), and a list of hosts sorted by number of attempts (the first column) can be found here (770 hosts, with up to 32 attempts each). Quite likely I'll be collecting more data and publishing updates when I have a few free moments.
A number of people were kind enough to contact me in the followup of the earlier articles, and from one of my correspendents (who asked not to be named) I learned that the likely culprit is a piece of Linux malware known as dt_ssh5. If you type dt_ssh5 into your favorite search engine, it will turn up a few hits, but significantly fewer than the number of hosts in my sample. A couple of those documents have some analysis of how a badly secured web application let the miscreants in.
We should not be surprised that whoever is behind this has more than one takeover method in their arsenal. Unfortunately it should not surprise anybody either that we're now seeing a third round of those attacks. Most likely the perpetrators keep going because they occasionally succeed, and when they do, it's because every now and then they luck on a Linux machine with either
You see where this is heading? Over the last few years we have seen Linux and other free systems take over ever more niches and even mission critical application areas in businesses and organizations all over the world. I for one think this move to free systems with source code accessible is a good thing.
But there is a downside: In our efforts to entice the suits into the wonderful new world of free software, we likely oversold the security part.
Yes, bugs are easier to find and fix when the source code is available, yes, the Unix security model or some variant thereof is vastly preferable to that disorienting hellhole system marketed from somewhere up Seattle way, and yes, there are a few hundred more good and valid reasons (technical and otherwise) why basing your business on free license software is generally a good idea. Security-wise you have a better starting point, and with a little effort you will gain control and stay in control. But it does take some degree of effort by one or more qualified persons who are willing and able to take charge and make sensible decisions.
Looking at the data earlier I can see that one of the hosts now trying to gain illegitimate access to one of my systems was originally set up as a "spam washer" (grep for spamvask in the data). Setting up a Linux box to filter spam is likely a good idea in many contexts, but please keep this in mind: The fact that your rig runs Linux does not mean you're home free. You need to keep paying attention. When your spam washer has been hijacked and tries to break into other people's systems, you urgently need to get your act together, right now.
We do not have self-propagating worms on Linux just yet (and in all fairness the likelihood of that ever happening is rather slim), but we certainly do not have a system that will stay secure if you ignore the basics of useful system administration. This recurring episode of dt_ssh5 and the slow bruteforcers should serve as a wakeup call for system administrators everywhere: Your system stays secure only as long as you pay proper attention to maintenance. Out there in the real world there are at least 770 machines where the maintenance regime slipped, either through incompetence or work overload or a combination of the two with a dash of bad planning thrown in.
The only way to make a fourth round of slow bruteforcers not happen is to make the people responsible for the 770 hosts listed here do the right thing. I suppose we will find out soon enough how many of those domains actually have the RFC2142-mandated mailboxes in place.
References: Previous posts on this topic include A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009).
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
The reason for my rather long silence in the blogosphere can be summed up in two words: Client Confidentiality. Some of the projects I've been working on might have been material for interesting articles, but separating the generally useful bits from what the customer could reasonably expect to be kept confidential has proved difficult, at least in the short run.
Update 2009-10-05 just before 10:00 CEST: Slightly newer log extracts uploaded, total attempts now 6590, from 775 hosts. I will likely upload fresher versions at intervals.
Update 2009-10-14 19:30ish: After a couple of days with literally none to three or four attempts per hour, my afternoon log check turned up activity almost at start of run levels, so total numbers are now 9405 attempts from a total of 946 unique hosts. I never bothered to change the file names, but the data have been updated a couple of times each day since I started publishing them.
Update 2009-10-29 00:10: Because a file transfer took a little longer than expected, I was awake to see what may actually be the start of the alphabetic phase -
More material later, for now I'd be happy to hear confirmation (reports of similar activity) if you see it in your logs.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
A new round of slow, distributed bruteforce attacks is in progress. Just like the other times we know about (see references later), the initial target is root. This time around I see only one of my ssh-contactable machines targeted, and the dribble started on September 30th.
I've put the raw data so far up for study here (a total of 6067 attempts), and a list of hosts sorted by number of attempts (the first column) can be found here (770 hosts, with up to 32 attempts each). Quite likely I'll be collecting more data and publishing updates when I have a few free moments.
A number of people were kind enough to contact me in the followup of the earlier articles, and from one of my correspendents (who asked not to be named) I learned that the likely culprit is a piece of Linux malware known as dt_ssh5. If you type dt_ssh5 into your favorite search engine, it will turn up a few hits, but significantly fewer than the number of hosts in my sample. A couple of those documents have some analysis of how a badly secured web application let the miscreants in.
We should not be surprised that whoever is behind this has more than one takeover method in their arsenal. Unfortunately it should not surprise anybody either that we're now seeing a third round of those attacks. Most likely the perpetrators keep going because they occasionally succeed, and when they do, it's because every now and then they luck on a Linux machine with either
- a maintenance regime that's disorganized enough that software with known and exploitable bugs is left in place for long enough to open the doors to undesirables, or
- at least one user (whoever is manning root or any of the other user IDs we know they will be sniffing out later) with a guessable password and a system administration regime that lets weak passwords exist in the first place.
You see where this is heading? Over the last few years we have seen Linux and other free systems take over ever more niches and even mission critical application areas in businesses and organizations all over the world. I for one think this move to free systems with source code accessible is a good thing.
But there is a downside: In our efforts to entice the suits into the wonderful new world of free software, we likely oversold the security part.
Yes, bugs are easier to find and fix when the source code is available, yes, the Unix security model or some variant thereof is vastly preferable to that disorienting hellhole system marketed from somewhere up Seattle way, and yes, there are a few hundred more good and valid reasons (technical and otherwise) why basing your business on free license software is generally a good idea. Security-wise you have a better starting point, and with a little effort you will gain control and stay in control. But it does take some degree of effort by one or more qualified persons who are willing and able to take charge and make sensible decisions.
Looking at the data earlier I can see that one of the hosts now trying to gain illegitimate access to one of my systems was originally set up as a "spam washer" (grep for spamvask in the data). Setting up a Linux box to filter spam is likely a good idea in many contexts, but please keep this in mind: The fact that your rig runs Linux does not mean you're home free. You need to keep paying attention. When your spam washer has been hijacked and tries to break into other people's systems, you urgently need to get your act together, right now.
We do not have self-propagating worms on Linux just yet (and in all fairness the likelihood of that ever happening is rather slim), but we certainly do not have a system that will stay secure if you ignore the basics of useful system administration. This recurring episode of dt_ssh5 and the slow bruteforcers should serve as a wakeup call for system administrators everywhere: Your system stays secure only as long as you pay proper attention to maintenance. Out there in the real world there are at least 770 machines where the maintenance regime slipped, either through incompetence or work overload or a combination of the two with a dash of bad planning thrown in.
The only way to make a fourth round of slow bruteforcers not happen is to make the people responsible for the 770 hosts listed here do the right thing. I suppose we will find out soon enough how many of those domains actually have the RFC2142-mandated mailboxes in place.
References: Previous posts on this topic include A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009).
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
The reason for my rather long silence in the blogosphere can be summed up in two words: Client Confidentiality. Some of the projects I've been working on might have been material for interesting articles, but separating the generally useful bits from what the customer could reasonably expect to be kept confidential has proved difficult, at least in the short run.
Update 2009-10-05 just before 10:00 CEST: Slightly newer log extracts uploaded, total attempts now 6590, from 775 hosts. I will likely upload fresher versions at intervals.
Update 2009-10-14 19:30ish: After a couple of days with literally none to three or four attempts per hour, my afternoon log check turned up activity almost at start of run levels, so total numbers are now 9405 attempts from a total of 946 unique hosts. I never bothered to change the file names, but the data have been updated a couple of times each day since I started publishing them.
Update 2009-10-29 00:10: Because a file transfer took a little longer than expected, I was awake to see what may actually be the start of the alphabetic phase -
Oct 28 23:58:35 rosalita sshd[61664]: Invalid user anthony from 62.225.63.99
Oct 28 23:58:35 rosalita sshd[61664]: error: PAM: authentication error for illegal user anthony from 62.225.63.99
Oct 28 23:58:35 rosalita sshd[61664]: Failed keyboard-interactive/pam for invalid user anthony from 62.225.63.99 port 58683 ssh2
Oct 28 23:59:43 rosalita sshd[61668]: Invalid user anthony from 217.136.253.239
Oct 28 23:59:43 rosalita sshd[61668]: error: PAM: authentication error for illegal user anthony from 239.253-136-217.adsl-static.isp.belgacom.be
Oct 28 23:59:43 rosalita sshd[61668]: Failed keyboard-interactive/pam for invalid user anthony from 217.136.253.239 port 54728 ssh2
Oct 29 00:00:26 rosalita sshd[61695]: Invalid user barbara from 112.78.124.159
Oct 29 00:00:27 rosalita sshd[61695]: error: PAM: authentication error for illegal user barbara from 112.78.124.159
Oct 29 00:00:27 rosalita sshd[61695]: Failed keyboard-interactive/pam for invalid user barbara from 112.78.124.159 port 56990 ssh2
Oct 29 00:03:42 rosalita sshd[61722]: Invalid user brian from 122.224.128.197
Oct 29 00:03:42 rosalita sshd[61722]: error: PAM: authentication error for illegal user brian from 122.224.128.197
Oct 29 00:03:42 rosalita sshd[61722]: Failed keyboard-interactive/pam for invalid user brian from 122.224.128.197 port 47058 ssh2
Oct 29 00:04:21 rosalita sshd[61725]: Invalid user brian from 218.69.27.138
Oct 29 00:04:21 rosalita sshd[61725]: error: PAM: authentication error for illegal user brian from 218.69.27.138
Oct 29 00:04:21 rosalita sshd[61725]: Failed keyboard-interactive/pam for invalid user brian from 218.69.27.138 port 41622 ssh2
Oct 29 00:05:00 rosalita sshd[61738]: Invalid user carol from 58.60.106.121
Oct 29 00:05:01 rosalita sshd[61738]: error: PAM: authentication error for illegal user carol from 58.60.106.121
Oct 29 00:05:01 rosalita sshd[61738]: Failed keyboard-interactive/pam for invalid user carol from 58.60.106.121 port 55916 ssh2
Oct 29 00:05:51 rosalita sshd[61744]: Invalid user carol from 200.248.242.218
Oct 29 00:05:52 rosalita sshd[61744]: error: PAM: authentication error for illegal user carol from 200.248.242.218
Oct 29 00:05:52 rosalita sshd[61744]: Failed keyboard-interactive/pam for invalid user carol from 200.248.242.218 port 54547 ssh2
Oct 29 00:07:24 rosalita sshd[61748]: Invalid user charles from 222.128.36.60
Oct 29 00:07:24 rosalita sshd[61748]: error: PAM: authentication error for illegal user charles from 222.128.36.60
Oct 29 00:07:24 rosalita sshd[61748]: Failed keyboard-interactive/pam for invalid user charles from 222.128.36.60 port 42322 ssh2
Oct 29 00:08:09 rosalita sshd[61755]: Invalid user christopher from 72.148.242.188
Oct 29 00:08:10 rosalita sshd[61755]: error: PAM: authentication error for illegal user christopher from adsl-072-148-242-188.sip.asm.bellsouth.net
Oct 29 00:08:10 rosalita sshd[61755]: Failed keyboard-interactive/pam for invalid user christopher from 72.148.242.188 port 37157 ssh2
Oct 29 00:09:06 rosalita sshd[61758]: Invalid user christopher from 58.223.251.232
Oct 29 00:09:06 rosalita sshd[61758]: error: PAM: authentication error for illegal user christopher from 58.223.251.232
Oct 29 00:09:06 rosalita sshd[61758]: Failed keyboard-interactive/pam for invalid user christopher from 58.223.251.232 port 46350 ssh2
Oct 29 00:09:48 rosalita sshd[61762]: Invalid user daniel from 64.69.114.115
Oct 29 00:09:49 rosalita sshd[61762]: error: PAM: authentication error for illegal user daniel from mulligan.softspike.org
Oct 29 00:09:49 rosalita sshd[61762]: Failed keyboard-interactive/pam for invalid user daniel from 64.69.114.115 port 39960 ssh2
Oct 29 00:11:29 rosalita sshd[61781]: Invalid user david from 80.255.179.150
Oct 29 00:11:29 rosalita sshd[61781]: error: PAM: authentication error for illegal user david from ppp-vpdn-80.255.179.150.yarnet.ru
Oct 29 00:11:29 rosalita sshd[61781]: Failed keyboard-interactive/pam for invalid user david from 80.255.179.150 port 49013 ssh2More material later, for now I'd be happy to hear confirmation (reports of similar activity) if you see it in your logs.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Sunday, April 12, 2009
The slow brute zombies are back
Real-life zombies feed off weak passwords.
Regular readers will remember that late last year we saw a peculiar form of distributed bruteforce attack on certain ssh servers. What made this particular batch of failed login attempts stand out was that unlike the traditional rapid-fire bruteforce attempts we were quite adept at heading off with state tracking tricks (such as the OpenBSD PF method described here or a slightly different Linux-specific method), the technique seemed to be specifically designed to slip past such defenses.
I described the method and the evidence at some length in a series of colums, A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008) and finally The slow brutes, a final roundup (January 22, 2009).
If you do not want to read through all of that, I'll recap: The traditional bruteforce attack is characterized by somebody trying again and again to find a combination of logon credentials (typically sets of user names and passwords) that will let them gain access to the system. The basic idea is that given enough tries, you will sooner or later find a user who has set a guessable password, and you have access.
One of the important weaknesses of the method is that the typical attacker would have gained access to only one or a few systems, and the attacker would then typically try to connect a lot more often than a typical user, and sysadmins learned to adapt their firewalls and other systems to lock out activity that fit the bruteforcer profile.
If you run a ssh service anywhere Internet-facing, you will be used to seeing a steady stream of failed logons for both existing and non-existing users. There's nothing new in seeing failed logons in your log files. However, what happened late last year was that we started seeing large numbers of failed ssh logon attempts, with the new twist that the same user would be trying to log on a large number of times, but never from the same place twice in rapid succession. This log data sample will give you an idea. The data will show you the pattern, as will the summary article.
Back then in January, my conclusion was that we had seen the conclusion of a largely failed experiment. After all, proceeding at a truly glacial pace and decreasing the number of attempts per user name as they went along hardly seemed like a recipe for success even if they could sneak past packet filtering based defenses.
It turns out I was wrong. Returning to my log summaries after taking a few days off from log data (yes, easter is big here, even moves geeks sometimes to take time off) I find data with almost exactly the same profile as last December's series.
This can only mean that there were enough successful attempts at guessing people's weak passwords in the last round that our unknown perpetrators found it worthwhile to start another round. For all I know they may have been at it all along, probing other parts of the Internet far from my unfashionable backwaters.
Anyway, they are most certainly back. A summary of the data so far follows in the concluding section. Please note that I would be very interested in communicating with other parties who see similar activity in their logs, and if you are responsible for one or more of the systems identified or you know who is, I urge you to take appropriate action.
The Data So Far
First, We Take root
The full log data reveal that they started rubbing up against my listening posts during the early hours of April 7th 2009 (as measured in CEST), concentrating first on the user root (attempts sorted by hostname here, hostnames here. Not terribly surprising that they tried to take on this one, as most Unix(ish) systems tend to at least have a root account.
Then We Take admin
After about twenty-four hours, the zombies in the cloud tired of root and turned their collective attention to admin (attempts sorted by hostname here, hostnames here. This is a user name I would not expect to be on a Unix syste by default. In my limited experience Microsoft systems tend to use this one more often, but then there may be enough of those systems out there with a SSH service and weak passwords out there to try.
And james Will Make Us Famous
Well into civilized lunchtime (CEST) on April 9th, our would-be guests turned their attention to james (attempts sorted by hostname here, hostnames here. Why this user name received so much attention is a complete mystery to me. They did tire in the end, though.
Before We Turn To 123456, aadi And The Rest Of The Rabble
As if to confirm that we are indeed seeing a rerun of last November and December's sequence of events, the robots started on their regular alphabetic attempts on April 11th:
Apr 11 17:44:15 rosalita sshd[51241]: error: PAM: authentication error for illegal user james from 221.130.177.154
Apr 11 17:50:47 rosalita sshd[51258]: error: PAM: authentication error for illegal user 123456 from 220.194.54.41
Apr 11 17:53:24 rosalita sshd[51262]: error: PAM: authentication error for illegal user 123456 from webmail.sistemafieg.org.br
Apr 11 17:58:25 rosalita sshd[51285]: error: PAM: authentication error for illegal user 123456 from 202.82.25.161
Apr 11 18:00:59 rosalita sshd[51307]: error: PAM: authentication error for illegal user aadi from mail.cgconsultoriacontabil.com.br
The various comments to my earlier columns tended to point out the near-futility of low-intensity bruteforcing, and I took the fact that the last series of attempts did not run to the end of the alphabet as an indication that the perpetrators themselves had reached the same conclusion.
Now we know that they just moved on, went elsewhere for a while and now they're back. In the meantime, they must have hit on enough weak minds and associated weak passwords that they were able to take over and zombiefy enough systems to be worth their while.
The data so far (collected up to April 12, 2009, roughly 21:00 CEST) indicates that the total number of hosts involved in the attempts at my listening posts is just over a thousand (a list of hosts available here).
It is probably worth repeating that in real life, zombies feed on both weak minds and weak passwords.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available for free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
Also worth noting, I will be doing a PF tutorial at BSDCan 2009 in May, and I will be staying around for the rest of the conference. I look forward to seeing you there.
Note: A Better Data Source Is Available
Update 2015-04-02: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Regular readers will remember that late last year we saw a peculiar form of distributed bruteforce attack on certain ssh servers. What made this particular batch of failed login attempts stand out was that unlike the traditional rapid-fire bruteforce attempts we were quite adept at heading off with state tracking tricks (such as the OpenBSD PF method described here or a slightly different Linux-specific method), the technique seemed to be specifically designed to slip past such defenses.
I described the method and the evidence at some length in a series of colums, A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008) and finally The slow brutes, a final roundup (January 22, 2009).
If you do not want to read through all of that, I'll recap: The traditional bruteforce attack is characterized by somebody trying again and again to find a combination of logon credentials (typically sets of user names and passwords) that will let them gain access to the system. The basic idea is that given enough tries, you will sooner or later find a user who has set a guessable password, and you have access.
One of the important weaknesses of the method is that the typical attacker would have gained access to only one or a few systems, and the attacker would then typically try to connect a lot more often than a typical user, and sysadmins learned to adapt their firewalls and other systems to lock out activity that fit the bruteforcer profile.
If you run a ssh service anywhere Internet-facing, you will be used to seeing a steady stream of failed logons for both existing and non-existing users. There's nothing new in seeing failed logons in your log files. However, what happened late last year was that we started seeing large numbers of failed ssh logon attempts, with the new twist that the same user would be trying to log on a large number of times, but never from the same place twice in rapid succession. This log data sample will give you an idea. The data will show you the pattern, as will the summary article.
Back then in January, my conclusion was that we had seen the conclusion of a largely failed experiment. After all, proceeding at a truly glacial pace and decreasing the number of attempts per user name as they went along hardly seemed like a recipe for success even if they could sneak past packet filtering based defenses.
It turns out I was wrong. Returning to my log summaries after taking a few days off from log data (yes, easter is big here, even moves geeks sometimes to take time off) I find data with almost exactly the same profile as last December's series.
This can only mean that there were enough successful attempts at guessing people's weak passwords in the last round that our unknown perpetrators found it worthwhile to start another round. For all I know they may have been at it all along, probing other parts of the Internet far from my unfashionable backwaters.
Anyway, they are most certainly back. A summary of the data so far follows in the concluding section. Please note that I would be very interested in communicating with other parties who see similar activity in their logs, and if you are responsible for one or more of the systems identified or you know who is, I urge you to take appropriate action.
The Data So Far
First, We Take root
The full log data reveal that they started rubbing up against my listening posts during the early hours of April 7th 2009 (as measured in CEST), concentrating first on the user root (attempts sorted by hostname here, hostnames here. Not terribly surprising that they tried to take on this one, as most Unix(ish) systems tend to at least have a root account.
Then We Take admin
After about twenty-four hours, the zombies in the cloud tired of root and turned their collective attention to admin (attempts sorted by hostname here, hostnames here. This is a user name I would not expect to be on a Unix syste by default. In my limited experience Microsoft systems tend to use this one more often, but then there may be enough of those systems out there with a SSH service and weak passwords out there to try.
And james Will Make Us Famous
Well into civilized lunchtime (CEST) on April 9th, our would-be guests turned their attention to james (attempts sorted by hostname here, hostnames here. Why this user name received so much attention is a complete mystery to me. They did tire in the end, though.
Before We Turn To 123456, aadi And The Rest Of The Rabble
As if to confirm that we are indeed seeing a rerun of last November and December's sequence of events, the robots started on their regular alphabetic attempts on April 11th:
Apr 11 17:44:15 rosalita sshd[51241]: error: PAM: authentication error for illegal user james from 221.130.177.154
Apr 11 17:50:47 rosalita sshd[51258]: error: PAM: authentication error for illegal user 123456 from 220.194.54.41
Apr 11 17:53:24 rosalita sshd[51262]: error: PAM: authentication error for illegal user 123456 from webmail.sistemafieg.org.br
Apr 11 17:58:25 rosalita sshd[51285]: error: PAM: authentication error for illegal user 123456 from 202.82.25.161
Apr 11 18:00:59 rosalita sshd[51307]: error: PAM: authentication error for illegal user aadi from mail.cgconsultoriacontabil.com.br
The various comments to my earlier columns tended to point out the near-futility of low-intensity bruteforcing, and I took the fact that the last series of attempts did not run to the end of the alphabet as an indication that the perpetrators themselves had reached the same conclusion.
Now we know that they just moved on, went elsewhere for a while and now they're back. In the meantime, they must have hit on enough weak minds and associated weak passwords that they were able to take over and zombiefy enough systems to be worth their while.
The data so far (collected up to April 12, 2009, roughly 21:00 CEST) indicates that the total number of hosts involved in the attempts at my listening posts is just over a thousand (a list of hosts available here).
It is probably worth repeating that in real life, zombies feed on both weak minds and weak passwords.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available for free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
Also worth noting, I will be doing a PF tutorial at BSDCan 2009 in May, and I will be staying around for the rest of the conference. I look forward to seeing you there.
Note: A Better Data Source Is Available
Update 2015-04-02: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Saturday, March 21, 2009
Oh yes, you signed up for this. You did. Honest.
Honesty in marketing. You may have heard of it.
It may come as a surprise to some, but I generally do not spend much time on spam related matters. Occasionally I need to do some manual labor to keep spamd and spamassasin in trim, but at most times my little robot helpers just keep running, leaving my desktop essentially spam free.
That changed slightly late last month. Messages hawking the oddest wares started arriving, with a largish number of messages claiming that I had actually signed up to receive them:
You are receiving this message because on 2/26/2009 at 3:57 PM peter@bsdly.net 64.12.116.10 registered to receive messages from e-researchcouncil.com and its partners. To change your preferences with e-researchcouncil.com, go to the website and select "Contact Us" to review your options.
Note: This piece is also available without trackers but classic formatting only here because a linkedin post by Paul Vixie in July 2025 reminded me I had written this way back when.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive or non-trivial assistance, please contact me (via email or other means) to make arrangements.
Note that the list of greytrapped addresses is updated ten past every full hour, fetching it every minute like some Americans have started doing is not a good use of your resources.
It may come as a surprise to some, but I generally do not spend much time on spam related matters. Occasionally I need to do some manual labor to keep spamd and spamassasin in trim, but at most times my little robot helpers just keep running, leaving my desktop essentially spam free.
That changed slightly late last month. Messages hawking the oddest wares started arriving, with a largish number of messages claiming that I had actually signed up to receive them:
You are receiving this message because on 2/26/2009 at 3:57 PM peter@bsdly.net 64.12.116.10 registered to receive messages from e-researchcouncil.com and its partners. To change your preferences with e-researchcouncil.com, go to the website and select "Contact Us" to review your options.
Note: This piece is also available without trackers but classic formatting only here because a linkedin post by Paul Vixie in July 2025 reminded me I had written this way back when.
To give you an idea how likely that statement is to be true, consider this: The 64.12.116.10 address resolves back to somewhere in America Online's network, pretty much an ocean and then some away from where I'm usually located.
I assume entering my address into a few web forms is somebody's idea of a joke, and the net effect was that a number of spammy messages started appearing in my mailbox, starting on February 27th. Only about third of the messages contained that particular claim, and a typical message would contain headers like these:
X-From-Line: eHarmonyDating@BranchSprint.com Fri Feb 27 16:30:36 2009
Return-path: <3f5.4.73479158-21937306@BranchSprint.com>
Envelope-to: peter@bsdly.net
Delivery-date: Fri, 27 Feb 2009 19:15:13 +0100
Received: from [99.198.152.161] (helo=dns7-cronomagic-biz.BranchSprint.com)
by skapet.bsdly.net with esmtp (Exim 4.69)
(envelope-from <3f5.4.73479158-21937306@BranchSprint.com>)
id 1Ld7Eu-00074N-NF
for peter@bsdly.net; Fri, 27 Feb 2009 19:15:13 +0100
X-Gnus-Mail-Source: pop:peter@bsdly.net
Message-Id: <KKcbjdhdagmcfbVN@BranchSprint.com>
Reply-To: <eHarmonyDating@BranchSprint.com<
From: eHarmonyDating <eHarmonyDating@BranchSprint.com>
Subject: eHarmony could match you with the right singles
Date: Fri, 27 Feb 2009 16:30:36 GMT
X-Information: 73479158_21937306 ListZA251
X-Complaints-To: <complaints@BranchSprint.com>
To: <peter@bsdly.net>
My first impulse was, in case this is an honest mistake somewhere, let's try and play nice at first. That meant sending messages to the X-Complaints-To: addresses and waiting to see what would happen.
You should not be terribly surprised to hear that those addresses all turned out to be invalid, the messages undeliverable.
In the meantime, I went on collecting messages, and the amount of data I had accumulated was large enough that I could reach some preliminary conclusions.
It's obvious that in order to reach me, the messages would need to clear greylisting and avoid triggering too many of my spamassassin rules. That meant in turn that the messages were sent using real mail servers. So I started collecting the messages with that claim for further study. The messages were almost all sent from a few distinct subnets, all of them apparently fairly well stocked with real mailservers.
Based on data from the spam messages and whois lookups and the larger groupings of messages, the professional spammers are, for your convenience in case you want to visit them:
NN, LLC
4001 Kennett Pike
Suite 134-910
Greenville, DE 19807
US
Spiesigma PLC
P.O. BOX 243, 2221 S Webster Ave
Green Bay, WI 54301
US
GreenButtonMedia.com
5580 La Jolla Blvd # 73
La Jolla, CA 92037
US
AdSelectMedia.com
5482 Wilshire Blvd. #302
Los Angeles, CA 90036
US
BestOnlineGreetings.com
5482 Wilshire Blvd. #302
Los Angeles, CA 90036
US
MyPromotionNetwork.com
970 West Valley Parkway
Suite 604
Escondido, CA 92025
GreatTechsOnline.com
5580 La Jolla Blvd # 73
La Jolla, CA 92037
US
CrownVenturesMedia.com
7127 Hollister Ave., Suite 25A, #145
Goleta, CA 93117
Top Notch Media, Inc.
1735 Market Street · Suite A · PMB 429
Philadelphia, PA 19103-7588
In addition, some of the domain names used in the spam messages were registered via an anonymizing service whose whois data comes out as:
Dynamic Dolphin Privacy Protect
5023 W 120th Ave #233
Broomfield
null,80020
The spam volume from all of them swelled at roughly the same time, so it is likely that they cooperate on keeping their lists up to date.
So we see spammers evolving: They buy or rent real mail servers now and they have even started coordinating. Using greylisting has actually increased the cost of becoming a successful spammer.
At our end of the game, we stay ahead of their game thanks to tools like spamd, and several of us dump and share our greytrap lists. It is even possible to collect IP addresses and feed a large number at the time to spamdb, but after a little while I grew tired of the increased manual work decided it was time for a counterprank. Cleaning up after spammers is no fun, unless you can have little robot helpers do the heavy lifting.
The Counterprank: A Feedback Loop
Regular readers will remember that I have a collection of known bad addresses in my domains that I use for my greytrapping, all generated elsewhere, that has come in handy at times. Run of the mill spam operators tend to just suck in anything that looks like email addresses, and keeping the list available on the web has served us extremely well here.
The professional spammers are apparently not quite that stupid, so the problem became a little different. They were able to sneak past greylisting and conventional content filtering. Also, they were apparently oblivious to email communication and as far as I can tell their Unsubscribe pages are not entirely believeable.
So it was a relief to find that places such as http://e-researchcouncil.com/ are very happy to accept any email address you can come up with. Time to enlist a few of our imaginary friends, drawn from the obvious source.
I did ponder the ethics for a few moments. After all, the forms included sentences such as "I certify that I am a US citizen", which was about as true as the assertion that I had signed up via an AOL proxy. But I did not ponder that matter for long. Moments later, most of the spam operators found themselves with new neighbors with odd names and foreign email addresses.
The net result is that the hosts start appearing automagically in the hourly dump of my list of greytrapped addresses and in the daily spamd activity report. With a little luck, we have succeeded in increasing the cost of spamming one tiny increment.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive or non-trivial assistance, please contact me (via email or other means) to make arrangements.
Note that the list of greytrapped addresses is updated ten past every full hour, fetching it every minute like some Americans have started doing is not a good use of your resources.
Saturday, March 14, 2009
What have the black boxes wrought
How compartmentalization turned into a security disaster. Greed, incompetence and dishonesty was involved.
IT security or the lack of any such thing has grabbed headlines lately here in Norway. A series of high profile public institutions have seen large scale worm infections on their Microsoft based networks.
Late last year the regional government agency responsible for essentially all health care in the western part of the country had a worm infection so bad they essentially shut down their network as a preventive measure.
During the last few weeks, the national police force, of all thinkable organizations, has seen not one, but two large-scale incidents.
Use of Microsoft products and sloppy system maintenance are both pervasive enough that similar incidents are likely happening right now elsewhere, somewhere near you too.
The news reports about the Norwegian police force's IT problems contained one item that was particularly shocking to IT types like me:
Apparently large parts of the bureaucracy that is responsible for the confidential and correct processing of criminal matters and all sorts of sensitive personal information associated with the crimes runs essential services on Microsoft Windows NT 4.0.
That version of the Microsoft product is so old it is officially abandonware, and early reports of the police network problems included the oldish news that even the antiware vendors have stopped supporting the system.
Later reports had police IT department officials claim that the worm infections were not that much of a security problem, since at this point all the worm actually did was spread.
Break out the popcorn, boys and girls: In an upcoming episode, we will see how the worm infected Windows machines the Royal Norwegian Police did not find or couldn't clean well enough are used in the perpetration of some cybercrime or other.
It's all pretty sickening, and at this point it would be rather tempting to spend the rest of the column ranting about the general stupidity of Windows users. But a smarter approach is to see if there is a lesson to be learned.
To do that, we need to backtrack quite a bit and look at the cult of the little black boxes.
The cult of the little black boxes, and Microsoft the 1980s corporation
We need to go back and take in what the world was like in the nineteen-eighties. This was back when the world was divided into real computers (from the likes of IBM, Digital, and regional quasi-monopolies like our own Norsk Data) and those annoying toys called 'personal' microcomputers, where the 'IBM PC compatibles' had emerged as the surprise leader of the pack.
Computer networks were usually private, corporate things and rarely interconnected much, with the rare exception of those institutions that were part of the US Department of Defense science experiment that was becoming known variously as ARPANET or 'the Internet'.
If you took your programming classes back in the nineteen-eighties, you likely know that we were taught that black boxes were good.
Compartmentalization was the order of the day, meaning that as a developer you were supposed to create code with a well defined inteface, so anybody interacting with your code would be presented with a cleanly predictable result to any given input.
Your code should be a black box and for any particular specificiation there could be written several interchangeable modules to fit the bill.
So far so good, predictability is nice and with compartmentalization comes, we hope at least, clear chains of responsibility.
But several factors combined to take the cult of the black boxes and turn it into a corporate culture at a corporation that was growing from a handful of furry hackers into a corporation at the time, namely Microsoft.
Microsoft' early successes all came from writing software for single-user systems that were so limited that working around the limitations of the hardware became much of a lifestyle.
At the start of the decade, networking on microcomputers in Microsoft's range was pretty much out of the question, and by the end of the eighties any sort of connectivity (even dial-up) was still an expensive optional extra that was still too hard to configure for most people.
On the storage side, we progressed from 128 kilobyte floppies to hard drives of just over a hundred megabytes for personal systems, with the 32 megabyte partition size still a very present limiting factor.
Amazing developments all, but both the applications and the organization grew faster than the hardware could keep up with.
The organization now had several levels of management, and each one demanded maximum productivity from their underlings.
Keeping in mind that each programmer or team would be writing little black boxes anyway, it made perfect sense to set up the software production line so each developer only had access to those parts of the system he or she was supposed to be working on.
That way developers would be concentrating on their main task and minimize time spent waiting for compiles to finish. At predetermined times the developers would then upload the source code for their little black boxes to a central build system.
The only people who had all the parts of the projects were in fact the custodians of the build system.
Source code version control systems were made part of the process, but there is anecdotal evidence that the transition from standalone hacking to a version control regime was a rough one for many early Microsoft developers.
Only a few days ago I offered pretty much the content of the last few paragraphs to a table of youngish free software developers over beer.
The reaction was quick an unanimous:
"That way, nobody really knows what's going on in the software".
That is a very valid point, and it proves how far we've come with free software.
At the same time there is every reason to believe that the extreme compartmentalization that Microsoft established for its product development in the 1980s was the way things were done there until very recently, if indeed it has changed at all.
By the mid-1990s when Microsoft had been dragged kicking and screaming into modern-day network environments, and the ongoing saga of internet-enabled malware started in earnest (I've written a summary in a malware paper -- an updated version is available here), with the company moving from early denial of any bugs whatsoever through a near-constant barrage of emergency hotfixes to today's monthly megapatch regime.
With the source code still a closely guarded (if occasionally leaked) secret, there is really no way for us to know if they've learned any lessons at all.
One indication that they still have some way to go is this Infoworld article about the state of their protocol documentation (summary: it's not to be trusted at all). As for the state of the source code, all we can do is to study the flow of urgent patches.
Much better then to learn how it should be done - Say from Theo de Raadt's AsiaBSDCon 2009 presentation about how OpenBSD's release process works, and if you want more of the gory details, do check his classic exploit mitigation presentation. Also, most likely you could do worse than read Damien Miller's OpenSSH security presentation (full text here).
It's all those little things we do, at FreeCode and in free software in general.
If you found this column useful, entertaining or irritating, please drop me a line.
Subscribe to:
Posts (Atom)