Good netizenship sometimes comes with a green tinge.
Taking in my daily Linuxtoday dose this morning, there was one item grabbed that my attention, with the headline "Botnets and You: Save the World--Install Linux", and the Linuxtoday entry in turn points to Ross Brunson's blog post with the same title. Do click the link to Ross' blog, it's well worth reading.
What I particularly like about the piece is that he makes the point that you can actually make a difference. More specifically, if you run Linux (being a Novellian, he naturally recommends SLES or SLED) and eliminate Microsoft from your system, you are not only gaining for yourself a safer and more reliable platform, you are also helping everybody else by making the probability of your machine ever joining a botnet a lot smaller.
As regular readers here will recognize, I rate being a good netizen (aka net citizen) as extremely important. Let others get on with their business while we tend to our own tasks, not interfering unless we really have to. If you opt to run your day to day business on the same software your machine most likely came with, the likelihood that somebody else will be taking control of your machine and using it for less than desirable purposes is in fact anything but negligible. I could have used stronger words ("reckless endangerment" comes to mind), but then Redmondians would have just shut off all remnants of rationality. I have argued earlier (article in Norwegian only, sorry) that a computer owner's responsibility should be roughly on par with a dog owner's, but it's possible I should return to that in a future column. And besides, any Linux I've touched for the last ten years is easier to install and operate than the Microsoft offering.
If you followed the Linuxtoday link earlier, you know that I could not resist making the suggestion there that it is in fact possible to be an even better netizen. As outlined in an earlier column (and its followups), if you do your greytrapping properly, you can keep the bad guys occupied and have fun at the same time, consuming next to no resources locally. How's that for green computing.
For example, the crew who started sending messages with headers like
From: "Mrs Maria Jose" <cordinator@euros-ukolympics.com>
Subject: Immediate Response Required.(Euro Award)
to various spamtrap addresses on May 15th are still patiently trying to deliver. A very superficial log analysis shows that there were originally four hosts sending those messages from the 217.70.178.0/24 network. There appears to be only one left now, but collectively these machines have so far made 476,787 attempts at delivery to my data collection points. Judging from a sample of some 21,000 connections from one of the hosts, the average connection time was 389.68 seconds, which in turn means that we've had those spam senders waste approximately 185792441 seconds, or time equal to 5.89 years.
Not bad in a little more than a month. On the downside, the predictions that spambots would sooner or later learn to do things in parallel have been proved true. My logs indicate that the current crop is able to handle at least sixty simultaneous delivery attempts. Even bogged down by a suboptimal operating system at the sending end, modern desktop computers are in fact powerful beasts. In my book it's just good netizenry to set up a machine to keep the garbage they send off your own network, and by extension off others since they don't get around to try delivering to others. By the way, that list is now almost 15,000 addresses long, all non-deliverable garbage. You could be excused for thinking it a twisted art project.
Wednesday, June 25, 2008
Tuesday, June 17, 2008
BSD Unix? That's purely historical
If you've ever bought something that ended up disappointing you to the point where you wanted to yell at somebody, you will recognize the frame of mind I was in after reading the book I ended up reviewing. With perfect hindsight I should of course have smelled the rat - anything written in this century about "BSD UNIX" is either a retrospective or ill informed. If you don't fancy a book review, tune in next time for something completely different instead.
Book Review: BSD Unix Toolbox
I came across this title while browsing an online bookstore for possible supporting literature for a course I'm planning. The teaser text boasts "1000+ Commands for FreeBSD, OpenBSD and NetBSD", and with a 2008 publication date I thought this one was definitely worth checking out.
At roughly 300 pages, covering 3-4 commands per page usefully would be a tall order for anyone, so I was a little surprised to find that the book sets aside two chapters to preliminaries, first "Starting with BSD systems" with a brief and not very complete overview of BSDish systems and some pointers to online resources, before moving on to an entire chapter on installing FreeBSD.
In a book that's supposedly about more than a thousand useful commands on FreeBSD, OpenBSD and NetBSD, setting aside an entire chapter to a rather superficial description of how to install one of the systems seems to me a very odd choice. Odder still, how to install either OpenBSD or NetBSD is not covered at all. Now installing any of those systems is not in fact too difficult, and I for one do not think the world needs yet another walkthrough of FreeBSD's or the other systems' install process. In my view, it would have been better if the authors had concentrated on getting around to describing those 1000+ commands, the sooner the better.
In the installation chapter, which also covers the ports and packages system, the authors seem to be unaware that each system has its own variety, and that on NetBSD the system goes under a different name. As the chapter title implies, this chapter is clearly FreeBSD specific, and would have been a lot more useful if the authors had noted at least some of the significant differences that exist between the systems the book sets out to cover.
Probably the most useful chapters in the book are chapters 3 through 6, where essentially all information is likely to be portable to all covered systems. However, Chapter 3, "Using the shell", is not without its oddities: The authors seem to assume that the user has installed Gnome as the preferred desktop environment and covers mainly using Bash as the shell. That is a slightly odd choice since to my knowledge Bash is not the default shell on any of the BSDs, but available as an optional extra through the package system.
Chapter 4, "Working with Files", walks through the basics of file types, file permissions, file system operations and commands such as cp, file, mount and a few others. After reading the chapter, you will be aware that these commands exist, if not much else.
Chapter 5, "Manipulating Text", offers the briefest treatment I've ever seen of regular expressions, mentions in passing vi and emacs as possible tools and then moves on to describing what appears to be the authors' favorite text editors (joe, nano and pico, neither of those are in the base system anywhere) and a brief mention of some graphical tools. The chapter then offers samples of using cat, head, tail, more, less, pr, grep, wc, sort, strings, sed, tr, diff, sdiff, awk, cut, od and finally unix2dos. Again, after reading the chapter you will be aware that the commands exist, but you will be looking elsewhere for detailed information.
Large chunks of Chapter 6, "Playing with Multimedia" would be useful on most unixlike systems (oggenc and convert likely perform much the same anywhere), but once again what little is offered as tips for getting sound or other functionality to work on your system is strictly FreeBSD-specific.
In Chapter 7, "Administering File Systems" the perspective is again distinctively FreeBSD-centric with little or no note of even potential differences across systems. For a FreeBSD user it may offer a useful if very brief and shallow walkthrough, though.
Chapter 8, "Backups and Removable Media", shows some examples of tar, gzip and rsync use, sometimes in combination, supplemented with brief mention of some common and less common file compression tools. The removable media section covers CD burning with cdrtools in more detail than most other software mentioned in this book, but fail to mention useful tidbits such as how to use OpenBSD's cdio command (which is in base) for similar tasks. In a book that claims to be up to date as of 2008, I find that a very curious omission.
Chapter 9, "Checking and Managing Running Processes" does little more than mention the names of some process management relevant commands such as nice, renice, fg, bg, kill and killall in passing before delving into a surprisingly detailed walkthrough of ps. It proceeds to describing top, pgrep, fuser (which the user is instructed to install via pkg_add), then returns to nice and renice and offering a couple of examples of fg and bg use before really picking up speed with kill and killall (fortunately with a list of signals), background processes started via either nohup or by appending an & character, and spending a couple of sentences each on at, gatch, atq, atrm and crontab.
Chapter 10, "Managing the System", weighs in at a little more than 20 pages, and characteristically slips back into FreeBSD-centric mode where it offers much detail at all. For some reason this is where the instructions about setting up your system for booting several operating systems turns up, along with a description of using GRUB as your boot loader.
Chapter 11, "Managing Network Connections" is again quite FreeBSD centric even if it does rattle off the names of the others at apparently random intervals. The information is as far as I can tell mostly correct for FreeBSD and some, if not all, commands will work elsewhere, but superficial enough that a user will have to turn elsewhere for help in resolving any problems that turn up.
Chapter 12, "Accessing Network Resources" covers anything from browsing the web (the authors state confidently that lynx 'has been supplanted [...] by the links browser, which was later replaced by elinks', apparently unaware that in OpenBSD at least, lynx is in fact part of the base system), fetching files with wget, curl, lftp (curiously recommending lftp even though in at least OpenBSD the base system's ftp client offers essentially all the required functionality), before spending four pages doing some handwaving about how to set up samba. IRC and mail clients are also mentioned, but I was a bit surprised that 'managing mail' apparently does not even touch on running a mail service.
Chapter 13, "Doing Remote System Administration", covers ssh, screen, tsclient (Gnome's windows remote desktop client), xhost, vnc and vino (another Gnome applet, this one for sharing your Gnome desktop) in that order, none of them in any great detail, bringing to mind the mantra 'now at least you know the commands exist'.
Chapter 14, "Locking Down Security", sprints through the basics of user and groups administration on FreeBSD, moves on to some tips about running services in general and via inetd, and also mentions firewalls.
The section "Configuring the Built-In Firewall" has me really baffled. The authors claim not only that ipfw has been ported to NetBSD and OpenBSD (where PF, mentioned here only as PacketFilter, has been the only packet filter since 2001), but the online reference they give (http://www.phildev.net/ipf) actually points to information about Darren Reed's IPFilter, also known as IPF.
It is unclear which firewall the authors think they have configured, but the actual rule sets they offer are ipfw scripts. It is likely that a user trying to run with the rc.conf snippet supplied and the rule set would in fact end up with both ipf and ipfw enabled, but likely with no working packet filtering (and depending on how the kernel with ipf and ipfw was compiled, the configuration would be either completely open or completely shut, another point apparently unknown or considered irrelevant by the authors). After a few examples of ipfw operations, the chapter then moves on to mention that yes, you can actually input your own information into the system logs, before recommending that you set up a centralized syslog server and instructing you to look into the third-party tools tripwire and chkrootkit.
The three appendixes have reference-style information (finally!) about using vi or vim, shell special characters and variables, and 'personal configuration files', aka dotfiles. All very brief, of course.
Unfortunately, this is a book I can not recommend. Large chunks of it or something very similar is available elsewhere, some of it for free and reasonably well written. If you're a FreeBSD user you will find yourself looking up the topics in the Handbook anyway, if you are a NetBSD or OpenBSD user, the relatively platform independent parts are addressed equally well in several 'Linux' books and other online resources.
There may in fact be more than a thousand monospace type 'command' examples in the book (I never counted), but other than that this title fails to live up to the expectations set up by the cover and other marketing. The book goes to some length to give the impression that it's current, with all dates I could see in examples set somewhere in the second half of 2008, but the authors appear to have been working with FreeBSD 6.3 as their current version.
The relatively frequent mention of 'BSD distributions' - a term that never entered the BSD vocabulary, mainly because the BSDs are maintained as separate systems - and various odd details such as the plainly wrong firewalls examples makes me suspect strongly that much of the material is in fact warmed over from a Linux book, slightly edited where the authors thought it was necessary. Unfortunately, they missed more than a few spots and for whatever reason the fact checking and testing did not get all the attention it should have. The result is a book that is very superficial where it's right and has enough spots that are anything from misleading to downright wrong that it gets rather irritating to an experienced reader. I can only imagine how frustrating it would be to use this as a resource for learning.
For a long time *BSD user (yes, all three), it seemed like a nice surprise that Wiley had discovered that there is such a thing as a BSD market. After a few hours looking at this entry, I hope they take the task a little more seriously the next time they try offering to sell us BSD literature.
Book info:
Title: BSD UNIX Toolbox: 1000+ Commands for FreeBSD, OpenBSD and NetBSD
Authors: Christopher Negus and Francois Caen
Publisher: Wiley Publishing, Inc. (Indianapolis)
Copyright: 2008
ISBN: 978-0-470-37603-4
Book Review: BSD Unix Toolbox
I came across this title while browsing an online bookstore for possible supporting literature for a course I'm planning. The teaser text boasts "1000+ Commands for FreeBSD, OpenBSD and NetBSD", and with a 2008 publication date I thought this one was definitely worth checking out.
At roughly 300 pages, covering 3-4 commands per page usefully would be a tall order for anyone, so I was a little surprised to find that the book sets aside two chapters to preliminaries, first "Starting with BSD systems" with a brief and not very complete overview of BSDish systems and some pointers to online resources, before moving on to an entire chapter on installing FreeBSD.
In a book that's supposedly about more than a thousand useful commands on FreeBSD, OpenBSD and NetBSD, setting aside an entire chapter to a rather superficial description of how to install one of the systems seems to me a very odd choice. Odder still, how to install either OpenBSD or NetBSD is not covered at all. Now installing any of those systems is not in fact too difficult, and I for one do not think the world needs yet another walkthrough of FreeBSD's or the other systems' install process. In my view, it would have been better if the authors had concentrated on getting around to describing those 1000+ commands, the sooner the better.
In the installation chapter, which also covers the ports and packages system, the authors seem to be unaware that each system has its own variety, and that on NetBSD the system goes under a different name. As the chapter title implies, this chapter is clearly FreeBSD specific, and would have been a lot more useful if the authors had noted at least some of the significant differences that exist between the systems the book sets out to cover.
Probably the most useful chapters in the book are chapters 3 through 6, where essentially all information is likely to be portable to all covered systems. However, Chapter 3, "Using the shell", is not without its oddities: The authors seem to assume that the user has installed Gnome as the preferred desktop environment and covers mainly using Bash as the shell. That is a slightly odd choice since to my knowledge Bash is not the default shell on any of the BSDs, but available as an optional extra through the package system.
Chapter 4, "Working with Files", walks through the basics of file types, file permissions, file system operations and commands such as cp, file, mount and a few others. After reading the chapter, you will be aware that these commands exist, if not much else.
Chapter 5, "Manipulating Text", offers the briefest treatment I've ever seen of regular expressions, mentions in passing vi and emacs as possible tools and then moves on to describing what appears to be the authors' favorite text editors (joe, nano and pico, neither of those are in the base system anywhere) and a brief mention of some graphical tools. The chapter then offers samples of using cat, head, tail, more, less, pr, grep, wc, sort, strings, sed, tr, diff, sdiff, awk, cut, od and finally unix2dos. Again, after reading the chapter you will be aware that the commands exist, but you will be looking elsewhere for detailed information.
Large chunks of Chapter 6, "Playing with Multimedia" would be useful on most unixlike systems (oggenc and convert likely perform much the same anywhere), but once again what little is offered as tips for getting sound or other functionality to work on your system is strictly FreeBSD-specific.
In Chapter 7, "Administering File Systems" the perspective is again distinctively FreeBSD-centric with little or no note of even potential differences across systems. For a FreeBSD user it may offer a useful if very brief and shallow walkthrough, though.
Chapter 8, "Backups and Removable Media", shows some examples of tar, gzip and rsync use, sometimes in combination, supplemented with brief mention of some common and less common file compression tools. The removable media section covers CD burning with cdrtools in more detail than most other software mentioned in this book, but fail to mention useful tidbits such as how to use OpenBSD's cdio command (which is in base) for similar tasks. In a book that claims to be up to date as of 2008, I find that a very curious omission.
Chapter 9, "Checking and Managing Running Processes" does little more than mention the names of some process management relevant commands such as nice, renice, fg, bg, kill and killall in passing before delving into a surprisingly detailed walkthrough of ps. It proceeds to describing top, pgrep, fuser (which the user is instructed to install via pkg_add), then returns to nice and renice and offering a couple of examples of fg and bg use before really picking up speed with kill and killall (fortunately with a list of signals), background processes started via either nohup or by appending an & character, and spending a couple of sentences each on at, gatch, atq, atrm and crontab.
Chapter 10, "Managing the System", weighs in at a little more than 20 pages, and characteristically slips back into FreeBSD-centric mode where it offers much detail at all. For some reason this is where the instructions about setting up your system for booting several operating systems turns up, along with a description of using GRUB as your boot loader.
Chapter 11, "Managing Network Connections" is again quite FreeBSD centric even if it does rattle off the names of the others at apparently random intervals. The information is as far as I can tell mostly correct for FreeBSD and some, if not all, commands will work elsewhere, but superficial enough that a user will have to turn elsewhere for help in resolving any problems that turn up.
Chapter 12, "Accessing Network Resources" covers anything from browsing the web (the authors state confidently that lynx 'has been supplanted [...] by the links browser, which was later replaced by elinks', apparently unaware that in OpenBSD at least, lynx is in fact part of the base system), fetching files with wget, curl, lftp (curiously recommending lftp even though in at least OpenBSD the base system's ftp client offers essentially all the required functionality), before spending four pages doing some handwaving about how to set up samba. IRC and mail clients are also mentioned, but I was a bit surprised that 'managing mail' apparently does not even touch on running a mail service.
Chapter 13, "Doing Remote System Administration", covers ssh, screen, tsclient (Gnome's windows remote desktop client), xhost, vnc and vino (another Gnome applet, this one for sharing your Gnome desktop) in that order, none of them in any great detail, bringing to mind the mantra 'now at least you know the commands exist'.
Chapter 14, "Locking Down Security", sprints through the basics of user and groups administration on FreeBSD, moves on to some tips about running services in general and via inetd, and also mentions firewalls.
The section "Configuring the Built-In Firewall" has me really baffled. The authors claim not only that ipfw has been ported to NetBSD and OpenBSD (where PF, mentioned here only as PacketFilter, has been the only packet filter since 2001), but the online reference they give (http://www.phildev.net/ipf) actually points to information about Darren Reed's IPFilter, also known as IPF.
It is unclear which firewall the authors think they have configured, but the actual rule sets they offer are ipfw scripts. It is likely that a user trying to run with the rc.conf snippet supplied and the rule set would in fact end up with both ipf and ipfw enabled, but likely with no working packet filtering (and depending on how the kernel with ipf and ipfw was compiled, the configuration would be either completely open or completely shut, another point apparently unknown or considered irrelevant by the authors). After a few examples of ipfw operations, the chapter then moves on to mention that yes, you can actually input your own information into the system logs, before recommending that you set up a centralized syslog server and instructing you to look into the third-party tools tripwire and chkrootkit.
The three appendixes have reference-style information (finally!) about using vi or vim, shell special characters and variables, and 'personal configuration files', aka dotfiles. All very brief, of course.
Unfortunately, this is a book I can not recommend. Large chunks of it or something very similar is available elsewhere, some of it for free and reasonably well written. If you're a FreeBSD user you will find yourself looking up the topics in the Handbook anyway, if you are a NetBSD or OpenBSD user, the relatively platform independent parts are addressed equally well in several 'Linux' books and other online resources.
There may in fact be more than a thousand monospace type 'command' examples in the book (I never counted), but other than that this title fails to live up to the expectations set up by the cover and other marketing. The book goes to some length to give the impression that it's current, with all dates I could see in examples set somewhere in the second half of 2008, but the authors appear to have been working with FreeBSD 6.3 as their current version.
The relatively frequent mention of 'BSD distributions' - a term that never entered the BSD vocabulary, mainly because the BSDs are maintained as separate systems - and various odd details such as the plainly wrong firewalls examples makes me suspect strongly that much of the material is in fact warmed over from a Linux book, slightly edited where the authors thought it was necessary. Unfortunately, they missed more than a few spots and for whatever reason the fact checking and testing did not get all the attention it should have. The result is a book that is very superficial where it's right and has enough spots that are anything from misleading to downright wrong that it gets rather irritating to an experienced reader. I can only imagine how frustrating it would be to use this as a resource for learning.
For a long time *BSD user (yes, all three), it seemed like a nice surprise that Wiley had discovered that there is such a thing as a BSD market. After a few hours looking at this entry, I hope they take the task a little more seriously the next time they try offering to sell us BSD literature.
Book info:
Title: BSD UNIX Toolbox: 1000+ Commands for FreeBSD, OpenBSD and NetBSD
Authors: Christopher Negus and Francois Caen
Publisher: Wiley Publishing, Inc. (Indianapolis)
Copyright: 2008
ISBN: 978-0-470-37603-4
Wednesday, June 4, 2008
More than 40,000 served
Today's blog comes to you from sunny Aalborg in northern Denmark, where our Danish friends had the good sense to put together a one-day conference. Go to the web site at http://www.foss-aalborg.dk/ for details of the programme, I certainly hope the organizers will start a tradition and put on another conference soon.
For my own part, the PF tutorial (the free predecessor to the book), saw its confirmed unique visitor number forty thousand today, apparently a user somewhere in Ukraine:
$ ./mystats.sh
Wed Jun 4 14:57:26 CEST 2008
Total PF tutorial visitors : 40000
I promise I won't bother you with updates of the number of visitors again until we reach fifty thousand. I'll do my very best to have produced some other interesting material before then.
For my own part, the PF tutorial (the free predecessor to the book), saw its confirmed unique visitor number forty thousand today, apparently a user somewhere in Ukraine:
$ ./mystats.sh
Wed Jun 4 14:57:26 CEST 2008
Total PF tutorial visitors : 40000
I promise I won't bother you with updates of the number of visitors again until we reach fifty thousand. I'll do my very best to have produced some other interesting material before then.
Sunday, May 25, 2008
I challenge your response, backscatterer
A few weeks before I left Datadok, some real user accounts there, including mine, were joejobbed. That is, somebody, somewhere started sending spam with the From: or return address set to a real, live mail account in one of our domains. There were several incidents, the backscatter output of one of the more recent ones is preserved here.
Of course there were a handful of the "please prove to me that you are a human" messages from challenge-response systems too, but more about that later.
I normally keep the composed, sceptical attitude you will have come to expect from system administrators, but at fairly random intervals I turn into a regular Goody Two-Shoes. So faced with a collection of bounces for messages I certainly did not send, I decided to take some time to contact the people responsible for the machines that sent the spam. Owners of spam sending machines (as distinct from their Pwners) generally are not aware what the machines are doing, and at times I turn into that soft-hearted guy who just wants to help.
As you will see from the collection, not all bounces contain enough of the original message to track down where the spam was actually sent from. Then there were others where it was possible to dechipher where the message came from, and I made a canned message and sent it off to whatever addresses seemed likely from the whois output.
You should not be surprised to hear that quite a handful of those produced regular bounces (yes, postmaster@ and several other RFC2142 mandated mailboxes are non-existent at several sites), but what struck me was that of those that did not bounce there were several that instead produced "Hi, I'm the challenge/response robot at site.nx, please follow my instructions to prove you are human" responses.
Seriously, folks, does anybody in this day and age actually believe that these systems work? It's possible that challenge-response adherents' claims that their system stops all incoming spam are true, but seen from the outside, the only guaranteed effect is that your site will produce backscatter, and lots of it. It would perhaps have been interesting to know just how many of the undeliverables I've squirted into my spammer bait list were actually produced by challenge-response systems, but unfortunately, we have no way of knowing.
There are a few other things that challenge-response systems do not protect you against, such as plain old stupidity and malicious clicking by those who receive your challenge-response backscatter. I take the spam bounces in my collection as proof of that.
So challenge-response people, yes, I agree that you had no way of knowing whether Hampe Ivancevic, Hamada Kirkegaard, Caixia Kluge, Bitte Kutschinski, Anant Johaneson, Sumol Przygocki or a few others actually worked for me at the time you sent your challenge. I will however promise those fine people that if they exist, if they have the right qualifications and I am in a position to offer them employment, I will consider their applications in all seriousness.
The rant about certain blacklists and certain unwise ways to use them will have to wait until later. There are other deadlines to keep, the house still needs the odd splash of paint, but I'll catch up with you all later.
Of course there were a handful of the "please prove to me that you are a human" messages from challenge-response systems too, but more about that later.
I normally keep the composed, sceptical attitude you will have come to expect from system administrators, but at fairly random intervals I turn into a regular Goody Two-Shoes. So faced with a collection of bounces for messages I certainly did not send, I decided to take some time to contact the people responsible for the machines that sent the spam. Owners of spam sending machines (as distinct from their Pwners) generally are not aware what the machines are doing, and at times I turn into that soft-hearted guy who just wants to help.
As you will see from the collection, not all bounces contain enough of the original message to track down where the spam was actually sent from. Then there were others where it was possible to dechipher where the message came from, and I made a canned message and sent it off to whatever addresses seemed likely from the whois output.
You should not be surprised to hear that quite a handful of those produced regular bounces (yes, postmaster@ and several other RFC2142 mandated mailboxes are non-existent at several sites), but what struck me was that of those that did not bounce there were several that instead produced "Hi, I'm the challenge/response robot at site.nx, please follow my instructions to prove you are human" responses.
Seriously, folks, does anybody in this day and age actually believe that these systems work? It's possible that challenge-response adherents' claims that their system stops all incoming spam are true, but seen from the outside, the only guaranteed effect is that your site will produce backscatter, and lots of it. It would perhaps have been interesting to know just how many of the undeliverables I've squirted into my spammer bait list were actually produced by challenge-response systems, but unfortunately, we have no way of knowing.
There are a few other things that challenge-response systems do not protect you against, such as plain old stupidity and malicious clicking by those who receive your challenge-response backscatter. I take the spam bounces in my collection as proof of that.
So challenge-response people, yes, I agree that you had no way of knowing whether Hampe Ivancevic, Hamada Kirkegaard, Caixia Kluge, Bitte Kutschinski, Anant Johaneson, Sumol Przygocki or a few others actually worked for me at the time you sent your challenge. I will however promise those fine people that if they exist, if they have the right qualifications and I am in a position to offer them employment, I will consider their applications in all seriousness.
The rant about certain blacklists and certain unwise ways to use them will have to wait until later. There are other deadlines to keep, the house still needs the odd splash of paint, but I'll catch up with you all later.
Wednesday, May 21, 2008
Fake Address Round Trip Time: 13 days
The results are in. Our adversaries really are mindless automata.
Regular readers will have noticed that I've been running a small scale experiment over the last few months, feeding one spammer byproduct back to them via a reasonably accessible web page. The hope was that I would learn a few things about spammer behavior in the process.
After collecting fake addresses in my domains generated elsewhere for a while, I started noticing that a short time after I'd put addresses on the traplist page, they would start appearing as To: addresses on what appeared to be spam message entries in my logs. After a while more, I was certain that the round trip time was down to a few days, but my notes did not include exact dates for when each individual harvested address made it into my spamtrap list. Meaning, of course, I had no way of telling just how fast the process is.
Time to cheat slightly, or, as they say in the trade, perform a controlled experiment. Instead of sticking to the original plan of strictly collecting addresses generated elsewhere and feed them back to the harvesters via the web page, I decided to deviate slightly and plant an address at a specified date, and maybe add another few for data points later. I did the obvious thing and on October 11th, 2007, I slipped put-here-2007-10-11@datadok.no into one of that day's batches of collected addresses.
Needless to say, my attention wandered from the project in the meantime. After all, in October there was still that book to finish, and after the book was done, other developments had my attention or at least drained my energy for quite a while. So it was only last Sunday it struck me that by now I should have at least some data on what actually happened. One other reason it was suddenly quite appropriate to sum up the data was that the datadok.no domain had been turned over to new owners, and it will likely move out of my sphere of responsibility in the near future.
So here's the result, the fake address round trip time:
$ grep put-here-2007-10-11@datadok.no /var/log/spamd
Oct 24 03:40:40 skapet spamd[20795]: (BLACK) 60.50.174.129:
<pepgyoygq@boisdelan.com> -> <put-here-2007-10-11@datadok.no>
That is, the first time my artificially inserted address was used as a spam target was thirteen days after I put it on the traplist page. Since then, something, somewhere, has tried
$ grep -c put-here-2007-10-11@datadok.no /var/log/spamd
300
to deliver email to our imaginary friend a total of 300 times. Data taken from the spamd in front of that domain's secondary mail exchanger, of course. As always, I would love to hear from you about any related experiences.
In upcoming columns we will see, er, actually I find myself with such a selection of tempting topics to choose from, it is really hard to decide what to cover next. But the next one will appear here shortly.
Update 2015-08-01: In a totally unrelated article, posted on the afternoon of July 24th, 2015, the string razz@skapet.bsdly.net appeared. it took only three days, two hours and forty-eight minutes (approximately) before my spamd(8) logged this attempt at delivering mail to that address:
Thanks to the script that also prepares the downloadable list of trapped IP addresses, I was alerted of this happening, and the address was duly added to the spamtraps list and the accompanying web page as part of the batch that took the list to its current count of 29135 entries.
From this accidental anecdotal evidence, we can conclude that the time from when random string containing an at sign appears on a web site to the time it's used as spam target has now shrunk to about three days.
The attempts seem to be a little less energetic this time though: greping the relevant logs turns up only 27 attempts at delivery. It's possible this is down to the fact that there are now so many more imaginary friends to choose from in that long list.
Note: The
Regular readers will have noticed that I've been running a small scale experiment over the last few months, feeding one spammer byproduct back to them via a reasonably accessible web page. The hope was that I would learn a few things about spammer behavior in the process.
After collecting fake addresses in my domains generated elsewhere for a while, I started noticing that a short time after I'd put addresses on the traplist page, they would start appearing as To: addresses on what appeared to be spam message entries in my logs. After a while more, I was certain that the round trip time was down to a few days, but my notes did not include exact dates for when each individual harvested address made it into my spamtrap list. Meaning, of course, I had no way of telling just how fast the process is.
Time to cheat slightly, or, as they say in the trade, perform a controlled experiment. Instead of sticking to the original plan of strictly collecting addresses generated elsewhere and feed them back to the harvesters via the web page, I decided to deviate slightly and plant an address at a specified date, and maybe add another few for data points later. I did the obvious thing and on October 11th, 2007, I slipped put-here-2007-10-11@datadok.no into one of that day's batches of collected addresses.
Needless to say, my attention wandered from the project in the meantime. After all, in October there was still that book to finish, and after the book was done, other developments had my attention or at least drained my energy for quite a while. So it was only last Sunday it struck me that by now I should have at least some data on what actually happened. One other reason it was suddenly quite appropriate to sum up the data was that the datadok.no domain had been turned over to new owners, and it will likely move out of my sphere of responsibility in the near future.
So here's the result, the fake address round trip time:
$ grep put-here-2007-10-11@datadok.no /var/log/spamd
Oct 24 03:40:40 skapet spamd[20795]: (BLACK) 60.50.174.129:
<pepgyoygq@boisdelan.com> -> <put-here-2007-10-11@datadok.no>
That is, the first time my artificially inserted address was used as a spam target was thirteen days after I put it on the traplist page. Since then, something, somewhere, has tried
$ grep -c put-here-2007-10-11@datadok.no /var/log/spamd
300
to deliver email to our imaginary friend a total of 300 times. Data taken from the spamd in front of that domain's secondary mail exchanger, of course. As always, I would love to hear from you about any related experiences.
In upcoming columns we will see, er, actually I find myself with such a selection of tempting topics to choose from, it is really hard to decide what to cover next. But the next one will appear here shortly.
Update 2015-08-01: In a totally unrelated article, posted on the afternoon of July 24th, 2015, the string razz@skapet.bsdly.net appeared. it took only three days, two hours and forty-eight minutes (approximately) before my spamd(8) logged this attempt at delivering mail to that address:
Jul 27 20:16:01 skapet spamd[1520]: (GREY) 183.79.28.71: <esther1jomkoma1ej@yahoo.co.jp> -> <razz@skapet.bsdly.net>
Thanks to the script that also prepares the downloadable list of trapped IP addresses, I was alerted of this happening, and the address was duly added to the spamtraps list and the accompanying web page as part of the batch that took the list to its current count of 29135 entries.
From this accidental anecdotal evidence, we can conclude that the time from when random string containing an at sign appears on a web site to the time it's used as spam target has now shrunk to about three days.
The attempts seem to be a little less energetic this time though: greping the relevant logs turns up only 27 attempts at delivery. It's possible this is down to the fact that there are now so many more imaginary friends to choose from in that long list.
Note: The
@datadok.no addresses were removed from the traplist in late 2008 when that domain was handed over to the care of a different organization.
Wednesday, May 14, 2008
Network devices that lie
Do some network devices lie about their config, or are they just talking past each other?
One of my tasks recently has been to start integrating the networks of two companies, because one of the companies bought the other.
At my end you would have guessed already that anything exposed to the Internet would be running OpenBSD, while at the other end they were strangely attached to products from a certain software marketer based in the north-western part of the USA. But for the IPSEC part, they had decided that a certain proprietary "Internet Security Appliance" would fill their needs, and at each of their locations they had identical appliances running for "VPN" purposes.
Therein lies a story. After a few working days of debugging my way to an almost working configuration, my main conclusion is, that it is quite likely that proprietary security appliance lies. That is, it will let its administrator choose a wide range of very specific configuation options, but will actually override them, at least if the device at the other end does not do the "I'm one of your kind" secret handshake.
Here's what it looks like in the logs on the OpenBSD end, the part that is supposed to be a negotiation but actually isn't and the options on the OpenBSD side matched the options given in the screendumps of the other side's GUI that I had to work with:
and so on, through the various options, each time expecting what matched the screendumps, getting something else entirely, up until the inevitable
or as they say elsewhere, Do not pass GO.
Moving on from there to actual tunnel setup reveals troublesome vendor IDs and other strangeness. This could very well be an attempt at vendor lock-in, but then the conventional wisdom anyway seems to be that the only way to set up a VPN is to have identical devices at every endpoint. There is no good reason why it should be like that, but I may be discovering the less good ones.
The main reason I'm not naming names or stating flat-out that "this device lies about its configuration" is that my analysis is not complete yet. If you have data you want to share with me on this or similar experiences, I would love to hear from you, at peter@bsdly.net. It's likely I'll write a followup based on what I hear and what happens.
During my silent period blogging wise, some notable events did occur, and I may return to them in later posts.
UKUUG Spring 2008
In late March I went to Birmingham to attend the UKUUG Sping 2008 conference - with a PF tutorial much like the one in Riga in February and a refreshed malware talk. A number of very good talks, and a number of Perl Mongers there, notably Barbie, who told me about his "Understanding Malware" talk that actually overlaps quite a bit with my paper. The social parts were good too, and yes, in Birmingham it is possible to find Real Ale.
BSD Magazine #1 is out
Not too long after the UKUUG Spring Conference, the first issue of BSD Magazine was published. Readers of this blog may find one article to be slightly familiar, but there's a nice selection of articles and reviews, covering all the BSD systems and a wide range of topics. And I should mention, they are looking for articles for upcoming issues.
OpenBSD 4.3 is out
OpenBSD 4.3 was released on schedule on May 1st, and as usual those who pre-ordered got their copies early. As always, exciting new stuff as well as a number of incremental improvements. There's interesting stuff happening post-4.3 as well, watch undeadly for updates.
Next up: FOSS Aalborg, June 4th
Our Danish friends are kind enough to give us yet another excuse to come visit. A group of BSD and Linux people have been hosting good conferences in Copenhagen for years, last year's EuroBSDCon being one.
This time it's Aalborg's turn, with a one day conference on June 4th. Yours truly will be on the speaker list. I hope to see you there, I will be bringing a small stack of books.
One of my tasks recently has been to start integrating the networks of two companies, because one of the companies bought the other.
At my end you would have guessed already that anything exposed to the Internet would be running OpenBSD, while at the other end they were strangely attached to products from a certain software marketer based in the north-western part of the USA. But for the IPSEC part, they had decided that a certain proprietary "Internet Security Appliance" would fill their needs, and at each of their locations they had identical appliances running for "VPN" purposes.
Therein lies a story. After a few working days of debugging my way to an almost working configuration, my main conclusion is, that it is quite likely that proprietary security appliance lies. That is, it will let its administrator choose a wide range of very specific configuation options, but will actually override them, at least if the device at the other end does not do the "I'm one of your kind" secret handshake.
Here's what it looks like in the logs on the OpenBSD end, the part that is supposed to be a negotiation but actually isn't and the options on the OpenBSD side matched the options given in the screendumps of the other side's GUI that I had to work with:
May 13 14:00:28 delilah isakmpd[13547]: attribute_unacceptable: ENCRYPTION_ALGORITHM: got DES_CBC, expected 3DES_CBC May 13 14:00:28 delilah isakmpd[13547]: attribute_unacceptable: HASH_ALGORITHM: got MD5, expected SHA
and so on, through the various options, each time expecting what matched the screendumps, getting something else entirely, up until the inevitable
May 13 14:00:28 delilah isakmpd[13547]: message_negotiate_sa: no compatible proposal found
or as they say elsewhere, Do not pass GO.
Moving on from there to actual tunnel setup reveals troublesome vendor IDs and other strangeness. This could very well be an attempt at vendor lock-in, but then the conventional wisdom anyway seems to be that the only way to set up a VPN is to have identical devices at every endpoint. There is no good reason why it should be like that, but I may be discovering the less good ones.
The main reason I'm not naming names or stating flat-out that "this device lies about its configuration" is that my analysis is not complete yet. If you have data you want to share with me on this or similar experiences, I would love to hear from you, at peter@bsdly.net. It's likely I'll write a followup based on what I hear and what happens.
During my silent period blogging wise, some notable events did occur, and I may return to them in later posts.
UKUUG Spring 2008
In late March I went to Birmingham to attend the UKUUG Sping 2008 conference - with a PF tutorial much like the one in Riga in February and a refreshed malware talk. A number of very good talks, and a number of Perl Mongers there, notably Barbie, who told me about his "Understanding Malware" talk that actually overlaps quite a bit with my paper. The social parts were good too, and yes, in Birmingham it is possible to find Real Ale.
BSD Magazine #1 is out
Not too long after the UKUUG Spring Conference, the first issue of BSD Magazine was published. Readers of this blog may find one article to be slightly familiar, but there's a nice selection of articles and reviews, covering all the BSD systems and a wide range of topics. And I should mention, they are looking for articles for upcoming issues.
OpenBSD 4.3 is out
OpenBSD 4.3 was released on schedule on May 1st, and as usual those who pre-ordered got their copies early. As always, exciting new stuff as well as a number of incremental improvements. There's interesting stuff happening post-4.3 as well, watch undeadly for updates.
Next up: FOSS Aalborg, June 4th
Our Danish friends are kind enough to give us yet another excuse to come visit. A group of BSD and Linux people have been hosting good conferences in Copenhagen for years, last year's EuroBSDCon being one.
This time it's Aalborg's turn, with a one day conference on June 4th. Yours truly will be on the speaker list. I hope to see you there, I will be bringing a small stack of books.
Sunday, April 13, 2008
Does anybody here remember Artie Eff?
© 2008 Peter N. M. Hansteen
That's what you thought you heard, right? Actually, this is not Uncle BSDly's nostalgia for a backstreet hoodlum he might have known way back when. It's Uncle BSDly trying to point out what happened the last time a standard was entrusted to that little Seattle company called Microsoft. R-T-F. The Rich Text format.
So the question is,
Does anybody here remember RTF?
Note: This piece is also available without trackers but classic formatting only here.
It's been a long, long time, so you probably do not remember. But take my word for it, way back when OS/2 was to be the Next Operating System, the Rich Text Format was generally hailed as a Very Good Thing. It was a document and data format which came with a published specification, and the plan was that this was to be the vendor neutral data exchange format for all applications which would talk RTF compatibly and henceforth data destruction due to inane incompatibilities and fuzzy interpretations would be a thing of the past.
The original RTF 1.0 specification has been preserved at sourceforge as http://latex2rtf.sourceforge.net/RTF-Spec-1.0.txt, and our friends at WikiPedia have a short but nice article at http://en.wikipedia.org/wiki/Rich_Text_Format.
I originally started writing that piece at the end of July 2007, before the ISO voting process over Microsofts "Office Open XML" specification had started. I was working on a book at the time, and never got very far with my planned overview with a very personal slant on that file format. The RTF format was actually useful in its time, and how it is constructed and how it was maintained can show us a lot of useful things. Now that it's clear that Microsoft succeeded in getting ISO approval for its specification, it's time to return to that subject.
What a process it has been. Microsoft has succeeded in buying itself an ISO standard. Piece by piece, in a process that included ballot-stuffing, bribes, and the administrative overruling of the relevant technical committee's decision by a national standards body (right here in Norway), Microsoft bought itself or is very close to getting its wholly-owned ISO standard. Nevermind that there is at least one formal investigation by the EU into the process and that the chairman of one national body's technical committee (once again, in Norway) has lodged a formal protest over procedure with ISO.
Microsoft won, and we will all have to learn to live with the consequences.
One of the very effective tactics was to have Microsoft partners sign up as voting members of national standards bodies, with clear instructions on how to vote. This lead to an influx of new voting members in various parts of the standards organization so large that other standardization efforts where Microsoft has no interest at the moment have reportedly ground to a halt. The reason is that what remains of formal procedure in ISO dictates that if enough members of a national body fail to vote on a proposed standard, the standard is not approved and fails by default. So the first victim of Microsoft's ISO takeover is no new standards. Who would want more ISO standards anyway?
And oh yes, Microsoft's main OOXML propagandist in Norway has offered up this document as proof that OOXML has been implemented and is in fact useable in OpenOffice too. It did not open at all in my OpenOffice 2.3 running on OpenBSD 4.3-current, (see here and here for the two stages of the results), and informed sources tell me that the much-ballyhooed formula is not editable when the document is imported into other applications such as Microsoft's own Office 2003 plus the compatibility pack.
And as if to emphasize the fact that all you can guarantee when you call something "XML" is that the data will start with a < and end with a >, this XML document in fact handles the equation part as an embedded binary object, intimately tied to Microsoft's equation editor. I suppose this means that even at 9,000 pages and counting, the spanking new ISO standard does not cover handling equations in a satisfactory manner. All in all, just how much more weirdness this will lead to is hard to predict.
But I digress. As I said earlier, RTF was thought to be a very good thing. The specification was a lot less ambitious and a lot slimmer than the XML based one, and since RTF was the source format for the Microsoft Windows Help file format, recommended for all apps to run on Windows, there were even usable examples documented in one of the several manuals you would get with the Microsoft programming tools.
As you would expect, word processors and other applications were rewritten to be able to read and write RTF. I worked in documentation and localization at the time, and over the years I had a fair amount of exposure to the format and the tools. Basically, any Windows application worth documenting would require an online help system and the source format would be RTF.
Just like an XML file has to start with a < and end with >, RTF files are made up of elements that are delimited by matching { and } curly braces. In principle, you could write valid RTF using any text editor, and the Microsoft documentation would show you how. That fact came in handy a few times when Word managed to mangle some other document with its 'fast save' feature or as punishment for sending the document to be edited at one machine too many during a technical review. The solution to essentially any problem, as long as the .DOC file was possible to open at all, would be to save the document as .RTF, count the number of {s and }s and see if the numbers were equal. If they were not, you would add one of the missing kind at either the top or the bottom, using any straight text editor. The magic would work, or at least put you in a position where you could extract useful data.
Over time you would upgrade to newer tools, and Microsoft's development discipline (amply documentend in the OOXML tomes) would show up at every Word upgrade if you had not been paying attention.
It would go like this: You get the fresh Word version with all the new bells and whistles, and do another revision of the help system for an app. The help compile would break horribly, and after some hair-pulling and tabletop-thumping you would eventually remember that this had in fact happened before.
The definition of RTF had changed.
After a while you would discover that hidden somewhere deep in the new Word's online help or README you would find Microsoft noting that application developers would need to install a newer version of the help compiler to get their RTF to compile. Or to put it slightly differently, the operative definition of what "RTF" is would always be "whatever Word produces when you save as RTF". To their credit, the RTF people at Microsoft would usually publish a new version of the specification some time after a new Word release.
So, we've been there before. "Whatever Microsoft {Word,Excel,PowerPoint} produces" is likely to be the operative definition of what OOXML is, even if the 9,000 pages and counting specification says something subtly different.
With a specification that lets dates be represented in radically different ways depending on context, say if you're saving from a word processor, a spreadsheet or a presentation program, we're in for a lot of entertainment, I'm sure.
The other things I'm sure about are that there will never be an implementation of OOXML as currently written (even if the OpenOffice.org people have made noises that they might try), and that a full implementation of anything vaguely similar will have to come from Microsoft.
Real gluttons for punishment could try the new hobby (or profession, if you can get a sponsor) of tracking the the day to day development in the set of differences between OOXML-as-published and OOXML-as-implemented.
Stuff I'll be getting back to: OpenBSD 4.3 is about to hit a mirror near you and hopefully your mailbox very soon, and more spam follies.
Update 2015-04-04: The demonstration document appears to load correctly in LibreOffice 4.3 -- I just tried it, and a screenshot is available here. I have no useful data on when the document became readable for anything not produced by Microsoft. In other news, Microsoft has now announced that an upcoming release of their Office product will be able to read and save ODF files.
Does anybody here remember Artie Eff? is © 2008 Peter N. M. Hansteen (published 2025-04-13)
You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).
At EuroBSDcon 2025, there will be a Network Management with the OpenBSD Packet Filter Toolset session, a full day tutorial starting at 2025-09-25 10:30 CET. You can register for the conference and tutorial by following the links from the conference Registration and Prices page.
Separately, pre-orders of The Book of PF, 4th edition are now open. For a little background, see the blog post Yes, The Book of PF, 4th Edition Is Coming Soon (also here). We are hoping to have physical copies of the book available in time for the conference, and hopefully you will be able to find it in good book stores by then.
Subscribe to:
Posts (Atom)
Posts
