I Must Be Living in a Parallel Universe, Then

It's Sunday morning, and I'm having my morning coffee while getting ready for a long session of editing my OpenCON presentation. By working on adapting the presentation tailored to the tutorial I've been rediscovering just how much work went into making the book, so a long Sunday session is needed, if not more.

Then courtesy of Groklaw's news picks comes the USA today piece called Despite filters, tidal wave of spam bears down on e-mailers.

A tidal wave of spam, no less. Well, we're seeing a lot of attempts at sending, like the sequence here (text link, formatting it would take too long) that I captured from the xterm running a tail -f on my spamd log a little while back. That sequence tells me, for one thing, that the naive spambot thinks my spamd looks like an open relay.

The other interesting thing about the sequence there is the pattern you can see in the From: addresses. It may have dawned on some of the spammers that generating random addresses in other people's domains might end up poisoning their own well, so they started introducing patterns to be able to weed out their own made up addresses from their lists. I take that as a confirmation that our harvesting and republishing efforts here and elsewhere have been working rather well.

Here the method seems to be that they take the victim domain name, prepend "dw" and append "m" to make up the local part and then append the domain, so starting from sia.com we get dwsiam@sia.com.

There is one other common variation on that theme, where the prepend string is "lin" and the append string is "met", producing addresses like linhrimet@hri.de, used just a few minutes ago to try to spam malseeinvmk@bsdly.net from the apparently Polish adress 89.228.40.80. This is of course very interesting, as is the fact that right now about two and a half thousand machines are in my spamd-greytrap list . That's where they end up, making no waves at all.

On the subject of patterns, earlier this month the address capitalgain02@gmail.com started appearing frequently enough that it caught my attention in my greylist dumps and log files.

The earliest contact as far as I can see was at Nov 10 14:30:57, trying to spam wkzp0jq0n6.fsf@datadok.no from 193.252.22.241 (apparently a France Telecom customer). The last attempt seems to have been ten days later, at Nov 20 15:20:31, from the Swedish machine 217.10.96.36.

My logs show me that during that period 6531 attempts had been made to deliver mail from capitalgain02@gmail.com via bsdly.net, from 35 different IP addresses, to 131 different recipients in our domains. Those recipients included three deliverable addresses, mine or aliases I receive mail for. None of those attempts actually succeeded, of course. With a little more time on my hands I'm sure I could have made a good regular expression to calculate to the second how much time those spam senders wasted here, too.

So where's the tidal wave? Back when PDF spam was the new horror, it actually took three weeks for one to reach me, and then only via an alias on a machine I really don't have much control over anymore. The number of spam sending machines does seem to be increasing, though.

Bob Beck's uatraps list is a good indicator, and the tendency is clear from the graph in my malware paper. The number did dip just below 100,000 addresses earlier this month, and it now seems to have stabilized in the 110,000 to 120,000 range.

From my perspective, it looks like a reasonably configured spamd is really all we need to observe the tidal wave at a safe distance and have fun all the while.

It's almost like living in a parallel universe.

Of Course, It Had To Be A Webshield

In an earlier blog post, I mentioned that I would buy a round of drinks the first time I saw an attempt to deliver a message with both the From: and To: addresses already on my spammer baiting list.

In fact it happened very soon afterwards, and as luck, misfortune or just plain old incompetence would have it, that message apparently came from a WebShield appliance not too far from here:

Oct 17 23:03:52 skapet spamd[20795]: 194.54.96.18: connected (6/4)
Oct 17 23:04:03 skapet spamd[20795]: (GREY) 194.54.96.18:
<capitulations7@datadok.no> -> <capitulations7@datadok.no>
Oct 17 23:04:03 skapet spamd[20795]: 194.54.96.18: disconnected
after 11 seconds.
Oct 17 23:19:21 skapet spamd[20795]: 194.54.96.18: connected (4/3)
Oct 17 23:19:32 skapet spamd[20795]: (GREY) 194.54.96.18:
<capitulations7@datadok.no> -> <capitulations7@datadok.no>
Oct 17 23:19:32 skapet spamd[20795]: 194.54.96.18: disconnected
after 11 seconds.
Oct 17 23:30:30 skapet spamd[20795]: 194.54.96.18: connected (4/4),
lists: spamd-greytrap
Oct 17 23:34:14 skapet spamd[20795]: (BLACK) 194.54.96.18:
<capitulations7@datadok.no> -> <capitulations7@datadok.no>
Oct 17 23:35:58 skapet spamd[20795]: 194.54.96.18: From:
Webshield.SMTP.V4.5.MR1a.Mail.Service@vs4.bgnett.no
Oct 17 23:35:58 skapet spamd[20795]: 194.54.96.18:
To: <capitulations7@datadok.no>
Oct 17 23:35:58 skapet spamd[20795]: 194.54.96.18:
Subject: Returned Mail: Error During Delivery
Oct 17 23:37:00 skapet spamd[20795]: 194.54.96.18:
disconnected after 390 seconds. lists: spamd-greytrap
Oct 17 23:57:18 skapet spamd[20795]: 194.54.96.18:
connected (6/6), lists: spamd-greytrap

I sent the operators at that site a polite message right away, pointing out the misconfiguration. Two weeks later I have seen no response other than the automatic acknowledgement, but it looks like the machine has managed to get itself automatically whitelisted in the meantime. So perhaps they found the button that actually does something.

Since my last blog post I have completed the book, and I expect the last bit of proofing to be done during the coming week. Then a few other necessary processes, and physical copies available for mid December if all goes well. With the cover in place, it looks like it's become attractive and popular over at amazon.com in its various categories. The BSD category there looks pretty No Starch dominated at the moment.

That can not be a bad thing. It's been a real pleasure working with the people at No Starch Press. If you think you want write a tech book, they should be on the list of publishers to contact with your proposal.

While all this was happening, the spammer baiting operation seems to have reached a critical mass of sorts. With roughly 7,200 addresses in the spamtrap list there are several hundred bait addresses for each real one in those domains taken together, so it's extremely unlikely that the spammers will ever get a chance to try delivery to a real address before they hit the tar pit. Over the last couple of weeks, my gateways have had anywhere between 2,500 and 4,000 hosts in the local spamd-greytrap, and anywhere from 0 to about 300 spambots pushing bytes into the tar pits at any time. It's fun to watch (some of the bots labor through the bait list from top to bottom), and the net effect is, well, we're not seeing much spam.

I think I've mentioned it before, but it bears repeating: To naive spammers and the tools they use, spamd looks like an open relay. Spamd never actually delivers any messages, but this


GREY|201.250.57.147|sofia|<vdaegkoxgk@bonana.com>|
<brad.james.anderson@jhg.com.au>|1193105605|1193127205|1193127205|1|0

says that whoever operates 201.250.57.147 (according to whois, likely located in or near Buenos Aires, Argentina), is unable to tell the difference between an open relay and spamd's 451 and subsequent "this is going to hurt you more than it hurts me" messages.

Another variation on that theme is what I think is some sort of amateurish relay testing, which typically produces anywhere from five hundred to a thousand greylist entries of the type


GREY|59.35.4.51|UATIM-F7E7949C7|<adgjnq@194.54.103.104>|
<ariel5268@yahoo.com.tw>|1193084672|1193113472|1193113472|2|0
GREY|59.35.4.51|UATIM-F7E7949C7|<xaehkn@rosalita.datadok.no>|
<ariel5268@yahoo.com.tw>|1193084675|1193113475|1193113475|2|0
GREY|59.35.4.51|UATIM-F7E7949C7|<qswyd@brutha.datadok.no>|
<ariel5268@yahoo.com.tw>|1193084691|1193113491|1193113491|2|0
GREY|59.35.4.51|UATIM-F7E7949C7|<nqtw@monalisa.datadok.no>|
<ariel5268@yahoo.com.tw>|1193084733|1193113533|1193113533|2|0

where the From parts are made up of host names and IP addresses in our local net, including in this case, the host name for one of our laser printers. Those floods have tended to swell the bait list a bit, even if I strip out the invalid @<IP address> ones.

Spamd makes the naive relay testers think we have a whole network of open relays, and we harvest the noise they generate to lead the spambots to the tarpit. That's pretty close to a hands-off spammer repellent for us, and a serious auto-LART for the spammers.

OpenCON is sneaking up on us in a month's time, and we're heading for Venice with a refreshed tutorial session. See you there!

PS - [non-IT PS coming up] Bergen's football (soccer) team SK Brann has just won the national league for the first time in 44 years. With one game to go before end of season they are so far ahead in points there is no way any other team will be able to catch up. The town is predictably going gaga over the event, and we joined the thousands at the central Festplassen square for the city sponsored celebration tonight. I'm surprised how many songs have been written about that team and how everybody around me seened to know every last word of the lyrics. Good fun, ending with fireworks.

Always a pleasure to be wasting your time, guv

This week has been a little unusual around the BSDly household. So far I've generally been doing my regular job in the daytime (with longish office hours), only working on the book evenings and weekends. That the arrangement would lead to "Exhaustion is my middle name" status was obvious to everyone except me, but I finally saw where it could be going. So for a little more than the past week I've been working on the book full time.

The state of perpetual exhaustion has had some not too happy consequences. Of course the general progress on the book suffered, but it also lead to me missing the monthly BLUG meeting in August. Of course much of that particular day I had spent persuading somebody not too bright that it indeed had to be a reconfiguration they said had never happend at their end which ended up breaking things at our end, and I was just too tired and missed what I assume was a well executed lecture on networking basics by Vegard Engen (of RFC1149 implementation fame).

This week with only one job I needed to tackle, I was there for an enjoyable one and a half hours of Bacula, well presented by Bård Aase (aka elzapp). Off to Henrik (the regular BLUG pub) for a few beers afterwards, and with Johan Riise volunteering to put together a 'Unix and time' lecture for next month, the BLUG calender seems to be in order after all, with Jill Walker doing the end of semester talk in November, on whatever interesting stuff she has been up to lately. Unfortunately it looks like the last Thursday of November is close enough to OpenCON that I'll likely miss Jill's session.

In the meantime, there are signs that the greytrapping and my bait list is working. Looking over the spamd logs today I found quite a few entries like these:

Sep 29 15:29:23 skapet spamd[20795]: (BLACK) 84.76.177.159: 
<royaleuromillion2007@yahoo.es> -> <211hgsreliart7@datadok.no>
Sep 29 15:29:32 skapet spamd[20795]: (BLACK) 84.76.177.159: 
<royaleuromillion2007@yahoo.es> -> <00b27f18@datadok.no>

which looks strikingly like the Spanish lottery scam spammers patiently and methodically working their way through my list of bait addresses, all the way from top to bottom, at roughly 3000 addresses it's going be a while. All I can say is, we are extremely pleased to be wasting your time, senor.

Also while the girls were off to the Raptus comics festival (an annual event, and one of the big things here in Bergen), I found enough trash backscatter to non-existent bsdly.net addresses that it's likely that the same weekend spambot operators who spewed their spam with @ehtrib.org and @skapet.datadok.no addresses earlier (both times at weekends) have now discovered bsdly.net and are doing their damnedest.

Why they prefer to generate a few hundred fake addresses and use them all in one go is beyond me. The other groups seem to generate only a handful of new addresses each every day, and for good measure at least one of them sort of reuse the generated addresses by using a forward and a reverse (such as in this morning's preserved greylist dumps, there was a potterv76@datadok.no as well as the reverse 67VRETTOP3@datadok.no). This lot just dumps all they have in one go, mainly contributing to swelling that file in my home directory with the totally unprintable file name which is the temporary storage before they go to into the traplist and on to the bait page.

Distractions of that kind from my main task is never entirely welcome, but with a larger influx of new addresses to be added to the bait list I made some small changes to make the maintenance of that page a bit more sane, rediscovering server-side includes and redirects along the
way. And the data I keep collecting may become the basis for other projects later.

Anyway, it is increasingly clear that the spammers are including the generated fake addresses in their "known good" lists. Consider the spambot at 210.111.190.216 (apparently in Korea), which insists on delivering to an address somebody generated in early July:

peter@skapet:~/www_sider$ grep  210.111.190.216 /var/log/spamd
Sep 29 15:58:07 skapet spamd[20795]: 210.111.190.216: 
connected (5/4)
Sep 29 15:58:21 skapet spamd[20795]: (GREY) 210.111.190.216: 
<jim.vance@presentsmadeeasy.com> -> 
<careersogt2083@datadok.no>
Sep 29 15:58:22 skapet spamd[20795]: 210.111.190.216: 
disconnected after 15 seconds.
Sep 29 15:58:35 skapet spamd[20795]: 210.111.190.216: 
onnected (4/3)
Sep 29 15:58:49 skapet spamd[20795]: (GREY) 210.111.190.216: 
<tbaker@groupecdb.com> -> 
lt;careersogt2083@datadok.no>
Sep 29 15:58:50 skapet spamd[20795]: 210.111.190.216: 
disconnected after 15 seconds.
Sep 29 15:59:03 skapet spamd[20795]: 210.111.190.216: 
connected (5/3)
Sep 29 15:59:17 skapet spamd[20795]: (GREY) 210.111.190.216: 
<wotan@4vsi.com> -> <careersogt2083@datadok.no>
Sep 29 15:59:18 skapet spamd[20795]: 210.111.190.216: 
disconnected after 15 seconds.
Sep 29 15:59:30 skapet spamd[20795]: 210.111.190.216: 
connected (6/5), lists: spamd-greytrap
Sep 29 16:03:14 skapet spamd[20795]: (BLACK) 210.111.190.216: 
<sylviacastleman@alltypecalligraphy.com> -> 
<careersogt2083@datadok.no>
Sep 29 16:04:59 skapet spamd[20795]: 210.111.190.216: 
From: "Marguerite Casey" <sylviacastleman@alltypecalligraphy.com>
Sep 29 16:04:59 skapet spamd[20795]: 210.111.190.216: 
To: <careersogt2083@datadok.no>
Sep 29 16:04:59 skapet spamd[20795]: 210.111.190.216: 
Subject: 100mg x 60 pills US $ 129.95 buy now
Sep 29 16:06:04 skapet spamd[20795]: 210.111.190.216: 
disconnected after 394 seconds. lists: spamd-greytrap

I have no real opinion on the validity of the From: addresses, but the address they are trying their best to deliver spam to here never actually existed, of course. The first record of it at datadok.no was this bounce from a Russian site:

Jul 12 23:38:52 delilah spamd[29851]: (GREY) 81.177.34.190: 
<> -> <careersogt2083@datadok.no>

Dumping their trash back at them is good for a laugh, and I am quite amazed how shortsighted the spambot operators appear to be. They get yelled at for spamming, so to avoid detection, they start using fake addresses. This in turn means they have no feedback whatsoever on the quality of their address lists, and with well pissers like me in action, they are getting less effectitive each day, reducing themselves to background noise in the network.

Now with this blog post done I will go back and finish the edits on the logs chapter. With the early parts of the book about to enter the layout phase while the last bits get written over the next few days, there is a chance that there will be a physical copies of the book to pass around at OpenCON. Not quite there yet, but the fulltime push is certainly helping. The preface with a list of thanks is part of what is entering layout; I think a few people who did not expect to be in there will soon have a pleasant surprise.

Also this week, the PF tutorial saw its unique visitor number 19,000 since EuroBSDCon 2006 on Thursday morning (September 27th). We certainly hope at least some of them will come back for the book.

The Great SCO Swindle Winding Down, But Will They All Get Away With It?

Poor Dan Lyons. He thought like a bookmaker and wrote what he thought was right.
You see, a few years back, when Caldera was still Caldera, that company had successfully sued a large corporation and won. Then Caldera changed its name to SCO and sued another huge corporation. Dan the bookie thought it was a sure bet, and started cheering them on. Four years on, the sure bet went south on a technicality. They did not actually own the code they had accused others of stealing. At least that's the way I read his Snowed by SCO article over at Forbes.

My take on this is, Dan, you only had to look at the facts. Knowing a bit of IT history is also a plus. When the SCOX matter came up, I like most people thought that you can never rule out the possibility that some code might have been copied. After all, Unix source code was never particularly hard to get you hands on and was widely used as classroom examples all over the world.

Then if that code was just identified, it would be ripped out and replaced. It's happened before. In the free software world, whole subsystems get replaced when there's a good reason to, and if the reason is copyright violation it gets somewhat urgent. The problem is, in the SCO matter, no code was ever identified.

Some journalists went through an elaborate procedure involving non-disclosure agreements and were, we are told, showed code from Linux and somewhere else which showed remarkable similarity. When Darl McBride used the SCOForum 2003 conference to show something he passed off as ripped off code, it took only hours to identify the exact chunks through the obfuscation (yep, formatting comments in the Symbol font) and the code proved to be irrelevant.

None of these events helped convince techies of their claims, but for me the tipping point was when they claimed to have a reason to sue the BSDs as well. Anyone who had been paying any attention at all to Unix history knew that the ATT vs BSD lawsuit was finally settled in 1994, with most of the terms sealed, but one of the few things that was made public was that the parties had forfeited any right to sue each other over the Unix code base. To me and quite a few others, this was proof positive that they were 'misguided or dishonest', as a commentator put it at the time.

One of my favorite summaries of the facts of the case was written by Greg Lehey (of The Complete FreeBSD fame), who looked at the various announcements from the technical side. He stopped maintaining it after a while, but it's still there at his website, with as far as I tested with all links intact.

Most people seem to be relieved that the matter seems to be over. I beg to differ.

For one thing, the main characteristic of this matter has been the amazing ability of the SCO crowd to drag out the proceedings over irrelevant, mainly procedural matters. They will have more tricks up their sleeves, for certain.

The other thing is, with Dan's friends out on the technicality that they did in fact not have the legal standing to sue, we will never get that detailed walkthrough of the code where Darl and his covert experts are supposed to point out the infringing code. I, for one would have looked forward to that. Then we would have had a chance of getting to know their real motivation too, and possibly some solid leads on the planning and funding. Now that will just not happen.

Then of course there's the stockholder lawsuits and possibly the FTC. If you were one of those chumps who bought SCOX stock at roughly twenty dollars a share based on Dan Lyons' recommendations, wouldn't you feel a little sore now that your investment is about a cent to your original dollar? That is, if you can unload it before SCOX are finally kicked out of NASDAQ for good?

So poor Dan Lyons for not seeing this coming. And damn the technicalities for cancelling the main event.

For those of you eager for news of the book, we're working hard to get it out there.

Update 2007-09-25: Another non-apology, this one from Rob Enderle.

According to linuxtoday, Rob Enderle claims he was tricked by (wait for it) both SCOX and those ever-bullying Linux people.

Actually, there's not much to see there, You can read it as just another non-apologizing apology, with some tall tales about death threats and DOS attacks thrown in (yes, really).

As I've said a few times earlier, enough facts were on the table right from the start of this timewasting story to show that more than likely the SCOX crowd did in fact not have a case.

Now I wonder what, if anything we will be hearing from
John Parkinson, who wrote in CIO:

"a lot of the intellectual property in Linux is actually owned by companies that never officially agreed to make it available under an open-source license."

Interestingly enough, that came without any qualification at all.

That irritated me enough at the time that I wrote to them (pasted into some inane feedback form):

Alleged intellectual property theft

In the article called "The End of Idealism" (http://www.cio.com/archive/070103/et_pundit.html), John Parkinson writes, "a lot of the intellectual property in Linux is actually owned by companies that never officially agreed to make it available under an open-source license."

Please take a moment to consider the seriousness of this allegation. What Parkinson actually says here is, "large parts of Linux consist of stolen property".

Reading such allegations in an article written by a senior executive of Cap Gemini Ernst & Young is quite shocking in itself.

It is only reasonable that Mr Parkinson or Cap Gemini Ernst & Young specify which parts of the Linux kernel they consider to consist of stolen property.

All versions of the Linux kernel, along with detailed change logs and archives of the developer mailing lists are available to the public. Using these resources, all parts of the code base can be traced to the individual who submitted them for inclusion.

In other words, it is quite easy to pinpoint who did what, and Mr Parkinson and Cap Gemini Ernst & Young would be doing the public a great disservice by refusing to help point out code which was illegally included in the open source operating system.

Quite a few articles, well informed and otherwise, have been written about the SCO vs IBM lawsuit and SCO's allegations. I suggest interested readers browse FreeBSD Core member Greg Lehey's overview at http://www.lemis.com/grog/SCO/index.html while we wait for more details from Mr Parkinson or Cap Gemini Ernst & Young.

Slashdot It!

EuroBSDCon was great, disks dying and some scary Windows stuff

This Monday finds me safely back from EuroBSDCon and trying to do useful things while the file server gets restored.

Of course it had to be that way. With me off to EuroBSDCon to do the tutorial and other refreshing geekiness, in the first batch of mail I retrieved after arriving in Copenhagen was a log summary from the machine which holds pretty much everything datadok is working on at any time, with these nuggets:

> > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=410884031
> > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=410912703
> > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=410884575
> > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=410905887
> > ad6: FAILURE - READ_DMA status=51 error=40 LBA=410857151
> > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=446104667
> > ad6: TIMEOUT - READ_DMA retrying (1 retry left) LBA=446104667
> > ad6: TIMEOUT - READ_DMA retrying (2 retries left) LBA=522840603

ouch.

This is what disks say when they've run out of space to map bad sectors into. The disk wasn't quite dead yet, but definitely time to plan a replacement. Not much to be done about that right away except alert the colleagues that there would be file server downtime on the Monday afternoon. Disks will die, and sysadmins end up with the task of replacing them.

My brief summary of EuroBSDCon is that it was an excellent conference, lots of good talks, interesting people to see and in a good, clean location with a network connectivity which worked, most of the time. update: finally my eurobsdcon pictures are on flickr

For my own part the PF tutorial went reasonably well, with 24 people signed up and I think one or two sit-ins. People were paying attention and there were a few good questions which made the session more interesting with a little more improv than the last few times I did this tutorial. Answers were had, though, and I believe a good time with useful info for the people who had signed up for the session. Not too many hours after we were done, the number of unique visitors (aka host names or addresses) to the tutorial tree since last EuroBSDCon rolled past 18,000.

After lunch Marco Zec's session about virtualizing the FreeBSD network stack was really interesting. Unfortunately none of the Thinkpads present were able to boot from the FreeBSD-current image Marco had prepared and supplied on USB thumbdrives, actually producing pretty much the same crash (illustrated here). But a very interesting topic and session. I'm glad I stuck around for it.

The Wednesday I had the choice of sightseeing, sitting in on Kirk's session and holing up in the hostel basement's hacker room to get some writing done, and I ended up going for the latter option, getting significant parts of the logging chapter done. There is of course a limit to how long you will avoid interruption in a semi-public area, but that session was certainly useful.

The EuroBSDCon hacker area with both wired and wireless networks was available to conference attendees all conference and tutorial days. Naturally it took on a social function in addition to being a convenient way to surf and fetch your email.

For the conference itself, it was sometimes hard to choose which talks to go to. I still think Ike's jails talk (pix here, here, here) was my favorite (similar but not identical to the one he gave at AsiaBSDCon in Tokyo), but there were a lot of good ones. I ended up managing to miss Pierre-Yves Ritschard's Load Balancing talk since they'd switched the schedule around. I hope there's a chance to pick up the essentials at some later date.

Fortunately Wim and Machtelt turned up to organize the OpenBSD booth (convenient for restocking your clothes cupboard) and some news about OpenCON - there will be an OpenCon 2007, but there's still some organizing to do. I hope to be seeing you there, Venice November 30th through December 2nd.

From the Windows Is Scary department, one episode from a few weeks back which I suddenly remembered when I realized the guy quietly hacking to the left of me was FreeBSD USB guru Hans Petter Selasky:

When I saw 4GB USB thumb drives priced at just under NOK 300 (USD 55), I decided I needed one. The drive mounted with no trouble at all in in OpenBSD (mount /dev/sd1i at the location of your choice), and I thought good, I'll just delete those .exe files to make room. A few days later I needed to retrieve som files which turned out were most easily accessible from my Windows machine at work. So I plugged in the new 4GB thumb drive.

Windows machines always do strange things and take a while to recognize new hardware, but this time it claimed to have found a new CD drive. A few confusing minutes later, with various message boxes flashing across the screen, the machine begged for a reboot. I let it have that, slightly puzzled but not entirely surprised that Windows wanted the user to jump through a few extra hoops to make something work.

I was able to retrieve the files eventually, while trying to avoid yet another quirky Windows application which wanted to handle my files. As it turns out, the device actually emulates a CD drive as well as USB mass storage. Here's what it looks like in /var/log/messages on my OpenBSD laptop:

Sep 17 22:23:23 thingy /bsd: umass0: SanDisk Corporation U3
Cruzer Micro, rev 2.00/0.10, addr 2
Sep 17 22:23:23 thingy /bsd: umass0: using SCSI over Bulk-Only
Sep 17 22:23:23 thingy /bsd: scsibus2 at umass0: 2 targets
Sep 17 22:23:23 thingy /bsd: sd1 at scsibus2 targ 1 lun 0:
SCSI2 0/direct removable
Sep 17 22:23:23 thingy /bsd: sd1: 3913MB, 498 cyl, 255 head,
63 sec, 512 bytes/sec, 8015502 sec total
Sep 17 22:23:23 thingy /bsd: cd1 at scsibus2 targ 1 lun 1:
SCSI2 5/cdrom removable

The reason all the strange and scary things happened with the Windows machine is that the emulated CD contains Windows Autorun files, which it seems there is no easy way to turn off or is at least enabled by default in that operating system. What I find slightly disturbing is that, as Hans Petter explained, this behavior is part of the device's firmware and you can't get rid of that five or six megabytes of useless software in these devices. The best you can do is use a system which ignores such silliness.

Returning to the file server, the box is a few years old and has by now probably had most of the original components replaced. The last time we replaced the motherboard, we were still thinking that SCSI was the only way to go for storage, disks and tape both. Not too long after that, we decided that actually SATA was OK for that little office of ours, but when the time came to replace that disk, I discovered that actually the motherboard had only two SATA ports on it, one for the system disk and one for the dying data disk. So copying across from one SATA disk to another had to be done via Ethernet instead. Fortunately installing a useful operating system takes only about twenty minutes, and the some tens of gigabytes transferred while I was writing this article. Far faster than restoring the same data via rsync from our offline backup, though.

Among the things announced in Copenhagen were that there will be an AsiaBSDCon in March 2008, NYCBSDCon will maybe be next year in the fall, and the next EuroBSDCon will be in Strasbourg. I hope to be at several of those, time and money allowing. But now on to finish that book.

Wanna help science? Study your greylists' innards!

If somebody, say five years ago, had told me that I would be spending a little time, every day, studying data about what invalid addresses some unknown miscreants are making up in my domains, I would have thought them to be slighly off their rockers.

Yet here I am, actually maintaining a publicly available list of addresses which do not stand a chance of becoming valid, ever. It all started with a log data anomaly - I noticed an increase in the number of failed delivery messages to non-existent addresses in our domains. I had expected that the bounces to invalid addresses would appear for a short period only, but for one reason or the other it looks like it's here to stay, with some dips and peaks like the ehtrib.org flood.

The list is apparently working as intended too. These addresses are on my local greytrap list, and I have started seeing addresses I put in there as all uppercase turn up in my logs in all lowercase variants. Fun to watch, sort of.

Anyway, the supply of new bogus addresses proved to be larger than I had expected. So to get a handle on just what is happening I ended up doing periodic dumps of the live greylist data. This is really easy to do if you're using spamd as your greylister, your basic command is

$ sudo spamdb | grep GREY

and you redirect to a file, pipe to mail, or whatever you like.

Now if you're a bit like me, looking for patterns in the noise like this makes you feel a little weirder than usual and possibly lead you to think of a Clive Barker novel (specifically the bits about the dead letter file in The Great and Secret Show) and you wonder why this is worth doing at all. After all there is precious little spam that actuall reaches my users, so like I said earlier, for us spamd users it really looks like spam is a solved problem. I guess I'm just a bit fascinated by the pure irrationality of the spammers' behavior.

From the data I collect here in my tiny corner of the world to browse when time allows there may be useful information lurking somewhere.

Typical entries show things like the host 202.152.33.43 tried to send with a From: address jcejft@charter.com to dkqvujfn@datadok.no and sdenuuu@datadok.no. Using a few common networking commands we see that there is no reason why charter.com email should come from the IP range belonging to idola.net.id, and as the admin of datadok.no I know these two addresses have never been deliverable. Most likely the admin at charter.com can tell you if that from address is deliverable, but I keep wondering how much of the spam out there is stuffed into the pipe with bogus From: and To: addresses both. Or in other words, purely useless noise, never to be delivered anywhere.

On a side note, with one or more of the spammer operations trying to sneak through using sender and recipient addresses in the target domain, I assume it is just a matter of time before I see a tuple with both sender and recipient addresses already in my spamtraps list. When that happens, I think I will feel inclined to let my friends have a round of refreshments on my tab.

It's obvious that there are a handful of spammer operations that have decided to use datadok.no (and to a lesser extent, dataped.no and ehtrib.org) From: addresses on the spam they send, apparently in an attempt to cover their tracks. I will probably never know why they decided to do that, but I wonder why they keep it up and for that matter how many other domains are seeing this, with bounces from strange places, directed at non-existent, fairly obviously generated bogus addresses.

So if you are seeing similar stupidity in your logs and if you are running a sensible greylister such as spamd, I would be interested in hearing from you so we can compare notes.

Out there in meatspace, EuroBSDCon 2007 is coming up. I'll be there with the PF tutorial on Wednesday. This Friday's deadline for an updated manuscript had totally slipped my mind (I blame the book and a few other, less rational, factors), but hopefully the 24 who signed up for the session will find it useful anyhow - there will be new bits and as much interesting stuff as I can manage. I'll be around for the rest of the conference too, but unfortunately I'll have to give the Legonland trip a miss.

Be seeing you in Copenhagen! The book is getting closer to finished, I promise!

A Lady in Distress; or Then Again, Maybe Not

A two user domain gets bounces for seven hundred, grep and sed to the rescue, spamd saves the day

The past week moved along with only minor disturbances on the keep-systems-running front. The time consuming frustrations were generated elsewhere, and (un?)fortunately I am not at liberty to discuss the details. Incompetence was involved, next week it's somebody else's problem.

All the while, the spammer trapping experiment has been moving along at a leisurely pace.

Generally keeping the lists (both the web version and the live one) updated would cost me a few minutes' browsing of greylist dumps two or three times a day or whenever I felt like it, with a typical catch of maybe fifteen new bogus addresses to feed to the trap list each day.

For the last three or four days the haul has been smaller, with essentially no new captures yesterday, for example. Now I've found out why. They have moved on, alpabetically.

Done with bsdly.net, the dominant group of spammers moved on to generating addresses in the D domains including datadok.no and dataped.no. I'm bound to have missed a few, since the grand total by this morning had yet to reach a full thousand. By now, they seem to have reached the Es. This morning I noticed the overnight greylist dumps were bigger than usual.

The reason: ehtrib.org, the domain we set up mainly for my wife's use (read: her email), appears to be the current home of made up From: addresses, with roughly seven hundred accumulated by the time I was done with morning routines of breakfast with coffee and browsing the overnight incoming mail.

That is by far the largest addition to the flypaper list ever.

Fortunately, with only two active addresses in the domain (I'm not telling what either other one is) it's fairly trivial to extract the bogus ones.

Up to now I've been integrating the noise into the traplist page manually, for now I've put this batch up at http://www.bsdly.net/~peter/ehtrib-1stbatch.

They're all in the active traplist at the gateways, of course. It's the editing into the page the spammers will slurp via unattended robot I'm putting off for a little more while I'm doing some other writing. [not any more. all there now, but the original list is preserved too]

Just why this time we are seeing this number of addresses over a short period of time, and not a handful each day over several months is an open question. One likely explanation is that one of the chickenboners fell asleep at the wheel and let the junk generator run longer than they actually intended. Time will show if this means they move on more quickly.

When I have more time, I will probably analyse the data I am accumulating at the moment and tell the tales of the silly lamer tricks the spammers try to pull.

In the meantime, following up on earlier posts, there are still a few people who Just Don't Get It:

Aug 19 13:28:03 delilah spamd[3712]: 217.159.231.230: connected 
(9/9), lists: spamd-greytrap
Aug 19 13:31:49 delilah spamd[3712]: (BLACK) 217.159.231.230:
<> -> <armrest10@datadok.no>
Aug 19 13:33:32 delilah spamd[3712]: 217.159.231.230: Subject:
Considered UNSOLICITED BULK EMAIL, apparently from you
Aug 19 13:33:32 delilah spamd[3712]: 217.159.231.230: From:
"Content-filter at linux.byroomaailm.ee" 
<postmaster@linux.byroomaailm.ee>
Aug 19 13:33:32 delilah spamd[3712]: 217.159.231.230: To: 
<armrest10@datadok.no>
Aug 19 13:34:38 delilah spamd[3712]: 217.159.231.230: disconnected 
after 395 seconds. lists: spamd-greytrap

And it looks like the published list is having the effect I was hoping for. I keep seeing quite a few of the addresses in ALLCAPS (with numbers tacked on) I put on the web page a few weeks back beginning to appear in lowercase but otherwise identical in my greylist dumps.

In other news, the PF tutorial session at EuroBSDCon is now a definite.

See you in Copenhagen, if not before!

Now for that other bit of writing. The Book of PF page now refers to the tutorial page at bsdly.net. Now let's get that baby done.

The lady is, in fact, not too distressed.

And in case you were wondering - Yes, you can use my auto-generated list of trapped hosts for your own blacklisting purposes if you like. Here it's just a supplement to Bob Beck's traplist, and most likely you're better off using the Beck/UofA list along with your own greytrapping, but if you really want to use mine, be my guest. It gets updated ten past every hour.