Is one of your machines secretly a spambot?

Some times we just need facts on the table, automated.

In my previous blog post, I wondered aloud about publishing data about the machines that verifiably tried to spam us. The response was other than overwhelming, and with the script running once per day anyway, I now publish the results via the Name And Shame Robot page.

The annoucement below is very close to the text there, so by way of explanation, here is a gift to all my fellow spamd junkies out there:

We started actively greytrapping and publishing our list of greytrap addresses (almost exclusively addresses generated or made up elsewhere and harvested from our logs) during July 2007. The list of greytrap addresses is published on the Traplist page along with some commentary. You can find related comments in this blog post and its followups.

One byproduct of the greytrapping is a list of IP addresses that has tried to deliver mail to one or more of our greytrap addresses during the last 24 hours. The reasoning is, none of these addresses are valid, and any attempts at delivering to those addresses is more likely than not spam. You can download that list here as a raw list of IP addresses, or as a DNS zone file intended as a DNS blacklist here.

In early August 2008, I wrote a small script that copies (rsyncs, actually) the current list of trapped IP addresses as well as the spamd log off the firewall and for each IP address collects all log entries from the spamd log. The resulting file is rsynced back to the webserver, and you can view the latest version here.

The material here is useful mainly to the system administrators responsible for the machines that appear in it, or people who are interested in studying spammer or spambot behavior. Times are given according to the Europe/Oslo time zone (CET or CEST according to season), and if a date appears several times for an IP address entry, the reason is simply that the log data spans several years. The default syslog settings do not record the year.

In the data you will find several kinds of entries, most of them are pretty obvious and straightforward, others less so. The likely FAQ is, "what are the entries with no log data?". The answer is, the spamd here synchronizes with a spamds at other sites. The entries without log data entered our traplist through a sync operation, but the host did not attempt direct contact here.

The other likely question is, "what is that becks list?". It's what the rest of the world refers to as uatraps. I copied the data for that list into my config from Bob Beck's message on OpenBSD-misc and didn't notice that the list had an official name until much later.

Please note that this is not an up-to-the minute list. Depending on the number of hosts currently in the list of trapped addresses, the script's run time could be anything up to several hours. For that reason, the script starts at the time stated at the beginning of the report file and runs until it finishes generating. The last thing the script does is to rsync the report file to the webserver. For the time being, I archive older versions off-line.

This is now a totally hands-off, automated operation. The report is currently generated on a Pentium IV-class computer with few and only occasional other duties. If you have any comments or concerns, the address in the next sentence is the one I use for day to day email. If you find this data useful, donations of faster hardware or money (paypal to peter@bsdly.net or contact me for bank information) is of course welcome.

Now that we have their addresses, do we name and shame?

The legal owners of botnet-controlled spam senders are quite likely unaware what their machines are doing. Do they deserve to be outed, named and shamed?

Earlier this week a friendly Australian who I think had been reading my blog sent me a few questions about spam, spammers and what to do with them. Would it for example be useful to forward the IP addresses in the local traplist to law enforcement? After all, I publish a dump of IP addresses from my local-greytrap once per hour, and apparently at least some people are fetching and using that as a valid blacklist on a regular basis.

(On a side note: if you do fetch that list regularly, keep in mind that the data is dumped ten past every hour, that's when the data is fresh. If you fetch at every full hour, the data is already fifty minutes old).

Anyway, my initial reaction to the question about forwarding the list of IP addresses to law enforcement was, along the lines of "Well, a raw list of IP addresses doesn't really add up to a lot of evidence, but if you can extract the log entries for each one, you may have something". My actual answer was phrased a little differently, but even while I was writing my reply I started fiddling with a script to read my list of trapped IP addresses and grep the spamd log for all entries for each IP address.

My complete collection of spamd logs goes back a few years, so searching for a complete history does take a while. (For techies: for each IP address, a grep of the entire log takes at least a few seconds (s) , total time is s is times number of entries (N), typically a few thousand, and grepping in parallel is difficult, because you want the output per IP address, not interlaced like in the raw log data).

After a while, you can see output roughly like this:


Host 81.183.80.187:
Aug 7 07:24:16 skapet spamd[13548]: 81.183.80.187: connected (12/9)
Aug 7 07:24:30 skapet spamd[13548]: (GREY) 81.183.80.187:
<akstcabrushwithafricamnsdgs@abrushwithafrica.com> -> <bennett-gauvin@ehtrib.org>
Aug 7 07:24:31 skapet spamd[13548]: 81.183.80.187: disconnected after 15 seconds.
Aug 7 07:24:44 skapet spamd[13548]: 81.183.80.187: connected (9/7)
Aug 7 07:25:06 skapet spamd[13548]: (GREY) 81.183.80.187:
<akstcamplepleasuremnsdgs@amplepleasure.net> -> <bennett-gauvin@ehtrib.org>
Aug 7 07:25:07 skapet spamd[13548]: 81.183.80.187: disconnected after 23 seconds.
Aug 7 07:25:08 skapet spamd[13548]: 81.183.80.187: connected (11/9)
Aug 7 07:25:23 skapet spamd[13548]: (GREY) 81.183.80.187:
<akstcaesamnsdgs@aesa.ch> -> <bennett-gauvin@ehtrib.org>
Aug 7 07:25:24 skapet spamd[13548]: 81.183.80.187: disconnected after 16 seconds.
Aug 7 07:26:16 skapet spamd[13548]: 81.183.80.187: connected (11/9), lists: spamd-greytrap
Aug 7 07:30:00 skapet spamd[13548]: (BLACK) 81.183.80.187:
-> <bennett-gauvin@ehtrib.org> -> <bennett-gauvin@ehtrib.org>
Aug 7 07:31:43 skapet spamd[13548]: 81.183.80.187: From: "Frances Ballard"
-> <bennett-gauvin@ehtrib.org>
Aug 7 07:31:43 skapet spamd[13548]: 81.183.80.187: To: <bennett-gauvin@ehtrib.org>
Aug 7 07:31:43 skapet spamd[13548]: 81.183.80.187: Subject: Extraordinary Narcotic Deals
Aug 7 07:32:47 skapet spamd[13548]: 81.183.80.187: disconnected after 391 seconds.
lists: spamd-greytrap


That's rougly what I would have expected to see: A host tries to send obvious spam to one of the trap addresses (one I harvested from incoming noise earlier), is added to spamd-greytrap and on the next attempts gets stuck for a few minutes. (Notice that this spammer has another version of grepable From: addresses - prepend akstc and append mnsdgs to the basename so abrushwithafrica.com becomes the junk address akstcabrushwithafricamnsdgs@abrushwithafrica.com. Content and header filterers, please take note.) I thought that this would be the typical behavior, but browsing the output from my script, entries of this kind seems to be more of the norm:

Host 81.192.185.9:
Aug 6 12:47:15 skapet spamd[13548]: 81.192.185.9: connected (12/8)
Aug 6 12:47:27 skapet spamd[13548]: (GREY) 81.192.185.9:
<jacowen@teaneckschools.org> -> <hevadcouture@bsdly.net>
Aug 6 12:47:27 skapet spamd[13548]: 81.192.185.9: disconnected after 12 seconds.

Here, the spambot tries exactly once, never to return. It's possible they detect the stuttering (our side answers one byte per second for the first ten seconds) and give up for that reason, but it could equally well be that it's classic fire-and-forget, the reason why greylisting still works. Or both, for that matter.

But back to the real question: Now that we have the data, what do we do with it?

With the script I have now, extracting the history for each of several thousand IP addresses takes some hours. The output is enlightening, but by the time the run is complete, it could be significantly more than twenty-four hours since the machines listed tried to send spam.

Should we name and shame anyway? If we forward the data to law enforcement, would they care?

For the time being, I'll try to think of a quicker way to extract the data. Any input on how to make the process more efficient is welcome, as is considered (learned or otherwise) opinion on the ethical up- or downside of publishing spamd log data.

Wanna help science? Study your greylists' innards!

If somebody, say five years ago, had told me that I would be spending a little time, every day, studying data about what invalid addresses some unknown miscreants are making up in my domains, I would have thought them to be slighly off their rockers.

Yet here I am, actually maintaining a publicly available list of addresses which do not stand a chance of becoming valid, ever. It all started with a log data anomaly - I noticed an increase in the number of failed delivery messages to non-existent addresses in our domains. I had expected that the bounces to invalid addresses would appear for a short period only, but for one reason or the other it looks like it's here to stay, with some dips and peaks like the ehtrib.org flood.

The list is apparently working as intended too. These addresses are on my local greytrap list, and I have started seeing addresses I put in there as all uppercase turn up in my logs in all lowercase variants. Fun to watch, sort of.

Anyway, the supply of new bogus addresses proved to be larger than I had expected. So to get a handle on just what is happening I ended up doing periodic dumps of the live greylist data. This is really easy to do if you're using spamd as your greylister, your basic command is

$ sudo spamdb | grep GREY

and you redirect to a file, pipe to mail, or whatever you like.

Now if you're a bit like me, looking for patterns in the noise like this makes you feel a little weirder than usual and possibly lead you to think of a Clive Barker novel (specifically the bits about the dead letter file in The Great and Secret Show) and you wonder why this is worth doing at all. After all there is precious little spam that actuall reaches my users, so like I said earlier, for us spamd users it really looks like spam is a solved problem. I guess I'm just a bit fascinated by the pure irrationality of the spammers' behavior.

From the data I collect here in my tiny corner of the world to browse when time allows there may be useful information lurking somewhere.

Typical entries show things like the host 202.152.33.43 tried to send with a From: address jcejft@charter.com to dkqvujfn@datadok.no and sdenuuu@datadok.no. Using a few common networking commands we see that there is no reason why charter.com email should come from the IP range belonging to idola.net.id, and as the admin of datadok.no I know these two addresses have never been deliverable. Most likely the admin at charter.com can tell you if that from address is deliverable, but I keep wondering how much of the spam out there is stuffed into the pipe with bogus From: and To: addresses both. Or in other words, purely useless noise, never to be delivered anywhere.

On a side note, with one or more of the spammer operations trying to sneak through using sender and recipient addresses in the target domain, I assume it is just a matter of time before I see a tuple with both sender and recipient addresses already in my spamtraps list. When that happens, I think I will feel inclined to let my friends have a round of refreshments on my tab.

It's obvious that there are a handful of spammer operations that have decided to use datadok.no (and to a lesser extent, dataped.no and ehtrib.org) From: addresses on the spam they send, apparently in an attempt to cover their tracks. I will probably never know why they decided to do that, but I wonder why they keep it up and for that matter how many other domains are seeing this, with bounces from strange places, directed at non-existent, fairly obviously generated bogus addresses.

So if you are seeing similar stupidity in your logs and if you are running a sensible greylister such as spamd, I would be interested in hearing from you so we can compare notes.

Out there in meatspace, EuroBSDCon 2007 is coming up. I'll be there with the PF tutorial on Wednesday. This Friday's deadline for an updated manuscript had totally slipped my mind (I blame the book and a few other, less rational, factors), but hopefully the 24 who signed up for the session will find it useful anyhow - there will be new bits and as much interesting stuff as I can manage. I'll be around for the rest of the conference too, but unfortunately I'll have to give the Legonland trip a miss.

Be seeing you in Copenhagen! The book is getting closer to finished, I promise!

We see your every move, spammer

My logs tell me that the spamtrap topic is a favorite, and more likely than not somebody who read the announcement will also take a peek at the traplist itself. So while I'm slowly preparing a post about something else entirely (which what I feel is actually a lot more interesting), it can't hurt to fill you in on what I've been doing to keep track of spammer behavior.

It's a quiet life, at least by surface appearances. In between the steady stream of mainly confidential tasks handled at Datadok and the odd request to bsdly.net for services of one kind or the other, I focus on getting the book done, chapter by chapter.

The traplist is slowly expanding. The collection process itself is automated for all the tedious tasks. The "Unknown user" entries from my mail server logs as a source of traplist material almost dried up, so I started looking at the greylists directly.

After sampling my greylists at random intervals for a while, a short shell script now dumps the data to somewhere safe ten past every full hour, notes the number of grey entries and TRAPPED entries, and dumps the TRAPPED IP addresses to a file which is available to the world from the traplist page. The list is comfortably short at most times. I imagine somebody with beefier bandwidth or a more widely known domain would have more hosts trapped at any time.

The file with currently trapped hosts gets overwritten each time the script runs. There is an outside chance that the other generated data might be useful in future research, and storage is cheap these days, so I keep the data around.

Observing the greylists reveal some odd things, like a certain Taiwanese host which tried, on August 1st, 2007, to send roughly a thousand messages to one address in a domain elsewhere, using generated From: addresses at every host name and IP address in our local network. They probably thought they'd found an open relay. Spamd's "250 This is hurting you more than it is hurting me." probably did not register with them as an outright rejection, much like it fools a number of web available open relay detectors.

The conclusions still stand, though. They echo the conclusions from the malware paper (*): the spammers are working harder at sending their trash mainly because we are as close as does not matter to always correctly detecting and dealing with their junk traffic.

I keep wondering if even the few minutes' worth of work a day updating the traplist is worth it, since we are catching essentially all spam anyway. Then at intervals, one or more of the generated, made up addresses from the list actually turns up in my greylist dumps.

(*) Whenever the "The silent network" paper comes up in discussions, it looks like depending on who you are, it's either way too long or too short. At twenty-few pages it's too long for the attention span of the loudmouth self-appointed SMTP experts you may encounter on web forums and mailing lists, and too short (read: not a book) to carry much weight with a decision maker who will not read much more than the executive summary anyway. Making that article morph into a book is on my list of Things To Look Into Later If Time Allows And It Still Makes Sense Then.

If you're still there after reading all this: Click the ads already. Make somebody else pay for your entertainment.

Linux is easier than Windows, hands down

It all started a few weeks back when my wife and I decided to get her a new laptop. It was also conveniently close to her July 14th birthday and Dell had a huge campaign going in the newspapers. Dell had just started selling laptops with various Linuxes preinstalled, and Ubuntu was the environment of choice, so I felt reasonably sure that the machine would be usable even if the Linux preinstalled option is not available in Norway yet. After all, our daughter had essentially no trouble at all getting her laptop (a sleek little LG number) going with Ubuntu earlier this year.

So I ended up clicking my way on to the Dell web site and ordering an Inspiron 1520 with 1440x900 resolution and the Ruby Red option (actually it's just the lid, but it does look distinctive).

As luck and Dell's suppliers would have it, the color added some time to the delivery. The machine apparently shipped earlier than expected, and arrived here last Monday afternoon, most likely very close to when the plane with my wife and daughter touched down at Madeira.

I had already burned a 7.04 desktop iso to CD and booted from that. As with anything that comes with Microsoft preinstalled, you need to be really careful to hit the right key at the right moment unless you want to do battle with the Microsoft installer.

Eventually I succeeded, though, the thing booted, went into the traditional graphic Ubuntu startup and dropped me to a shell. Odd. Fortunately from my OpenBSD laptop I was able to find this ubuntuforums.org post which explains in some detail what needs to be done. In particular, the piece about how to fix the graphics driver is quite instructive.

Getting things done right when the graphics card and the wireless network card both need proprietary drivers loaded is a little puzzling when the system is supposed to be all graphic and your default network is wireless, but by following the instructions from the forum I made it eventually.

The short version is, in addition to the oddities involved in getting proprietary drivers the machine requires, the Feisty installer kernel for some reason does not have the proper support for the SATA controller in this Dell model.

Going back to the earlier version lets you do a clean upgrade, and to get all the bits working you need to have both the universe and the multiverse repositories in your package manager's configuration. Not hard for somebody who has not yet recycled all the grey cells with Debian etched into them, just a bit tedious, and even though all steps of the process was accompanied by sensible messages from the system, our only live Ethernet is in the attic where the servers live.

Anyway by elevenish in the evening I had the system all installed with native resolution and 32 bit color depth, on the wireless network and Just Working. The process took significantly less time than putting Windows from restore CDs back on the system it came with.

Yet another evening spent on other things than I had intended (such as writing those crucial bits of the book), but seriously folks, you must never miss an opportunity to make your wife happy.

With that out of the way, I can state with even more confidence:

Linux is easier and more user friendly than Windows.

By now I consider that to be documented beyond question.

But for the things I care about, I really prefer OpenBSD or FreeBSD.

Updates: The spamtrap is growing by a few new addresses a day. I sometimes spot them flying by in the xterm where my spamd log is tail -f'd, sometimes I grep them out of the mail server's logs. Some patterns are emerging, but more later when I have more data.

Spam is a solved problem

Executive summary: Spam is a solved problem, email works again. There are a few knuckle draggers out there who haven't noticed yet, but we'll get around to dealing with those shortly.

I've been looking over my log summaries again. My regular logs get rotated out of existence after seven days, but from the summaries I do keep around, it looks like various made up @datadok.no addresses have been used as spammers' fake From: addresses for about a month. I was too busy with other Very! Urgent! Things! to notice at first, but it finally dawned on me when I searched my mail server logs for "Unknown" as in "Unknown user" and saw from the results that somebody, somewhere, was using that domain for generating sender addresses.

Note: This piece is also available without trackers but classic formatting only here.

After about two weeks of observation and collecting made up or generated addresses for my traplist, my conclusion is what the title of this post says. Spam is not a problem anymore. I know, of course, that "how to cope with email and spam" self help guides are best sellers, and a recent Salon.com piece even went so far as saying about email,

Now, it seems, we're drowning. There's simply too much e-mail. The tide of spam buries valuable messages.

That kind of surprises me, because it's not what we're seeing here at all. Of course we know that there's a lot of junk being sent, but ask any of the people on the sites I run on any given day how much spam they've received recently and they have to look up the date of the last one in their "Junk mail" folders.

I do see some spam myself, mainly because I still fetch and read mail I receive at an ISP address I've used a lot for USENET and mailing lists over the years. And since unfortunately no method ever has a zero error rate, occasionally a spam message or two trickles through that shouldn't have on the systems I run myself. But if the tide of spam buries valuable messages, you haven't kept up with the technology, plain and simple.

By and large, from the perspective of somebody who has been the purported sender of an unknown portion of the tide which drowns out the Salon.com writer's messages, it looks like spam is treated correctly or at least in ways that do not annoy others unnecessarily at most sites. (In all fairness that piece is more about email versus other types of writing than technical matters, and certainly worth reading for that reason. The same writer has written a number of other articles which are worth your time too.)

At the last count, our main spamd running gateway had all of 316 addresses in the local spamd-greytrap table, meaning that only that many hosts have actually tried to send mail to one or more of the addresses listed at our spamtrap page during the last 24 hours. Some of the trapped machines would have been active spam senders, and most of the rest seem to have been sites which were configured to receive spam and bounce back to the From: address when the spam was not deliverable.

That is an important point to note. If your system sends a 'message undeliverable' bounce message for spam sent to a non-existent user, it is configured to deliver spam to the users you do have, and there are certainly ways to avoid that. I've decided not to plug any of my other writing directly in this post, but you should be able to find the references easily enough if you're interested.

Reading the spamd logs is sometimes quite entertaining if you're that kind of guy or girl. Here is one example of a site with clearly deficient spam and/or malware filtering, possibly their own homebrew:

Jul 14 09:13:28 delilah spamd[29851]: 66.35.252.70: From postmaster@trendmicro.com
Jul 14 09:13:28 delilah spamd[29851]: 66.35.252.70: To: jonson3846@datadok.no
Jul 14 09:13:28 delilah spamd[29851]: 66.35.252.70: Subject: Delivery Status Notification (Failure)

It does not matter much to us, but they'll be unable to get mail through to us for the next 24 hours.

The next one is clearly problematic, since whoever set up the system appears to have left back in the time when there still was a chance that spammers used real addresses. Or maybe the poor wretch stayed on and now suffers from delusions, incompentence or both:

Jul 13 14:36:50 delilah spamd[29851]: 212.154.213.228: Subject: Considered UNSOLICITED BULK EMAIL, apparently from you
Jul 13 14:36:50 delilah spamd[29851]: 212.154.213.228: From: "Content-filter at srv77.kit.kz" <postmaster@srv77.kit.kz>
Jul 13 14:36:50 delilah spamd[29851]: 212.154.213.228: To:
<skulkedq58@datadok.no>


Would SPF have helped? Possibly. We have our records set up, but clearly these guys are not using it in any meaningful way, and after -- what is it -- five years it's still not clear which of the competing RFCs with varying degrees of proprietary content is going to come out on top.

Staying out of our traplist would have saved some resources on their side. On our side, well, we have a working system. Email works. Mail from us has to go through our mail server. Incoming mail needs to clear spamd's greylisting (and really needs to come from IP addresses which are not in any of the blocklists we use) and pass content filtering inspection by spamassassin and clamav, all of it conveniently within reach on any freely available BSD system. The content filtering packages are available on your favorite Linux as well, but on our sites, we use OpenBSD and FreeBSD.

Living spam free and unworried by malware is possible. If you make a few right choices it's actually easy, it doesn't cost much and just imagine how much of your time you stop wasting.

Hey, spammer! Here's a list for you!

Last week I started noticing from my log summaries that my mail servers had seen a lot more mail to non-existent users than usual. This usually happens when somebody has picked one of our domains as the home of their made-up return addresses for their spam run. This time, from the looks of it, the spam runs were mainly targeted at Russian and Ukrainian users. At least that's where most of the backscatter appears to have come from.

As I've written before in the PF tutorial and the malware paper (updated version available as this blog post -- that's the end of today's plugging, I promise), I've used the "Unknown user" messages as a valuable data source for my spamtrap list, just quitely adding addresses that looked really unlikely to ever become valid. After a brief airing on the OpenBSD-misc mailing list and running it by my colleagues at Datadok and Dataped, I've decided to take it a bit further.

Now that I've got a list of addresses which will never receive any legitimate mail, I really want spammers to try to send mail to those addresses. After all, if they send anything to an address which consists of a random string with one of our domains stuck on after the '@', we know it's all spam from there on.

We don't care about the rest, for the next 24 hours. Your SMTP dialogue with us (actually our spamd) will be all a-stutter, receiving answers one byte at the time until you give up. For the record, that usually takes about 400 seconds, with the really imbecile ones taking a lot longer. See the paper or the tutorial for some numbers.

The other possibility is of course that your system is set up in a way which makes it actually receive and try to deliver spam. Some of the spam will be addressed to non-existent users in your domain, so if your users receive spam, you will be trying to send bounce messages back to the purported sender for spam to non-existent users. That's tough, kid. If you're set up that way, your machine will be treated to the tarpit here for the next 24 hours. All a-stutter and all that. Repeat offenders stay there longer.

Now for the spamtrap list, I've checked that my colleagues and associates have never actually wanted to use those addresses for anything, and I made this page which wraps it all in a bit of explanation. For some reason, the list keeps growing each time I look at my log summaries.

When I get around to it and find a visually not-too-horrible way to do it, I'll include links to that page where they fit naturally on our web sites. In the meantime, here's hoping that the spammers' address harvesting robots find this list and put it to good use.

The chapter, it's improving. More later.

UPDATE 12-jul-2007: The softer side of me ponders the possibility of sending email form letters to the various postmaster@s with the URL to this blog post. On the other hand, I'm not sure I'm ready for another round of finding out that postmaster@ is in fact not deliverable at a surprising number of sites around the world.

One other thing I've noticed since I published the traplist is that bounces to addresses like mixt.apex.dp.ua-1184227575-testing@datadok.no have started appearing in the logs. I don't see how messages like these could be useful by themselves, but the addresses are of course obvious traplist material.

13-jul-2007: Oddly enough, there's still a stream of backscatter, and my logs tell me a few new addresses turn up every day. This morning's fresh ones were careersogt2083@datadok.no, phalanxesxb88@datadok.no and retryingvtt@datadok.no. Another few bytes to help weed out the bad ones early, thanks to the robots out there.