Rickrolled? Get Ready for the Hail Mary Cloud!

If you publish your user name and password, somebody who is not you will use it, sooner or later.

It's been a fun few weeks. Over in Microsoft land, it must have been a big issue that according to malware hunters Sophos, the newly released Windows 7 with no extras is roughly as vulnerable as its older siblings. No great surprises there, I suppose.

For those of us with a more Unixish leaning, the more interesting piece of news involved Apple iPhones. These phones apparently run a version of MacOS that has enough Unix in it that with a bit of tinkering, it is possible to install a variety of Unix software, such as the ubiquitous secure shell daemon sshd. To some, there is a certain attraction in knowing that you have an SSH server in your pocket or handbag. Too bad, then that enough of those adventurous iPhone owners never read up on the instructions and chose to run their toy with the default password for the root account and were vulnerable to a wonderful prank perpetrated by a programmer down under.

The prank (described in the inimitable The Register style here) demonstrated just how bad an idea it is to publish your user name and password. If you followed the news around last weekend you would notice that a large segment of the Microsoft-attached instapunditry got their facts wrong as usual with the this proves that Apple (and by extension any Unix and of course Linux) is just as vulnerable as Microsoft mantra repeated over and over.

In fact, there are two historical incidents that point to Unix being no silver bullet: the 2002 Linux Slapper Worm and the original network-enabled worm, the 1988 Morris Worm. Those two prove mainly that yes, some bugs are exploitable, and the way forward is to fix bugs and make them harder to exploit in the first place. Now these two famous exploits is possibly to be joined by a third, the rickrolling prank.

I beg to differ. The rickroller is about bad passwords, no more, no less. I've spent considerable time ranting about passwords in earlier columns, and this incident only underscores what we've been repeating until your eardrums wear thin an my vocal cords swell from exhaustion:

Publishing your username and password is a really bad idea.
It's almost as bad as picking a guessable password.

Add to this that the fact, as we've noted here earlier, there is a whole cloud of hijacked machines out there beavering away at guessing passwords right now, and they have been at it for quite a while.

The most remarkable of these botnets is the one that tries to avoid detection by distributing the password guessing for any target across a large number of hosts, so each guesser never shows high enough rates of activity to trigger any of the traditional bruteforce detection mechanism. The attempts look something like this in your authentication log:

Nov 13 21:10:14 rosalita sshd[50401]: error: PAM: authentication error for illegal user mars from 125.40.69.208
Nov 13 21:10:14 rosalita sshd[50401]: Failed keyboard-interactive/pam for invalid user mars from 125.40.69.208 port 38052 ssh2
Nov 13 21:11:20 rosalita sshd[50419]: reverse mapping checking getaddrinfo for 115-186-131-90.nayatel.pk [115.186.131.90] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:11:20 rosalita sshd[50419]: Invalid user mars from 115.186.131.90
Nov 13 21:11:21 rosalita sshd[50419]: error: PAM: authentication error for illegal user mars from 115.186.131.90
Nov 13 21:11:21 rosalita sshd[50419]: Failed keyboard-interactive/pam for invalid user mars from 115.186.131.90 port 42235 ssh2
Nov 13 21:13:43 rosalita sshd[50428]: Invalid user mars from 58.247.222.163
Nov 13 21:13:43 rosalita sshd[50428]: error: PAM: authentication error for illegal user mars from 58.247.222.163
Nov 13 21:13:43 rosalita sshd[50428]: Failed keyboard-interactive/pam for invalid user mars from 58.247.222.163 port 35134 ssh2
Nov 13 21:15:59 rosalita sshd[50440]: Invalid user mars from 89.76.186.99
Nov 13 21:16:00 rosalita sshd[50440]: error: PAM: authentication error for illegal user mars from chello089076186099.chello.pl
Nov 13 21:16:00 rosalita sshd[50440]: Failed keyboard-interactive/pam for invalid user mars from 89.76.186.99 port 52156 ssh2
Nov 13 21:17:16 rosalita sshd[50448]: Invalid user mars2 from 88.134.166.31
Nov 13 21:17:16 rosalita sshd[50448]: error: PAM: authentication error for illegal user mars2 from 88-134-166-31-dynip.superkabel.de
Nov 13 21:17:16 rosalita sshd[50448]: Failed keyboard-interactive/pam for invalid user mars2 from 88.134.166.31 port 39254 ssh2
Nov 13 21:18:14 rosalita sshd[50452]: Invalid user room from 62.198.66.107
Nov 13 21:18:14 rosalita sshd[50452]: error: PAM: authentication error for illegal user room from 62.198.66.107
Nov 13 21:18:14 rosalita sshd[50452]: Failed keyboard-interactive/pam for invalid user room from 62.198.66.107 port 47557 ssh2
Nov 13 21:19:27 rosalita sshd[50458]: Invalid user room from 61.74.75.43
Nov 13 21:19:27 rosalita sshd[50458]: error: PAM: authentication error for illegal user room from 61.74.75.43
Nov 13 21:19:27 rosalita sshd[50458]: Failed keyboard-interactive/pam for invalid user room from 61.74.75.43 port 57794 ssh2
Nov 13 21:21:41 rosalita sshd[50472]: Invalid user room from 212.243.41.9
Nov 13 21:21:41 rosalita sshd[50472]: error: PAM: authentication error for illegal user room from 212.243.41.9
Nov 13 21:21:41 rosalita sshd[50472]: Failed keyboard-interactive/pam for invalid user room from 212.243.41.9 port 38396 ssh2
Nov 13 21:22:58 rosalita sshd[50491]: reverse mapping checking getaddrinfo for static-ip-cr1901468058.cable.net.co [190.146.80.58] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:22:58 rosalita sshd[50491]: Invalid user room from 190.146.80.58
Nov 13 21:22:58 rosalita sshd[50491]: error: PAM: authentication error for illegal user room from 190.146.80.58
Nov 13 21:22:58 rosalita sshd[50491]: Failed keyboard-interactive/pam for invalid user room from 190.146.80.58 port 4420 ssh2
Nov 13 21:24:01 rosalita sshd[50509]: Invalid user room from 62.23.130.173
Nov 13 21:24:01 rosalita sshd[50509]: error: PAM: authentication error for illegal user room from host.173.130.23.62.rev.coltfrance.com
Nov 13 21:24:01 rosalita sshd[50509]: Failed keyboard-interactive/pam for invalid user room from 62.23.130.173 port 3904 ssh2
Nov 13 21:25:21 rosalita sshd[50517]: reverse mapping checking getaddrinfo for hn.kd.ny.adsl [125.40.69.208] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:25:21 rosalita sshd[50517]: Invalid user room from 125.40.69.208
Nov 13 21:25:21 rosalita sshd[50517]: error: PAM: authentication error for illegal user room from 125.40.69.208
Nov 13 21:25:21 rosalita sshd[50517]: Failed keyboard-interactive/pam for invalid user room from 125.40.69.208 port 3294 ssh2

and so on.

I put it to you: What you see here is the cybercrime equivalent of the Hail Mary Pass".

Each attempt in theory has monumental odds against succeeding, but occasionally the guess will be right and they have scored a login. As far as we know, this is at least the third round of password guessing from the Hail Mary Cloud (see the archives for earlier postings about slow bruteforcers), but there could have been earlier rounds that escaped our attention.

The fact that we see the Hail Mary Cloud keeping up the guessing is a strong indicator that there are a lot of guessable passwords and possibly badly maintained systems out there, and that even against the very long odds they are succeeding often enough in their attempts to gain a foothold somewhere that it is worth keeping up the efforts. For one thing, the cost of using other people's equipment is likely to be quite low.

There are a lot of things about the Hail Mary Cloud and its overseers that we do not know. People who responded to the earlier articles with reports of similar activity also reported pretty consistently something like a sixty to seventy percent match in hosts making the attempts.

With 1767 hosts in the current sample it is likely that we have a cloud of at least several thousand, and most likely no single guessing host in the cloud ever gets around to contacting every host in the target list. The busier your SSH deamon is with normal traffic, the harder it will be to detect the footprint of Hail Mary activity, and likely a lot of this goes undetected.

The data, as I am sure you have been waiting for it, is available in these forms: Raw log data, with 3-4 lines per attempt (as illustrated above and requested by some correspondents), one line per attempt (shows the pattern a little more clearly), a list of the hosts participating in the Hail Mary Cloud sorted by number of attempts, and the user names attempted, sorted by number of attempts.

The pattern is fairly familiar by now, but this time the alphabetic cycles are shorter and at times the coordination seems to have broken down. My guess is that the apparent breakdowns are due to silly factors like the guessing machines running without time synchronization or other signs of incompetence.

And finally, some words of advice for those of you who want to avoid both rickrolling and getting cracked by other password guessing.

You should at least consider setting a password policy and enforcing it with something like John the ripper, which more than likely is available at the cost of a few keystrokes from your package system. And of course there is the fine art of sshd configuration. Some of the things you could do are, in no particular order:

• disable root logins over the network
  
• use packet filtering or other means to restrict where users can log in from
  
• disable password logins entirely allowing only key-based logins
  
• set up your sshd to listen on a non-standard port

whatever your users can bear to live with.

If you see traces of the Hail Mary Cloud's activity in your logs and you want to share and study, I would very much like to hear from you. I will most likely be updating the log data and extracts at intervals.

---

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.

---

Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

A Third Time, Uncharmed

Spamwashers hijacked, a wake-up call for lazy sysadmins everywhere. The slow bruteforcers are back for another round.

A new round of slow, distributed bruteforce attacks is in progress. Just like the other times we know about (see references later), the initial target is root. This time around I see only one of my ssh-contactable machines targeted, and the dribble started on September 30th.

I've put the raw data so far up for study here (a total of 6067 attempts), and a list of hosts sorted by number of attempts (the first column) can be found here (770 hosts, with up to 32 attempts each). Quite likely I'll be collecting more data and publishing updates when I have a few free moments.

A number of people were kind enough to contact me in the followup of the earlier articles, and from one of my correspendents (who asked not to be named) I learned that the likely culprit is a piece of Linux malware known as dt_ssh5. If you type dt_ssh5 into your favorite search engine, it will turn up a few hits, but significantly fewer than the number of hosts in my sample. A couple of those documents have some analysis of how a badly secured web application let the miscreants in.

We should not be surprised that whoever is behind this has more than one takeover method in their arsenal. Unfortunately it should not surprise anybody either that we're now seeing a third round of those attacks. Most likely the perpetrators keep going because they occasionally succeed, and when they do, it's because every now and then they luck on a Linux machine with either

• a maintenance regime that's disorganized enough that software with known and exploitable bugs is left in place for long enough to open the doors to undesirables, or


• at least one user (whoever is manning root or any of the other user IDs we know they will be sniffing out later) with a guessable password and a system administration regime that lets weak passwords exist in the first place.

You see where this is heading? Over the last few years we have seen Linux and other free systems take over ever more niches and even mission critical application areas in businesses and organizations all over the world. I for one think this move to free systems with source code accessible is a good thing.

But there is a downside: In our efforts to entice the suits into the wonderful new world of free software, we likely oversold the security part.

Yes, bugs are easier to find and fix when the source code is available, yes, the Unix security model or some variant thereof is vastly preferable to that disorienting hellhole system marketed from somewhere up Seattle way, and yes, there are a few hundred more good and valid reasons (technical and otherwise) why basing your business on free license software is generally a good idea. Security-wise you have a better starting point, and with a little effort you will gain control and stay in control. But it does take some degree of effort by one or more qualified persons who are willing and able to take charge and make sensible decisions.

Looking at the data earlier I can see that one of the hosts now trying to gain illegitimate access to one of my systems was originally set up as a "spam washer" (grep for spamvask in the data). Setting up a Linux box to filter spam is likely a good idea in many contexts, but please keep this in mind: The fact that your rig runs Linux does not mean you're home free. You need to keep paying attention. When your spam washer has been hijacked and tries to break into other people's systems, you urgently need to get your act together, right now.

We do not have self-propagating worms on Linux just yet (and in all fairness the likelihood of that ever happening is rather slim), but we certainly do not have a system that will stay secure if you ignore the basics of useful system administration. This recurring episode of dt_ssh5 and the slow bruteforcers should serve as a wakeup call for system administrators everywhere: Your system stays secure only as long as you pay proper attention to maintenance. Out there in the real world there are at least 770 machines where the maintenance regime slipped, either through incompetence or work overload or a combination of the two with a dash of bad planning thrown in.

The only way to make a fourth round of slow bruteforcers not happen is to make the people responsible for the 770 hosts listed here do the right thing. I suppose we will find out soon enough how many of those domains actually have the RFC2142-mandated mailboxes in place.

References: Previous posts on this topic include A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009).

---

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.

---

The reason for my rather long silence in the blogosphere can be summed up in two words: Client Confidentiality. Some of the projects I've been working on might have been material for interesting articles, but separating the generally useful bits from what the customer could reasonably expect to be kept confidential has proved difficult, at least in the short run.

Update 2009-10-05 just before 10:00 CEST: Slightly newer log extracts uploaded, total attempts now 6590, from 775 hosts. I will likely upload fresher versions at intervals.

Update 2009-10-14 19:30ish: After a couple of days with literally none to three or four attempts per hour, my afternoon log check turned up activity almost at start of run levels, so total numbers are now 9405 attempts from a total of 946 unique hosts. I never bothered to change the file names, but the data have been updated a couple of times each day since I started publishing them.

Update 2009-10-29 00:10: Because a file transfer took a little longer than expected, I was awake to see what may actually be the start of the alphabetic phase -


Oct 28 23:58:35 rosalita sshd[61664]: Invalid user anthony from 62.225.63.99
Oct 28 23:58:35 rosalita sshd[61664]: error: PAM: authentication error for illegal user anthony from 62.225.63.99
Oct 28 23:58:35 rosalita sshd[61664]: Failed keyboard-interactive/pam for invalid user anthony from 62.225.63.99 port 58683 ssh2
Oct 28 23:59:43 rosalita sshd[61668]: Invalid user anthony from 217.136.253.239
Oct 28 23:59:43 rosalita sshd[61668]: error: PAM: authentication error for illegal user anthony from 239.253-136-217.adsl-static.isp.belgacom.be
Oct 28 23:59:43 rosalita sshd[61668]: Failed keyboard-interactive/pam for invalid user anthony from 217.136.253.239 port 54728 ssh2
Oct 29 00:00:26 rosalita sshd[61695]: Invalid user barbara from 112.78.124.159
Oct 29 00:00:27 rosalita sshd[61695]: error: PAM: authentication error for illegal user barbara from 112.78.124.159
Oct 29 00:00:27 rosalita sshd[61695]: Failed keyboard-interactive/pam for invalid user barbara from 112.78.124.159 port 56990 ssh2
Oct 29 00:03:42 rosalita sshd[61722]: Invalid user brian from 122.224.128.197
Oct 29 00:03:42 rosalita sshd[61722]: error: PAM: authentication error for illegal user brian from 122.224.128.197
Oct 29 00:03:42 rosalita sshd[61722]: Failed keyboard-interactive/pam for invalid user brian from 122.224.128.197 port 47058 ssh2
Oct 29 00:04:21 rosalita sshd[61725]: Invalid user brian from 218.69.27.138
Oct 29 00:04:21 rosalita sshd[61725]: error: PAM: authentication error for illegal user brian from 218.69.27.138
Oct 29 00:04:21 rosalita sshd[61725]: Failed keyboard-interactive/pam for invalid user brian from 218.69.27.138 port 41622 ssh2
Oct 29 00:05:00 rosalita sshd[61738]: Invalid user carol from 58.60.106.121
Oct 29 00:05:01 rosalita sshd[61738]: error: PAM: authentication error for illegal user carol from 58.60.106.121
Oct 29 00:05:01 rosalita sshd[61738]: Failed keyboard-interactive/pam for invalid user carol from 58.60.106.121 port 55916 ssh2
Oct 29 00:05:51 rosalita sshd[61744]: Invalid user carol from 200.248.242.218
Oct 29 00:05:52 rosalita sshd[61744]: error: PAM: authentication error for illegal user carol from 200.248.242.218
Oct 29 00:05:52 rosalita sshd[61744]: Failed keyboard-interactive/pam for invalid user carol from 200.248.242.218 port 54547 ssh2
Oct 29 00:07:24 rosalita sshd[61748]: Invalid user charles from 222.128.36.60
Oct 29 00:07:24 rosalita sshd[61748]: error: PAM: authentication error for illegal user charles from 222.128.36.60
Oct 29 00:07:24 rosalita sshd[61748]: Failed keyboard-interactive/pam for invalid user charles from 222.128.36.60 port 42322 ssh2
Oct 29 00:08:09 rosalita sshd[61755]: Invalid user christopher from 72.148.242.188
Oct 29 00:08:10 rosalita sshd[61755]: error: PAM: authentication error for illegal user christopher from adsl-072-148-242-188.sip.asm.bellsouth.net
Oct 29 00:08:10 rosalita sshd[61755]: Failed keyboard-interactive/pam for invalid user christopher from 72.148.242.188 port 37157 ssh2
Oct 29 00:09:06 rosalita sshd[61758]: Invalid user christopher from 58.223.251.232
Oct 29 00:09:06 rosalita sshd[61758]: error: PAM: authentication error for illegal user christopher from 58.223.251.232
Oct 29 00:09:06 rosalita sshd[61758]: Failed keyboard-interactive/pam for invalid user christopher from 58.223.251.232 port 46350 ssh2
Oct 29 00:09:48 rosalita sshd[61762]: Invalid user daniel from 64.69.114.115
Oct 29 00:09:49 rosalita sshd[61762]: error: PAM: authentication error for illegal user daniel from mulligan.softspike.org
Oct 29 00:09:49 rosalita sshd[61762]: Failed keyboard-interactive/pam for invalid user daniel from 64.69.114.115 port 39960 ssh2
Oct 29 00:11:29 rosalita sshd[61781]: Invalid user david from 80.255.179.150
Oct 29 00:11:29 rosalita sshd[61781]: error: PAM: authentication error for illegal user david from ppp-vpdn-80.255.179.150.yarnet.ru
Oct 29 00:11:29 rosalita sshd[61781]: Failed keyboard-interactive/pam for invalid user david from 80.255.179.150 port 49013 ssh2

More material later, for now I'd be happy to hear confirmation (reports of similar activity) if you see it in your logs.

Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

The slow brute zombies are back

Real-life zombies feed off weak passwords.

Regular readers will remember that late last year we saw a peculiar form of distributed bruteforce attack on certain ssh servers. What made this particular batch of failed login attempts stand out was that unlike the traditional rapid-fire bruteforce attempts we were quite adept at heading off with state tracking tricks (such as the OpenBSD PF method described here or a slightly different Linux-specific method), the technique seemed to be specifically designed to slip past such defenses.

I described the method and the evidence at some length in a series of colums, A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008) and finally The slow brutes, a final roundup (January 22, 2009).

If you do not want to read through all of that, I'll recap: The traditional bruteforce attack is characterized by somebody trying again and again to find a combination of logon credentials (typically sets of user names and passwords) that will let them gain access to the system. The basic idea is that given enough tries, you will sooner or later find a user who has set a guessable password, and you have access.

One of the important weaknesses of the method is that the typical attacker would have gained access to only one or a few systems, and the attacker would then typically try to connect a lot more often than a typical user, and sysadmins learned to adapt their firewalls and other systems to lock out activity that fit the bruteforcer profile.

If you run a ssh service anywhere Internet-facing, you will be used to seeing a steady stream of failed logons for both existing and non-existing users. There's nothing new in seeing failed logons in your log files. However, what happened late last year was that we started seeing large numbers of failed ssh logon attempts, with the new twist that the same user would be trying to log on a large number of times, but never from the same place twice in rapid succession. This log data sample will give you an idea. The data will show you the pattern, as will the summary article.

Back then in January, my conclusion was that we had seen the conclusion of a largely failed experiment. After all, proceeding at a truly glacial pace and decreasing the number of attempts per user name as they went along hardly seemed like a recipe for success even if they could sneak past packet filtering based defenses.

It turns out I was wrong. Returning to my log summaries after taking a few days off from log data (yes, easter is big here, even moves geeks sometimes to take time off) I find data with almost exactly the same profile as last December's series.

This can only mean that there were enough successful attempts at guessing people's weak passwords in the last round that our unknown perpetrators found it worthwhile to start another round. For all I know they may have been at it all along, probing other parts of the Internet far from my unfashionable backwaters.

Anyway, they are most certainly back. A summary of the data so far follows in the concluding section. Please note that I would be very interested in communicating with other parties who see similar activity in their logs, and if you are responsible for one or more of the systems identified or you know who is, I urge you to take appropriate action.

The Data So Far

First, We Take root
The full log data reveal that they started rubbing up against my listening posts during the early hours of April 7th 2009 (as measured in CEST), concentrating first on the user root (attempts sorted by hostname here, hostnames here. Not terribly surprising that they tried to take on this one, as most Unix(ish) systems tend to at least have a root account.

Then We Take admin
After about twenty-four hours, the zombies in the cloud tired of root and turned their collective attention to admin (attempts sorted by hostname here, hostnames here. This is a user name I would not expect to be on a Unix syste by default. In my limited experience Microsoft systems tend to use this one more often, but then there may be enough of those systems out there with a SSH service and weak passwords out there to try.

And james Will Make Us Famous
Well into civilized lunchtime (CEST) on April 9th, our would-be guests turned their attention to james (attempts sorted by hostname here, hostnames here. Why this user name received so much attention is a complete mystery to me. They did tire in the end, though.

Before We Turn To 123456, aadi And The Rest Of The Rabble
As if to confirm that we are indeed seeing a rerun of last November and December's sequence of events, the robots started on their regular alphabetic attempts on April 11th:

Apr 11 17:44:15 rosalita sshd[51241]: error: PAM: authentication error for illegal user james from 221.130.177.154
Apr 11 17:50:47 rosalita sshd[51258]: error: PAM: authentication error for illegal user 123456 from 220.194.54.41
Apr 11 17:53:24 rosalita sshd[51262]: error: PAM: authentication error for illegal user 123456 from webmail.sistemafieg.org.br
Apr 11 17:58:25 rosalita sshd[51285]: error: PAM: authentication error for illegal user 123456 from 202.82.25.161
Apr 11 18:00:59 rosalita sshd[51307]: error: PAM: authentication error for illegal user aadi from mail.cgconsultoriacontabil.com.br
The various comments to my earlier columns tended to point out the near-futility of low-intensity bruteforcing, and I took the fact that the last series of attempts did not run to the end of the alphabet as an indication that the perpetrators themselves had reached the same conclusion.

Now we know that they just moved on, went elsewhere for a while and now they're back. In the meantime, they must have hit on enough weak minds and associated weak passwords that they were able to take over and zombiefy enough systems to be worth their while.

The data so far (collected up to April 12, 2009, roughly 21:00 CEST) indicates that the total number of hosts involved in the attempts at my listening posts is just over a thousand (a list of hosts available here).

It is probably worth repeating that in real life, zombies feed on both weak minds and weak passwords.

---

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available for free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.

---

Also worth noting, I will be doing a PF tutorial at BSDCan 2009 in May, and I will be staying around for the rest of the conference. I look forward to seeing you there.


Note: A Better Data Source Is Available
Update 2015-04-02: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

Oh yes, you signed up for this. You did. Honest.

Honesty in marketing. You may have heard of it.

It may come as a surprise to some, but I generally do not spend much time on spam related matters. Occasionally I need to do some manual labor to keep spamd and spamassasin in trim, but at most times my little robot helpers just keep running, leaving my desktop essentially spam free.

That changed slightly late last month. Messages hawking the oddest wares started arriving, with a largish number of messages claiming that I had actually signed up to receive them:

You are receiving this message because on 2/26/2009 at 3:57 PM peter@bsdly.net 64.12.116.10 registered to receive messages from e-researchcouncil.com and its partners. To change your preferences with e-researchcouncil.com, go to the website and select "Contact Us" to review your options.

To give you an idea how likely that statement is to be true, consider this: The 64.12.116.10 address resolves back to somewhere in America Online's network, pretty much an ocean and then some away from where I'm usually located.

I assume entering my address into a few web forms is somebody's idea of a joke, and the net effect was that a number of spammy messages started appearing in my mailbox, starting on February 27th. Only about third of the messages contained that particular claim, and a typical message would contain headers like these:

X-From-Line: eHarmonyDating@BranchSprint.com Fri Feb 27 16:30:36 2009
Return-path: <3f5.4.73479158-21937306@BranchSprint.com>
Envelope-to: peter@bsdly.net
Delivery-date: Fri, 27 Feb 2009 19:15:13 +0100
Received: from [99.198.152.161] (helo=dns7-cronomagic-biz.BranchSprint.com)
by skapet.bsdly.net with esmtp (Exim 4.69)
(envelope-from <3f5.4.73479158-21937306@BranchSprint.com>)
id 1Ld7Eu-00074N-NF
for peter@bsdly.net; Fri, 27 Feb 2009 19:15:13 +0100
X-Gnus-Mail-Source: pop:peter@bsdly.net
Message-Id: <KKcbjdhdagmcfbVN@BranchSprint.com>
Reply-To: <eHarmonyDating@BranchSprint.com<
From: eHarmonyDating <eHarmonyDating@BranchSprint.com>
Subject: eHarmony could match you with the right singles
Date: Fri, 27 Feb 2009 16:30:36 GMT
X-Information: 73479158_21937306 ListZA251
X-Complaints-To: <complaints@BranchSprint.com>
To: <peter@bsdly.net>
My first impulse was, in case this is an honest mistake somewhere, let's try and play nice at first. That meant sending messages to the X-Complaints-To: addresses and waiting to see what would happen.

You should not be terribly surprised to hear that those addresses all turned out to be invalid, the messages undeliverable.

In the meantime, I went on collecting messages, and the amount of data I had accumulated was large enough that I could reach some preliminary conclusions.

It's obvious that in order to reach me, the messages would need to clear greylisting and avoid triggering too many of my spamassassin rules. That meant in turn that the messages were sent using real mail servers. So I started collecting the messages with that claim for further study. The messages were almost all sent from a few distinct subnets, all of them apparently fairly well stocked with real mailservers.

Based on data from the spam messages and whois lookups and the larger groupings of messages, the professional spammers are, for your convenience in case you want to visit them:

NN, LLC
4001 Kennett Pike
Suite 134-910
Greenville, DE 19807
US

Spiesigma PLC
P.O. BOX 243, 2221 S Webster Ave
Green Bay, WI 54301
US

GreenButtonMedia.com
5580 La Jolla Blvd # 73
La Jolla, CA 92037
US

AdSelectMedia.com
5482 Wilshire Blvd. #302
Los Angeles, CA 90036
US

BestOnlineGreetings.com
5482 Wilshire Blvd. #302
Los Angeles, CA 90036
US

MyPromotionNetwork.com
970 West Valley Parkway
Suite 604
Escondido, CA 92025

GreatTechsOnline.com
5580 La Jolla Blvd # 73
La Jolla, CA 92037
US

CrownVenturesMedia.com
7127 Hollister Ave., Suite 25A, #145
Goleta, CA 93117

Top Notch Media, Inc.
1735 Market Street · Suite A · PMB 429
Philadelphia, PA 19103-7588

In addition, some of the domain names used in the spam messages were registered via an anonymizing service whose whois data comes out as:

Dynamic Dolphin Privacy Protect
5023 W 120th Ave #233
Broomfield
null,80020

The spam volume from all of them swelled at roughly the same time, so it is likely that they cooperate on keeping their lists up to date.

So we see spammers evolving: They buy or rent real mail servers now and they have even started coordinating. Using greylisting has actually increased the cost of becoming a successful spammer.

At our end of the game, we stay ahead of their game thanks to tools like spamd, and several of us dump and share our greytrap lists. It is even possible to collect IP addresses and feed a large number at the time to spamdb, but after a little while I grew tired of the increased manual work decided it was time for a counterprank. Cleaning up after spammers is no fun, unless you can have little robot helpers do the heavy lifting.


The Counterprank: A Feedback Loop
Regular readers will remember that I have a collection of known bad addresses in my domains that I use for my greytrapping, all generated elsewhere, that has come in handy at times. Run of the mill spam operators tend to just suck in anything that looks like email addresses, and keeping the list available on the web has served us extremely well here.

The professional spammers are apparently not quite that stupid, so the problem became a little different. They were able to sneak past greylisting and conventional content filtering. Also, they were apparently oblivious to email communication and as far as I can tell their Unsubscribe pages are not entirely believeable.

So it was a relief to find that places such as http://e-researchcouncil.com/ are very happy to accept any email addresss you can come up with. Time to enlist a few of our imaginary friends, drawn from the obvious source.

I did ponder the ethics for a few moments. After all, the forms included sentences such as "I certify that I am a US citizen", which was about as true as the assertion that I had signed up via an AOL proxy. But I did not ponder that matter for long. Moments later, most of the spam operators found themselves with new neighbors with odd names and foreign email addresses.

The net result is that the hosts start appearing automagically in the hourly dump of my list of greytrapped addresses and in the daily spamd activity report. With a little luck, we have succeeded in increasing the cost of spamming one tiny increment.

---

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive or non-trivial assistance, please contact me (via email or other means) to make arrangements.

Note that the list of greytrapped addresses is updated ten past every full hour, fetching it every minute like some Americans have started doing is not a good use of your resources.

What have the black boxes wrought

How compartmentalization turned into a security disaster. Greed, incompetence and dishonesty was involved. 

IT security or the lack of any such thing has grabbed headlines lately here in Norway. A series of high profile public institutions have seen large scale worm infections on their Microsoft based networks. 

Late last year the regional government agency responsible for essentially all health care in the western part of the country had a worm infection so bad they essentially shut down their network as a preventive measure. 

During the last few weeks, the national police force, of all thinkable organizations, has seen not one, but two large-scale incidents. Use of Microsoft products and sloppy system maintenance are both pervasive enough that similar incidents are likely happening right now elsewhere, somewhere near you too. 

The news reports about the Norwegian police force's IT problems contained one item that was particularly shocking to IT types like me: 

Apparently large parts of the bureaucracy that is responsible for the confidential and correct processing of criminal matters and all sorts of sensitive personal information associated with the crimes runs essential services on Microsoft Windows NT 4.0. 

That version of the Microsoft product is so old it is officially abandonware, and early reports of the police network problems included the oldish news that even the antiware vendors have stopped supporting the system. 

Later reports had police IT department officials claim that the worm infections were not that much of a security problem, since at this point all the worm actually did was spread. Break out the popcorn, boys and girls: In an upcoming episode, we will see how the worm infected Windows machines the Royal Norwegian Police did not find or couldn't clean well enough are used in the perpetration of some cybercrime or other. 

 It's all pretty sickening, and at this point it would be rather tempting to spend the rest of the column ranting about the general stupidity of Windows users. But a smarter approach is to see if there is a lesson to be learned. 

To do that, we need to backtrack quite a bit and look at the cult of the little black boxes. 

The cult of the little black boxes, and Microsoft the 1980s corporation 

We need to go back and take in what the world was like in the nineteen-eighties. This was back when the world was divided into real computers (from the likes of IBM, Digital, and regional quasi-monopolies like our own Norsk Data) and those annoying toys called 'personal' microcomputers, where the 'IBM PC compatibles' had emerged as the surprise leader of the pack. 

Computer networks were usually private, corporate things and rarely interconnected much, with the rare exception of those institutions that were part of the US Department of Defense science experiment that was becoming known variously as ARPANET or 'the Internet'. 

If you took your programming classes back in the nineteen-eighties, you likely know that we were taught that black boxes were good. 

Compartmentalization was the order of the day, meaning that as a developer you were supposed to create code with a well defined inteface, so anybody interacting with your code would be presented with a cleanly predictable result to any given input. 

Your code should be a black box and for any particular specificiation there could be written several interchangeable modules to fit the bill. 

So far so good, predictability is nice and with compartmentalization comes, we hope at least, clear chains of responsibility. 

But several factors combined to take the cult of the black boxes and turn it into a corporate culture at a corporation that was growing from a handful of furry hackers into a corporation at the time, namely Microsoft. 

Microsoft' early successes all came from writing software for single-user systems that were so limited that working around the limitations of the hardware became much of a lifestyle. 

At the start of the decade, networking on microcomputers in Microsoft's range was pretty much out of the question, and by the end of the eighties any sort of connectivity (even dial-up) was still an expensive optional extra that was still too hard to configure for most people. 

On the storage side, we progressed from 128 kilobyte floppies to hard drives of just over a hundred megabytes for personal systems, with the 32 megabyte partition size still a very present limiting factor.

Amazing developments all, but both the applications and the organization grew faster than the hardware could keep up with. 

The organization now had several levels of management, and each one demanded maximum productivity from their underlings. 

Keeping in mind that each programmer or team would be writing little black boxes anyway, it made perfect sense to set up the software production line so each developer only had access to those parts of the system he or she was supposed to be working on. 

That way developers would be concentrating on their main task and minimize time spent waiting for compiles to finish. At predetermined times the developers would then upload the source code for their little black boxes to a central build system. 

The only people who had all the parts of the projects were in fact the custodians of the build system. 

Source code version control systems were made part of the process, but there is anecdotal evidence that the transition from standalone hacking to a version control regime was a rough one for many early Microsoft developers. 

Only a few days ago I offered pretty much the content of the last few paragraphs to a table of youngish free software developers over beer. 

The reaction was quick an unanimous: 

"That way, nobody really knows what's going on in the software". 

That is a very valid point, and it proves how far we've come with free software. 

At the same time there is every reason to believe that the extreme compartmentalization that Microsoft established for its product development in the 1980s was the way things were done there until very recently, if indeed it has changed at all. 

By the mid-1990s when Microsoft had been dragged kicking and screaming into modern-day network environments, and the ongoing saga of internet-enabled malware started in earnest (I've written a summary in a malware paper -- an updated version is available here), with the company moving from early denial of any bugs whatsoever through a near-constant barrage of emergency hotfixes to today's monthly megapatch regime. 

With the source code still a closely guarded (if occasionally leaked) secret, there is really no way for us to know if they've learned any lessons at all. 

One indication that they still have some way to go is this Infoworld article about the state of their protocol documentation (summary: it's not to be trusted at all). As for the state of the source code, all we can do is to study the flow of urgent patches. 

Much better then to learn how it should be done - Say from Theo de Raadt's AsiaBSDCon 2009 presentation about how OpenBSD's release process works, and if you want more of the gory details, do check his classic exploit mitigation presentation. Also, most likely you could do worse than read Damien Miller's OpenSSH security presentation (full text here). 

It's all those little things we do, at FreeCode and in free software in general. If you found this column useful, entertaining or irritating, please drop me a line.

The slow brutes, a final roundup

The slow brutes stopped their churning. Their last call was for sophia.

Over the last few columns, we have followed the progress of what appears to be a botnet cloud's attempt at gaining access to a couple of FreeBSD machines I have in my care. One of my predictions about the distributed, slow ssh bruteforce attempts we started seeing in November of 2008 was that at the rate they were going at the time, it would be well into the new year before we would see the end of their alphabetic progression. As it turns out, they stopped just before year end, before even reaching the 'T's. The last attempt recorded was this:

Dec 30 11:09:03 filehut sshd[54981]: error: PAM: authentication error for illegal user sophia from static-98-119-110-139.lsanca.dsl-w.verizon.net

The full collection of raw data is available here, with a .csv summarising number of attempts, user names and hosts per day here.

With the incident apparently over, we can sit back and study the data and see what patterns emerge.

The anatomy of the attack
There are a number of ways to slice and dice the data. One useful way to view the collection is to do day to day statistics, such as the ones in this .csv file, numbers extracted by some simple greping and awkery. Based on the day to day data, I made this graph to illustrate the progression.



Then for your data overload cravings, I turned the same data to a log scale for enhancement and added number of attempts -



hopefully adding some insight into just what happened when and maybe supporting some guessing about what they were indeed trying to achieve.

It is possible that we missed the actual start of those coordinated attempts, but the data we do have show a few interesting points.

The earliest preserved data from November 19th shows the most attempts per user name (average 13.29), with 7 unique user names tried and a relatively low number of hosts (76).

On November 20th, the cloud turned its attention to one user, root, trying only that user name a total of 1697 times from 566 different hosts. It would be well into November 21st before the cloud moved on to admin (128 attempts, 107 different hosts) and an apparently coordinated alphabetic progression.

The absolute number of attempts and hosts involved per day fell quickly, with average number of attempts per user name stabilizing at a fairly low number after a few days. The exception is the peak on December 27th, which could perhaps be explained by owners of compromised computers returning from holiday celebrations and turning their computers back on. The sharp decline in all numbers over the next few days before the attempts stop seems consistent with what we assumed: That the botnet masters were allocating resources according to likelihood of success.

So why is this incident important, or even interesting? After all, attempts to gain access to services by brute force or dictionary based attacks are nothing new. I was rather intrigued to see clear evidence that miscreants were trying to find the way under the radar of intrusion detectors by distributing the intrusion task over a large number of hosts. If their success rate at my sites is anything to go by, this may be just a weird anomaly and an idea that did not lead anywhere. I haven't heard from anybody who was actually compromised by this particular set of clouds, but then again anybody who got bitten would likely be rather shy about telling the world at large or even fairly obscure researchers about the fact.

Always looking for patterns I even went to the trouble of extracting some data from the logs about the individual hosts that participated in the attack. After some basic shell gymnastics I ended up with a .csv of hostnames, number of attempts from the host as well as the date of first and last contact (available here). Next I tried (and failed - gnuplot gurus, here's your chance) to graph the data usefully in OpenOffice, but ended up with a sorted version (sorted by attempts, start and end date) that at least shows us that a surprising number of hosts actually hung on for most of the time the coordinated attepts went on.

The lessons learned: security the old-fashioned way

The general lesson of this incident is rather predictably that miscreants will occasionally try new and original ways to try to crack their way into your system. The slow method was a refreshing variation, and for all we know they may have succeeded in places where the people in charge remain blissfully unaware. Trying to catalogue and detect all kinds of variations on the theme of "attempts at unauthorized access" is the kind of activity that has kept "antivirus" people in beer money for quite a while, and if there is a lesson to be learned here, it is that trying to enumerate badness (Yes, do look it up using your favorite search) is a losing game. Make sure whatever system you run is sanely constructed, any bugs that do turn up are fixable within a reasonable time frame, and so on. I suppose I will come back later with a rant about how much damage the "black boxes" school of thinking about software has done, especially after it got elevated to practically religious dogma by certain major players. And yes, you can usefully look that up as well.

For those of you who are interested in the data, here are the now complete extracts for your perusal:

The full set of log data
The per day .csv file - and the same in an .ods sheet with some graphing attempts
Per host data in the "Host,Attempts,StartDate,EndDate" format and sorted by attempts, start and end

For those of you interested in learning about OpenBSD and related delights like PF, FreeCode is set to start offering courses featuring among others yours truly as well as the usual support and consulting offerings. Contact the good front end people at FreeCode for further details.

---

International readers are at liberty to ignore the following, but Norwegian online IT magazine digi.no are apparently in the process of setting up a sort of census of Norwegian bloggers. The following is there to make this blog show up in their listing. You can read their article about the initiative (Norwegian only, unfortunately).

Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.