Saturday, October 5, 2013

The Hail Mary Cloud And The Lessons Learned

Against ridiculous odds and even after gaining some media focus, the botnet dubbed The Hail Mary Cloud apparently succeeded in staying under the radar and kept compromising Linux machines for several years. This article, based on my BSDCan 2013 talk, sums up known facts about the botnet and suggests some common-sense measures to be taken going forward.

The Hail Mary Cloud was a widely distributed, low intensity password guessing botnet that targeted Secure Shell (ssh) servers on the public Internet.

The first activity may have been as early as 2007, but our first recorded data start in late 2008. Links to full data and extracts are included in this article.

We present the basic behavior and algorithms, and point to possible policies for staying safe(r) from similar present or future attacks.

But first, a few words about the devil we knew before the incidents that form the core of the narrative.

The Traditional SSH Bruteforce Attack

If you run an Internet-facing SSH service, you have seen something like this in your logs:

Sep 26 03:12:34 skapet sshd[25771]: Failed password for root from 200.72.41.31 port 40992 ssh2
Sep 26 03:12:34 skapet sshd[5279]: Failed password for root from 200.72.41.31 port 40992 ssh2
Sep 26 03:12:35 skapet sshd[5279]: Received disconnect from 200.72.41.31: 11: Bye Bye
Sep 26 03:12:44 skapet sshd[29635]: Invalid user admin from 200.72.41.31
Sep 26 03:12:44 skapet sshd[24703]: input_userauth_request: invalid user admin
Sep 26 03:12:44 skapet sshd[24703]: Failed password for invalid user admin from 200.72.41.31 port 41484 ssh2
Sep 26 03:12:44 skapet sshd[29635]: Failed password for invalid user admin from 200.72.41.31 port 41484 ssh2
Sep 26 03:12:45 skapet sshd[24703]: Connection closed by 200.72.41.31
Sep 26 03:13:10 skapet sshd[11459]: Failed password for root from 200.72.41.31 port 43344 ssh2


This is the classic, rapid-fire type of bruteforce attack, with rapid-fire login attempts from one source. (And yes, skapet is the Internet-facing host on my home network.)

The Likely Business Plan

These attempts are often preceded by a port scan, but in other cases it appears that the miscreants are just blasting away at random. In my experience, with the gateway usually at the lowest-numbered address, the activity usually turns up first there, before trying higher-numbered hosts. I'm not really in a mind to offer help or advice to the people running those scripts, but it might be possible to scan the internet from 255.255.255.255 downwards next time. Anyway, looking at the log excerpts, miscreants' likely plan is
  1. Try for likely user names, hope for guessable password, keep guessing until successful.
  2. PROFIT!
But then the attempts usually come in faster than most of us can type, so with a little help from toolmakers, we came up with an inexpensive first line of defense, easily implemented in perimeter packet filters (aka firewalls).

Traditional Anti-Bruteforce Rules

Rapid-fire bruteforce attacks are easy to head off. I tend to use OpenBSD on internet facing hosts, so first we present the technique as it has been available in OpenBSD since version 3.5 (released in 2005), where state tracking options are used to set limits we later act on:

In your /etc/pf.conf, you add a table to store addresses, block access for all traffic coming from members of that table, and finally amend your typical pass rule with some state tracking options. The result looks something like this:

table <bruteforce> persist
block quick from <bruteforce>
pass inet proto tcp to $int_if:network port $tcp_services \
        keep state (max-src-conn 100, max-src-conn-rate 15/5, \
         overload <bruteforce> flush global)

Here, max-src-conn is the maximum number of concurrent connections allowed from one host

max-src-conn-rate is the maximum allowed rate of new connections, here 15 connections per 5 seconds.

overload <bruteforce> means that any hosts that exceed either of these limits are have their adress added to this table

and, just for good measure, flush global means that for host that is added to our overload table, we kill all existing connections too.

Basically, problem solved - the noise from rapid-fire bruteforcers generally disappears instantly or after a very few attempts. If you are about to implement something like this (and many do -- the bruteforcer section in my PF tutorial appears to be among the more popular ones), you probably need to watch your logs to find useful numbers for your site, and tweak rules accordingly. I have yet to meet an admin who plausibly claims to never have been tripped up by their overload rules at some point. That's when you learn to appreciate having an alternative way in to your systems, such as a separate admin network.

Traditional Anti-Bruteforce Rules, Linux Style

For those not yet converted to the fine OpenBSD toolset (available in FreeBSD and other BSDs too, with only minor if any variations in details for this particular context), the Linux equivalent would be something like

sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 5 \ --hitcount 15 --rttl --name SSH -j DROP

But be warned: this will still be minus the maximum number of connections limit, plus the usual iptables warts. And you'd need a separate set of commands for ip6tables.

It's likely something similar is doable with other tools and products too, including possibly some proprietary ones. I've made something of an effort to limit my exposure to the non-free tools, so I can't offer you any more detail. To find out what your present product can do, please dive into the documentation for whichever product you are using. Or come back for some further OpenBSD goodness.

But as you can see, for all practical purposes the rapid-fire bruteforce or floods problem has been solved with trivial configuration tweaks.

But then something happened.

What's That? Something New!

On November 19th, 2008 (or shortly thereafter), I noticed this in my authentication logs:

Nov 19 15:04:22 rosalita sshd[40232]: error: PAM: authentication error for illegal user alias from s514.nxs.nl
Nov 19 15:07:32 rosalita sshd[40239]: error: PAM: authentication error for illegal user alias from c90678d3.static.spo.virtua.com.br
Nov 19 15:10:20 rosalita sshd[40247]: error: PAM: authentication error for illegal user alias from 207-47-162-126.prna.static.sasknet.sk.ca
Nov 19 15:13:46 rosalita sshd[40268]: error: PAM: authentication error for illegal user alias from 125-236-218-109.adsl.xtra.co.nz
Nov 19 15:16:29 rosalita sshd[40275]: error: PAM: authentication error for illegal user alias from 200.93.147.114
Nov 19 15:19:12 rosalita sshd[40279]: error: PAM: authentication error for illegal user alias from 62.225.15.82
Nov 19 15:22:29 rosalita sshd[40298]: error: PAM: authentication error for illegal user alias from 121.33.199.39
Nov 19 15:25:14 rosalita sshd[40305]: error: PAM: authentication error for illegal user alias from 130.red-80-37-213.staticip.rima-tde.net
Nov 19 15:28:23 rosalita sshd[40309]: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com
Nov 19 15:31:17 rosalita sshd[40316]: error: PAM: authentication error for illegal user alias from gate-dialog-simet.jgora.dialog.net.pl
Nov 19 15:34:18 rosalita sshd[40334]: error: PAM: authentication error for illegal user alias from 80.51.31.84
Nov 19 15:37:23 rosalita sshd[40342]: error: PAM: authentication error for illegal user alias from 82.207.104.34
Nov 19 15:40:20 rosalita sshd[40350]: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com
Nov 19 15:43:39 rosalita sshd[40354]: error: PAM: authentication error for illegal user alias from 200.20.187.222
Nov 19 15:46:41 rosalita sshd[40374]: error: PAM: authentication error for illegal user amanda from 58.196.4.2
Nov 19 15:49:31 rosalita sshd[40378]: error: PAM: authentication error for illegal user amanda from host116-164.dissent.birch.net
Nov 19 15:55:47 rosalita sshd[40408]: error: PAM: authentication error for illegal user amanda from robert71.lnk.telstra.net
Nov 19 15:59:08 rosalita sshd[40412]: error: PAM: authentication error for illegal user amanda from static-71-166-159-177.washdc.east.verizon.net

... and so on. The alphabetic progression of user names went on and on.

The pattern seemed to be that several hosts, in widely different networks, try to access our system as the same user, up to minutes apart. When any one host comes back it's more likely than not several user names later. The full sequence (it stopped December 30th), is available here.

Take a few minutes to browse the log data if you like. It's worth noting that rosalita was a server that had a limited set of functions for a limited set of users, and basically no other users than myself ever logged in there via SSH, even if they for various reason had the option open to them. So in contrast to busier sites where sequences like this might have drowned in the noise, here it really stood out. And I suppose after looking at the data, you can understand my initial reaction.

The Initial Reaction

My initial reaction was pure disbelief.

For the first few days I tried tweaking PF rules, playing with the attempts/second values and scratching my head, going, "How do I make this match?"

I spent way too much time on that, and the short version of the answer to that question is, you can't. With the simple and in fact quite elegant state tracking options, you will soon hit limits (especially time limits) that interfere with normal use, and you end up blocking legitimate traffic.

So I gave up on prevention (which really only would have rid me of a bit of noise in my authentication logs), and I started analyzing the data instead, trying to eyeball patterns that would explain what I was seeing. After a while it dawned on me that this could very well be a coordinated effort, using a widely distributed set of compromised hosts.

So there was a bit of reason in there after all. Maybe even a business plan or model. Next, I started analyzing my data, and came up with -

Bruteforcer Business Plan, Distributed Version

The Executive Summary would run something like this: Have more hosts take turns, round robin-ish, at long enough intervals to stay under the radar, guessing for weak passwords.

The plan is much like before, but now we have more host on the attacking side, so
  1. Pick a host from our pool, assign a user name and password (picked from a list, dictionary or pool)
  2. For each host,
    1. Try logging in to the chosen target with the assigned user name and password
    2. If successful, report back to base (we theorize); else wait for instructions (again we speculate)
  3. Go to 1).
  4. For each success at 2.2), PROFIT!

You're The Target

Let's recap, and take step back. What have we learned?

To my mind at least, it all boils down to the basics
  • Your Unix computer (Linux, OpenBSD, FreeBSD or other) is a desirable, powerful thing.
  • If your password is weak, you will be 0WN3D, sooner rather than later.
  • There's a whole fleet out there, and they're coordinated.
At this point I thought I had something useful, so I started my first writeup for publication. I had just started a new job at the time, and I think I mentioned the oddities to some of my new colleagues (that company is unfortunately defunct, but the original linked articles give some information). Anyway, I wrote and published, hoping to generate a little public attention for myself and my employer. And who knows, maybe even move a few more copies of that book I'd written the year before.


Initial Public Reaction

On December 2, 2008, I published the first blog post in what would become a longish sequence, A low intensity, distributed bruteforce attempt, where I summarized my findings. It's slightly more wordy than this piece, but if I've piqued your interest so far, please go ahead and read. And as to a little public attention, I got my wish. The post ended up slashdotted, the first among my colleagues to end up with their name on the front page of Slashdot.

That brought
  • more disbelief, (see slashdot and other comments) but also
  • confirmation, via comments and email, that others were seeing the same thing, and that the first occurrences may have been seen up to a year earlier (November-ish 2007).

The slow bruteforcers were not getting in, so I just went on collecting data. I estimated they'd be going on well past new year's if they were going to reach the end of the alphabet.


On December 30th, 2008, The Attempts Stopped

The attempts came to an end, conveniently while I was away on vacation. The last entries were:

Dec 30 11:03:08 rosalita sshd[51108]: error: PAM: authentication error for illegal user sophia from 201.161.28.9
Dec 30 11:05:08 filehut sshd[54932]: error: PAM: authentication error for illegal user sophia from 201.161.28.9
Dec 30 11:06:35 rosalita sshd[51116]: error: PAM: authentication error for illegal user sophia from static-98-119-110-139.lsanca.dsl-w.verizon.net
Dec 30 11:09:03 filehut sshd[54981]: error: PAM: authentication error for illegal user sophia from static-98-119-110-139.lsanca.dsl-w.verizon.net


That is, not even completing a full alphabetic cycle.

By then they had made 29916 attempts, all failed. You can find the full listing at here).

Trying 6100 user IDs (list by frequency here). More than likely you can guess the top one without even looking.

From a total of 1193 different hosts (list by frequency here).

As I said earlier, there were no successful penetrations. Zero.


Common characteristics

The slashdot story brought comments and feedback, with some observations from other sites. Not a lot of data, but enough that the patterns we had observed were confirmed. The attempts were all password authentication attempts, no other authentication methods attempted.

For the most part the extended incident consisted of attempts on an alphabetic sequence of 'likely' user names, but all sites also saw at least one long run of root only attempts. This pattern was to repeat itself, and also show up in data from other sources.

There would be anything from seconds to minutes between attempts, but attempts from any single host would come at much longer intervals.

First Round Observations, Early Conclusions

Summing up what we had so far, here are a few observations and attempts at early conclusions.

At the site where I had registered the distributed attempts, the Internet-reachable machines all ran either OpenBSD or FreeBSD. Only two FreeBSD boxes were contacted.

The attackers were hungry for root, so having PermitRootLogin no in our sshd config anywhere Internet facing proved to be a good idea.

We hadn't forced our users to keys only, but a bit of luck and John the Ripper (/usr/ports/security/john) saved our behinds.

The number of attempts per user name had decreased over time (as illustrated by this graph), so we speculated in the second article Into a new year, slowly pounding the gates (on slashdot as The Slow Bruteforce Botnet(s) May Be Learning) that success or not was measured at a command and control site, with resources allocated accordingly.

With the sequence not completed, we thought they'd given up. After all, the odds against succeeding seemed monumental.

After all, a couple of slashdotted blog posts couldn't have hurt, could they?


But Of Course They Came Back

As luck would have it, whoever was out there had not totally admitted defeat just yet. In the early hours CET, April 7th, 2009, the slow brutes showed up again:

Apr 7 05:02:07 rosalita sshd[4739]: error: PAM: authentication error for root from ruth.globalcon.net
Apr 7 05:02:15 rosalita sshd[4742]: error: PAM: authentication error for root from ip-206-83-192-201.sterlingnetwork.net
Apr 7 05:02:54 rosalita sshd[4746]: error: PAM: authentication error for root from cyscorpions.com
Apr 7 05:02:59 rosalita sshd[4745]: error: PAM: authentication error for root from smtp.bancomorada.com.br
Apr 7 05:03:10 rosalita sshd[4751]: error: PAM: authentication error for root from 82.192.86.217
Apr 7 05:03:25 rosalita sshd[4754]: error: PAM: authentication error for root from 66.135.60.203
Apr 7 05:03:52 rosalita sshd[4757]: error: PAM: authentication error for root from rainha.florianonet.com.br
Apr 7 05:04:00 rosalita sshd[4760]: error: PAM: authentication error for root from 72.11.144.34
Apr 7 05:04:34 rosalita sshd[4763]: error: PAM: authentication error for root from s1.serverhex.com
Apr 7 05:04:38 rosalita sshd[4765]: error: PAM: authentication error for root from mail.pitnet.com.br



Starting with 2318 attempts at root before moving on to admin and proceeding with the alphabetic sequence. The incident played out pretty much like the previous one, only this time I was sure I had managed to capture all relevant data before my logs were rotated out of existence.

The data is available in the following forms: Full log here, one line per attempt here, users by frequency here, hosts by frequency here.

I couldn't resist kicking up some more publicity, and indeed we got another slashdot storm out of the article The slow brute zombies are back, on slashdot as The Low Intensity Brute-Force Zombies Are Back.

And shortly afterwards, we learned something new -

Introducing dt_ssh5, Linux /tmp Resident

Of course there was a piece of malware involved.

A Linux binary called dt_ssh5 did the grunt work.

The dt_ssh5 file was found installed in /tmp on affected systems. The reason our perpetrators chose to target that directory is likely because the /tmp directory tends to be world readable and world writeable.

Again, this points us to the three basic lessons:




  • Stay away from guessable passwords
  • Watch for weird files (stuff you didn't put there yourself) anywhere in your file system, even in /tmp.
  • Internalize the fact that PermitRootLogin yes is a bad idea.

  • dt_ssh5: Basic Algorithm

    The discovery of dt_ssh5 made for a more complete picture. A rough algorithm suggested itself:


    1. Pick a new host from our pool, assign a user name and password
    2. For each host,
      1. Try user name and password
      2. if successful
        1. drop the dt_ssh5 binary in /tmp; start it
        2. report back to base
        else wait for instructions
    3. Go to 1.
    4. For each success at 2.2, PROFIT!

    I never got myself a copy, so the actual mechanism for communicating back to base remains unclear.


    The Waves We Saw, 2008 - 2012

    We saw eight sequences (complete list of articles in the References section at the end),

    From - To AttemptsUser IDsHostsSuccessful Logins
    2008-11-19 15:04:22 - 2008-12-30 11:09:0329916610011930
    2009-04-07 03:56:25 - 2009-04-12 21:01:371264124911040
    2009-09-30 21:15:36 - 2009-10-15 13:42:079998110710
    2009-10-28 23:58:35 - 2010-01-22 09:56:2444513811041580
    2010-06-17 01:55:34 - 2010-08-11 13:23:0123014388755680
    2011-10-23 04:13:00 - 2011-10-29 05:40:0747739443380
    2011-11-03 20:56:18 - 2011-11-26 17:42:19490724742520
    2012-04-01 12:33:04 - 2012-04-06 14:52:1147571081230


    The 2009-09-30 sequence was notable for trying only root, the 2012-04-01 sequence for being the first to attempt access to OpenBSD hosts.

    We may have missed earlier sequences, early reports place the first similar attempts as far back as 2007.


    For A While, The Botnet Grew

    From our point of view, the swarm stayed away for a while and came back stronger, for a couple of iterations, possibly after tweaking their code in the meantime. Or rather, the gaps in our data represent times when it focused elsewhere.

    Clearly, not everybody was listening to online rants about guessable passwords.

    For a while, the distributed approach appeared to be working.

    It was (of course) during a growth period I coined the phrase "The Hail Mary Cloud".

    Instantly, a myriad of "Hail Mary" experts joined the insta-punditry on slashdot and elsewhere.

    It Went Away Or Dwindled

    Between August 2010 and October 2010, things either started going badly for The Hail Mary Cloud, or possibly they focused elsewhere.

    I went on collecting data.

    There wasn't much to write about, except possibly that the botnet's command and control was redistributing effort based on past success. Aiming at crackable hosts elsewhere.


    And Resurfaced In China?

    Our last sighting so far was in April 2012. The data is preserved here.

    This was the first time we saw Hail Mary Cloud style attempts at accessing OpenBSD systems.

    The majority of attempts were spaced at at least 10 seconds apart, and until I revisited the data recently, I thought only two hosts in China were involved.

    In fact, 23 hosts made a total of 4757 attempts at 1081 user IDs, netting 0 successful logins.

    The new frequency data I thought interesting enough to write about, so I wrote up If We Go One Attempt Every Ten Seconds, We're Under The Radar, and netted another slashdotting. I took another look at the data later and slightly amended the conclusions, the article has been corrected with proper data extracted.


    Then What To Do?

    The question anybody reading this far will be asking is, what should we do in order to avoid compromise by the password guessing swarms? To my mind, it all boils down to common sense systems administration:

    Mind your logs. You can read them yourself, or train a robot to. I use logsentry, other monitoring tools can be taught to look for anomalies (failed logins, etc)

    Keep your system up to date. If not OpenBSD, check openssh.com for the latest version, check what your system has and badger the maintainer if it's outdated.

    And of course, configure your applications such as sshd properly -

    sshd_config: 'PermitRootLogin no' and a few other items

    These two settings in your sshd_config will give you the most bang for the buck:

    PermitRootLogin no
    PasswordAuthentication no
    

    Make your users generate keys, add the *.pub to their ~/.ssh/authorized_keys files.

    For a bit of background, Michael W. Lucas: SSH Mastery (Tilted Windmill Press 2013) is a recent and very readable guide to configuring your SSH (server and clients) sensibly. It's compact and affordable too.


    Keep Them Out, Keep Them Guessing

    At this point, most geeks would wax lyrical about the relative strengths of different encryption schemes and algorithms.

    Being a simpler mind, I prefer a different metric for how good your scheme is, or effectivness of obfuscation (also see entropy):

    How many bytes does a would-be intruder have to get exactly right?
    I've summed up the answer to that question in this table:

    Authentication methodNumber of bytes
    PasswordPassword length (varies, how long is yours?)
    Alternate PortPort number (2 bytes, it's a 16 bit value, remember)
    Port KnockingNumber of ports in sequence * 2 (still a 16 bit value)
    Single Packet Authentication2 bytes (the port) plus Max 1440 (IPv4/Ethernet) or 1220 (IPv6/Ethernet)
    Key OnlyNumber of bytes in key (depending on key strength, up to several kB)


    You can of course combine several methods (with endless potential for annoying your users), or use two factor authentication (OpenSSH supports several schemes).



    Keys. You've Got To have Keys!

    By far the most effective measure is to go keys only for your ssh logins. In your sshd_config, add or uncomment

    PasswordAuthentication no
    

    Restart you sshd, and have all users generate keys, like this:

    $ ssh-keygen -C "userid@domain.tld"
    

    There are other options to play with, see ssh-keygen(1) for inspiration.

    Then add the *.pub to their ~/.ssh/authorized_keys files.

    And I'll let you in on a dirty little secret: you can even match on interface in your sshd config for things like these



    Why Not Use Port Knocking?

    Whenever I mention the Hail Mary Cloud online, two suggestions always turn up: The iptables example I mentioned earlier (or link to the relevant slide), and "Why not use port knocking?". Well, consider this:

    Port knocking usually means having all ports closed, but with a daemon reading your firewalls logs for a predetermined sequence of ports. Knock on the correct ports in sequence, your're in.

    Another dirty little secret: It's possible to implement port knocking with only the tools in an OpenBSD base system. No, I won't tell you how.

    Executive Summary: Don't let this keep you from keeping your system up to date.

    To my mind port knocking gives you:
    1. Added complexity or, one more thing that will go wrong. If the daemon dies, you've bricked your system.
    2. An additional password that's hard to change. There's nothing magical about TCP/UDP ports. It's a 16 bit number, and in our context, it's just another alphabet. The swarm will keep guessing. And it's likely the knock sequence (aka password) is the same for all users.
    3. You won't recognize an attack until it succeeds, if even then. Guessing attempts will be indistinguishable from random noise (try a raw tcpdump of any internet-facing interface to see the white noise you mostly block drop anyway), so you will have no early warning.
    Port knocking proponents seem to have sort of moved on to single packet authentication instead, but even those implementations still contain the old port knocking code intact.

    If you want a longer form or those arguments, my November 4, 2012 rant Why Not Use Port Knocking? was my take (with some inaccuracies, but you'll live).



    There's No Safety In High Ports Anymore

    Another favorite suggestion is to set your sshd to listen on some alternate port instead of the default port 22/TCP.

    People who did so have had a few years of quiet logs, but recent reports show that whoever is out there have the resources to scan alternate ports too.

    Once again, don't let running your sshd on an alternate port keep you from keeping your system up to date.

    Of course I've ranted about this too, in February 2013, There's No Protection In High Ports Anymore. If Indeed There Ever Was. (which earned me another slashdotting).

    Reports with logs trickle in from time to time of such activity at alternate ports, but of course on any site with a default deny packet filtering policy will not see any traces of such scans unless you go looking specifically at the mass of traffic that gets dropped at the perimeter.



    Final thoughts, for now

    Microsoftish instapundits were quick to assert that ssh is insecure.

    They're wrong. OpenSSH (which is what essentially everyone uses) is maintained as an integral part of the OpenBSD project, and as such is a very thoroughly audited mass of code. And it keeps improving with every release.

    I consider the Hail Mary Cloud an example of distributed, parallel problem solving, conceptually much like SETI@Home but with different logic and of course a more sinister intent.

    Computing power is cheap now, getting cheaper, and even more so when you can leverage other people's spare cycles.

    The huge swarm of attackers concept is as I understand it being re-used in the recent WordPress attacks. We should be prepared for swarm attacks on other applications as soon as they reach a critical mass of users.

    There may not be a bullseye on your back yet (have you looked lately?), but you are an attractive target.

    Fortunately, sane system administration practices will go a long way towards thwarting intrusion attempts, as in
    • keep your system up to date,
    • allow only what's necessary for the task at hand and
    • keep watching your logs for weirdness.
    Keep it simple, stay safe.

    UPDATE 2013-11-21: A recent ACM Conference on Computer and Communication Security paper, "Detecting stealthy, distributed SSH brute-forcing," penned by Mobin Javed and Vern Paxson, references a large subset of the data and offers some real analysis, including correlation with data from other sites (Spoiler alert: in some waves, almost total overlap of participating machines). One interesting point from the paper is that apparently attack matching our profile were seen at the Lawrence Berkeley National Laboratory as early as 2005.

    And in other news, it appears that GitHub has been subject to an attack that matches the characteristics we have described. A number of accounts with weak passwords were cracked. Investigations appears to be still ongoing. Fortunately, GitHub appear to have started offering other authentication methods.

    UPDATE 2014-09-28: Since early July 2014, we have been seeing similar activity aimed at our POP3 service, with usernames taken almost exclusively from our spamtrap list. The article Password Gropers Take the Spamtrap Bait has all the details and log data as well as references to the spamtrap list.

    UPDATE 2014-12-10: My Passwords14 presentation, Distributed, Stealthy Brute Force Password Guessing Attempts - Slicing and Dicing Data from Recent Incidents has some further data as well as further slicing and dicing of the earlier data (with slightly different results). 

    UPDATE 2016-08-10: The POP3 gropers never went away entirely and soon faded into a kind of background noise. In June of 2016, however, they appeared to have hired themselves out to a systematic hunt for Chinese user names. The article Chinese Hunting Chinese over POP3 in Fjord Country has further details, and as always, links to log data and related files.

    UPDATE 2020-04-28: Somewhat belatedly added a link to the more recent (August 2018) article Badness, Enumerated by Robots which has updates on data we currently collect with some further explanation. Also see links therein.

    References

    The slides for the talk this article is based on live at http://home.nuug.no/~peter/hailmary2013/, with a zipped version including all data at http://home.nuug.no/~peter/hailmary2013.zip (approx. 26MB) for your convenience.

    Mobin Javed and Vern Paxson, "Detecting stealthy, distributed SSH brute-forcing," ACM International Conference on Computer and Communication Security (CCS), November 2013.

    The blog posts (field notes) of the various incidents, data links within:

    Peter N. M. Hansteen, (2008-12-02) A low intensity, distributed bruteforce attempt (slashdotted)

    Peter N. M. Hansteen, (2008-12-06) A Small Update About The Slow Brutes

    Peter N. M. Hansteen, (2008-12-21) Into a new year, slowly pounding the gates (slashdotted)

    Peter N. M. Hansteen, (2009-01-22) The slow brutes, a final roundup

    Peter N. M. Hansteen, (2009-04-12) The slow brute zombies are back (slashdotted)

    Peter N. M. Hansteen, (2009-10-04) A Third Time, Uncharmed (slashdotted)

    Peter N. M. Hansteen, (2009-11-15) Rickrolled? Get Ready for the Hail Mary Cloud! (slashdotted)

    Peter N. M. Hansteen, (2011-10-23) You're Doing It Wrong, Or, The Return Of The Son Of The Hail Mary Cloud

    Peter N. M. Hansteen, (2011-10-29) You're Doing It Wrong, Returning Scoundrels

    Peter N. M. Hansteen, (2012-04-06) If We Go One Attempt Every Ten Seconds, We're Under The Radar (slashdotted)

    Peter N. M. Hansteen, (2012-04-11) Why Not Use Port Knocking?

    Peter N. M. Hansteen, (2013-02-16) There's No Protection In High Ports Anymore. If Indeed There Ever Was. (slashdotted)

    Peter N. M. Hansteen (2018-08-13) Badness, Enumerated by Robots (updates on data we currently collect, also see links therein).

    Other Useful Texts

    Marcus Ranum: The Six Dumbest Ideas in Computer Security, September 1, 2005

    Michael W. Lucas: SSH Mastery, Tilted Windmill Press 2013 (order direct from the OpenBSD bookstore here)

    Michael W. Lucas: Absolute OpenBSD, 2nd edition No Starch Press 2013 (order direct from the OpenBSD bookstore here)

    Peter N. M. Hansteen, The Book of PF, 3rd edition, No Starch Press 2014, also the online PF tutorial it grew out of, several formats http://home.nuug.no/~peter/pf/, more extensive slides matching the most recent session at http://home.nuug.no/~peter/pf/newest/. More recently I've done a rewrite of the PF tutorial with Max Stucchi co-authoring and co-presenting, the lastest slides from one of these sessions are available at https://home.nuug.no/~peter/pftutorial/ (sorry, labs available for real attendees only).

    OpenBSDs web http://www.openbsd.org/ -- lots of useful information.


    If you enjoyed this: Support OpenBSD!

    If you have enjoyed reading this, please buy OpenBSD CDs and other items, and/or donate!

    Useful links for this are:

    OpenBSD.org Orders Page: http://www.openbsd.org/orders.html

    OpenBSD Donations Page: http://www.openbsd.org/donations.html.

    OpenBSD Hardware Wanted Page: http://www.openbsd.org/want.html.

    Remember: Free software takes real work and real money to develop and maintain.

    If you want to support me, buy the book! (if you want to give the OpenBSD project a cut of that, this is the link you want).

    34 comments:

    1. "I'm not really in a mind to offer help or advice to the people running those scripts, but it might be possible to scan the internet from 255.255.255.255 downwards next time."

      I'm sure hitting all the multicast addresses first will be highly effective ;)

      ReplyDelete
      Replies
      1. I think it would be a better idea for those script kiddies to start in the middle of the internet: 127.0.0.1

        Delete
    2. I've seen the same pattern happening on RDP ports, I am not surprised they are trying with SSH these days..

      ReplyDelete
    3. There is a tool called denyhosts that solves your problem trivially. I myself wrote similar tool in awk and shell and used it for many years. It is funny how same problem comes up over and over again and people look for their own solution and analysis instead of using existing tools.

      ReplyDelete
    4. Or pick a nonstandard port for sshd.

      ReplyDelete
      Replies
      1. Did you even read the article?

        Delete
      2. simple port scan away your port is known. Only harasses legitimate users. Obfuscation is not a form of security.

        Delete
    5. Couldn't & shouldn't you set the 'noexec' flag on /tmp? Or even fake a partition for /tmp like

      shell> dd if=/dev/zero of=tmp.iso bs=516096 seek=9182 count=0
      shell> mkfs -t ext4 /tmp
      shell> mount -o loop tmp.iso /tmp
      shell> echo "/path/to/tmp.iso /tmp ext4 nosuid,noexec,nodev 0 0" >> /etc/fstab

      ReplyDelete
    6. Glancing through the log, it looks like this "low and slow" detection algorithm could have helped too. (e.g, I see multiple tries from "static-98-119-110-139.lsanca.dsl-w.verizon.net")

      http://www.filetransferconsulting.com/how-to-detect-and-prevent-low-and-slow-brute-force-attacks/

      ReplyDelete
    7. This is why I disable password logins and only allow auth via public key for any internet-facing servers.

      ReplyDelete
    8. We see similar slow attacks on passwords for SMTP AUTH, pop3/imap, ftp, etc. Any protocol that uses username/password for authentication is vulnerable to such a slow distributed attack.

      ReplyDelete
    9. I don't agree that root login is a bad idea. It is a bad idea to have root login with a guessable password. Just use one that these f****** cannot guess in 1000 years and you are fine.

      ReplyDelete
      Replies
      1. Sorry, there's no useful reason to allow remote root logins on *ANY* server, whether it's facing the Intardwebz or not. Any password you can come up with can probably be broken, it may just take some effort. Why open yourself to the risk? There's no benefit to it whatsoever. Sudo is your friend. Learn it, use it, love it.

        ssh root@ should never be allowed.

        (Oh, and disable password auth, as well).

        Delete
      2. You do know that a ssh key is just a long password, right?

        Delete
      3. I used to create an alternate root account and ssh altroot@, with password. I now use sudo. Using the formula above, security is measured in how many bytes the attackers have to get exactly right. It turns out that renaming the root account (to a username as long as your non-root account) is just as secure as signing on as yourself and using sudo; after all, you're just typing the same password over again at the sudo prompt.

        I would switch to ssh keys only, except that I want to be able to sign on in the event I don't have my ssh key with me.

        We should log what password these slow-brutes are using, because they're effectively broadcasting the ip addresses, usernames and passwords of all of the hosts in their botnet.

        Delete
      4. An ssh key is, indeed, just a long password; but remember that it's in two parts: the passphrase, and the private key.

        Together, those are usually quite long.

        Delete
    10. I've seen the same issues. I use sshblack. It should work in your case too.

      ReplyDelete
    11. I've got a SSH server on a different port + check the logs daily, and in the last 5 years of running it, I can't think of a time that we've had anyone trying to connect to it. The other ssh server (chrooted/sftponly) on the standard port on that machine gets lots of attempts though.
      Perhaps that's part of the solution? - what amounts to a honey pot on a standard port and the actual SSH server somewhere else?

      ReplyDelete
    12. I don't see why a server needs to allow everyone to try to login. I limited access to a handful of specific addresses, and everyone else gets ignored.

      ReplyDelete
      Replies
      1. Travelling.. Hotels, unknown locations, etc..

        They will all have IP address that you cannot know in advance, and in many cases will operate with dhcp so even if you did know the address, it may change.

        Delete
      2. VPN back to the office and connect from there. As a backup I can VPN to my home and connect from there to. Only 2 ip-addresses are allowed ssh login.

        Delete
    13. A little thing that helped me, since I'm administrating my servers from a randomly changing dialin IP:
      Only allow IPs access to your services, which you are on. Create an iptables entry for that that updates regulary (wrote myself a script for that).

      For that, use dyndns of any kind. Of course, people like the NSA or other bigbiz could still force companies (if you use a commercial service) into changing your dyndns - ip into something of their preference. But even then all they have is a normal SSH-Login with pubkey authentication and "PermitRootLogin No". AND you'd instantly know if something went astray, if you suddenly can't access your servers via SSH (or other services like eg. NRPE) anymore.

      In short: Allow access to services on your public IP addresses only to known-probably-ok IP addresses, and most of your botnet-related problems are gone. Of course there are still exploits out there, so you are not completely in the green. Good luck :)

      HiSPeed

      ReplyDelete
    14. I use DenyHosts and Google Authenticator PAM module for SSH logins on all my internet facing hosts. Both are open source and fairly easy to setup on various OS'.

      ReplyDelete
    15. Besides changing the port, on the Linux side; I use fail2ban. It takes care of this automatically with iptables ( for any that do find the port ). I believe that the BSD flavors have it as well.

      ReplyDelete
    16. I use port knocking via my Mikrotik router to allow access to port 22. I know your dislike of Port Knocking, but it pretty much stops these random IP scans. You really need multiple layers of security to protect your servers, and its best if all that traffic is stopped before it even reaches your server.

      With my setup, you have to hit random ports with specific data (the router does Layer 7 inspection) on 5 different non-sequential ip addresses on my subnet all within 1 second to enable access. Hit any other ip/port with any packet of data, and you are going to be starting over. This of course then just gives you access to the NATed port (not 22) to ssh to my servers (key only, no pass auth)

      There doesn't have to even be ports or even servers actually listening on those IPs, its all handled in the router itself. I then have rules in place that will black hole your IP for 24 hours if you attempt to brute force it.

      ReplyDelete
    17. Just a quick note, with iptables why not use the "hashlimit" module? That should achieve the same as the pf filter.

      ReplyDelete
    18. Hi Peter, I enjoyed reading your thorough and informative presentation of this topic. Thank you.

      ReplyDelete
    19. There is a free cloud based service at https://jumpcloud.com that does a good job of identifying and blocking SSH brute attacks that could possibly succeed

      ReplyDelete
    20. Hey Peter... I've been seeing these style attacks in my logs as well... also running OpenBSD and started noticing them back when you wrote that first article! They are still there, plugging away day after day.

      ReplyDelete
    21. Keys are good, but only as good as your endpoint security. If the user doesn't password-protect his key, and his laptop gets stolen, the thief/attacker is in unless the admin gets notified in time and revoked the key. If the user does password his key, there's no way to verify password strength without some sort oglf agent running on the laptop.

      On the other hand, brute-forcing s key passsword may be harder due to lack of plain text in the key file (or maybe not, I don't know). And I suppose it's still less risk than brute-forcers over the net.

      ReplyDelete
    22. I also saw the same slow distributed attacks on my OpenBSD box back in 2008. I always wanted to reply back to these attacks. One day I tried to log into one of those by guessing the login/pass. It took me few tries to get admin/admin. Then I deleted the executable in /tmp and killed the daemon in memory. I also changed the admin password before logging out.

      It gives me the idea to write a daemon to do this automatically based on wrong usernames (the ones not in passwd) or overload table. I never managed to get the motivation to write this, but it could have been an interesting project, put aside the illegal nature of this project.

      ReplyDelete
    23. What strikes me in the first usernames list is that they are mostly french woman names. There seems to be a pattern in second one too, but can't identify it (the first one was easy cause I'm french).

      I wonder, does people having spotted the attack in the same time on their hosts report about the same list ?

      If not, that could mean an even more sophisticated (albeit totally failed here) attack trying to guess sex / location of host before selecting a dictionary.

      ReplyDelete
    24. Hi, I would like to seek your opinion on which is best to protect my server.

      1) Disable root login, create new user, create SSH for new user, disable new user password

      2) Disable root login, create new user, create Password for new user, no SSH for new user.

      Please advise! I'm in sooooooooo much confusion! Thanks :))

      ReplyDelete
    25. Four years late to the party...but still here are my comments.

      Excellent article, not just the technical content but also the writing style. Reminds me of the Cuckoo's Egg.

      Why not build a table Hail Mary IPs and block them in PF? Perhaps the IPs can also be shared using BGP (like Hessler's presentation for spamd). I am not a networking expert - would it be possible to build a spamd equivalent for sshd and redirect HM IPs to a fakeSSHD to just waste their time?

      ReplyDelete

    Note: Comments are moderated. On-topic messages will be liberated from the holding queue at semi-random (hopefully short) intervals.

    I invite comment on all aspects of the material I publish and I read all submitted comments. I occasionally respond in comments, but please do not assume that your comment will compel me to produce a public or immediate response.

    Please note that comments consisting of only a single word or only a URL with no indication why that link is useful in the context will be immediately recycled so those poor electrons get another shot at a meaningful existence.

    If your suggestions are useful enough to make me write on a specific topic, I will do my best to give credit where credit is due.