The Hail Mary Cloud was a widely distributed, low intensity password guessing botnet that targeted Secure Shell (ssh) servers on the public Internet.
The first activity may have been as early as 2007, but our first recorded data start in late 2008. Links to full data and extracts are included in this article.
We present the basic behavior and algorithms, and point to possible policies for staying safe(r) from similar present or future attacks.
But first, a few words about the devil we knew before the incidents that form the core of the narrative.
The Traditional SSH Bruteforce Attack
If you run an Internet-facing SSH service, you have seen something like this in your logs:
Sep 26 03:12:34 skapet sshd: Failed password for root from 126.96.36.199 port 40992 ssh2
Sep 26 03:12:34 skapet sshd: Failed password for root from 188.8.131.52 port 40992 ssh2
Sep 26 03:12:35 skapet sshd: Received disconnect from 184.108.40.206: 11: Bye Bye
Sep 26 03:12:44 skapet sshd: Invalid user admin from 220.127.116.11
Sep 26 03:12:44 skapet sshd: input_userauth_request: invalid user admin
Sep 26 03:12:44 skapet sshd: Failed password for invalid user admin from 18.104.22.168 port 41484 ssh2
Sep 26 03:12:44 skapet sshd: Failed password for invalid user admin from 22.214.171.124 port 41484 ssh2
Sep 26 03:12:45 skapet sshd: Connection closed by 126.96.36.199
Sep 26 03:13:10 skapet sshd: Failed password for root from 188.8.131.52 port 43344 ssh2
This is the classic, rapid-fire type of bruteforce attack, with rapid-fire login attempts from one source. (And yes, skapet is the Internet-facing host on my home network.)
The Likely Business Plan
These attempts are often preceded by a port scan, but in other cases it appears that the miscreants are just blasting away at random. In my experience, with the gateway usually at the lowest-numbered address, the activity usually turns up first there, before trying higher-numbered hosts. I'm not really in a mind to offer help or advice to the people running those scripts, but it might be possible to scan the internet from 255.255.255.255 downwards next time. Anyway, looking at the log excerpts, miscreants' likely plan is
- Try for likely user names, hope for guessable password, keep guessing until successful.
Traditional Anti-Bruteforce Rules
Rapid-fire bruteforce attacks are easy to head off. I tend to use OpenBSD on internet facing hosts, so first we present the technique as it has been available in OpenBSD since version 3.5 (released in 2005), where state tracking options are used to set limits we later act on:
In your /etc/pf.conf, you add a table to store addresses, block access for all traffic coming from members of that table, and finally amend your typical pass rule with some state tracking options. The result looks something like this:
table <bruteforce> persist block quick from <bruteforce> pass inet proto tcp to $int_if:network port $tcp_services \ keep state (max-src-conn 100, max-src-conn-rate 15/5, \ overload <bruteforce> flush global)
Here, max-src-conn is the maximum number of concurrent connections allowed from one host
max-src-conn-rate is the maximum allowed rate of new connections, here 15 connections per 5 seconds.
overload <bruteforce> means that any hosts that exceed either of these limits are have their adress added to this table
and, just for good measure, flush global means that for host that is added to our overload table, we kill all existing connections too.
Traditional Anti-Bruteforce Rules, Linux Style
For those not yet converted to the fine OpenBSD toolset (available in FreeBSD and other BSDs too, with only minor if any variations in details for this particular context), the Linux equivalent would be something like
sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
sudo iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 5 \ --hitcount 15 --rttl --name SSH -j DROP
But be warned: this will still be minus the maximum number of connections limit, plus the usual iptables warts. And you'd need a separate set of commands for ip6tables.
It's likely something similar is doable with other tools and products too, including possibly some proprietary ones. I've made something of an effort to limit my exposure to the non-free tools, so I can't offer you any more detail. To find out what your present product can do, please dive into the documentation for whichever product you are using. Or come back for some further OpenBSD goodness.
But as you can see, for all practical purposes the rapid-fire bruteforce or floods problem has been solved with trivial configuration tweaks.
But then something happened.
What's That? Something New!
On November 19th, 2008 (or shortly thereafter), I noticed this in my authentication logs:
Nov 19 15:04:22 rosalita sshd: error: PAM: authentication error for illegal user alias from s514.nxs.nl
Nov 19 15:07:32 rosalita sshd: error: PAM: authentication error for illegal user alias from c90678d3.static.spo.virtua.com.br
Nov 19 15:10:20 rosalita sshd: error: PAM: authentication error for illegal user alias from 207-47-162-126.prna.static.sasknet.sk.ca
Nov 19 15:13:46 rosalita sshd: error: PAM: authentication error for illegal user alias from 125-236-218-109.adsl.xtra.co.nz
Nov 19 15:16:29 rosalita sshd: error: PAM: authentication error for illegal user alias from 184.108.40.206
Nov 19 15:19:12 rosalita sshd: error: PAM: authentication error for illegal user alias from 220.127.116.11
Nov 19 15:22:29 rosalita sshd: error: PAM: authentication error for illegal user alias from 18.104.22.168
Nov 19 15:25:14 rosalita sshd: error: PAM: authentication error for illegal user alias from 130.red-80-37-213.staticip.rima-tde.net
Nov 19 15:28:23 rosalita sshd: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com
Nov 19 15:31:17 rosalita sshd: error: PAM: authentication error for illegal user alias from gate-dialog-simet.jgora.dialog.net.pl
Nov 19 15:34:18 rosalita sshd: error: PAM: authentication error for illegal user alias from 22.214.171.124
Nov 19 15:37:23 rosalita sshd: error: PAM: authentication error for illegal user alias from 126.96.36.199
Nov 19 15:40:20 rosalita sshd: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com
Nov 19 15:43:39 rosalita sshd: error: PAM: authentication error for illegal user alias from 188.8.131.52
Nov 19 15:46:41 rosalita sshd: error: PAM: authentication error for illegal user amanda from 184.108.40.206
Nov 19 15:49:31 rosalita sshd: error: PAM: authentication error for illegal user amanda from host116-164.dissent.birch.net
Nov 19 15:55:47 rosalita sshd: error: PAM: authentication error for illegal user amanda from robert71.lnk.telstra.net
Nov 19 15:59:08 rosalita sshd: error: PAM: authentication error for illegal user amanda from static-71-166-159-177.washdc.east.verizon.net
... and so on. The alphabetic progression of user names went on and on.
The pattern seemed to be that several hosts, in widely different networks, try to access our system as the same user, up to minutes apart. When any one host comes back it's more likely than not several user names later. The full sequence (it stopped December 30th), is available here.
Take a few minutes to browse the log data if you like. It's worth noting that rosalita was a server that had a limited set of functions for a limited set of users, and basically no other users than myself ever logged in there via SSH, even if they for various reason had the option open to them. So in contrast to busier sites where sequences like this might have drowned in the noise, here it really stood out. And I suppose after looking at the data, you can understand my initial reaction.
The Initial Reaction
My initial reaction was pure disbelief.
For the first few days I tried tweaking PF rules, playing with the attempts/second values and scratching my head, going, "How do I make this match?"
I spent way too much time on that, and the short version of the answer to that question is, you can't. With the simple and in fact quite elegant state tracking options, you will soon hit limits (especially time limits) that interfere with normal use, and you end up blocking legitimate traffic.
So I gave up on prevention (which really only would have rid me of a bit of noise in my authentication logs), and I started analyzing the data instead, trying to eyeball patterns that would explain what I was seeing. After a while it dawned on me that this could very well be a coordinated effort, using a widely distributed set of compromised hosts.
So there was a bit of reason in there after all. Maybe even a business plan or model. Next, I started analyzing my data, and came up with -
Bruteforcer Business Plan, Distributed Version
The Executive Summary would run something like this: Have more hosts take turns, round robin-ish, at long enough intervals to stay under the radar, guessing for weak passwords.
The plan is much like before, but now we have more host on the attacking side, so
- Pick a host from our pool, assign a user name and password (picked from a list, dictionary or pool)
- For each host,
- Try logging in to the chosen target with the assigned user name and password
- If successful, report back to base (we theorize); else wait for instructions (again we speculate)
- Go to 1).
- For each success at 2.2), PROFIT!
You're The Target
Let's recap, and take step back. What have we learned?
To my mind at least, it all boils down to the basics
- Your Unix computer (Linux, OpenBSD, FreeBSD or other) is a desirable, powerful thing.
- If your password is weak, you will be 0WN3D, sooner rather than later.
- There's a whole fleet out there, and they're coordinated.
Initial Public Reaction
On December 2, 2008, I published the first blog post in what would become a longish sequence, A low intensity, distributed bruteforce attempt, where I summarized my findings. It's slightly more wordy than this piece, but if I've piqued your interest so far, please go ahead and read. And as to a little public attention, I got my wish. The post ended up
- more disbelief, (see slashdot and other comments) but also
- confirmation, via comments and email, that others were seeing the same thing, and that the first occurrences may have been seen up to a year earlier (November-ish 2007).
The slow bruteforcers were not getting in, so I just went on collecting data. I estimated they'd be going on well past new year's if they were going to reach the end of the alphabet.
On December 30th, 2008, The Attempts Stopped
The attempts came to an end, conveniently while I was away on vacation. The last entries were:
Dec 30 11:03:08 rosalita sshd: error: PAM: authentication error for illegal user sophia from 220.127.116.11
Dec 30 11:05:08 filehut sshd: error: PAM: authentication error for illegal user sophia from 18.104.22.168
Dec 30 11:06:35 rosalita sshd: error: PAM: authentication error for illegal user sophia from static-98-119-110-139.lsanca.dsl-w.verizon.net
Dec 30 11:09:03 filehut sshd: error: PAM: authentication error for illegal user sophia from static-98-119-110-139.lsanca.dsl-w.verizon.net
That is, not even completing a full alphabetic cycle.
By then they had made 29916 attempts, all failed. You can find the full listing at here).
Trying 6100 user IDs (list by frequency here). More than likely you can guess the top one without even looking.
From a total of 1193 different hosts (list by frequency here).
As I said earlier, there were no successful penetrations. Zero.
The slashdot story brought comments and feedback, with some observations from other sites. Not a lot of data, but enough that the patterns we had observed were confirmed. The attempts were all password authentication attempts, no other authentication methods attempted.
For the most part the extended incident consisted of attempts on an alphabetic sequence of 'likely' user names, but all sites also saw at least one long run of root only attempts. This pattern was to repeat itself, and also show up in data from other sources.
There would be anything from seconds to minutes between attempts, but attempts from any single host would come at much longer intervals.
First Round Observations, Early Conclusions
Summing up what we had so far, here are a few observations and attempts at early conclusions.
At the site where I had registered the distributed attempts, the Internet-reachable machines all ran either OpenBSD or FreeBSD. Only two FreeBSD boxes were contacted.
The attackers were hungry for root, so having PermitRootLogin no in our
We hadn't forced our users to keys only, but a bit of luck and John the Ripper (/usr/ports/security/john) saved our behinds.
The number of attempts per user name had decreased over time (as illustrated by this graph), so we speculated in the second article Into a new year, slowly pounding the gates (on slashdot as The Slow Bruteforce Botnet(s) May Be Learning) that success or not was measured at a command and control site, with resources allocated accordingly.
With the sequence not completed, we thought they'd given up. After all, the odds against succeeding seemed monumental.
After all, a couple of slashdotted blog posts couldn't have hurt, could they?
But Of Course They Came Back
As luck would have it, whoever was out there had not totally admitted defeat just yet. In the early hours CET, April 7th, 2009, the slow brutes showed up again:
Apr 7 05:02:07 rosalita sshd: error: PAM: authentication error for root from ruth.globalcon.net
Apr 7 05:02:15 rosalita sshd: error: PAM: authentication error for root from ip-206-83-192-201.sterlingnetwork.net
Apr 7 05:02:54 rosalita sshd: error: PAM: authentication error for root from cyscorpions.com
Apr 7 05:02:59 rosalita sshd: error: PAM: authentication error for root from smtp.bancomorada.com.br
Apr 7 05:03:10 rosalita sshd: error: PAM: authentication error for root from 22.214.171.124
Apr 7 05:03:25 rosalita sshd: error: PAM: authentication error for root from 126.96.36.199
Apr 7 05:03:52 rosalita sshd: error: PAM: authentication error for root from rainha.florianonet.com.br
Apr 7 05:04:00 rosalita sshd: error: PAM: authentication error for root from 188.8.131.52
Apr 7 05:04:34 rosalita sshd: error: PAM: authentication error for root from s1.serverhex.com
Apr 7 05:04:38 rosalita sshd: error: PAM: authentication error for root from mail.pitnet.com.br
Starting with 2318 attempts at root before moving on to admin and proceeding with the alphabetic sequence. The incident played out pretty much like the previous one, only this time I was sure I had managed to capture all relevant data before my logs were rotated out of existence.
The data is available in the following forms: Full log here, one line per attempt here, users by frequency here, hosts by frequency here.
I couldn't resist kicking up some more publicity, and indeed we got another slashdot storm out of the article The slow brute zombies are back, on slashdot as The Low Intensity Brute-Force Zombies Are Back.
And shortly afterwards, we learned something new -
Introducing dt_ssh5, Linux /tmp Resident
Of course there was a piece of malware involved.
A Linux binary called dt_ssh5 did the grunt work.
The dt_ssh5 file was found installed in /tmp on affected systems. The reason our perpetrators chose to target that directory is likely because the /tmp directory tends to be world readable and world writeable.
Again, this points us to the three basic lessons:
dt_ssh5: Basic Algorithm
The discovery of dt_ssh5 made for a more complete picture. A rough algorithm suggested itself:
- Pick a new host from our pool, assign a user name and password
For each host,
- Try user name and password
- drop the dt_ssh5 binary in /tmp; start it
- report back to base
Go to 1.
- For each success at 2.2, PROFIT!
I never got myself a copy, so the actual mechanism for communicating back to base remains unclear.
The Waves We Saw, 2008 - 2012
We saw eight sequences (complete list of articles in the References section at the end),
|From - To||Attempts||User IDs||Hosts||Successful Logins|
|2008-11-19 15:04:22 - 2008-12-30 11:09:03||29916||6100||1193||0|
|2009-04-07 03:56:25 - 2009-04-12 21:01:37||12641||249||1104||0|
|2009-09-30 21:15:36 - 2009-10-15 13:42:07||9998||1||1071||0|
|2009-10-28 23:58:35 - 2010-01-22 09:56:24||44513||8110||4158||0|
|2010-06-17 01:55:34 - 2010-08-11 13:23:01||23014||3887||5568||0|
|2011-10-23 04:13:00 - 2011-10-29 05:40:07||4773||944||338||0|
|2011-11-03 20:56:18 - 2011-11-26 17:42:19||4907||2474||252||0|
|2012-04-01 12:33:04 - 2012-04-06 14:52:11||4757||1081||23||0|
The 2009-09-30 sequence was notable for trying only root, the 2012-04-01 sequence for being the first to attempt access to OpenBSD hosts.
We may have missed earlier sequences, early reports place the first similar attempts as far back as 2007.
For A While, The Botnet Grew
From our point of view, the swarm stayed away for a while and came back stronger, for a couple of iterations, possibly after tweaking their code in the meantime. Or rather, the gaps in our data represent times when it focused elsewhere.
Clearly, not everybody was listening to online rants about guessable passwords.
For a while, the distributed approach appeared to be working.
It was (of course) during a growth period I coined the phrase "The Hail Mary Cloud".
Instantly, a myriad of "Hail Mary" experts joined the insta-punditry on slashdot and elsewhere.
It Went Away Or Dwindled
Between August 2010 and October 2010, things either started going badly for The Hail Mary Cloud, or possibly they focused elsewhere.
I went on collecting data.
There wasn't much to write about, except possibly that the botnet's command and control was redistributing effort based on past success. Aiming at crackable hosts elsewhere.
And Resurfaced In China?
Our last sighting so far was in April 2012. The data is preserved here.
This was the first time we saw Hail Mary Cloud style attempts at accessing OpenBSD systems.
The majority of attempts were spaced at at least 10 seconds apart, and until I revisited the data recently, I thought only two hosts in China were involved.
In fact, 23 hosts made a total of 4757 attempts at 1081 user IDs, netting 0 successful logins.
The new frequency data I thought interesting enough to write about, so I wrote up If We Go One Attempt Every Ten Seconds, We're Under The Radar, and netted another slashdotting. I took another look at the data later and slightly amended the conclusions, the article has been corrected with proper data extracted.
Then What To Do?
The question anybody reading this far will be asking is, what should we do in order to avoid compromise by the password guessing swarms? To my mind, it all boils down to common sense systems administration:
Mind your logs. You can read them yourself, or train a robot to. I use
Keep your system up to date. If not OpenBSD, check openssh.com for the latest version, check what your system has and badger the maintainer if it's outdated.
And of course, configure your applications such as
sshd_config: 'PermitRootLogin no' and a few other items
These two settings in your
PermitRootLogin no PasswordAuthentication no
Make your users generate keys, add the *.pub to their ~/.ssh/authorized_keys files.
For a bit of background, Michael W. Lucas: SSH Mastery (Tilted Windmill Press 2013) is a recent and very readable guide to configuring your SSH (server and clients) sensibly. It's compact and affordable too.
Keep Them Out, Keep Them Guessing
At this point, most geeks would wax lyrical about the relative strengths of different encryption schemes and algorithms.
Being a simpler mind, I prefer a different metric for how good your scheme is, or effectivness of obfuscation (also see entropy):
How many bytes does a would-be intruder have to get exactly right?I've summed up the answer to that question in this table:
|Authentication method||Number of bytes|
|Password||Password length (varies, how long is yours?)|
|Alternate Port||Port number (2 bytes, it's a 16 bit value, remember)|
|Port Knocking||Number of ports in sequence * 2 (still a 16 bit value)|
|Single Packet Authentication||2 bytes (the port) plus Max 1440 (IPv4/Ethernet) or 1220 (IPv6/Ethernet)|
|Key Only||Number of bytes in key (depending on key strength, up to several kB)|
You can of course combine several methods (with endless potential for annoying your users), or use two factor authentication (OpenSSH supports several schemes).
Keys. You've Got To have Keys!
By far the most effective measure is to go keys only for your ssh logins. In your sshd_config, add or uncomment
Restart you sshd, and have all users generate keys, like this:
$ssh-keygen -C "firstname.lastname@example.org"
There are other options to play with, see ssh-keygen(1) for inspiration.
Then add the *.pub to their ~/.ssh/authorized_keys files.
And I'll let you in on a dirty little secret: you can even match on interface in your sshd config for things like these
Why Not Use Port Knocking?
Whenever I mention the Hail Mary Cloud online, two suggestions always turn up: The iptables example I mentioned earlier (or link to the relevant slide), and "Why not use port knocking?". Well, consider this:
Port knocking usually means having all ports closed, but with a daemon reading your firewalls logs for a predetermined sequence of ports. Knock on the correct ports in sequence, your're in.
Another dirty little secret: It's possible to implement port knocking with only the tools in an OpenBSD base system. No, I won't tell you how.
Executive Summary: Don't let this keep you from keeping your system up to date.
To my mind port knocking gives you:
- Added complexity or, one more thing that will go wrong. If the daemon dies, you've bricked your system.
- An additional password that's hard to change. There's nothing magical about TCP/UDP ports. It's a 16 bit number, and in our context, it's just another alphabet. The swarm will keep guessing. And it's likely the knock sequence (aka password) is the same for all users.
- You won't recognize an attack until it succeeds, if even then. Guessing attempts will be indistinguishable from random noise (try a raw tcpdump of any internet-facing interface to see the white noise you mostly block drop anyway), so you will have no early warning.
If you want a longer form or those arguments, my November 4, 2012 rant Why Not Use Port Knocking? was my take (with some inaccuracies, but you'll live).
There's No Safety In High Ports Anymore
Another favorite suggestion is to set your sshd to listen on some alternate port instead of the default port 22/TCP.
People who did so have had a few years of quiet logs, but recent reports show that whoever is out there have the resources to scan alternate ports too.
Once again, don't let running your
Of course I've ranted about this too, in February 2013, There's No Protection In High Ports Anymore. If Indeed There Ever Was. (which earned me another slashdotting).
Reports with logs trickle in from time to time of such activity at alternate ports, but of course on any site with a default deny packet filtering policy will not see any traces of such scans unless you go looking specifically at the mass of traffic that gets dropped at the perimeter.
Final thoughts, for now
Microsoftish instapundits were quick to assert that
They're wrong. OpenSSH (which is what essentially everyone uses) is maintained as an integral part of the OpenBSD project, and as such is a very thoroughly audited mass of code. And it keeps improving with every release.
I consider the Hail Mary Cloud an example of distributed, parallel problem solving, conceptually much like SETI@Home but with different logic and of course a more sinister intent.
Computing power is cheap now, getting cheaper, and even more so when you can leverage other people's spare cycles.
The huge swarm of attackers concept is as I understand it being re-used in the recent WordPress attacks. We should be prepared for swarm attacks on other applications as soon as they reach a critical mass of users.
There may not be a bullseye on your back yet (have you looked lately?), but you are an attractive target.
Fortunately, sane system administration practices will go a long way towards thwarting intrusion attempts, as in
- keep your system up to date,
- allow only what's necessary for the task at hand and
- keep watching your logs for weirdness.
UPDATE 2013-11-21: A recent ACM Conference on Computer and Communication Security paper, "Detecting stealthy, distributed SSH brute-forcing," penned by Mobin Javed and Vern Paxson, references a large subset of the data and offers some real analysis, including correlation with data from other sites (Spoiler alert: in some waves, almost total overlap of participating machines). One interesting point from the paper is that apparently attack matching our profile were seen at the Lawrence Berkeley National Laboratory as early as 2005.
And in other news, it appears that GitHub has been subject to an attack that matches the characteristics we have described. A number of accounts with weak passwords were cracked. Investigations appears to be still ongoing. Fortunately, GitHub appear to have started offering other authentication methods.
UPDATE 2014-09-28: Since early July 2014, we have been seeing similar activity aimed at our POP3 service, with usernames taken almost exclusively from our spamtrap list. The article Password Gropers Take the Spamtrap Bait has all the details and log data as well as references to the spamtrap list.
UPDATE 2014-12-10: My Passwords14 presentation, Distributed, Stealthy Brute Force Password Guessing Attempts - Slicing and Dicing Data from Recent Incidents has some further data as well as further slicing and dicing of the earlier data (with slightly different results).
UPDATE 2016-08-10: The POP3 gropers never went away entirely and soon faded into a kind of background noise. In June of 2016, however, they appeared to have hired themselves out to a systematic hunt for Chinese user names. The article Chinese Hunting Chinese over POP3 in Fjord Country has further details, and as always, links to log data and related files.
UPDATE 2020-04-28: Somewhat belatedly added a link to the more recent (August 2018) article Badness, Enumerated by Robots which has updates on data we currently collect with some further explanation. Also see links therein.
The slides for the talk this article is based on live at http://home.nuug.no/~peter/hailmary2013/, with a zipped version including all data at http://home.nuug.no/~peter/hailmary2013.zip (approx. 26MB) for your convenience.
Mobin Javed and Vern Paxson, "Detecting stealthy, distributed SSH brute-forcing," ACM International Conference on Computer and Communication Security (CCS), November 2013.
The blog posts (field notes) of the various incidents, data links within:
Peter N. M. Hansteen, (2008-12-02) A low intensity, distributed bruteforce attempt (slashdotted)
Peter N. M. Hansteen, (2008-12-06) A Small Update About The Slow Brutes
Peter N. M. Hansteen, (2008-12-21) Into a new year, slowly pounding the gates (slashdotted)
Peter N. M. Hansteen, (2009-01-22) The slow brutes, a final roundup
Peter N. M. Hansteen, (2009-04-12) The slow brute zombies are back (slashdotted)
Peter N. M. Hansteen, (2009-10-04) A Third Time, Uncharmed (slashdotted)
Peter N. M. Hansteen, (2009-11-15) Rickrolled? Get Ready for the Hail Mary Cloud! (slashdotted)
Peter N. M. Hansteen, (2011-10-23) You're Doing It Wrong, Or, The Return Of The Son Of The Hail Mary Cloud
Peter N. M. Hansteen, (2011-10-29) You're Doing It Wrong, Returning Scoundrels
Peter N. M. Hansteen, (2012-04-06) If We Go One Attempt Every Ten Seconds, We're Under The Radar (slashdotted)
Peter N. M. Hansteen, (2012-04-11) Why Not Use Port Knocking?
Peter N. M. Hansteen, (2013-02-16) There's No Protection In High Ports Anymore. If Indeed There Ever Was. (slashdotted)
Peter N. M. Hansteen (2018-08-13) Badness, Enumerated by Robots (updates on data we currently collect, also see links therein).
Other Useful Texts
Marcus Ranum: The Six Dumbest Ideas in Computer Security, September 1, 2005
Michael W. Lucas: SSH Mastery, Tilted Windmill Press 2013 (order direct from the OpenBSD bookstore here)
Michael W. Lucas: Absolute OpenBSD, 2nd edition No Starch Press 2013 (order direct from the OpenBSD bookstore here)
Peter N. M. Hansteen, The Book of PF, 3rd edition, No Starch Press 2014, also the online PF tutorial it grew out of, several formats http://home.nuug.no/~peter/pf/, more extensive slides matching the most recent session at http://home.nuug.no/~peter/pf/newest/. More recently I've done a rewrite of the PF tutorial with Max Stucchi co-authoring and co-presenting, the lastest slides from one of these sessions are available at https://home.nuug.no/~peter/pftutorial/ (sorry, labs available for real attendees only).
OpenBSDs web http://www.openbsd.org/ -- lots of useful information.
If you enjoyed this: Support OpenBSD!
If you have enjoyed reading this, please buy OpenBSD CDs and other items, and/or donate!
Useful links for this are:
OpenBSD.org Orders Page: http://www.openbsd.org/orders.html
OpenBSD Donations Page: http://www.openbsd.org/donations.html.
OpenBSD Hardware Wanted Page: http://www.openbsd.org/want.html.
Remember: Free software takes real work and real money to develop and maintain.
If you want to support me, buy the book! (if you want to give the OpenBSD project a cut of that, this is the link you want).
"I'm not really in a mind to offer help or advice to the people running those scripts, but it might be possible to scan the internet from 255.255.255.255 downwards next time."ReplyDelete
I'm sure hitting all the multicast addresses first will be highly effective ;)
I think it would be a better idea for those script kiddies to start in the middle of the internet: 127.0.0.1Delete
I've seen the same pattern happening on RDP ports, I am not surprised they are trying with SSH these days..ReplyDelete
There is a tool called denyhosts that solves your problem trivially. I myself wrote similar tool in awk and shell and used it for many years. It is funny how same problem comes up over and over again and people look for their own solution and analysis instead of using existing tools.ReplyDelete
Or pick a nonstandard port for sshd.ReplyDelete
Did you even read the article?Delete
simple port scan away your port is known. Only harasses legitimate users. Obfuscation is not a form of security.Delete
Couldn't & shouldn't you set the 'noexec' flag on /tmp? Or even fake a partition for /tmp likeReplyDelete
shell> dd if=/dev/zero of=tmp.iso bs=516096 seek=9182 count=0
shell> mkfs -t ext4 /tmp
shell> mount -o loop tmp.iso /tmp
shell> echo "/path/to/tmp.iso /tmp ext4 nosuid,noexec,nodev 0 0" >> /etc/fstab
Glancing through the log, it looks like this "low and slow" detection algorithm could have helped too. (e.g, I see multiple tries from "static-98-119-110-139.lsanca.dsl-w.verizon.net")ReplyDelete
This is why I disable password logins and only allow auth via public key for any internet-facing servers.ReplyDelete
We see similar slow attacks on passwords for SMTP AUTH, pop3/imap, ftp, etc. Any protocol that uses username/password for authentication is vulnerable to such a slow distributed attack.ReplyDelete
I don't agree that root login is a bad idea. It is a bad idea to have root login with a guessable password. Just use one that these f****** cannot guess in 1000 years and you are fine.ReplyDelete
Sorry, there's no useful reason to allow remote root logins on *ANY* server, whether it's facing the Intardwebz or not. Any password you can come up with can probably be broken, it may just take some effort. Why open yourself to the risk? There's no benefit to it whatsoever. Sudo is your friend. Learn it, use it, love it.Delete
ssh root@ should never be allowed.
(Oh, and disable password auth, as well).
You do know that a ssh key is just a long password, right?Delete
I used to create an alternate root account and ssh altroot@, with password. I now use sudo. Using the formula above, security is measured in how many bytes the attackers have to get exactly right. It turns out that renaming the root account (to a username as long as your non-root account) is just as secure as signing on as yourself and using sudo; after all, you're just typing the same password over again at the sudo prompt.Delete
I would switch to ssh keys only, except that I want to be able to sign on in the event I don't have my ssh key with me.
We should log what password these slow-brutes are using, because they're effectively broadcasting the ip addresses, usernames and passwords of all of the hosts in their botnet.
An ssh key is, indeed, just a long password; but remember that it's in two parts: the passphrase, and the private key.Delete
Together, those are usually quite long.
I've seen the same issues. I use sshblack. It should work in your case too.ReplyDelete
I've got a SSH server on a different port + check the logs daily, and in the last 5 years of running it, I can't think of a time that we've had anyone trying to connect to it. The other ssh server (chrooted/sftponly) on the standard port on that machine gets lots of attempts though.ReplyDelete
Perhaps that's part of the solution? - what amounts to a honey pot on a standard port and the actual SSH server somewhere else?
I don't see why a server needs to allow everyone to try to login. I limited access to a handful of specific addresses, and everyone else gets ignored.ReplyDelete
Travelling.. Hotels, unknown locations, etc..Delete
They will all have IP address that you cannot know in advance, and in many cases will operate with dhcp so even if you did know the address, it may change.
VPN back to the office and connect from there. As a backup I can VPN to my home and connect from there to. Only 2 ip-addresses are allowed ssh login.Delete
A little thing that helped me, since I'm administrating my servers from a randomly changing dialin IP:ReplyDelete
Only allow IPs access to your services, which you are on. Create an iptables entry for that that updates regulary (wrote myself a script for that).
For that, use dyndns of any kind. Of course, people like the NSA or other bigbiz could still force companies (if you use a commercial service) into changing your dyndns - ip into something of their preference. But even then all they have is a normal SSH-Login with pubkey authentication and "PermitRootLogin No". AND you'd instantly know if something went astray, if you suddenly can't access your servers via SSH (or other services like eg. NRPE) anymore.
In short: Allow access to services on your public IP addresses only to known-probably-ok IP addresses, and most of your botnet-related problems are gone. Of course there are still exploits out there, so you are not completely in the green. Good luck :)
I use DenyHosts and Google Authenticator PAM module for SSH logins on all my internet facing hosts. Both are open source and fairly easy to setup on various OS'.ReplyDelete
Besides changing the port, on the Linux side; I use fail2ban. It takes care of this automatically with iptables ( for any that do find the port ). I believe that the BSD flavors have it as well.ReplyDelete
I use port knocking via my Mikrotik router to allow access to port 22. I know your dislike of Port Knocking, but it pretty much stops these random IP scans. You really need multiple layers of security to protect your servers, and its best if all that traffic is stopped before it even reaches your server.ReplyDelete
With my setup, you have to hit random ports with specific data (the router does Layer 7 inspection) on 5 different non-sequential ip addresses on my subnet all within 1 second to enable access. Hit any other ip/port with any packet of data, and you are going to be starting over. This of course then just gives you access to the NATed port (not 22) to ssh to my servers (key only, no pass auth)
There doesn't have to even be ports or even servers actually listening on those IPs, its all handled in the router itself. I then have rules in place that will black hole your IP for 24 hours if you attempt to brute force it.
Just a quick note, with iptables why not use the "hashlimit" module? That should achieve the same as the pf filter.ReplyDelete
Hi Peter, I enjoyed reading your thorough and informative presentation of this topic. Thank you.ReplyDelete
There is a free cloud based service at https://jumpcloud.com that does a good job of identifying and blocking SSH brute attacks that could possibly succeedReplyDelete
Hey Peter... I've been seeing these style attacks in my logs as well... also running OpenBSD and started noticing them back when you wrote that first article! They are still there, plugging away day after day.ReplyDelete
Keys are good, but only as good as your endpoint security. If the user doesn't password-protect his key, and his laptop gets stolen, the thief/attacker is in unless the admin gets notified in time and revoked the key. If the user does password his key, there's no way to verify password strength without some sort oglf agent running on the laptop.ReplyDelete
On the other hand, brute-forcing s key passsword may be harder due to lack of plain text in the key file (or maybe not, I don't know). And I suppose it's still less risk than brute-forcers over the net.
I also saw the same slow distributed attacks on my OpenBSD box back in 2008. I always wanted to reply back to these attacks. One day I tried to log into one of those by guessing the login/pass. It took me few tries to get admin/admin. Then I deleted the executable in /tmp and killed the daemon in memory. I also changed the admin password before logging out.ReplyDelete
It gives me the idea to write a daemon to do this automatically based on wrong usernames (the ones not in passwd) or overload table. I never managed to get the motivation to write this, but it could have been an interesting project, put aside the illegal nature of this project.
What strikes me in the first usernames list is that they are mostly french woman names. There seems to be a pattern in second one too, but can't identify it (the first one was easy cause I'm french).ReplyDelete
I wonder, does people having spotted the attack in the same time on their hosts report about the same list ?
If not, that could mean an even more sophisticated (albeit totally failed here) attack trying to guess sex / location of host before selecting a dictionary.
Hi, I would like to seek your opinion on which is best to protect my server.ReplyDelete
1) Disable root login, create new user, create SSH for new user, disable new user password
2) Disable root login, create new user, create Password for new user, no SSH for new user.
Please advise! I'm in sooooooooo much confusion! Thanks :))
Four years late to the party...but still here are my comments.ReplyDelete
Excellent article, not just the technical content but also the writing style. Reminds me of the Cuckoo's Egg.
Why not build a table Hail Mary IPs and block them in PF? Perhaps the IPs can also be shared using BGP (like Hessler's presentation for spamd). I am not a networking expert - would it be possible to build a spamd equivalent for sshd and redirect HM IPs to a fakeSSHD to just waste their time?