It's been a fun few weeks. Over in Microsoft land, it must have been a big issue that according to malware hunters Sophos, the newly released Windows 7 with no extras is roughly as vulnerable as its older siblings. No great surprises there, I suppose.
For those of us with a more Unixish leaning, the more interesting piece of news involved Apple iPhones. These phones apparently run a version of MacOS that has enough Unix in it that with a bit of tinkering, it is possible to install a variety of Unix software, such as the ubiquitous secure shell daemon sshd. To some, there is a certain attraction in knowing that you have an SSH server in your pocket or handbag. Too bad, then that enough of those adventurous iPhone owners never read up on the instructions and chose to run their toy with the default password for the root account and were vulnerable to a wonderful prank perpetrated by a programmer down under.
The prank (described in the inimitable The Register style here) demonstrated just how bad an idea it is to publish your user name and password. If you followed the news around last weekend you would notice that a large segment of the Microsoft-attached instapunditry got their facts wrong as usual with the this proves that Apple (and by extension any Unix and of course Linux) is just as vulnerable as Microsoft mantra repeated over and over.
In fact, there are two historical incidents that point to Unix being no silver bullet: the 2002 Linux Slapper Worm and the original network-enabled worm, the 1988 Morris Worm. Those two prove mainly that yes, some bugs are exploitable, and the way forward is to fix bugs and make them harder to exploit in the first place. Now these two famous exploits is possibly to be joined by a third, the rickrolling prank.
I beg to differ. The rickroller is about bad passwords, no more, no less. I've spent considerable time ranting about passwords in earlier columns, and this incident only underscores what we've been repeating until your eardrums wear thin an my vocal cords swell from exhaustion:
Publishing your username and password is a really bad idea.
It's almost as bad as picking a guessable password.
Add to this that the fact, as we've noted here earlier, there is a whole cloud of hijacked machines out there beavering away at guessing passwords right now, and they have been at it for quite a while.
The most remarkable of these botnets is the one that tries to avoid detection by distributing the password guessing for any target across a large number of hosts, so each guesser never shows high enough rates of activity to trigger any of the traditional bruteforce detection mechanism. The attempts look something like this in your authentication log:
Nov 13 21:10:14 rosalita sshd[50401]: error: PAM: authentication error for illegal user mars from 125.40.69.208
Nov 13 21:10:14 rosalita sshd[50401]: Failed keyboard-interactive/pam for invalid user mars from 125.40.69.208 port 38052 ssh2
Nov 13 21:11:20 rosalita sshd[50419]: reverse mapping checking getaddrinfo for 115-186-131-90.nayatel.pk [115.186.131.90] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:11:20 rosalita sshd[50419]: Invalid user mars from 115.186.131.90
Nov 13 21:11:21 rosalita sshd[50419]: error: PAM: authentication error for illegal user mars from 115.186.131.90
Nov 13 21:11:21 rosalita sshd[50419]: Failed keyboard-interactive/pam for invalid user mars from 115.186.131.90 port 42235 ssh2
Nov 13 21:13:43 rosalita sshd[50428]: Invalid user mars from 58.247.222.163
Nov 13 21:13:43 rosalita sshd[50428]: error: PAM: authentication error for illegal user mars from 58.247.222.163
Nov 13 21:13:43 rosalita sshd[50428]: Failed keyboard-interactive/pam for invalid user mars from 58.247.222.163 port 35134 ssh2
Nov 13 21:15:59 rosalita sshd[50440]: Invalid user mars from 89.76.186.99
Nov 13 21:16:00 rosalita sshd[50440]: error: PAM: authentication error for illegal user mars from chello089076186099.chello.pl
Nov 13 21:16:00 rosalita sshd[50440]: Failed keyboard-interactive/pam for invalid user mars from 89.76.186.99 port 52156 ssh2
Nov 13 21:17:16 rosalita sshd[50448]: Invalid user mars2 from 88.134.166.31
Nov 13 21:17:16 rosalita sshd[50448]: error: PAM: authentication error for illegal user mars2 from 88-134-166-31-dynip.superkabel.de
Nov 13 21:17:16 rosalita sshd[50448]: Failed keyboard-interactive/pam for invalid user mars2 from 88.134.166.31 port 39254 ssh2
Nov 13 21:18:14 rosalita sshd[50452]: Invalid user room from 62.198.66.107
Nov 13 21:18:14 rosalita sshd[50452]: error: PAM: authentication error for illegal user room from 62.198.66.107
Nov 13 21:18:14 rosalita sshd[50452]: Failed keyboard-interactive/pam for invalid user room from 62.198.66.107 port 47557 ssh2
Nov 13 21:19:27 rosalita sshd[50458]: Invalid user room from 61.74.75.43
Nov 13 21:19:27 rosalita sshd[50458]: error: PAM: authentication error for illegal user room from 61.74.75.43
Nov 13 21:19:27 rosalita sshd[50458]: Failed keyboard-interactive/pam for invalid user room from 61.74.75.43 port 57794 ssh2
Nov 13 21:21:41 rosalita sshd[50472]: Invalid user room from 212.243.41.9
Nov 13 21:21:41 rosalita sshd[50472]: error: PAM: authentication error for illegal user room from 212.243.41.9
Nov 13 21:21:41 rosalita sshd[50472]: Failed keyboard-interactive/pam for invalid user room from 212.243.41.9 port 38396 ssh2
Nov 13 21:22:58 rosalita sshd[50491]: reverse mapping checking getaddrinfo for static-ip-cr1901468058.cable.net.co [190.146.80.58] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:22:58 rosalita sshd[50491]: Invalid user room from 190.146.80.58
Nov 13 21:22:58 rosalita sshd[50491]: error: PAM: authentication error for illegal user room from 190.146.80.58
Nov 13 21:22:58 rosalita sshd[50491]: Failed keyboard-interactive/pam for invalid user room from 190.146.80.58 port 4420 ssh2
Nov 13 21:24:01 rosalita sshd[50509]: Invalid user room from 62.23.130.173
Nov 13 21:24:01 rosalita sshd[50509]: error: PAM: authentication error for illegal user room from host.173.130.23.62.rev.coltfrance.com
Nov 13 21:24:01 rosalita sshd[50509]: Failed keyboard-interactive/pam for invalid user room from 62.23.130.173 port 3904 ssh2
Nov 13 21:25:21 rosalita sshd[50517]: reverse mapping checking getaddrinfo for hn.kd.ny.adsl [125.40.69.208] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 13 21:25:21 rosalita sshd[50517]: Invalid user room from 125.40.69.208
Nov 13 21:25:21 rosalita sshd[50517]: error: PAM: authentication error for illegal user room from 125.40.69.208
Nov 13 21:25:21 rosalita sshd[50517]: Failed keyboard-interactive/pam for invalid user room from 125.40.69.208 port 3294 ssh2and so on.
I put it to you: What you see here is the cybercrime equivalent of the Hail Mary Pass".
Each attempt in theory has monumental odds against succeeding, but occasionally the guess will be right and they have scored a login. As far as we know, this is at least the third round of password guessing from the Hail Mary Cloud (see the archives for earlier postings about slow bruteforcers), but there could have been earlier rounds that escaped our attention.
The fact that we see the Hail Mary Cloud keeping up the guessing is a strong indicator that there are a lot of guessable passwords and possibly badly maintained systems out there, and that even against the very long odds they are succeeding often enough in their attempts to gain a foothold somewhere that it is worth keeping up the efforts. For one thing, the cost of using other people's equipment is likely to be quite low.
There are a lot of things about the Hail Mary Cloud and its overseers that we do not know. People who responded to the earlier articles with reports of similar activity also reported pretty consistently something like a sixty to seventy percent match in hosts making the attempts.
With 1767 hosts in the current sample it is likely that we have a cloud of at least several thousand, and most likely no single guessing host in the cloud ever gets around to contacting every host in the target list. The busier your SSH deamon is with normal traffic, the harder it will be to detect the footprint of Hail Mary activity, and likely a lot of this goes undetected.
The data, as I am sure you have been waiting for it, is available in these forms: Raw log data, with 3-4 lines per attempt (as illustrated above and requested by some correspondents), one line per attempt (shows the pattern a little more clearly), a list of the hosts participating in the Hail Mary Cloud sorted by number of attempts, and the user names attempted, sorted by number of attempts.
The pattern is fairly familiar by now, but this time the alphabetic cycles are shorter and at times the coordination seems to have broken down. My guess is that the apparent breakdowns are due to silly factors like the guessing machines running without time synchronization or other signs of incompetence.
And finally, some words of advice for those of you who want to avoid both rickrolling and getting cracked by other password guessing.
You should at least consider setting a password policy and enforcing it with something like John the ripper, which more than likely is available at the cost of a few keystrokes from your package system. And of course there is the fine art of sshd configuration. Some of the things you could do are, in no particular order:
- disable root logins over the network
- use packet filtering or other means to restrict where users can log in from
- disable password logins entirely allowing only key-based logins
- set up your sshd to listen on a non-standard port
If you see traces of the Hail Mary Cloud's activity in your logs and you want to share and study, I would very much like to hear from you. I will most likely be updating the log data and extracts at intervals.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Posts
Thanks Peter - I've seen this activity often and along with your tips I have also used DenyHosts and certificates to prevent/reduce these types of attack.
ReplyDeleteA couple of comments:
ReplyDeleteFirst, I keep seeing the Morris Worm referred to as a Unix virus/worm/whatever. Actually, it isn't. It didn't depend upon any weakness in Unix, but on a Unix application (sendmail). If anything, it was sendmail worm that was exploited to give access to the underlying system.
The second comment is that the absolute simplest way to protect systems from this sort of attack is to turn off username/password access on all Internet facing sshd servers.
If you use sshd certificate access, they can try usernames and passwords until the cows come home. Yes, it does mean you need to have the certificate available to get into your systems from outside, but that really isn't a big deal. I have mine both on my laptop and also stored on a USB dongle, along with a copy of PuTTY that I can run from the dongle.
The linux netfilter module 'recent' helps out here (although it can't detect fully distributed attempts where each originating address tries only once).
ReplyDeleteJust add these rules:
iptables -N blacklist 2>/dev/null
iptables -F blacklist
iptables -A blacklist -m recent --name blacklist --set
iptables -A blacklist -j DROP
iptables -N ssh 2>/dev/null
iptables -F ssh
iptables -A ssh -m recent --update --name blacklist --seconds 600 --hitcount 1 -j DROP
iptables -A ssh -m recent --set --name counting1
iptables -A ssh -m recent --set --name counting2
iptables -A ssh -m recent --set --name counting3
iptables -A ssh -m recent --update --name counting1 --seconds 20 --hitcount 3 -j blacklist
iptables -A ssh -m recent --update --name counting2 --seconds 200 --hitcount 15 -j blacklist
iptables -A ssh -m recent --update --name counting3 --seconds 2000 --hitcount 20 -j blacklist
iptables -A ssh -j ACCEPT
Finally you need to link that into your INPUT table somewhere:
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j ssh
"Publishing your username and password is a really bad idea. It's almost as bad as picking a guessable password."
ReplyDeleteWait what, publishing your password is a better idea than picking a guessable password and keeping it secret? How does that make any sense at all?
There is a package in debian / ubuntu called fail2ban that automatically bans this kind of behavior using iptables.
ReplyDeleteIn less then a month ive had 33,232 failed logins on my little openbsd box thats sitting on a cable modem. I opened sshd on Oct 23rd and just closed it today because of this nonsense.
ReplyDeleteI agree with Justin. I install DenyHosts on every Linux/Mac system with sshd access from the Internet. It just makes sense.
ReplyDeleteAnd password protected certificates prevent stolen laptops or kids on home machines from casually accessing your network.
I have 2669 unique IP's I've collected over the past month or so from several servers across three departments at UMass Amherst, USA. I compared those with the numeric IP's among your list of addresses. Just doing a simple quick check here and didn't want to mess with nslookup to get all your list numeric. Anyway, I had 90% of the numbers on your list. Even though they are blocked, the logs show them continuing on about every minute or two on a couple of the servers. Less often on others. I saw about 3 different guessing strategies and assumed different botnets. But maybe not. Anyway, root was one. Common administrative names (radmind, mysql, oracle, amanda, etc.) was another. And common user names the third. They seemed to group together.
ReplyDeleteI've been seeing these login attempts for many, many months; one user, scores of passwords, rinse, repeat...to a system that never accepted password-based login (and any remaining access is in layered security). Just trying passwords tells me that the contact is bogus. I considered, once, configuring SSH to reply with a header of arbitrary length, but I don't need to DoS my own server :) ... or some hapless 3rd party, if the occasional attempts are spoofed.
ReplyDelete@mozart: It's actually worse to pick an easy password and keep it secret, because that's not even congruent with the marginally advisable tenet of "security by obscurity." If you publish your password, you are more likely to recognize that maybe you should change it, and also to more clearly understand what happens to you as a result.
If, however, you pick a sweet password you can find in the dictionary, your family may not be able to figure it out, but you set yourself up with a false sense security. It's the difference between having to bandage your forehead when your car (inevitably) hits a pothole, and striding confidently about claiming that neither cars nor potholes exist. While both are arguably less than ideal, the oblivious one is worse.