Sunday, April 12, 2009
The slow brute zombies are back
Real-life zombies feed off weak passwords.
Regular readers will remember that late last year we saw a peculiar form of distributed bruteforce attack on certain ssh servers. What made this particular batch of failed login attempts stand out was that unlike the traditional rapid-fire bruteforce attempts we were quite adept at heading off with state tracking tricks (such as the OpenBSD PF method described here or a slightly different Linux-specific method), the technique seemed to be specifically designed to slip past such defenses.
I described the method and the evidence at some length in a series of colums, A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008) and finally The slow brutes, a final roundup (January 22, 2009).
If you do not want to read through all of that, I'll recap: The traditional bruteforce attack is characterized by somebody trying again and again to find a combination of logon credentials (typically sets of user names and passwords) that will let them gain access to the system. The basic idea is that given enough tries, you will sooner or later find a user who has set a guessable password, and you have access.
One of the important weaknesses of the method is that the typical attacker would have gained access to only one or a few systems, and the attacker would then typically try to connect a lot more often than a typical user, and sysadmins learned to adapt their firewalls and other systems to lock out activity that fit the bruteforcer profile.
If you run a ssh service anywhere Internet-facing, you will be used to seeing a steady stream of failed logons for both existing and non-existing users. There's nothing new in seeing failed logons in your log files. However, what happened late last year was that we started seeing large numbers of failed ssh logon attempts, with the new twist that the same user would be trying to log on a large number of times, but never from the same place twice in rapid succession. This log data sample will give you an idea. The data will show you the pattern, as will the summary article.
Back then in January, my conclusion was that we had seen the conclusion of a largely failed experiment. After all, proceeding at a truly glacial pace and decreasing the number of attempts per user name as they went along hardly seemed like a recipe for success even if they could sneak past packet filtering based defenses.
It turns out I was wrong. Returning to my log summaries after taking a few days off from log data (yes, easter is big here, even moves geeks sometimes to take time off) I find data with almost exactly the same profile as last December's series.
This can only mean that there were enough successful attempts at guessing people's weak passwords in the last round that our unknown perpetrators found it worthwhile to start another round. For all I know they may have been at it all along, probing other parts of the Internet far from my unfashionable backwaters.
Anyway, they are most certainly back. A summary of the data so far follows in the concluding section. Please note that I would be very interested in communicating with other parties who see similar activity in their logs, and if you are responsible for one or more of the systems identified or you know who is, I urge you to take appropriate action.
The Data So Far
First, We Take root
The full log data reveal that they started rubbing up against my listening posts during the early hours of April 7th 2009 (as measured in CEST), concentrating first on the user root (attempts sorted by hostname here, hostnames here. Not terribly surprising that they tried to take on this one, as most Unix(ish) systems tend to at least have a root account.
Then We Take admin
After about twenty-four hours, the zombies in the cloud tired of root and turned their collective attention to admin (attempts sorted by hostname here, hostnames here. This is a user name I would not expect to be on a Unix syste by default. In my limited experience Microsoft systems tend to use this one more often, but then there may be enough of those systems out there with a SSH service and weak passwords out there to try.
And james Will Make Us Famous
Well into civilized lunchtime (CEST) on April 9th, our would-be guests turned their attention to james (attempts sorted by hostname here, hostnames here. Why this user name received so much attention is a complete mystery to me. They did tire in the end, though.
Before We Turn To 123456, aadi And The Rest Of The Rabble
As if to confirm that we are indeed seeing a rerun of last November and December's sequence of events, the robots started on their regular alphabetic attempts on April 11th:
Apr 11 17:44:15 rosalita sshd[51241]: error: PAM: authentication error for illegal user james from 221.130.177.154
Apr 11 17:50:47 rosalita sshd[51258]: error: PAM: authentication error for illegal user 123456 from 220.194.54.41
Apr 11 17:53:24 rosalita sshd[51262]: error: PAM: authentication error for illegal user 123456 from webmail.sistemafieg.org.br
Apr 11 17:58:25 rosalita sshd[51285]: error: PAM: authentication error for illegal user 123456 from 202.82.25.161
Apr 11 18:00:59 rosalita sshd[51307]: error: PAM: authentication error for illegal user aadi from mail.cgconsultoriacontabil.com.br
The various comments to my earlier columns tended to point out the near-futility of low-intensity bruteforcing, and I took the fact that the last series of attempts did not run to the end of the alphabet as an indication that the perpetrators themselves had reached the same conclusion.
Now we know that they just moved on, went elsewhere for a while and now they're back. In the meantime, they must have hit on enough weak minds and associated weak passwords that they were able to take over and zombiefy enough systems to be worth their while.
The data so far (collected up to April 12, 2009, roughly 21:00 CEST) indicates that the total number of hosts involved in the attempts at my listening posts is just over a thousand (a list of hosts available here).
It is probably worth repeating that in real life, zombies feed on both weak minds and weak passwords.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available for free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
Also worth noting, I will be doing a PF tutorial at BSDCan 2009 in May, and I will be staying around for the rest of the conference. I look forward to seeing you there.
Note: A Better Data Source Is Available
Update 2015-04-02: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Regular readers will remember that late last year we saw a peculiar form of distributed bruteforce attack on certain ssh servers. What made this particular batch of failed login attempts stand out was that unlike the traditional rapid-fire bruteforce attempts we were quite adept at heading off with state tracking tricks (such as the OpenBSD PF method described here or a slightly different Linux-specific method), the technique seemed to be specifically designed to slip past such defenses.
I described the method and the evidence at some length in a series of colums, A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008) and finally The slow brutes, a final roundup (January 22, 2009).
If you do not want to read through all of that, I'll recap: The traditional bruteforce attack is characterized by somebody trying again and again to find a combination of logon credentials (typically sets of user names and passwords) that will let them gain access to the system. The basic idea is that given enough tries, you will sooner or later find a user who has set a guessable password, and you have access.
One of the important weaknesses of the method is that the typical attacker would have gained access to only one or a few systems, and the attacker would then typically try to connect a lot more often than a typical user, and sysadmins learned to adapt their firewalls and other systems to lock out activity that fit the bruteforcer profile.
If you run a ssh service anywhere Internet-facing, you will be used to seeing a steady stream of failed logons for both existing and non-existing users. There's nothing new in seeing failed logons in your log files. However, what happened late last year was that we started seeing large numbers of failed ssh logon attempts, with the new twist that the same user would be trying to log on a large number of times, but never from the same place twice in rapid succession. This log data sample will give you an idea. The data will show you the pattern, as will the summary article.
Back then in January, my conclusion was that we had seen the conclusion of a largely failed experiment. After all, proceeding at a truly glacial pace and decreasing the number of attempts per user name as they went along hardly seemed like a recipe for success even if they could sneak past packet filtering based defenses.
It turns out I was wrong. Returning to my log summaries after taking a few days off from log data (yes, easter is big here, even moves geeks sometimes to take time off) I find data with almost exactly the same profile as last December's series.
This can only mean that there were enough successful attempts at guessing people's weak passwords in the last round that our unknown perpetrators found it worthwhile to start another round. For all I know they may have been at it all along, probing other parts of the Internet far from my unfashionable backwaters.
Anyway, they are most certainly back. A summary of the data so far follows in the concluding section. Please note that I would be very interested in communicating with other parties who see similar activity in their logs, and if you are responsible for one or more of the systems identified or you know who is, I urge you to take appropriate action.
The Data So Far
First, We Take root
The full log data reveal that they started rubbing up against my listening posts during the early hours of April 7th 2009 (as measured in CEST), concentrating first on the user root (attempts sorted by hostname here, hostnames here. Not terribly surprising that they tried to take on this one, as most Unix(ish) systems tend to at least have a root account.
Then We Take admin
After about twenty-four hours, the zombies in the cloud tired of root and turned their collective attention to admin (attempts sorted by hostname here, hostnames here. This is a user name I would not expect to be on a Unix syste by default. In my limited experience Microsoft systems tend to use this one more often, but then there may be enough of those systems out there with a SSH service and weak passwords out there to try.
And james Will Make Us Famous
Well into civilized lunchtime (CEST) on April 9th, our would-be guests turned their attention to james (attempts sorted by hostname here, hostnames here. Why this user name received so much attention is a complete mystery to me. They did tire in the end, though.
Before We Turn To 123456, aadi And The Rest Of The Rabble
As if to confirm that we are indeed seeing a rerun of last November and December's sequence of events, the robots started on their regular alphabetic attempts on April 11th:
Apr 11 17:44:15 rosalita sshd[51241]: error: PAM: authentication error for illegal user james from 221.130.177.154
Apr 11 17:50:47 rosalita sshd[51258]: error: PAM: authentication error for illegal user 123456 from 220.194.54.41
Apr 11 17:53:24 rosalita sshd[51262]: error: PAM: authentication error for illegal user 123456 from webmail.sistemafieg.org.br
Apr 11 17:58:25 rosalita sshd[51285]: error: PAM: authentication error for illegal user 123456 from 202.82.25.161
Apr 11 18:00:59 rosalita sshd[51307]: error: PAM: authentication error for illegal user aadi from mail.cgconsultoriacontabil.com.br
The various comments to my earlier columns tended to point out the near-futility of low-intensity bruteforcing, and I took the fact that the last series of attempts did not run to the end of the alphabet as an indication that the perpetrators themselves had reached the same conclusion.
Now we know that they just moved on, went elsewhere for a while and now they're back. In the meantime, they must have hit on enough weak minds and associated weak passwords that they were able to take over and zombiefy enough systems to be worth their while.
The data so far (collected up to April 12, 2009, roughly 21:00 CEST) indicates that the total number of hosts involved in the attempts at my listening posts is just over a thousand (a list of hosts available here).
It is probably worth repeating that in real life, zombies feed on both weak minds and weak passwords.
If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available for free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.
Also worth noting, I will be doing a PF tutorial at BSDCan 2009 in May, and I will be staying around for the rest of the conference. I look forward to seeing you there.
Note: A Better Data Source Is Available
Update 2015-04-02: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
23 comments:
Note: Comments are moderated. On-topic messages will be liberated from the holding queue at semi-random (hopefully short) intervals.
I invite comment on all aspects of the material I publish and I read all submitted comments. I occasionally respond in comments, but please do not assume that your comment will compel me to produce a public or immediate response.
Please note that comments consisting of only a single word or only a URL with no indication why that link is useful in the context will be immediately recycled so those poor electrons get another shot at a meaningful existence.
If your suggestions are useful enough to make me write on a specific topic, I will do my best to give credit where credit is due.
Subscribe to:
Post Comments (Atom)
I have had fail2ban on my machine for a couple of years now. I configured it with a peculiar "one-strike-your-out, permanently" policy. Basically if you attempt to log in to SSH and you type a bad password my box permanently bans your IP and won't talk SSH with your IP ever again.
ReplyDeleteThis is how I noticed the "Slow Brutes." My fail2ban policy effectively eliminated the single hosts trying every user and password combination, and highlighted something entirely different: The same user name tried by literally hundreds of hosts in a row.
Every now and again I reboot my computer as power outages or hardware adjustments sometimes necessary. I discovered that when fail2ban reset the lists - sometimes days after the initial entry - the same hosts were reappearing. The attack is very specifically geared to dodge the "normal" SSH ban rules: (fail x times and be locked out for y minutes)
After watching the same IP addresses keep coming back, I started stashing the iptables rules and bringing them back on reboot. I also built a ruby script to fetch IP's from my stored fail2ban email messages so I could have a more complete list. So far I have 4161 IP addresses in my DROP list.
About 1/4 of my list is new since 6/April/2009 athough fail2ban indicated that it had seen some of the IPs before (not sure how that happened... time for some digging, I suppose.)
Very similar pattern, though-
20090406: several attempts for "root"
20090407: switched to "admin"
20090409: a few attempts from "james"
20090411: more variety: abby, accalia, aderes, aderyn, adi, adia, adie, adiel, adina, adish, adita, adlai, admon, adolfo, adolph, adonai, adonis, ... and so on.
What I find interesting is that after watching the fail2ban email messages for the last year, this particular batch has a much broader penetration inside the US. Previous brute-force attempts were mostly in APNIC or RIPE blocks, or Brazil.
I'm considering adjusting the iptables rules to log so I can tell which hosts simply keep knocking, and how often. It is a rather fascinating development, however...
While using a blocking host (e.g. http://www.aczoom.com/cms/blockhosts) does a good job, it doesn't do much against the distributed attacks. What *does* seem to work is to simply move your ssh server to any other port. Thus far, the attacks have not been used in conjunction with a port scan, and I've seen all attempts drop to 0 by moving to a different and non-traditional port.
ReplyDelete$.02
Neil
note that 'james' is the most common first name in the United States that would explain why they try james first.
ReplyDeletehttp://names.mongabay.com/male_names.htm
Jeez, look at that log!
ReplyDeleteYeah only first names makes perfect sense if your trying to get into someone's system with the most privilege. The hypothesis here is that people prefer usernames that are 'their' names, and they like them to be simple, hence the first name. Usually you only get that option as username if it's 'your system', I bet that's what they're going on.
Thank you for all of the useful information. I wonder if this has anything to do with the latest Conficker update. This seems interesting, I'm almost tempted to set up a dummy ssh server and see what happens. Again, thanks for the info, I'll be back to see if there is any new news.
ReplyDeleteOut at my end of the backwaters the brute force attempts started up again on April 8 and have been running strong since then. At first almost all the attempts were on the 'james' username but in the last 24 hours they have moved on to an alphabetical list trying each username up to 5 times but never more than 5.
ReplyDeleteVery curious indeed.
While not a cure-all, on my shell servers I have always blocked ssh (and sftp, going although way back to the days of FTP and telnet) from IP addresses that do not have matching forward/reverse DNS entries. Just from the log entries above, 3 out of 4 didn't have reverse DNS.
ReplyDeleteFor old style rapid attacks, it limits the number of hosts that can pound on you, mitigating the DOS threat. For this style I would think blocking even half the hosts doing the slow-speed scan would limit it's effectiveness.
I've seen a lot of failed login attempts on SSH as well. Probably the most entertaining thing is that not only are all the user names (except) very wrong, but root login is disabled on my machine anyway.
ReplyDeleteSample attempted users:
ryan
joshua
michael
eaguilar
payala
estudiante
alex
gvera
vacaciones
housingp
...
Some seem to be simply first names, and I have to wonder what sysadmin would name accounts like that. Some seem to be odder combinations.
I'm seeing pretty much exactly the same events as you do on my FreeBSD systems, and I also noticed the same events at the end of last year.
ReplyDeleteAnd since i have a strict principle of banning an IP on the first failed attempt and sending a notification email, it gets pretty annoying after a while. Today i received around 200 of those emails within 24 hours.
The times are pretty much the same as yours, with 2 hours difference at most (I'm also CEST timezone).
If you'd like I can send you the logs of the attacks to help with the data.
I installed my first Ubuntu server last summer. The first time I looked at the auth.log file I almost had a heart attack, so I immediately decided to restrain as much as possible that kind of brute force attack. I installed "denyhosts" which works pretty well. any suggestion on this?
ReplyDeleteHi, got here from Slashdot. Started poking. This:
ReplyDeletegrep "Failed password for invalid" auth.log.0 | cut -d ' ' -f 11 | sort -u | wc
yields 325 unique, invalid username attempts on one
of my machines from 5-apr to 12-apr.
This machine is running Debian 5.0 (Lenny).
I can email you the list of usernames if you wish.
gregb
laserlab
com
I am noticing the same exact behavior for my ftp server. It seems this wave i snot limited to the one vector.
ReplyDeleteNot sure if you have control over HOW your users log in, but I moved SSH to a different port and on hosts I care about, removed password authentication in favor of certs.
ReplyDeletePeter,
ReplyDeleteNice post!
I haven't checked until now the auth.log for a Ubuntu driven site I have. I use fail2ban and denyhosts on it.
Running a little command (thanks Greg) gave me some interesting results:
grep "Failed password for invalid" auth.log.0 | cut -d ' ' -f 11 | sort -u | wc
3673 3673 22842
grep "Failed password for" auth.log.1 | cut -d ' ' -f 11 | sort -u | wc
3676 3676 22886
The last count show the results of failed login attempts that include the root user.
I will upload a copy of this stuff and hopefully have time to go them through with you next week.
R.
I have noticed a lot of network gear with an "admin" account. Some of them might have ssh enabled. Firewalls, wireless routers, cameras, printers...
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteSame with my box, just started ramping up last week, to the tune of .28 load on a 8 CPU server. I have to have 22 open, my company blocks to many other ports for me to try knocking, so i just did what most users did here. 2 strike rule for me, as even i enter the password incorectly once and a while.
ReplyDeleteI ran some of those grep scripts on my CENTOS box, only on the 11th do I have a large number. I see 1102, 1102, 7703
ReplyDeletehttp://www.evildaystar.net/banned.html
ReplyDeleteThis is a list of IPs I'm blocking in my firewall thanks to sshdfilter. It is only about 2 days worth of addresses thanks to my accidentally resetting the SSHD chain.
I'm working on interfacing sshdfilter with denyhosts to allow it to send and receive lists of blocked hosts.
When I have some more time I'll write something to pull the actual user names they are attempting and how many attempts per.
Since I can enumerate ahead of time the list of sources of acceptable SSH connections, I use TCP wrappers to help the zombies out:
ReplyDeleteIn /etc/hosts.allow (extended TCP wrapper syntax):
sshd : validnet/validmask \
127.0.0.0/255.0.0.0 \
: ALLOW
sshd : ALL : banners /var/db/banners \
: twist /bin/sleep 60
The other component:
% cat /var/db/banners/sshd
SSH-2.0-OpenSSH_5.1p1 FreeBSD-20080901
Legitimate connections from within the IP space defined by validnet/validmask are passed to sshd normally. Everything else gets something that looks valid, the TCP connection is held open for up to 60 seconds, and then it closes. It's analogous to PF-spamd's blacklisting behavior.
http://stats.sickness.mooo.com/hosts.deny
ReplyDeletedenyhosts for the win!!
I see this very often; and yep it's on the increase. I've seen the login attempt as the user 'fluffy' 20 different times since March 13th. fluffy makes me laugh.
ReplyDeleteI've been seeing anything up to a couple of thousand attempted login attempts over the last two weeks, starting with admin and james before going all alphabetical. Yesterday it stopped, suddenly at "chantoya", at around 2232 UTC.
ReplyDeleteI also had the same behaviour around December of last year, which also stopped suddenly (although the logs from that period have just expired, so I can't check the exact details).