Tuesday, December 2, 2008

A low intensity, distributed bruteforce attempt

We have seen the future of botnets, and it is a distributed, low-key affair. Are sites running free software finally becoming malware targets?

Note: This piece describes illegal activity I detected in 2008, targeting SSH servers. Later pieces in this series would hint at the existence of a specific piece of Linux malware, which I had not identified at the time this piece was written.


NOTE: A version without trackers but “classical” formatting is available here.

Phase 1: “That's odd …”
During the last few weeks, I noticed an anomaly in the authentication logs on one of my listening posts. There were a larger than usual number of ssh login attempts overall, a higher than usual number of attempts for non-existent user names as well as some failures for a few that actually exist as well.

Looking at the log directly a typical progression would look like this:

Nov 19 15:04:22 rosalita sshd[40232]: error: PAM: authentication error for illegal user alias from s514.nxs.nl
Nov 19 15:07:32 rosalita sshd[40239]: error: PAM: authentication error for illegal user alias from c90678d3.static.spo.virtua.com.br
Nov 19 15:10:20 rosalita sshd[40247]: error: PAM: authentication error for illegal user alias from 207-47-162-126.prna.static.sasknet.sk.ca
Nov 19 15:13:46 rosalita sshd[40268]: error: PAM: authentication error for illegal user alias from 125-236-218-109.adsl.xtra.co.nz
Nov 19 15:16:29 rosalita sshd[40275]: error: PAM: authentication error for illegal user alias from 200.93.147.114
Nov 19 15:19:12 rosalita sshd[40279]: error: PAM: authentication error for illegal user alias from 62.225.15.82
Nov 19 15:22:29 rosalita sshd[40298]: error: PAM: authentication error for illegal user alias from 121.33.199.39
Nov 19 15:25:14 rosalita sshd[40305]: error: PAM: authentication error for illegal user alias from 130.red-80-37-213.staticip.rima-tde.net
Nov 19 15:28:23 rosalita sshd[40309]: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com
Nov 19 15:31:17 rosalita sshd[40316]: error: PAM: authentication error for illegal user alias from gate-dialog-simet.jgora.dialog.net.pl
Nov 19 15:34:18 rosalita sshd[40334]: error: PAM: authentication error for illegal user alias from 80.51.31.84
Nov 19 15:37:23 rosalita sshd[40342]: error: PAM: authentication error for illegal user alias from 82.207.104.34
Nov 19 15:40:20 rosalita sshd[40350]: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com
Nov 19 15:43:39 rosalita sshd[40354]: error: PAM: authentication error for illegal user alias from 200.20.187.222
Nov 19 15:46:41 rosalita sshd[40374]: error: PAM: authentication error for illegal user amanda from 58.196.4.2
Nov 19 15:49:31 rosalita sshd[40378]: error: PAM: authentication error for illegal user amanda from host116-164.dissent.birch.net
Nov 19 15:55:47 rosalita sshd[40408]: error: PAM: authentication error for illegal user amanda from robert71.lnk.telstra.net
Nov 19 15:59:08 rosalita sshd[40412]: error: PAM: authentication error for illegal user amanda from static-71-166-159-177.washdc.east.verizon.net
Nov 19 16:02:06 rosalita sshd[40455]: error: PAM: authentication error for illegal user amanda from host87-163-static.30-87-b.business.telecomitalia.it
Nov 19 16:05:08 rosalita sshd[40459]: error: PAM: authentication error for illegal user amanda from 213.150.184.70
Nov 19 16:08:16 rosalita sshd[40465]: error: PAM: authentication error for illegal user amanda from mail.pddsl.de
Nov 19 16:11:24 rosalita sshd[40486]: error: PAM: authentication error for illegal user amanda from abu66.internetdsl.tpnet.pl
Nov 19 16:15:00 rosalita sshd[40491]: error: PAM: authentication error for illegal user amanda from 125.77.106.246
Nov 19 16:17:43 rosalita sshd[40497]: error: PAM: authentication error for illegal user amanda from 217.76.34.230
Nov 19 16:20:54 rosalita sshd[40506]: error: PAM: authentication error for illegal user amanda from robert71.lnk.telstra.net
Nov 19 16:24:09 rosalita sshd[40529]: error: PAM: authentication error for illegal user amanda from p578b4f0b.dip0.t-ipconnect.de
Nov 19 16:28:11 rosalita sshd[40538]: error: PAM: authentication error for illegal user amanda from mail.carena-ci.com
Nov 19 16:30:15 rosalita sshd[40551]: error: PAM: authentication error for illegal user amavis from 87.229.3.89
Nov 19 16:34:31 rosalita sshd[40567]: error: PAM: authentication error for illegal user amavis from 218.248.79.251
Nov 19 16:36:40 rosalita sshd[40574]: error: PAM: authentication error for illegal user amavis from 83-103-70-170.ip.fastwebnet.it
Nov 19 16:40:05 rosalita sshd[40596]: error: PAM: authentication error for illegal user amavis from 75-49-251-71.lightspeed.snjsca.sbcglobal.net

- and so on, with a striking regularity. See for example the attempts to log on as the alias user, 14 attempts are made from 13 different hosts, with only 70-46-140-187.orl.fdn.com trying more than once. Then thirteen attempts are made for the amanda user, from 13 other hosts. The pattern repeats again for users amavis, apache, at, and goes on with others, apparently trying users in an alphabetic sequence.

Phase 2: Not your run of the mill screwup, the data say
Repeated login attempts for non-existing users are nothing new (in fact the bruteforce avoidance section is one of the more popular parts of the PF tutorial), but I was a bit surprised to see the attempts actually reaching this machine, which is on a local network behind a PF gateway with a configuration that is in fact closely related to the one in the tutorial (and the book for that matter). Then looking at the log entries, I noticed a few more things: The attempts are never less than a minute apart, and the attempts from a single host are separated by much longer intervals. The full data set I extracted from the point I started noticing those anomalies sum up to these figures can be found here, in case you want to look at it and draw you own conclusions

Some one-liners give us illustrative numbers:

peter@thingy:~$ wc -l slowbrutes.txt
16727 slowbrutes.txt

That is, over this period there were 16727 failed ssh login attempts at this host. A large number for this particular machine, but not enough to raise eyebrows by itself at larger or busier sites.

More than sixteen thousand attempts, but for how many invalid user names?

peter@thingy:~$ grep illegal slowbrutes.txt | awk '{print $13}' | sort -u | wc -l
2962
peter@thingy:~$ grep illegal slowbrutes.txt | awk '{print $15}' | sort -u | wc -l
671

That is, approaching three thousand unlucky guesses, coming from 671 different hosts.

How many valid user names did they stumble upon?

peter@thingy:~$ grep -v illegal slowbrutes.txt | awk '{print $11}' | sort -u | wc -l
2

A grand total of two, one of them the rather obvious root, for a total of

peter@thingy:~$ grep -vc illegal slowbrutes.txt
1698

1698 attempts, coming from

peter@thingy:~$ grep -v illegal slowbrutes.txt | awk '{print $13}' | sort -u | wc -l
566

566 different hosts.

The patterns that emerge from the data, with the alphabetical ordering and apparent coordination, point to a botnet herder trying out new methods. Intrusion detection systems and adaptive firewalls are generally tuned to detecting things like large numbers of simultaneous connecions or a high rate of new connections from a host. Distributing the task of bruteforcing passwords to several hosts could seem like an inspired way to come in under the radar wherever relatively smart systems are in place. Setting the herd to attempt at a low frequency would likely mean that those failed attempts simply drown in the noise at higher volume sites, and will not be noticed.

Phase 3: Are you one of their guinea pigs, too?
There are indications that the method has not been quite perfected yet. At the start of this run, the bots would make at least ten attempts before moving on down the alphabet. Now it seems enough bots have been taken out of circulation that the typical number of attempts per user name is closer to three, with some tried only once:

Dec 2 11:45:59 rosalita sshd[55775]: error: PAM: authentication error for illegal user heaven from cpe001217e403b3-cm000f9fa6157c.cpe.net.cable.rogers.com
Dec 2 11:48:16 rosalita sshd[55778]: error: PAM: authentication error for illegal user heaven from 90.190.96.46
Dec 2 11:50:39 rosalita sshd[55791]: error: PAM: authentication error for illegal user heaven from static-71-117-126-102.snloca.dsl-w.verizon.net
Dec 2 11:55:26 rosalita sshd[55811]: error: PAM: authentication error for illegal user heavynne from dsl-217-155-184-54.zen.co.uk
Dec 2 11:57:57 rosalita sshd[55814]: error: PAM: authentication error for illegal user heavynne from pd907ee1e.dip0.t-ipconnect.de
Dec 2 12:00:20 rosalita sshd[55836]: error: PAM: authentication error for illegal user heba from 201-26-172-213.dial-up.telesp.net.br
Dec 2 12:07:37 rosalita sshd[55879]: error: PAM: authentication error for illegal user hector from 75.145.16.83
Dec 2 12:09:58 rosalita sshd[55882]: error: PAM: authentication error for illegal user hector from ppp-69-217-30-214.dsl.applwi.ameritech.net
Dec 2 12:12:33 rosalita sshd[55901]: error: PAM: authentication error for illegal user hector from 75-49-251-71.lightspeed.snjsca.sbcglobal.net
Dec 2 12:14:51 rosalita sshd[55905]: error: PAM: authentication error for illegal user hedda from 201.218.231.142
Dec 2 12:17:21 rosalita sshd[55911]: error: PAM: authentication error for illegal user hedda from 75.147.27.85
Dec 2 12:19:48 rosalita sshd[55914]: error: PAM: authentication error for illegal user hedda from 203.70.179.113

From where I'm sitting it's hard to tell whether the lower number of attempts means that the machines have cleaned up by their legal owners or whether they have simply taken out of rotation by their herders. Even with the initial 14 attempts per user name the chance of actually finding a valid combination of user names and passwords would be slim but not non-existent, but decreasing the number of attempts per time unit will necessarily make the chance of eventually finding a valid pair even smaller.

Apparently I'm not the only one seeing the slow brutes, as this post to openbsd-misc indicates. The sensible countermeasure could be to disallow shh password logins and allow only key logins, probably easier to set up and enforce than network-level measures. With the slow rate of attempts and the relatively large number of hosts involved, the undesirable traffic here is relatively hard to distinguish automatically from innocent errors unless you make have any attempt to log in with an invalid user name a sufficent reason for blocking traffic from that host.

Phase N: The shape of things to come
In the longer term view, this may very well be the shape of botnets to come. With a large enough pool of compromised hosts under their control, future botnet herders can afford to organize their activity so any one host only participates in undesirable activity at intervals long enough that malware detectors do not trigger (and thinking further ahead, if the world ever does go IPv6 wholesale and you can expect any one network interface to have dozens of IP addresses, think again how much more interesting detecting botnets becomes).

Antiware vendors will likely put their spin on this too when their marketing departments start noticing columns (Hey! It's Linux they're targeting!), but then as regular readers know, the more productive approach is always to reduce malware masters' target area by using systems that are less vulnerable because they have been extensively audited and whose makers are unafraid to make source code available for public inspection and experimentation.



As people who ran into me at the recent PF tutorial in London or at OpenCON will already know, have I joined FreeCode, the Norwegian free software consultancy. We expect to keep doing fun and useful things with free software for customers and friends, and some of it may be interesting enough to be the topics of future columns.


Note:
 A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

24 comments:

  1. Hello,

    I've noticed this on my machine as well. I'm running FreeBSD 7.0 on my webserver. I always have sshblack running and it was remarkably successful at blocking brute-force attacks which come from a single host. But over the past few weeks I've been noticing attacks exactly like the ones you describe, coming from different IP's.

    ReplyDelete
  2. Same here. I noticed it a few weeks ago. I ended up blocking port 22 from all but a few trusted hosts (I have ssh open on another port, for earlier reasons).

    My big fear is that this will turn from low-key to high-key -- and it'll be a damned-near unstoppable DDoS.

    ReplyDelete
  3. Yeah, I noticed this a few weeks ago as well. Quite annoying. I use iptables to filter out the offending TCP/IP addresses.

    ReplyDelete
  4. that's why denyhosts is so great. not only does it lock out hosts that are trying to often, but it synchronizes data with others so that everyone can benefit once a host goes above that threshold.

    denyhosts.sourceforge.net

    ReplyDelete
  5. I noticed this same thing just yesterday on my pfsense box running 1.2.1 RC (based on 7.0). There's no ssh listening on the external interface, but that isn't keeping it from trying.

    ReplyDelete
  6. Nice post.

    I've just come found you through the Slashdot post.

    Nice blog too.

    ReplyDelete
  7. Nice article! I've been seeing a staggering number of these on my public-IP gentoo box the past few weeks. It abated slightly after I left it powered off for five days, but picked up again about 24 hours later. Having these attempts buzz through my logs is the new background noise of the internet.

    ReplyDelete
  8. Nice article! I've been seeing a staggering number of these on my public-IP gentoo box the past few weeks. It abated slightly after I left it powered off for five days, but picked up again about 24 hours later. Having these attempts buzz through my logs is the new background noise of the internet.

    ReplyDelete
  9. Yeah, we get this on wefixtech.co.uk too. We have something that actually blocks your IP after the third failed ssh attempt. I have to go in a back way when I mistype my own password too much ;-)

    ReplyDelete
  10. I've been noticing this on a FreeBSD machine I use as a firewall/NAT for my home network. I wrote a script to harvest the IP addresses. It can also drop them into an "ipfw2" table I use to block with.

    Some hosts showed up only once, some showed up as much as 35 times in 3 days.

    ReplyDelete
  11. This went on with my server for months on end. I run the most generic, common configuration possible, and I am far from an expert. (In other words, I worry a lot.) I responded first of all by disabling ssh for any but local logins--and that includes myself. I also acquired and installed a software firewall to trap, log, and block IP addresses. I feel like the Dutch boy with his finger in the dike.

    The reason I'm responding to your comment is that I noticed the exact situation you describe well over a year ago and have had these two small remedies in place for almost as long.

    ReplyDelete
  12. I remember back when I had FreeBSD as my server OS, there was a trick in pf that I could use to very quickly add a given host to a list of blocked IP's. The method escapes me now as I've been ubuntu-bound for a while. That said...

    Noticing some "hard and fast" SSHD brute force attempts in my logs, I created a little script to grab the last 1000 or so lines of the auth log, grep out whatever pattern identified a brute attack, extract IP addresses, sort, and get a report of number of bogus/valid login attempts per IP.

    Now just add to the "blacklist candidates" any IP that has more than n bogus attempts (I used 10, because I was feeling lazy and it meant I could use a regex to check for more than 1 digit). Add to the "whitelist candidates" any IP that has more than m successful logins (say, 3). Prune the white entries from the black and add the result to pf with a drop rule. Now run that every 5 minutes from Cron.

    Leaving this script in place, I cut down on maybe 95% of the SSH errors I would see in the log, and was able to give better attention to those incidents which needed it. Granted it's a primitive approach, but it would appear to work here, so long as a given node tried more than the threshold number of attempts, regardless of which username was chosen in each attempt. YMMV.

    ReplyDelete
  13. Checking for the total number of *unique* incorrect passwords should help.

    When a legit user mistypes a password, or enters the incorrect password for a particular login, it'll only typically be one of a few things.


    I can't imagine a legit user attempting more than 10 unique incorrect passwords, and brute-forcers aren't going to attempt the same incorrect password twice.

    ReplyDelete
  14. Okay, this is just about the last straw. I am ready to move all my ssh servers to an alternate port and TCP tarpit all traffic to 22/tcp...

    ReplyDelete
  15. Exactly what I've seen on my web server.

    See my story for the resolution to my brute force ssh attacks.

    All has been quiet since making the changes.

    Good story!

    Dietrich T. Schmitz
    Linux IT Consultant
    http://www.dtschmitz.com

    ReplyDelete
  16. I can't help but notice that a number of the comments here partially miss the point. We're seeing this massively distributed SSH brute force attack too, and it's clear that reactive blocking by IP can be only marginally effective. Botnets are becoming sufficiently large that attackers now have a virtually inexhaustible pool of hosts at their disposal. Our solution was to finally implement knockd.

    Of course implementing port knocking only shifts the problem rather than solving it -- we'll probably eventually see brute-force port knocking attacks. But, for the moment, it's satisfying to have detritus-free sshd logs.

    ReplyDelete
  17. I solved my ssh problems by writing a wrapper: ssh-faker. Nobody gets to connect to my sshd until they telnet to port 22 and type a password. At that point, their ip address is added to /etc/hosts.allow permanently. It's a rock-solid solution with few moving parts, and no bot is going to bypass it or buffer-overflow it.

    ReplyDelete
  18. Knock is an interesting suggestion. While in effect it isn't that much more than another password it has certain advantages:

    - potentially much more random than passwords using the full 65000 range against a lot of alphanumericals in typical passwords. Discourages dictionary attacks.

    - Adding firewall rules for hosts costs cycles both for the work and then potentionally for all further network traffic, exponentially. Using knock instead, while knock adds overhead it is linear. The firewall on its end can be set up to generally consume less resources to compensate. Knock offers better victim economics.

    - It combines well with keys, even if the key or the laptop it is on is stolen, the risk is limited (if the knock isn't recorded in bash history ...)

    On the flip side it is vulnerable to replay attacks, but replay attacks are somewhat costly and likely not found worthwhile if what they open is a ssh login prompt.

    Leading to my second point;

    The problem is that it costs very little for the client to try ssh passwords all day long. If the ssh logon needed you to solve increasingly complex challenges for each try, it could pose significant problem for botnet managers since it would make the bot slow down, possibly triggering response with the owner of the computer. It is tolerable to users to wait 5 seconds more for external ssh login, (even if it would be 30 seconds on an iphone).

    A possible third failed try challenge could be a sudoku puzzle, perhaps in a pdf form. Cheap to generate a reasonable batch of these. Would be preferrable to be locked out for users or admins. With some randomization it wouldn't checksum. Numbers could be gifs. Now obviously it wouldn't stop any determined attacker, but the cost of the attack would be up with a very high factor. And each success would only yield a very meager chance of a return on the investment.

    Perhaps a possible project for his grumppyness?

    ReplyDelete
  19. I have encountered the same issue for the past 6 months. The first wave was approximately 500 hosts and lasted 2 weeks, and each successive wave adds new IP addresses.
    I have used denyhosts and its regex to add these hosts to /etc/hosts.deny, and to sync to the denyhosts project.
    Each successive wave adds an additional 40-120 new hosts, which then are added to /etc/hosts.deny
    One thing that the /etc/hosts.deny does not allow me to do is block ranges of IP addresses (or I'm doing something wrong). Since only local US IP addresses should be allowed to attempt to ssh into port 22 on one of our machines, I wanted to block .cn. .cz, .hu. .ru. etc...
    I've encounted brute force attacks but actually now enjoy tailing syslog to watch all the "refused connect" errors.

    ReplyDelete
  20. Instead of rejecting, let them in!

    What about giving for each invalid user/password an answer that the login was successful (I mean a fake login)?
    Now that would be quite frustrating to the hackers to have so many false positives. They will probably either have to change their scanning methods or just dump those servers who simulate a successful login.

    This would need a special ssh daemon of course.

    ReplyDelete
  21. Firewalling is a great first step but, for me, I think that further action is needed.. hackers need to be made accountable for their actions.

    Recently I made a small mod to the OpenSSH code to allow it to call a script with the ip and attempted username from each hack attempt. This script then does a whois lookup to extract the abuse contact for the offending netblock owner and then send an abuse email, this has been met with great success from abuse admins and has actually caused positive action towards those that are either comprmised or directly making hack attampts.

    Please feel free to check it out if you'd like, http://daemons.tog.net

    ReplyDelete
  22. My system was subjected to the same attack for some time, and it seems to have just stopped abruptly very recently. Have you seen the same?
    I was expecting that the attack would use names from A-Z, but they didn't even make it to names starting with T before it stopped...

    ReplyDelete
  23. Love your site. I put a link to you on my blog...

    ReplyDelete

Note: Comments are moderated. On-topic messages will be liberated from the holding queue at semi-random (hopefully short) intervals.

I invite comment on all aspects of the material I publish and I read all submitted comments. I occasionally respond in comments, but please do not assume that your comment will compel me to produce a public or immediate response.

Please note that comments consisting of only a single word or only a URL with no indication why that link is useful in the context will be immediately recycled so those poor electrons get another shot at a meaningful existence.

If your suggestions are useful enough to make me write on a specific topic, I will do my best to give credit where credit is due.