A low intensity, distributed bruteforce attempt

We have seen the future of botnets, and it is a distributed, low-key affair. Are sites running free software finally becoming malware targets?

Note: This piece describes illegal activity I detected in 2008, targeting SSH servers. Later pieces in this series would hint at the existence of a specific piece of Linux malware, which I had not identified at the time this piece was written.

NOTE: A version without trackers but “classical” formatting is available here.

Phase 1: “That's odd …”
During the last few weeks, I noticed an anomaly in the authentication logs on one of my listening posts. There were a larger than usual number of ssh login attempts overall, a higher than usual number of attempts for non-existent user names as well as some failures for a few that actually exist as well.

Looking at the log directly a typical progression would look like this:

Nov 19 15:04:22 rosalita sshd[40232]: error: PAM: authentication error for illegal user alias from s514.nxs.nl
Nov 19 15:07:32 rosalita sshd[40239]: error: PAM: authentication error for illegal user alias from c90678d3.static.spo.virtua.com.br
Nov 19 15:10:20 rosalita sshd[40247]: error: PAM: authentication error for illegal user alias from 207-47-162-126.prna.static.sasknet.sk.ca
Nov 19 15:13:46 rosalita sshd[40268]: error: PAM: authentication error for illegal user alias from 125-236-218-109.adsl.xtra.co.nz
Nov 19 15:16:29 rosalita sshd[40275]: error: PAM: authentication error for illegal user alias from 200.93.147.114
Nov 19 15:19:12 rosalita sshd[40279]: error: PAM: authentication error for illegal user alias from 62.225.15.82
Nov 19 15:22:29 rosalita sshd[40298]: error: PAM: authentication error for illegal user alias from 121.33.199.39
Nov 19 15:25:14 rosalita sshd[40305]: error: PAM: authentication error for illegal user alias from 130.red-80-37-213.staticip.rima-tde.net
Nov 19 15:28:23 rosalita sshd[40309]: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com
Nov 19 15:31:17 rosalita sshd[40316]: error: PAM: authentication error for illegal user alias from gate-dialog-simet.jgora.dialog.net.pl
Nov 19 15:34:18 rosalita sshd[40334]: error: PAM: authentication error for illegal user alias from 80.51.31.84
Nov 19 15:37:23 rosalita sshd[40342]: error: PAM: authentication error for illegal user alias from 82.207.104.34
Nov 19 15:40:20 rosalita sshd[40350]: error: PAM: authentication error for illegal user alias from 70-46-140-187.orl.fdn.com
Nov 19 15:43:39 rosalita sshd[40354]: error: PAM: authentication error for illegal user alias from 200.20.187.222
Nov 19 15:46:41 rosalita sshd[40374]: error: PAM: authentication error for illegal user amanda from 58.196.4.2
Nov 19 15:49:31 rosalita sshd[40378]: error: PAM: authentication error for illegal user amanda from host116-164.dissent.birch.net
Nov 19 15:55:47 rosalita sshd[40408]: error: PAM: authentication error for illegal user amanda from robert71.lnk.telstra.net
Nov 19 15:59:08 rosalita sshd[40412]: error: PAM: authentication error for illegal user amanda from static-71-166-159-177.washdc.east.verizon.net
Nov 19 16:02:06 rosalita sshd[40455]: error: PAM: authentication error for illegal user amanda from host87-163-static.30-87-b.business.telecomitalia.it
Nov 19 16:05:08 rosalita sshd[40459]: error: PAM: authentication error for illegal user amanda from 213.150.184.70
Nov 19 16:08:16 rosalita sshd[40465]: error: PAM: authentication error for illegal user amanda from mail.pddsl.de
Nov 19 16:11:24 rosalita sshd[40486]: error: PAM: authentication error for illegal user amanda from abu66.internetdsl.tpnet.pl
Nov 19 16:15:00 rosalita sshd[40491]: error: PAM: authentication error for illegal user amanda from 125.77.106.246
Nov 19 16:17:43 rosalita sshd[40497]: error: PAM: authentication error for illegal user amanda from 217.76.34.230
Nov 19 16:20:54 rosalita sshd[40506]: error: PAM: authentication error for illegal user amanda from robert71.lnk.telstra.net
Nov 19 16:24:09 rosalita sshd[40529]: error: PAM: authentication error for illegal user amanda from p578b4f0b.dip0.t-ipconnect.de
Nov 19 16:28:11 rosalita sshd[40538]: error: PAM: authentication error for illegal user amanda from mail.carena-ci.com
Nov 19 16:30:15 rosalita sshd[40551]: error: PAM: authentication error for illegal user amavis from 87.229.3.89
Nov 19 16:34:31 rosalita sshd[40567]: error: PAM: authentication error for illegal user amavis from 218.248.79.251
Nov 19 16:36:40 rosalita sshd[40574]: error: PAM: authentication error for illegal user amavis from 83-103-70-170.ip.fastwebnet.it
Nov 19 16:40:05 rosalita sshd[40596]: error: PAM: authentication error for illegal user amavis from 75-49-251-71.lightspeed.snjsca.sbcglobal.net
- and so on, with a striking regularity. See for example the attempts to log on as the alias user, 14 attempts are made from 13 different hosts, with only 70-46-140-187.orl.fdn.com trying more than once. Then thirteen attempts are made for the amanda user, from 13 other hosts. The pattern repeats again for users amavis, apache, at, and goes on with others, apparently trying users in an alphabetic sequence.

Phase 2: Not your run of the mill screwup, the data say
Repeated login attempts for non-existing users are nothing new (in fact the bruteforce avoidance section is one of the more popular parts of the PF tutorial), but I was a bit surprised to see the attempts actually reaching this machine, which is on a local network behind a PF gateway with a configuration that is in fact closely related to the one in the tutorial (and the book for that matter). Then looking at the log entries, I noticed a few more things: The attempts are never less than a minute apart, and the attempts from a single host are separated by much longer intervals. The full data set I extracted from the point I started noticing those anomalies sum up to these figures can be found here, in case you want to look at it and draw you own conclusions

Some one-liners give us illustrative numbers:

peter@thingy:~$ wc -l slowbrutes.txt
16727 slowbrutes.txt
That is, over this period there were 16727 failed ssh login attempts at this host. A large number for this particular machine, but not enough to raise eyebrows by itself at larger or busier sites.

More than sixteen thousand attempts, but for how many invalid user names?

peter@thingy:~$ grep illegal slowbrutes.txt | awk '{print $13}' | sort -u | wc -l
2962
peter@thingy:~$ grep illegal slowbrutes.txt | awk '{print $15}' | sort -u | wc -l
671
That is, approaching three thousand unlucky guesses, coming from 671 different hosts.

How many valid user names did they stumble upon?

peter@thingy:~$ grep -v illegal slowbrutes.txt | awk '{print $11}' | sort -u | wc -l
2
A grand total of two, one of them the rather obvious root, for a total of

peter@thingy:~$ grep -vc illegal slowbrutes.txt
1698
1698 attempts, coming from

peter@thingy:~$ grep -v illegal slowbrutes.txt | awk '{print $13}' | sort -u | wc -l
566
566 different hosts.

The patterns that emerge from the data, with the alphabetical ordering and apparent coordination, point to a botnet herder trying out new methods. Intrusion detection systems and adaptive firewalls are generally tuned to detecting things like large numbers of simultaneous connecions or a high rate of new connections from a host. Distributing the task of bruteforcing passwords to several hosts could seem like an inspired way to come in under the radar wherever relatively smart systems are in place. Setting the herd to attempt at a low frequency would likely mean that those failed attempts simply drown in the noise at higher volume sites, and will not be noticed.

Phase 3: Are you one of their guinea pigs, too?
There are indications that the method has not been quite perfected yet. At the start of this run, the bots would make at least ten attempts before moving on down the alphabet. Now it seems enough bots have been taken out of circulation that the typical number of attempts per user name is closer to three, with some tried only once:

Dec 2 11:45:59 rosalita sshd[55775]: error: PAM: authentication error for illegal user heaven from cpe001217e403b3-cm000f9fa6157c.cpe.net.cable.rogers.com
Dec 2 11:48:16 rosalita sshd[55778]: error: PAM: authentication error for illegal user heaven from 90.190.96.46
Dec 2 11:50:39 rosalita sshd[55791]: error: PAM: authentication error for illegal user heaven from static-71-117-126-102.snloca.dsl-w.verizon.net
Dec 2 11:55:26 rosalita sshd[55811]: error: PAM: authentication error for illegal user heavynne from dsl-217-155-184-54.zen.co.uk
Dec 2 11:57:57 rosalita sshd[55814]: error: PAM: authentication error for illegal user heavynne from pd907ee1e.dip0.t-ipconnect.de
Dec 2 12:00:20 rosalita sshd[55836]: error: PAM: authentication error for illegal user heba from 201-26-172-213.dial-up.telesp.net.br
Dec 2 12:07:37 rosalita sshd[55879]: error: PAM: authentication error for illegal user hector from 75.145.16.83
Dec 2 12:09:58 rosalita sshd[55882]: error: PAM: authentication error for illegal user hector from ppp-69-217-30-214.dsl.applwi.ameritech.net
Dec 2 12:12:33 rosalita sshd[55901]: error: PAM: authentication error for illegal user hector from 75-49-251-71.lightspeed.snjsca.sbcglobal.net
Dec 2 12:14:51 rosalita sshd[55905]: error: PAM: authentication error for illegal user hedda from 201.218.231.142
Dec 2 12:17:21 rosalita sshd[55911]: error: PAM: authentication error for illegal user hedda from 75.147.27.85
Dec 2 12:19:48 rosalita sshd[55914]: error: PAM: authentication error for illegal user hedda from 203.70.179.113
From where I'm sitting it's hard to tell whether the lower number of attempts means that the machines have cleaned up by their legal owners or whether they have simply taken out of rotation by their herders. Even with the initial 14 attempts per user name the chance of actually finding a valid combination of user names and passwords would be slim but not non-existent, but decreasing the number of attempts per time unit will necessarily make the chance of eventually finding a valid pair even smaller.

Apparently I'm not the only one seeing the slow brutes, as this post to openbsd-misc indicates. The sensible countermeasure could be to disallow shh password logins and allow only key logins, probably easier to set up and enforce than network-level measures. With the slow rate of attempts and the relatively large number of hosts involved, the undesirable traffic here is relatively hard to distinguish automatically from innocent errors unless you make have any attempt to log in with an invalid user name a sufficent reason for blocking traffic from that host.

Phase N: The shape of things to come
In the longer term view, this may very well be the shape of botnets to come. With a large enough pool of compromised hosts under their control, future botnet herders can afford to organize their activity so any one host only participates in undesirable activity at intervals long enough that malware detectors do not trigger (and thinking further ahead, if the world ever does go IPv6 wholesale and you can expect any one network interface to have dozens of IP addresses, think again how much more interesting detecting botnets becomes).

Antiware vendors will likely put their spin on this too when their marketing departments start noticing columns (Hey! It's Linux they're targeting!), but then as regular readers know, the more productive approach is always to reduce malware masters' target area by using systems that are less vulnerable because they have been extensively audited and whose makers are unafraid to make source code available for public inspection and experimentation.

---

As people who ran into me at the recent PF tutorial in London or at OpenCON will already know, have I joined FreeCode, the Norwegian free software consultancy. We expect to keep doing fun and useful things with free software for customers and friends, and some of it may be interesting enough to be the topics of future columns.


Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.