A Small Update About The Slow Brutes

Slow and steady might actually do it, eventually.

The reactions to my December 2nd column hit me with a bit of surprise. The column was taken on by slashdot and Linux Today both, producing a largish number of page views, but only two clicks on my featured ads. But while my clickthrough rate is not particularly interesting to others, the comments to the columns sometimes are.

If you look at the comments at slashdot and elsewhere, most of the commenters most likely did not actually read the column in full or did not take the time to digest what it actually said, with some notable exceptions. And yes, there were others, some also wrote in via email with informed comment - thanks!

For the benefit of those who did not get the point the first time around, I'll try once more to explain what the observations are and what they may in fact mean.

A number of commenters offered well meant advice to use packages like fail2ban, denyhosts or a few others.

The common denominator for all of them is that they track single hosts that make a larger than usual number of connections or are the source of a number of failed logins higher than a certain threshold value over a set time period. I appreciate your concerns, but the subject of the column did not fit well with the way those the packages work.

In fact, a similar scheme was already in place at the site that provided the data. The machines that provided the ssh logs are FreeBSD ones (as the sharper ones have observed already, and the reasons may possibly be revealed over beer sometime), but any gateway under my control will run OpenBSD, and by extension, PF (and yes, there is a book you might want to order from one (North America) or the other (Europe and elsewhere) of the OpenBSD project's sites). For a quick fix of background, the online PF tutorial may be worth a look.

Anyway, the /etc/pf.conf at that site's gateway contains the lines

table <abusive_hosts> persist
block log quick from <abusive_hosts>
and

pass log (all) quick proto { tcp, udp } from any to any port ssh flags S/SA keep state \
(max-src-conn 15, max-src-conn-rate 7/3, overload <abusive_hosts> flush global)

Those lines provide a variation on the logic that those posters recommended. Essentially, any host that tries 15 or more simultaneous ssh connections, or come in at a rate of more than seven over the span of three seconds, will be added to the table <abusive_hosts>, and the block quick rule blocks any further access from those hosts. Yes, flush global means what you think it does.

This works at the network level. For a gateway with a potentially large number of hosts on either side, the success or otherwise of eventual authentication may not be relevant and may be better dealt with elsewhere. Anyway, at the time I started working on this column, the table <abusive_hosts> on the gateway contained only two hosts:

:~$ sudo pfctl -t abusive_hosts -v show
194.204.37.93
201.57.187.114
I keep offenders in that table for 24 hours only, I do not believe in the permanent bans that some commenters advocate. After all, there is such a thing as DHCP, and entire netblocks are reallocated with amazingly short intervals.

Anyway, looking at the authentication log on the gateway reveals how those hosts got added to the the table in the first place:

:~$ grep 194.204.37.93 /var/log/authlog
Dec 5 22:50:30 delilah sshd[15266]: Did not receive identification string from 194.204.37.93
Dec 5 22:50:37 delilah sshd[6106]: Did not receive identification string from 194.204.37.93
Dec 6 01:29:58 delilah sshd[30359]: Failed password for root from 194.204.37.93 port 47071 ssh2
Dec 6 01:29:58 delilah sshd[7293]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:29:59 delilah sshd[4395]: Failed password for root from 194.204.37.93 port 47296 ssh2
Dec 6 01:29:59 delilah sshd[27615]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:00 delilah sshd[12248]: Failed password for root from 194.204.37.93 port 47330 ssh2
Dec 6 01:30:00 delilah sshd[24579]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:01 delilah sshd[3434]: Failed password for root from 194.204.37.93 port 47380 ssh2
Dec 6 01:30:01 delilah sshd[32737]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:02 delilah sshd[11984]: Failed password for root from 194.204.37.93 port 47425 ssh2
Dec 6 01:30:02 delilah sshd[27059]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:03 delilah sshd[13345]: Failed password for root from 194.204.37.93 port 47459 ssh2
Dec 6 01:30:03 delilah sshd[1858]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:04 delilah sshd[12739]: Failed password for root from 194.204.37.93 port 47516 ssh2
Dec 6 01:30:04 delilah sshd[16843]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:05 delilah sshd[13796]: Failed password for root from 194.204.37.93 port 47564 ssh2
Dec 6 01:30:05 delilah sshd[16789]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:06 delilah sshd[628]: Failed password for root from 194.204.37.93 port 47602 ssh2
Dec 6 01:30:06 delilah sshd[6162]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:07 delilah sshd[2579]: Failed password for root from 194.204.37.93 port 47646 ssh2
Dec 6 01:30:07 delilah sshd[12461]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:08 delilah sshd[12725]: Failed password for root from 194.204.37.93 port 47685 ssh2
Dec 6 01:30:08 delilah sshd[29909]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:09 delilah sshd[16560]: Failed password for root from 194.204.37.93 port 47724 ssh2
Dec 6 01:30:09 delilah sshd[1690]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:09 delilah sshd[1600]: Failed password for root from 194.204.37.93 port 47771 ssh2
Dec 6 01:30:09 delilah sshd[28882]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:10 delilah sshd[29953]: Failed password for root from 194.204.37.93 port 47807 ssh2
Dec 6 01:30:10 delilah sshd[15349]: Received disconnect from 194.204.37.93: 11: Bye Bye
Dec 6 01:30:11 delilah sshd[2962]: Failed password for root from 194.204.37.93 port 47845 ssh2
Dec 6 01:30:11 delilah sshd[27557]: Received disconnect from 194.204.37.93: 11: Bye Bye
and

:~$ grep 201.57.187.114 /var/log/authlog
Dec 5 23:55:30 delilah sshd[24338]: Did not receive identification string from 201.57.187.114
Dec 5 23:55:35 delilah sshd[23570]: Did not receive identification string from 201.57.187.114
Dec 5 23:59:58 delilah sshd[10216]: Invalid user raimundo from 201.57.187.114
Dec 5 23:59:58 delilah sshd[10216]: Failed password for invalid user raimundo from 201.57.187.114 port 35776 ssh2
Dec 5 23:59:58 delilah sshd[18515]: Received disconnect from 201.57.187.114: 11: Bye Bye
Dec 6 00:00:01 delilah sshd[17353]: Invalid user joan from 201.57.187.114
Dec 6 00:00:01 delilah sshd[17353]: Failed password for invalid user joan from 201.57.187.114 port 37570 ssh2
Dec 6 00:00:02 delilah sshd[30314]: Received disconnect from 201.57.187.114: 11: Bye Bye

which again shows that these were the old-fashioned, rapid-fire kind of bots. Looking a bit closer even reveals that they kept trying after they were put in the doghouse:

:~$ sudo pfctl -t abusive_hosts -vT show
194.204.37.93
Cleared: Sat Dec 6 01:30:11 2008
In/Block: [ Packets: 18985 Bytes: 835336 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]
201.57.187.114
Cleared: Sat Dec 6 00:00:02 2008
In/Block: [ Packets: 800 Bytes: 48268 ]
In/Pass: [ Packets: 0 Bytes: 0 ]
Out/Block: [ Packets: 0 Bytes: 0 ]
Out/Pass: [ Packets: 0 Bytes: 0 ]

And of course there were no traces of those IP addresses or corresponding host names in the authentication logs in the machines where I have collected the data about slow bots. My best guess at why is that the gateway's IP address is at the low end of the routable range for that site. Note to bruteforcers: try working the Internet in reverse next time. (Not that it would help much here, but that's another story.)

The reason why I don't see much activity for other services is simply that those machines do not run all that many services, and only the services they actually run for the world's benefit are in fact available to the outside.

My log data shows a definite pattern, and the alphabetic progression points to a degree of coordination. The slow bots are, I theorize, operated by a botnet herder who has a large pool of compromised hosts available and who also believes that given enough time, sooner or later you will find a correct combination of user names and passwords for a given host. Statisticians tell me that assumption is in fact valid, at least to some extent.

By setting up the necessarily large number of attempts to come from a sizable number of hosts in either round-robin or pseudo-random order and intervals for each individual host's attempts in the several minutes to hours range, there is a very real possibility that the slow but determined campaign for control of any single system will drown in the noise.

It is useful to keep in mind that malware moved out of the hands of pranksters and vandals years ago. Mass destruction of systems or data might still make the occasional headline, but staying out of the limelight is likely to be a lot more profitable. Modern malware masters want their creations and charges to stay undetected. What we may be seeing right at this moment is that they have realized the herd may only be sustainable if they grow it slowly.

---

If you are interested in researching the phenomena I've blogged about, you're welcome to contact me directly for more information or raw data.


Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.