Over the last few columns, we have followed the progress of what appears to be a botnet cloud's attempt at gaining access to a couple of FreeBSD machines I have in my care. One of my predictions about the distributed, slow ssh bruteforce attempts we started seeing in November of 2008 was that at the rate they were going at the time, it would be well into the new year before we would see the end of their alphabetic progression. As it turns out, they stopped just before year end, before even reaching the 'T's. The last attempt recorded was this:
Dec 30 11:09:03 filehut sshd[54981]: error: PAM: authentication error for illegal user sophia from static-98-119-110-139.lsanca.dsl-w.verizon.net
The full collection of raw data is available here, with a .csv summarising number of attempts, user names and hosts per day here.
With the incident apparently over, we can sit back and study the data and see what patterns emerge.
The anatomy of the attack
There are a number of ways to slice and dice the data. One useful way to view the collection is to do day to day statistics, such as the ones in this .csv file, numbers extracted by some simple greping and awkery. Based on the day to day data, I made this graph to illustrate the progression.
Then for your data overload cravings, I turned the same data to a log scale for enhancement and added number of attempts -
hopefully adding some insight into just what happened when and maybe supporting some guessing about what they were indeed trying to achieve.
It is possible that we missed the actual start of those coordinated attempts, but the data we do have show a few interesting points.
The earliest preserved data from November 19th shows the most attempts per user name (average 13.29), with 7 unique user names tried and a relatively low number of hosts (76).
On November 20th, the cloud turned its attention to one user, root, trying only that user name a total of 1697 times from 566 different hosts. It would be well into November 21st before the cloud moved on to admin (128 attempts, 107 different hosts) and an apparently coordinated alphabetic progression.
The absolute number of attempts and hosts involved per day fell quickly, with average number of attempts per user name stabilizing at a fairly low number after a few days. The exception is the peak on December 27th, which could perhaps be explained by owners of compromised computers returning from holiday celebrations and turning their computers back on. The sharp decline in all numbers over the next few days before the attempts stop seems consistent with what we assumed: That the botnet masters were allocating resources according to likelihood of success.
So why is this incident important, or even interesting? After all, attempts to gain access to services by brute force or dictionary based attacks are nothing new. I was rather intrigued to see clear evidence that miscreants were trying to find the way under the radar of intrusion detectors by distributing the intrusion task over a large number of hosts. If their success rate at my sites is anything to go by, this may be just a weird anomaly and an idea that did not lead anywhere. I haven't heard from anybody who was actually compromised by this particular set of clouds, but then again anybody who got bitten would likely be rather shy about telling the world at large or even fairly obscure researchers about the fact.
Always looking for patterns I even went to the trouble of extracting some data from the logs about the individual hosts that participated in the attack. After some basic shell gymnastics I ended up with a .csv of hostnames, number of attempts from the host as well as the date of first and last contact (available here). Next I tried (and failed - gnuplot gurus, here's your chance) to graph the data usefully in OpenOffice, but ended up with a sorted version (sorted by attempts, start and end date) that at least shows us that a surprising number of hosts actually hung on for most of the time the coordinated attepts went on.
The lessons learned: security the old-fashioned way
The general lesson of this incident is rather predictably that miscreants will occasionally try new and original ways to try to crack their way into your system. The slow method was a refreshing variation, and for all we know they may have succeeded in places where the people in charge remain blissfully unaware. Trying to catalogue and detect all kinds of variations on the theme of "attempts at unauthorized access" is the kind of activity that has kept "antivirus" people in beer money for quite a while, and if there is a lesson to be learned here, it is that trying to enumerate badness (Yes, do look it up using your favorite search) is a losing game. Make sure whatever system you run is sanely constructed, any bugs that do turn up are fixable within a reasonable time frame, and so on. I suppose I will come back later with a rant about how much damage the "black boxes" school of thinking about software has done, especially after it got elevated to practically religious dogma by certain major players. And yes, you can usefully look that up as well.
For those of you who are interested in the data, here are the now complete extracts for your perusal:
The full set of log data
The per day .csv file - and the same in an .ods sheet with some graphing attempts
Per host data in the "Host,Attempts,StartDate,EndDate" format and sorted by attempts, start and end
For those of you interested in learning about OpenBSD and related delights like PF, FreeCode is set to start offering courses featuring among others yours truly as well as the usual support and consulting offerings. Contact the good front end people at FreeCode for further details.
International readers are at liberty to ignore the following, but Norwegian online IT magazine digi.no are apparently in the process of setting up a sort of census of Norwegian bloggers. The following is there to make this blog show up in their listing. You can read their article about the initiative (Norwegian only, unfortunately).
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.