Regular readers of this column know that I pay attention to my logs, as any sysadmin worth his or her salt would. We read logs, or at least skim summaries because in between the endless sequence of messages that were successfuly delivered, users who logged in without a hitch and web pages served, from time to time unexpected things turn up. Every now and then, the unexpected log entries lead to discoveries that are useful, entertaining or even a bit bizarre.
In no particular order, my favorite log discoveries over the years have been
- the ssh bruteforcer password guessers, a discovery that in turn lead to some smart people developing smart countermeasures that will shut each one of them down, automatically.
- the faked sender addresses on spam, leading to bounces to non-existent users. Once again, thanks to the OpenBSD developers, I and others have been feeding those obviously fake addresses back to spamdb, slowing spammers down and generating publishable blacklists.
- and never forgetting the relatively recent slow, distributed ssh bruteforcers, also known as "The Hail Mary Cloud", who in all likelihood had expected to avoid detection by spreading their access attempts over a long time and a large amount of hosts in coordination that was only detectable by reading the logs in detail.
After the Hail Mary Cloud cycle of attempted attacks, I thought I'd seen it all.
But of course I was wrong. Now we have more likely than not found evidence of another phenomenon which by perverse coincidence or fate appears to combine features of all these earlier activities.
Early in July 2014, my log summaries started turning up failed logins to
the pop3 service (yes, I still run one, for the same bad reasons more
than a third of my readers probably do: inertia).
At our site (which by now serves mainly as my personal lab in addition to serving a few old friends and family), pop3 activity besides successful logins by the handful of users who still use the service had been quite low for quite a while. And if we for now ignore the ssh bruteforcers who somehow never seem to tire of trying to log in as root, my log summaries had been quite boring for quite a while. But then the failed pop3 logins starting this July were for the unlikely targets admin, sales and info, user names that have never actually existed on that system, so I just kept ignoring the irregular and infrequent attempts.
My log summaries seemed to indicate that whoever is in charge of superonline.net (see the log summary) should tighten up a few things, but then it's really not my problem.
But then on July 18th, this log entry presented itself:
Jul 18 17:01:14 skapet spop3d[28606]: authentication failed: no such user: malseeinvmk - host-92-45-149-176.reverse.superonline.net (92.45.149.176)
At our site (which by now serves mainly as my personal lab in addition to serving a few old friends and family), pop3 activity besides successful logins by the handful of users who still use the service had been quite low for quite a while. And if we for now ignore the ssh bruteforcers who somehow never seem to tire of trying to log in as root, my log summaries had been quite boring for quite a while. But then the failed pop3 logins starting this July were for the unlikely targets admin, sales and info, user names that have never actually existed on that system, so I just kept ignoring the irregular and infrequent attempts.
My log summaries seemed to indicate that whoever is in charge of superonline.net (see the log summary) should tighten up a few things, but then it's really not my problem.
But then on July 18th, this log entry presented itself:
That user name malseeinvmk is weird enough that I remembered adding it as part of one of the spamtrap addresses at one point. I had to check:
$ grep malseeinvmk sortlist malseeinvmk@bsdly.net
which means yes, I remembered correctly. That user name is part of one of the more than twenty-eight thousand addresses on my traplist page, added at some point between 2006 and now to my spamdb database as a spamtrap, and as such known to be non-deliverable. So I started paying a bit more attention, and sure enough, over the next few days the logs (preserved here) were still turning up more entries from the traplist page. We would see local parts (usernames) only, of course, but
grep would find them for us.Now, trying to log on to a system with user names that are known not to exist sounds more than a little counterproductive at the best of times. But from my perspective, the activity was in fact quite harmless and could be potentially fun to watch, so I adjusted my log rotation configuration to preserve the relevant logs for a little longer than the default seven days.
Coming back to peek at the results every now and then, I noticed fairly soon that the attempts in almost all cases were concentrated in the first two minutes past every full hour. There are a couple of hour long periods with attempts spread more or less evenly with a few minutes in between, but the general pattern is a anything from one to maybe ten attempts in the :00:00-:02:00 interval.
The pattern was not exactly the Hail Mary Cloud one, but similar enough in that the long silent intervals could very well be an attempt at hiding in between other log noise.
But that returns us to the question, Why would anybody treat a list of known to be non-existent user names as if they actually offered a small chance of access?
Let's put each of those hypotheses to the test.
First, when you try to log in to the service, do you get any indication whether the user you attempt to log in as exists?
Here's what it looks like when a valid user logs in:
First, when you try to log in to the service, do you get any indication whether the user you attempt to log in as exists?
Here's what it looks like when a valid user logs in:
$ telnet skapet.bsdly.net pop3 Trying 213.187.179.198... Connected to skapet.bsdly.net. Escape character is '^]'. +OK Solid POP3 server ready USER peter +OK username accepted PASS n0neof.y3RB1Z +OK authentication successful
At this point, I would have been able to list or retrieve messages or even delete them. But since my regular mail client does that better than I do by hand, I close the session instead:
quit +OK session ended Connection closed by foreign host.
And in case you were wondering, that string is not my current password, if it ever was.
Next, let us compare with what happens when you try logging in as a user that does not exist:
$ telnet skapet.bsdly.net pop3 Trying 213.187.179.198... Connected to skapet.bsdly.net. Escape character is '^]'. +OK Solid POP3 server ready USER jallaballa +OK username accepted PASS Gakkazoof -ERR authentication failed Connection closed by foreign host.
Here, too, the user name is tentatively accepted, but the login fails anyway without disclosing whether the password was the only thing that was invalid. If weeding out bad entries from the list of more than twenty-eight thousand was the objective, they're not getting any closer using this method.
Unless somebody actually bothered to compromise several hundred machines in order to pull off a joke that would be funny to a very limited set of people, the inescapable conclusion is that we are faced with would-be password guessers who
- go about their business slowly and in short bursts, hoping to avoid detection
- base their activity on a list that was put together with the explicit purpose of providing no valid information
If throwing semi-random but 'likely' user names and passwords at random IP addresses in slow motion had monumental odds against succeeding, I'm somewhat at a loss to describe the utter absurdity of this phenomenon. With trying to sneak under the radar to guess the passwords of users that have never existed, I think we're at the point where the Internet's bottom-feeding password gropers have finally hit peak stupidity.
More likely than not, this is the result of robots feeding robots with little or no human intervention, also known as excessive automation. Automation in IT is generally a good thing, but but I have a feeling somebody is about to discover the limits of this particular automation's usefulness.
I hate to break it to you, kids, but your imaginary friends, listed here, never actually existed, and they're not coming back.
And if this is actually a joke that has somebody, somewhere rolling on the floor laughing, now is a good time to 'fess up. There is still the matter of a few hundred compromised hosts to answer for, which may be a good idea to clear up as soon as possible.
As usual, I'll be tracking the activities of the miscreants and will refresh these resources at semi-random intervals as long as the activity persists:
- List of compromised hosts participating, sorted by number of attempts
- List of compromised host participating, sorted by number of attempts and with date and time (CEST and when the time comes, CET) of 'first seen' and 'last seen'
- List of attempted user names, sorted by number of attempts
- Log of access attempts in chronological order
At the rate they're going at the moment, we could be seeing them hang on for quite a while. And keep in mind that the list generally expands by a few new finds every week.
Update 2014-08-19: The attempts appear to have stopped. After some 3798 access attempts by 849 hosts trying 2093 user IDs, the last attempt so far was
I take this as an indication that these access attempts are at least to some extent monitored, and with those numbers of attempts with a total of 0 successes, any reasonably constructed algorithm would have found reason to divert resources elsewhere. We can of course hope that some of the participating hosts have been cleaned up (although nobody actually wrote me back about doing that), and of course you can't quite rule out the possibility that whoever runs the operations reads slashdot.
But then again, the fact that the pop3 password gropers have moved on from my site should not lead you to believe that your site is no longer a target.
Update 2014-08-21: I spoke too soon about the pop3 password gropers giving up and moving on. In fact, only a few hours after the last update (in the early morning CEST hours of August 20th), two more attempts occured:
Aug 20 05:07:41 skapet spop3d[2943]: authentication failed: no such user: info - 94.102.63.160
Aug 20 05:31:58 skapet spop3d[15882]: authentication failed: no such user: info - host-92-45-150-182.reverse.superonline.net (92.45.150.182)
before another long silence set in and I decided to wait a bit before making any new announcements.
But at just after ten in the morning CEST on August 21st, the slow and distributed probing with usernames lifted from my spamtraps page resumed. At the time of this writing, the total number of attempts has reached 3822, with 856 participating hosts and attempts on 2103 distinct user names. I'll publish refreshed files at quasi-random intervals, likely once per day if not more frequently.
Update 2014-09-01: Statistics improved, estimates revised.
A couple of days ago, I tweeted:
Based on 54 d of observation, the pop3 password gropers will run out of imaginary friends to try in approx 267 days http://t.co/c8BNHfFwHc
— Peter N. M. Hansteen (@pitrh) August 30, 2014
which, based on some quick calculations at the time, seemed a reasonable number. Now we've reached the end of the first full month of this extended incident, and it's time to present some preliminary statistics, lightly polished.For the estimated end date, which will be loosely based on the rate of attempts at new user names, the calculation is as follows: On the spamtraps page, roughly half the addresses belong to domains whose primary MX is not skapet.bsdly.net (we're secondary or further out). Stripping those addresses from our total, we're left with 14809 possible usernames. Of course, we have no idea when the list they're working from was sucked in, but this is our latest data. Next, by the time I started writing this update, a total of 2523 usernames had been attempted. We're now well into the 57th day, so dividing 2523 usernames by 57, we get a little more than 44 usernames per day on average. Dividing our average per day with total number of usernames, we get approximately 334 days to try them all, which means that the first attempt on the last username in the list is likely 277 (344 - 57) days in the future. This means that at the current rate, it will be early June 2015 before they run out of usernames. How long they stick around will likely depend on their strategy for selecting passwords to match.
If you want to see some more detail on the attackers' progress, here are some data I generated while actively procrastinating and putting off other tasks with a defined and fixed deadline:
(larger .png here). I also tried feeding LibreOffice this .csv file with per hour data, but getting the data graphed in any sensible manner seems to require more effort than I'm inclined to put into that particular task.
Hosts participating with first and last seen dates: Possibly more useful is this .csv file, which has participating hosts in the same order as the List of compromised hosts participating but expanded to comprise the fields Attempts,Host,User Names,First Seen,Last Seen, where the last two are dates in formats that your spreadsheet should be able to parse and use as a sort key with only minor efforts. The host with the most attempts as of the last dump was only first seen on August 29th, while several of the top 10 have been with us since some time in July. And of course, at the other end of the scale, quite a few have made only one attempt to date. I'll be updating the data at semi-random intervals, likely at least once a day. Raw data and generated files can be found here, along with scripts used and even a few temporary files. The data can be used for any purpose as long as proper attribution is included. See the Hail Mary Cloud Data page for details, and the Hail Mary Cloud overview article for background information.
Update 2014-09-04: A separate effort detected Perceptive readers will have noticed that the data now includes traces of activity from hosts in the dynamic.adsl.gvt.net.br domain starting on September 2, 2014, which deviates far enough from the general distributed pattern that it's likely it represents a separate, if not overly successful, effort.
These four (so far) hosts have tried relatively long sequences of attempts at single user names at a time, targeting what I assume are 'probable' user names for that part of the world, 22 user IDs so far. None of those user IDs expand to valid addresses in our domains, and just because I can, I've now added these user IDs to the spamtraps page as presumed members of our nxdomain.no domain. If you feel those hosts should be removed from the data for purposes of your own analyses, it's fairly easy to strip them away. In fact, a simple
$ grep -v dynamic.adsl.gvt.net.br filename
where filename is either the raw authentication log or the extracted files that include hostnames will remove them from sight.
Update 2014-12-10: Data from this incident as well as some others have been included in my Passwords14 presentation, PDF version available too, here.
Update 2015-08-24: Since this article was originally posted, there have been several shorter episodes of pop3 groping activity, but mostly very low volume and short lived enough to be not really worth noting. However, at the very end of July 2015, the intensity increased sligtly, and I've preserved the log entries starting July 27 in a convenient place - raw log data, overview .csv file, user names by frequency and finally hosts by frequency.
A very preliminary analysis of the data says we have a total of 268 hosts attempting access at least once, and apparent coordination (several hosts processing the same user name for a short while before moving on to the next) in several long sequences. We also see long sequences of one single host attempting one user name or an assortment of user names, and while we have a number of 'likely' user names (including my first name) as attempted user IDs, we also see an assortment of user IDs apparently lifted from the spamtraps page.
If you're seeing something like this in your logs, I'm interested in hearing from you. And if you're responsible for one or more of the systems that appear in those logs, you have a bit of cleaning up to do.
Book of PF News: All ten chapters and the two appendixes have been through layout and proofing, front and back matter is still a work in progress, expected arrival for physical copies is still not set. Those most eager to read the thing could try early access or preorder (a fourth edition is now available).
It's almost certain that physical copies will not be available at EuroBSDCon in Sofia, but If you make it to one of my sessions there, a special discount code worth 40% off on both the ebook and print version will be disclosed to attendees. (And I'm told it's active already, so if you guess what it is, you can use it.)
(And of course, as this blog post shows, the book came out and was well received, at least in some quarters.)
Update 2018-05-10: It appears that my spamtraps have entered a canon of sorts. On our freshly configured imapd, this happened. A few dozen login attemps to spamtrap IDs, earning of course only a place in the permanent blocks along with the popflooders as described in this article.
