Showing posts with label bruteforcers. Show all posts
Showing posts with label bruteforcers. Show all posts

Sunday, June 8, 2025

Should I Stop Caring and Let IP Address Reputation Sort Them Out?

© 2025 Peter N. M. Hansteen

How long does data on misbehaving hosts on the Internet stay relevant in an IP Address Reputation context?

log of pop3 gropers in action, failing of course

My main presence on the Internet also serves, for all practical purposes, as a honeypot, and is seen as mainly that by what appears to be a small but significant number of people who run IP Reputation services (yes, link to SpamHaus because apparently Wikipedia does not have a page dedicated to the topic yet). The article Badness, Enumerated by Robots (also here) describes the data collected by the honeypot, with links to the data as well as to other relevant resources.

But there is a thing about putting something on the Internet for free so anybody at all can download it: People will not necessarily read the instructions.


Note: This piece is also available without trackers but classic formatting only here.


The honeypot service has been collecting and sharing data for years, as will be clear from the articles linked in a previous paragraph. All the lists have their consumers and get downloaded regularly.

Although there are signs that the list data is further processed into various services, including those that provide IP reputation rankings, the only people who seem to care enough to actually contact me about specific entries in the data are people who own one or more IP addresses that have for one reason or other been included in the lists.

Recently I was contacted by somebody who claimed that some of their traffic seemed to be filtered due to IP reputation, and they had tracked down the problem to the POP3 Gropers list we publish. The big one, that is.

I was a bit surprised by this, give that I had provided a fairly clear description of the lists and their expiry times in the published material. That material also clearly stated, in my view at least, that the big POP3 gropers list does not have an expiry set, and should for that reason be used with caution, if at all.

But apparently one or more operators of IP address reputation services did not actually read that far.

I am still pondering what is the correct action here, so I created a fediverse poll,

Post by @pitrh@mastodon.social
View on Mastodon

(For those without working fediverse links, the question posed is "Should I", with the options: "Stop publishing the BIG pop3gropers list", "Stop caring and let IP reputation sort them out", or "No opinion, show results", prepended with a shorter version of the description in the first part of this article) which may of course have run its course when you read this.

If the poll has run its course and you don't get to vote, you are of course welcome to contact me or comment where you find reference to this article.

I am genuinely interested in hearing informed opinions on how to deal with data collected in the context of contributing to IP address reputation context.

In addition to the poll, I added a note to the Badness, Enumerated by Robots (also here) article as well as to the exported text files:


NOTE: The BIG pop3 gropers list is for history only, use the sixweeks one for IP reputation evaluations instead
As stated earlier, the big list of pop3 gropers was intended as a collection of all hosts that had ever tried and failed in guessing passwords (see Password Gropers Take the Spamtrap Bait for background). This means that the list only exists as a historical collection of sorts, and if you are intrerested in seeing when a particular host entered the data set, you can look it up in the pop3 gropers archive directory.

For any reasonably current IP Reputation purposes, you will be better served with the pop3 gropers during the last six weeks list, which conveniently is also archived for those who wish to study developments.

For what it's worth, there is an archive of the greytrapped hosts list available too, along with a separate archive of the SSH bruteforcers list, all kept around for as long as I find it at least a little useful to do so.

For reference on just what triggered the inclusion, see the log extracts preserved in the pop3logs directory, which has entries going back to February of 2024.


I hope at least some of the relevant people -- people running IP address reputation services -- take the time to read that little piece of text and have a think.

For my own part, I will be pondering the ethics and practicalities of blocklists much along the same lines as I wrote about in the 2013 piece Maintaining A Publicly Available Blacklist - Mechanisms And Principles (also here).

If you found this piece to be useful, informative, annoying or would for some other reason like to contact me or comment, please do.

You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).


Upcoming events:

Ottawa, Canada: BSDCan 2025 has tutorials June 11-12, 2025 and talks June 13-14. A new version of Network Management with the OpenBSD Packet Filter Toolset will go ahead there.

A little later on in 2025, the EuroBSDcon 2025 conference is still accepting submissions for papers and tutorials, so if you have an interesting BSD-related topic you want the world to know about, your submissions will be welcome at the EuroBSDcon submissions system, where the deadline is 2025-06-21, or June 21st, 2025 (full disclosure: I'm on the program committee). This year's conference is set in beautiful Zagreb, Croatia in late September.

At EuroBSDcon 2025, there will be a Network Management with the OpenBSD Packet Filter Toolset session, a full day tutorial starting at 2025-09-25 10:30 CET. You can register for the conference and tutorial by following the links from the conference Registration and Prices page.

Separately, pre-orders of The Book of PF, 4th edition are now open. For a little background, see the blog post Yes, The Book of PF, 4th Edition Is Coming Soon. We are hoping to have physical copies of the book available in time for the conference, and hopefully you will be able to find it in good book stores by then.


Monday, August 13, 2018

Badness, Enumerated by Robots

A condensed summary of the blocklist data generated from traffic hitting bsdly.net and cooperating sites.

After my runbsd.info entry (previously bsdjobs.com) was posted, there has been an uptick in interest about the security related data generated at the bsdly.net site. I have written quite extensively about these issues earlier so I'll keep this piece short. If you want to go deeper, the field note-like articles I reference and links therein will offer some further insights.

There are three separate sets of downloadable data, all automatically generated and with only very occasional manual intervention.


Known spam sources during the last 24 hours

This is the list directly referenced in the runbsd.info piece.

This is a greytrapping based list, where the conditions for inclusion are simple: Attempts at delivery to known-bad addresses (download link here) in domains we handle mail for have happened within the last 24 hours.

In addition there will occasionally be some addresses added by cron jobs I run that pick the IP addresses of hosts that sent mail that made it through greylisting performed by our spamd(8) but did not pass the subsequent spamassassin or clamav treatment. The bsdly.net system is part of the bgp-spamd cooperation.

The traplist has a home page and at one point was furnished with a set of guidelines.

A partial history (the log starts 2017-05-20) of when spamtraps were added and from which sources can be found in this log directory (or at this alternate location). Read on for a bit of information on the alternate sources.

Note: The list is generated at ten past every full hour by a script that uses essentially the one-liner

    spamdb | grep TRAPPED | awk -F\| '{print $2}' >bsdly.net.traplist

to generate the body of the list.


For those interested in the entire history of the greytrapping-based blocklist, the 2025 article Eighteen Years of Greytrapping - Is the Weirdness Finally Paying Off? (2025), also available tracked, prettified will be of interest.

Misc other bots: SSH Password bruteforcing, malicious web activity, POP3 Password Bruteforcing.

The bruteforcers list is really a combination of several things, delivered as one file but with minimal scripting ability you should be able to dig out the distinct elements, described in this piece.

The (usually) largest chunk is a list of hosts that hit the rate limit for SSH connections described in the article or that was caught trying to log on as a non-existent user or other undesirable activity aimed at my sshd(8) service. Some as yet unpublished scriptery helps me feed the miscreants that the automatic processes do not catch into the table after a manual quality check. For a more thorough treatment of ssh bruteforcers, see the The Hail Mary Cloud and the Lessons Learned overview article which links to several other articles in the sequence.

The second part is a list of IP addresses that tried to access our web service in undesirable ways, including trying for specific URLs or files that will never be found at any world-facing part of our site.

After years of advocating short lifetimes (typically 24 hours) for blocklist entries only to see my logs fill up with attempts made at slightly slower speeds, I set the lifetime for entries in this data set to 28 days (since expanded to 2419200 seconds, or if you will, six weeks). The background including some war stories of monitoring SSH password groping can be found in this piece, while the more recent piece here covers some of the weeding out bad web activity.

The POP3 gropers list comes in two variations. Again lists of IP addresses caught trying to access a service, most of those accesses are to non-existent user names with an almost perfect overlap with the spamtraps list, local-part only (the part before the @ sign).

The big list is a complete corpus of IP addresses that have tried these kinds of accesses since I started recording and trapping them (see this piece for some early experience and this one for the start of the big collection).

There is also a smaller set, produced from the longterm table described in this piece. For much the same reason I did not stick to 24-hour expiry for the SSH list, this one has six-week expiry. With some minimal scriptery I run by hand one or two times per day, any invalid POP3 accesses to valid accounts get their IP adresses added to the longterm table and the exported list.


NOTE: The BIG pop3 gropers list is for history only, use the sixweeks one for IP reputation evaluations instead
As stated earlier, the big list of pop3 gropers was intended as a collection of all hosts that had ever tried and failed in guessing passwords (see Password Gropers Take the Spamtrap Bait for background). This means that the list only exists as a historical collection of sorts, and if you are intrerested in seeing when a particular host entered the data set, you can look it up in the pop3 gropers archive directory.

For any reasonably current IP Reputation purposes, you will be better served with the pop3 gropers during the last six weeks list, which conveniently is also archived for those who wish to study developments.

For what it's worth, there is an archive of the greytrapped hosts list available too, along with a separate archive of the SSH bruteforcers list, all kept around for as long as I find it at least a little useful to do so.

For reference on just what triggered the inclusion, see the log extracts preserved in the pop3logs directory, which has entries going back to February of 2024.

The most recent exports of all lists generated here can be found in this directory. Before making any inguiries on removal from any of the lists, check all files in this directory for occurences or not of the IP address in question.



Note: The lists generated by table exports are generated by variations of pfctl's show table subcommand. At ruleset reload such as reboots after a sysupgrade, the tables are re-initialized from these same exported files.

If you're wondering about the title, the term "enumerating badness" stems from Marcus Ranum's classic piece The Six Dumbest Ideas in Computer Security. Please do read that one.

Here are a few other references other than those referenced in the paragraphs above that you might find useful:

The Book of PF, 4th edition
Hey, spammer! Here's a list for you! which contains the announcement of the bsdly.net traplist.
Effective Spam and Malware Countermeasures, a more complete treatment of those keywords

If you're interested in further information on any of this, the most useful contact information is in the comment blocks in the exported lists.

Update 2020-07-29: I added a direct link to the complete list of spamtraps, since the web page seemed a bit crowded to at least one visitor. Direct link again here for your convenience.


Update 2021-01-15: Note that at some point after the article was written I cranked up expiry for the bruteforce tables to six weeks (sorry, I forgot to note the exact date).

Update 2021-03-11: In light of recent Microsoft Exchange exploits it might interest some that any request to bsdly.net for "GET /owa/" lands the source in the webtrash table, exported as part of the bruteforcers list.

Update 2021-08-03: Added notes about how the lists are generated and table maintenance.

Update 2025-03-23: Addresses matching a jumble of regexps for "silly web things" are now also exported separately as webtrash.

Update 2025-07-19: If you found this piece to be useful, informative, annoying or would for some other reason like to contact me or comment, please do.

You might also be interested in reading selected pieces via That Grumpy BSD Guy: A Short Reading List (also here).


Saturday, February 16, 2013

There's No Protection In High Ports Anymore. If Indeed There Ever Was.

Running your service on a non-standard port buys you an equivalent of two more characters in your password. Will that save you any trouble at all?

Looking back at the past week, the most interesting communication received here was not about meteorites hitting Russia (although some of the dashcam videos were really exciting), but rather about an old and very simple prediction of mine that appears to have come true.

It came via an email message from a friend and somewhat infrequent correspondent, who wrote on February 15 2013,
"[...] Here's something mildly curious, though. Between now and Dec 26, the current start of my authlog, the below tried to get in via port 2222 on my openbsd box. I'm surprised both that there are not many yet at the same time that there are any at all."
And the log excerpt he sent me looked like this:

Jan 11 16:02:01 net5501 sshd[3170]: Invalid user admin from 66.135.32.170
Jan 11 17:58:43 net5501 sshd[2611]: Invalid user aadriano from 66.135.32.170
Feb 10 17:16:33 net5501 sshd[23679]: Invalid user alex from 66.133.188.86
Feb 10 17:18:32 net5501 sshd[20330]: Invalid user cactiuser from 66.133.188.86
Feb 10 17:20:31 net5501 sshd[19432]: Invalid user www-data from 66.133.188.86
Feb 10 17:24:28 net5501 sshd[29455]: Invalid user user from 66.133.188.86
Feb 10 17:26:27 net5501 sshd[82]: Invalid user tomcat from 66.133.188.86
Feb 10 17:28:26 net5501 sshd[31011]: Invalid user test from 66.133.188.86
Feb 10 17:30:25 net5501 sshd[20057]: Invalid user temp from 66.133.188.86
Feb 10 17:32:23 net5501 sshd[19848]: Invalid user service from 66.133.188.86
Feb 10 17:34:22 net5501 sshd[31209]: Invalid user samba from 66.133.188.86
Feb 10 17:38:20 net5501 sshd[12045]: Invalid user postgres from 66.133.188.86
Feb 10 17:40:20 net5501 sshd[1085]: Invalid user oracle from 66.133.188.86
Feb 10 17:42:19 net5501 sshd[12896]: Invalid user nagios from 66.133.188.86
Feb 10 17:44:19 net5501 sshd[10681]: Invalid user mysql from 66.133.188.86
Feb 10 17:46:19 net5501 sshd[27897]: Invalid user guest from 66.133.188.86
Feb 10 17:48:19 net5501 sshd[13400]: Invalid user tom from 66.133.188.86
Feb 10 17:50:18 net5501 sshd[5132]: Invalid user ftp from 66.133.188.86
Feb 10 17:54:15 net5501 sshd[28789]: Invalid user jboss from 66.133.188.86
Feb 10 17:56:14 net5501 sshd[20435]: Invalid user backup from 66.133.188.86


So just a sampling of likely service user IDs on a random system, right?

The reason why this is significant is that my friend, like a few others, tired of the apparently never ending password guessing attempts that were hitting the sshd (also known as the Secure Shell Daemon) listening on the standard port 22, and moved the service to listen on a different port instead.

The immediate effect in almost all cases was a much quieter authentication log, because the password guessers weren't yet trying to bother you on non-standard ports.

Those of us who could for one reason or other not be bothered to change the config to have our sshds run on other ports and using OpenBSD on anything Internet-facing soon found ourselves using the state-tracking features of OpenBSD (introduced in the May 2005 3.7 release). I wrote about it in the PF tutorial, specifically the Turning away the brutes part, and of course a slightly embellished version made it into The Book of PF. Please note the sentence somewhere down the page,
"You might consider moving the service to some other port, but then again, the ones flooding you on port 22 would probably be able to scan their way to port 22222 for a repeat performance."
The news is that this is now happening. The only surprise here to my mind is that signs of that exact activity has not been reported earlier. I put that down to mainly two factors. One is that the number of sites running their ssh service only on non-standard ports is probably quite small, another is that people who run services in general pay way too little attention to their logs.

To those of us who run our packet filtering with a default deny policy and drop anything that's not relevant to services we actually run, those attempts to access ports where we do not run anything are simply part of the background noise, so we drop and ignore unless we're looking for patterns in the connection attempts we drop. So we wouldn't necessarily notice the attempts either.

And for all I know (I really can't resist a dig at the high port refugees) it could be that for the most part, those who have moved their main login service to a non-standard port pay less attention to their logs than the rest of us, now that they have obscured the entrance to the kingdom to their own satisfaction if not so many else's.

As I've argued elsewhere, obscuring your login service via non-standard ports or even a requirement to try several ports in sequence really only buys you security equivalent to lengthening your password by two characters per port. After all, TCP and UDP port numbers are limited to 65536 distinct values, within the scope of a 16-bit value. Moving your service to a non-standard port means that the would-be intruder has to guess a two-byte value. Introducing a sequence of ports buys you the added obscurity of two bytes per port.

I have reason to believe that my correspondent here has done away with passwords altogether for logins to his systems, and therein lies the real solution: SSH offers out of the box the opportunity to allow only key based authentication, and with a little extra effort, you can implement more sophisticated authentication schemes. One useful reference for ssh in addition to the manual pages (starting with this one, all of the expected OpenBSD quality) is Michael Lucas' recent book SSH Mastery.

Stay safe, please move from passwords to passphrases, and apply those to your keys only, never to be transmitted in the clear over the net where you never know who could be listening in.

For anyone else seeing this kind of activity, as always I'll be very happy to hear from you, via comments or email.

Good luck and good night.

Thanks to Lars Noodén for valuable input for this column.



Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

Wednesday, April 11, 2012

Why Not Use Port Knocking?

The robots currently at work knocking around for your guessable password could easily be repurposed to guess your Unicode password currently known as your port knocking sequence, and quite likely have been already. Plus, we already have authpf(8) for network-level restrictions on access.

Whenever you write about security in general and SSH security in particular (and for good measure also get slashdotted for your efforts), the comments inevitably turn up a variety of useful and less useful suggestions.

Note: This piece is also available without trackers but classic formatting only here.

The April 6th, 2012 article about the possible preparations for a new round of slow bruteforcers was no different. Some of these recurring ideas are worthy of some more attention, and a more or less complete list is as follows -

  • Set up iptables with the recent module - a clear indication that the commenter did not, in fact read the article. That module implements a subset of the state tracking techniques from my PF rule set referenced in the article, but unlike the PF rules it is specific to the SSH protocol rather than a general technique. And inevitably, the actual configuration the poster presents would in fact not trigger on the kind of traffic we see from the Hail Mary Cloud with its signature slow brute force attempts. Misunderstandings of this kind keep cropping up, for no good reason.

  • Disable root logins over the network. Yes, this is obviously useful, and if I remember correctly, PermitRootLogin no is the default in recent OpenSSH versions. Do remember to check what the setting is on your kit, obviously.
     
  • Use packet filtering or other means to restrict where users can log in from. Once again, a very useful suggestion, but some sites may require shell access from semi-random locations, so practical issues (aka real life) can interfere with this one.

  • Disable password logins entirely, and allow only key-based logins. Another a very useful suggestion, and one that is useful for several reasons. You do get the downside of managing your keys securely, but key-only logins are generally recommended practice and should be encouraged.

  • Set up your sshd to listen on a non-standard port. Several sites report less log file noise after moving their sshd to a non-standard port. Easy to implement and possibly useful, but if a prospective attacker takes the time to do a real port scan for active services, they will find your sshd running on port 22222 fairly easily. I'd say moving your sshd to listen on a non-standard port postpones rather than solves the problem.

  • Use denyhosts to automatically add hosts with too many failed logins to a blacklist. Yes, an excellent idea in principle, but it comes with one caveat: Creating a blacklist automatically leaves you in charge of maintaining its content. Blacklist maintenance -- as in keeping your lists up to date and useful, is a complex enough issue that I'm fairly confident it is worth a separate column, and I most likely will return to the issue later. (Update: I did return to the subject, in a 2013 column called Maintaining A Publicly Available Blacklist - Mechanisms And Principles)

  • Why not set up port knocking? Well, as you may have guessed I'm about to tell you about a few whys not. I've been circling port knocking as an article subject for a long time, and now it seems that the time has come to outline why I think that port knocking emphatically not a good idea. Read below for my take on the subject.
Port Knocking: Threat Or Menace?
I must admit that the port knocking concept fascinated me at first, but all implementations had the downside of adding yet another piece of clearly experimental software to my system along with somewhat convoluted procedures for setting the thing up, so I left it for later.

At each of the later dates when i decided to look into the topic again, I discovered new reasons not to like the method (read this thread on OpenBSD-misc for a particularly disenchanting example -- how's single-packet unconditional remote reboot or remote kill your sshd for a feature?). Finally, it was the appearance of the slow bruteforcers (also known as the Hail Mary Cloud) that made me finally decide that port knocking is neither particularly elegant or actually useful.

One of the reasons the port knocking concept seems so deceptively attractive is probably that it is fairly easy to explain (see the Port Knocking article on Wikipedia for a fairly thorough treatment). The target system runs with no services accessible to the outside, and in order to gain access a remote system must send a specific, pre-determined sequence of packets aimed at specified ports. Unless you know (or are able to guess) the correct sequence of ports, you will not be able to gain any access to the target system.

And after this explanation, the naive listener goes
"Oh, how clever, we send packets to random ports!
Nobody will ever be able to guess our combination!".

But first impressions are not always correct. First, please take one step back and explain to me what problem this is supposed to solve. If the answer is the old canard "all the insecurities of that horrible SSH protocol", I will ask you to please point out to me just what those are, and if you will, describe which parts of what I am about to describe actually adds value (that's "security" to you) in a real life scenario.

So let's go straight to the the core of the matter, and consider what actual information content an attacker would need to get right in order to gain access to the system. The hopeful would need to know or guess a sequence of TCP or UDP ports. In both cases, the range of possible ports is a 16 bit number, with a total of 65536 possible values.

 Each value is a 16-bit number, with a size of two bytes, or equal to two ASCII characters or one Unicode character. Port knocking examples generally do not run to more than three packets, which means that the minimum amount of information a prospective attacker would need to get right in order to gain access is six bytes, equal to six ASCII characters or three Unicode characters.

Seen from this angle, all port knocking gets you is a somewhat cumbersome method for encoding your unicode password. In most implementations, that password would even be common to all users, with no easy way to change it.

And of course, in almost all other contexts where passwords are used, most sites dictate that you choose a personal password that is longer than six bytes. So that's at least two failures rolled into one: a password that's common to several or all users, and one that is hard or impossible to change, possibly even short enough to fail even basic guidelines.

The amount of information an attacker would need to get right, measured in number of bits or bytes is a fairly useful measure. I admit it's a fairly crude measure, but at least real data are easy to obtain, and since we are talking here about designing and maintaining systems, not writing them, how much cryptographic fun goes into generating the necessary data is not really relevant to the context. Proper cryptography is essential to maintain confidentiality and ensure the integrity of the communications, but we'll leave the details of the crypto bits for a later time.

One other factor that speaks against most implementations of port knocking is that the system runs with no ports open, and a daemon that parses firewall logs for anticipated sequences of port numbers contacted as the sole authority to determine whether access will be granted. We all know that all non-trivial software will contain bugs, so what are the chances that even a fairly simple deamon will at some point in the future be confronted with a situation that makes it terminate, making your system inaccessible (or as we say around here, "bricking your system")?

And one other thing, how do you spot an attacker? If you're looking for access attempts to closed ports, as you should be if you run a port knocking setup, how do you spot an attacker in the primal soup of noise that will anyway be arriving at your public interfaces? Do you, for example, record all sequences that could possibly be seen as unsuccessful attempts and put the source addresses in a "definitely banned" list?

I have not seen any useful answer to that one either. Doing a sudo tcpdump -nettti pflog0 action drop (that is, using tcpdump(8) to extract information on blocked traffic from my packet filtering setup, directly from the log device) on my gateway here certainly shows me access attempts to ports I didn't know much about -- what legitimate purpose, for example, is UDP port 287 actually used for? Once again it all boils down to the fact that if you rely on port knocking, you're really only implementing a cumbersome variant encoding of relatively short passwords.

On a side note (thanks to Adrian Close for reminding me), anybody who sits in the signal path between your knock target and somebody executing the correct knock sequence will be able to record the connections (using, for example, a netflow based tool -- expect to see Netflow-themed columns here in the future), making your closely guarded secret roughly equivalent to a plaintext password much like we used to see in old-style ftp setups.

Next, let us let us instead look at the thing port knocking is trying to protect, namely the sshd service. Typical SSH connection setup involves the transfer of at least the public host keys (typically in the some hundred bytes to several kilobytes range), followed by the user authentication which can involve passwords, keys or a number of cryptography based toys^H^Hols, in any case producing further transfer and validation of anything up to several more kilobytes of data (and in some setups, use of out-of-band methods like short lived keys generated by special-purpose devices) before any meaningful access is possible. The exchange and validation of host keys in itself involves more information that the attacker would have to get right in order to gain access than is involved in any port knocking setup I've seen demonstrated. And remember, host keys exchange is only one early steps of several on the way to authentication.

For the actual math on how data sizes and entropy is significant (yes, in some contexts, size does matter), see the Wikipedia entry and a somewhat shorter summary by Phil Ratcliffe.

All port knocking implementations share the problems I've outlined, and their proponents have tended to ignore or gloss over the issues rather than address them. It is an unfortunate and perhaps embarrasing fact that port knocking in practice comes down to implementing and maintaining a separate set of passwords, and in all likelihood you will be using tools that are less appropriate for the task than the ones that already come with the base system on any reasonable Unix-like system.

In an environment where we know there are large, distributed efforts underway to crack easily guessable passwords, it is naive not to assume that the same password guessing tools could be adapted to try sequences of TCP or UDP ports instead of character sequences. If there is in fact any significant number of systems that use port knocking today, it would surprise me if this isn't already happening. The way I see it, the belief that port knocking in fact offers an effective protection against attackers is a dangerous misunderstanding and probably only serves to divert your attention away from the real problems involved in keeping your systems running in a secure fashion.

If you're still not convinced, I'll show you a better way

Now, if you really, really want to implement port knocking anyway, I'll let you in on a dirty little secret: you don't need a write a new daemon or install anything besides what's already in the OpenBSD base system (or for that matter, on other PF-equipped BSD variants).

You can implement port knocking in a relatively straightforward manner via minor contortions in your PF rule set, and I know for a fact that people have done just that. The examples I was thinking of before I started writing this piece appear to have been removed from the public eye, but with a little effort using the obvious keywords, some well intended but actually quite bad advice on how to put together a seemingly elegant implementation will be on its way to you.

Be aware that you will be doing things the tools were not designed for, and more likely than not you will soon find yourself in a mess of rinky-dink workarounds that keep fighting back.

If, however, what you really want to do is create a separate line of defence with its own round of authentication required for access, your OpenBSD base system already contains a suitable tool in authpf(8). It's fairly easy to come up with a setup that lets your users log in to the authenticating gateway, using any authentication method sshd(8) supports.

You can even run your inital sshd(8) on a separate, non-standard port if you like, and a successful login at that point will only result in a set of packet filtering rules (possibly tailored to that specific user) being loaded. When the ssh session terminates, the rules are unloaded, and the system reverts to the state it had before the user authenticated.

The tools are as flexible and robust as you have come to expect from the OpenBSD system tools (even if you're not running OpenBSD at the moment, you are by a large margin more likely than not using OpenSSH as your ssh client and server, which is supplied to the world by the OpenBSD project), and you can tailor your access profiles and services to your heart's content so they fit your needs.

If you've read this far, you've probably also come to the conclusion that port knocking is really not the way to go. Other, more appropriate tools exist. But how to go about improving the situation?

My suggestions follow below, with some specific suggestions that may have a personal slant.

A Path To Self Improvement

As with any problem that is really rooted in a lack of information or knowledge, educating yourself is the key. Read up on relevant subjects, go back and determine what problem you are actually trying to solve, and then perhaps shop around for tools that will help you solve that problem.

My suggestion is to go to first to www.openbsd.org and browse the resources available there, including the online manual pages and the extensive FAQ document that actually serves quite well as a user manual.

The next logical step is to go to the Orders page and order at least one copy of the latest release (and if you're reading this not too long after publication, do pre-order the upcoming release as well). If you're too impatient to wait for the postal services' mules to load up and deliver, you could download the release (or a -current snapshot if you're feeling adventurous) from a suitable mirror site and start installing right away.

If you don't order any merchandise, you really should head over to the donations page and donate at least an amount equal to the cost of the CDs and other stuff you would otherwise have ordered. There are several fit and frightening ghosts primed and ready to come haunt you if you don't.

If you want some literature to back up the official OpenBSD documentation, you could do worse than order The Book of PF, 4th edition and Michael W. Lucas' 2012 book SSH Mastery. Using those links here will ensure the OpenBSD project gets a larger-than-otherwise cut of the money as net profit. There are also electronic versions available for both titles (see The Book of PF home page or the SSH Mastery home page for details). If you're really short of funds, even the free, online PF tutorial that the Book of PF evolved from will get you started.

And finally, there are few places better to collect useful information about OpenBSD and its sister operating systems than the various local and regional user groups and conferences such as the upcoming BSDCan in Ottawa, Canada May 9th through 12th, 2012 or EuroBSDCon, with this year's version to be held in Warsaw, Poland, October 18th through 21st.

These events attract both regular users (some of them quite proficient) as well as a useful subset of the people who write and maintain the tools.
Whatever you end up doing, buying or downloading, please keep in mind that it's your responsibility to keep your systems in the best possible shape, and make sure you monitor them properly, via log files or other means. Also see the Pledge of the Network Admin for my (slightly humorous) take on that responsibility. Along with the better parts of the tutorial, it made its way into the book, too.

Thanks
to Michael Dexter, Kristaps Dzonsons, Michael W. Lucas, Thordur Bjornsson and Henning Brauer for insightful comments and valuable input.

Copyright © 2012 Peter N. M. Hansteen


If you found this article inspiring, irritating, useful or otherwise moving, please let me know via the comments field just below here, or if you like, drop me a line at peter at bsdly dot eu.

Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation (also in article form as The Hail Mary Cloud And The Lessons Learned) which summarizes this series of articles and provides links to all the data. The links in the presentation and the article point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

Sunday, October 4, 2009

A Third Time, Uncharmed

Spamwashers hijacked, a wake-up call for lazy sysadmins everywhere. The slow bruteforcers are back for another round.

A new round of slow, distributed bruteforce attacks is in progress. Just like the other times we know about (see references later), the initial target is root. This time around I see only one of my ssh-contactable machines targeted, and the dribble started on September 30th.

I've put the raw data so far up for study here (a total of 6067 attempts), and a list of hosts sorted by number of attempts (the first column) can be found here (770 hosts, with up to 32 attempts each). Quite likely I'll be collecting more data and publishing updates when I have a few free moments.

A number of people were kind enough to contact me in the followup of the earlier articles, and from one of my correspendents (who asked not to be named) I learned that the likely culprit is a piece of Linux malware known as dt_ssh5. If you type dt_ssh5 into your favorite search engine, it will turn up a few hits, but significantly fewer than the number of hosts in my sample. A couple of those documents have some analysis of how a badly secured web application let the miscreants in.

We should not be surprised that whoever is behind this has more than one takeover method in their arsenal. Unfortunately it should not surprise anybody either that we're now seeing a third round of those attacks. Most likely the perpetrators keep going because they occasionally succeed, and when they do, it's because every now and then they luck on a Linux machine with either

  • a maintenance regime that's disorganized enough that software with known and exploitable bugs is left in place for long enough to open the doors to undesirables, or

  • at least one user (whoever is manning root or any of the other user IDs we know they will be sniffing out later) with a guessable password and a system administration regime that lets weak passwords exist in the first place.

You see where this is heading? Over the last few years we have seen Linux and other free systems take over ever more niches and even mission critical application areas in businesses and organizations all over the world. I for one think this move to free systems with source code accessible is a good thing.

But there is a downside: In our efforts to entice the suits into the wonderful new world of free software, we likely oversold the security part.

Yes, bugs are easier to find and fix when the source code is available, yes, the Unix security model or some variant thereof is vastly preferable to that disorienting hellhole system marketed from somewhere up Seattle way, and yes, there are a few hundred more good and valid reasons (technical and otherwise) why basing your business on free license software is generally a good idea. Security-wise you have a better starting point, and with a little effort you will gain control and stay in control. But it does take some degree of effort by one or more qualified persons who are willing and able to take charge and make sensible decisions.

Looking at the data earlier I can see that one of the hosts now trying to gain illegitimate access to one of my systems was originally set up as a "spam washer" (grep for spamvask in the data). Setting up a Linux box to filter spam is likely a good idea in many contexts, but please keep this in mind: The fact that your rig runs Linux does not mean you're home free. You need to keep paying attention. When your spam washer has been hijacked and tries to break into other people's systems, you urgently need to get your act together, right now.

We do not have self-propagating worms on Linux just yet (and in all fairness the likelihood of that ever happening is rather slim), but we certainly do not have a system that will stay secure if you ignore the basics of useful system administration. This recurring episode of dt_ssh5 and the slow bruteforcers should serve as a wakeup call for system administrators everywhere: Your system stays secure only as long as you pay proper attention to maintenance. Out there in the real world there are at least 770 machines where the maintenance regime slipped, either through incompetence or work overload or a combination of the two with a dash of bad planning thrown in.

The only way to make a fourth round of slow bruteforcers not happen is to make the people responsible for the 770 hosts listed here do the right thing. I suppose we will find out soon enough how many of those domains actually have the RFC2142-mandated mailboxes in place.

References: Previous posts on this topic include A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009).


If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.


The reason for my rather long silence in the blogosphere can be summed up in two words: Client Confidentiality. Some of the projects I've been working on might have been material for interesting articles, but separating the generally useful bits from what the customer could reasonably expect to be kept confidential has proved difficult, at least in the short run.

Update 2009-10-05 just before 10:00 CEST: Slightly newer log extracts uploaded, total attempts now 6590, from 775 hosts. I will likely upload fresher versions at intervals.

Update 2009-10-14 19:30ish: After a couple of days with literally none to three or four attempts per hour, my afternoon log check turned up activity almost at start of run levels, so total numbers are now 9405 attempts from a total of 946 unique hosts. I never bothered to change the file names, but the data have been updated a couple of times each day since I started publishing them.

Update 2009-10-29 00:10: Because a file transfer took a little longer than expected, I was awake to see what may actually be the start of the alphabetic phase -


Oct 28 23:58:35 rosalita sshd[61664]: Invalid user anthony from 62.225.63.99
Oct 28 23:58:35 rosalita sshd[61664]: error: PAM: authentication error for illegal user anthony from 62.225.63.99
Oct 28 23:58:35 rosalita sshd[61664]: Failed keyboard-interactive/pam for invalid user anthony from 62.225.63.99 port 58683 ssh2
Oct 28 23:59:43 rosalita sshd[61668]: Invalid user anthony from 217.136.253.239
Oct 28 23:59:43 rosalita sshd[61668]: error: PAM: authentication error for illegal user anthony from 239.253-136-217.adsl-static.isp.belgacom.be
Oct 28 23:59:43 rosalita sshd[61668]: Failed keyboard-interactive/pam for invalid user anthony from 217.136.253.239 port 54728 ssh2
Oct 29 00:00:26 rosalita sshd[61695]: Invalid user barbara from 112.78.124.159
Oct 29 00:00:27 rosalita sshd[61695]: error: PAM: authentication error for illegal user barbara from 112.78.124.159
Oct 29 00:00:27 rosalita sshd[61695]: Failed keyboard-interactive/pam for invalid user barbara from 112.78.124.159 port 56990 ssh2
Oct 29 00:03:42 rosalita sshd[61722]: Invalid user brian from 122.224.128.197
Oct 29 00:03:42 rosalita sshd[61722]: error: PAM: authentication error for illegal user brian from 122.224.128.197
Oct 29 00:03:42 rosalita sshd[61722]: Failed keyboard-interactive/pam for invalid user brian from 122.224.128.197 port 47058 ssh2
Oct 29 00:04:21 rosalita sshd[61725]: Invalid user brian from 218.69.27.138
Oct 29 00:04:21 rosalita sshd[61725]: error: PAM: authentication error for illegal user brian from 218.69.27.138
Oct 29 00:04:21 rosalita sshd[61725]: Failed keyboard-interactive/pam for invalid user brian from 218.69.27.138 port 41622 ssh2
Oct 29 00:05:00 rosalita sshd[61738]: Invalid user carol from 58.60.106.121
Oct 29 00:05:01 rosalita sshd[61738]: error: PAM: authentication error for illegal user carol from 58.60.106.121
Oct 29 00:05:01 rosalita sshd[61738]: Failed keyboard-interactive/pam for invalid user carol from 58.60.106.121 port 55916 ssh2
Oct 29 00:05:51 rosalita sshd[61744]: Invalid user carol from 200.248.242.218
Oct 29 00:05:52 rosalita sshd[61744]: error: PAM: authentication error for illegal user carol from 200.248.242.218
Oct 29 00:05:52 rosalita sshd[61744]: Failed keyboard-interactive/pam for invalid user carol from 200.248.242.218 port 54547 ssh2
Oct 29 00:07:24 rosalita sshd[61748]: Invalid user charles from 222.128.36.60
Oct 29 00:07:24 rosalita sshd[61748]: error: PAM: authentication error for illegal user charles from 222.128.36.60
Oct 29 00:07:24 rosalita sshd[61748]: Failed keyboard-interactive/pam for invalid user charles from 222.128.36.60 port 42322 ssh2
Oct 29 00:08:09 rosalita sshd[61755]: Invalid user christopher from 72.148.242.188
Oct 29 00:08:10 rosalita sshd[61755]: error: PAM: authentication error for illegal user christopher from adsl-072-148-242-188.sip.asm.bellsouth.net
Oct 29 00:08:10 rosalita sshd[61755]: Failed keyboard-interactive/pam for invalid user christopher from 72.148.242.188 port 37157 ssh2
Oct 29 00:09:06 rosalita sshd[61758]: Invalid user christopher from 58.223.251.232
Oct 29 00:09:06 rosalita sshd[61758]: error: PAM: authentication error for illegal user christopher from 58.223.251.232
Oct 29 00:09:06 rosalita sshd[61758]: Failed keyboard-interactive/pam for invalid user christopher from 58.223.251.232 port 46350 ssh2
Oct 29 00:09:48 rosalita sshd[61762]: Invalid user daniel from 64.69.114.115
Oct 29 00:09:49 rosalita sshd[61762]: error: PAM: authentication error for illegal user daniel from mulligan.softspike.org
Oct 29 00:09:49 rosalita sshd[61762]: Failed keyboard-interactive/pam for invalid user daniel from 64.69.114.115 port 39960 ssh2
Oct 29 00:11:29 rosalita sshd[61781]: Invalid user david from 80.255.179.150
Oct 29 00:11:29 rosalita sshd[61781]: error: PAM: authentication error for illegal user david from ppp-vpdn-80.255.179.150.yarnet.ru
Oct 29 00:11:29 rosalita sshd[61781]: Failed keyboard-interactive/pam for invalid user david from 80.255.179.150 port 49013 ssh2


More material later, for now I'd be happy to hear confirmation (reports of similar activity) if you see it in your logs.

Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

Sunday, December 21, 2008

Into a new year, slowly pounding the gates

The distributed but clearly coordinated bruteforcers are still at it. How long until they reach the end of the alphabet? And why are they staying away from my OpenBSD machines? Are we seeing the contours of a controlling intelligence?

As large parts of the Western world prepares for the holidays, the swarm of little robots that started trying to pry open the doors to my machines some weeks back are still at it. As far as we can tell, the coordinated attempts started some time in early November or perhaps late October (we don't keep logs around for long enough to be sure), with an alphabetic progression that has now progressed to somewere into the os. The complete listing from the time I started noticing up to the time I started writing this column can be found here.

I've written about this before, and in fact one of those columns was slashdotted, a pleasant surprise to me and a cause of some excitement among my colleagues at FreeCode.

After writing that article, I did some further research and found out that a precursor to what we are seeing now was observed as early as May 2008, as described in an Ars Technica article published at the time. That article also reveals, via Linuxtoday, that yours truly was among the many who failed to understand the problem, at least for a while. Then again, maybe actual log excerpts would have helped.

The problem, such as it is, seems to be that a somebody who herds a botnet has decided that the laws of big numbers favors those who keep trying for long enough. User names and passwords are generally far enough from random that if you are allowed to go on for long enough, you will sooner or later manage to guess a correct combination of username and password and get access to a machine somewhere.

Sysadmins have been seeing bruteforce attacks for years. The traditional brute force attack would be a rapid succession of login attempts from one host, and usable countermeasures were devised in short order. My favorite of course involves PF, and the description of how to thwart traditional bruteforcers is one of the more popular pages in my PF tutorial.

The distributed, slow bruteforcers are different. For one, the login attempts from each host out in the cloud are spaced far enough apart in time that intrusion attmpt detectors will not trigger. Next, it takes a keen eye to spot the common thread in the attempts spaced up to a number of minutes apart: a monotonously alphabetic progression of user names, with attempts coming in from different hosts. Some number of attemtps at a specific user name, before the cloud moves on the next one, in alphabetic order.

During the period we have been observing the slow brute activity, a total of 695 hosts have been involved. A total of 665 hosts made unsuccessful attempts at authenticating at the hosts we are observing during November, while the number for December so far is 346. The typical number of attempts per user name has decreased, too, from a typical ten do fifteen during the early days down to between one and four during the last couple of weeks.

I thought at first that the decrease in activity was just an indicator that compromised hosts were getting cleaned up, but my colleague Egil Möller was the first to suggest that since we know the attempts are coordinated, it is not too far fetched to assume that the controlling system measures the rates of success for each of the chosen targets and allocates resources accordingly.

If Egil's assumption is right, we are seeing the bad guys adapting. My systems do not run any services they do not need to, and apparently all attempts at gaining access have been futile so far. So, the controlling system shifts resources to elsewhere, even if the access attempts do not stop entirely. Come to think of it, I'm not seeing any attempts at all on my OpenBSD systems, so it is possible to speculate that whoever is behind this phenomenon has decided that OpenBSD systems are hardened enough to begin with and usually run by compentent paranoids as to be useless as targets. That would be a comforting thought at the end of a long and sometimes trying year.

Speaking of the new year, look for exciting announcements coming from FreeCode. We're working on some cool things. And with a bit of luck, I might run into you at one conference or the other during the coming year.

Happy holidays to everyone.


Note:
 A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.