Looking back at the past week, the most interesting communication received here was not about meteorites hitting Russia (although some of the dashcam videos were really exciting), but rather about an old and very simple prediction of mine that appears to have come true.
It came via an email message from a friend and somewhat infrequent correspondent, who wrote on February 15 2013,
"[...] Here's something mildly curious, though. Between now and Dec 26, the current start of my authlog, the below tried to get in via port 2222 on my openbsd box. I'm surprised both that there are not many yet at the same time that there are any at all."And the log excerpt he sent me looked like this:
Jan 11 16:02:01 net5501 sshd[3170]: Invalid user admin from 66.135.32.170
Jan 11 17:58:43 net5501 sshd[2611]: Invalid user aadriano from 66.135.32.170
Feb 10 17:16:33 net5501 sshd[23679]: Invalid user alex from 66.133.188.86
Feb 10 17:18:32 net5501 sshd[20330]: Invalid user cactiuser from 66.133.188.86
Feb 10 17:20:31 net5501 sshd[19432]: Invalid user www-data from 66.133.188.86
Feb 10 17:24:28 net5501 sshd[29455]: Invalid user user from 66.133.188.86
Feb 10 17:26:27 net5501 sshd[82]: Invalid user tomcat from 66.133.188.86
Feb 10 17:28:26 net5501 sshd[31011]: Invalid user test from 66.133.188.86
Feb 10 17:30:25 net5501 sshd[20057]: Invalid user temp from 66.133.188.86
Feb 10 17:32:23 net5501 sshd[19848]: Invalid user service from 66.133.188.86
Feb 10 17:34:22 net5501 sshd[31209]: Invalid user samba from 66.133.188.86
Feb 10 17:38:20 net5501 sshd[12045]: Invalid user postgres from 66.133.188.86
Feb 10 17:40:20 net5501 sshd[1085]: Invalid user oracle from 66.133.188.86
Feb 10 17:42:19 net5501 sshd[12896]: Invalid user nagios from 66.133.188.86
Feb 10 17:44:19 net5501 sshd[10681]: Invalid user mysql from 66.133.188.86
Feb 10 17:46:19 net5501 sshd[27897]: Invalid user guest from 66.133.188.86
Feb 10 17:48:19 net5501 sshd[13400]: Invalid user tom from 66.133.188.86
Feb 10 17:50:18 net5501 sshd[5132]: Invalid user ftp from 66.133.188.86
Feb 10 17:54:15 net5501 sshd[28789]: Invalid user jboss from 66.133.188.86
Feb 10 17:56:14 net5501 sshd[20435]: Invalid user backup from 66.133.188.86
So just a sampling of likely service user IDs on a random system, right?
The reason why this is significant is that my friend, like a few others, tired of the apparently never ending password guessing attempts that were hitting the sshd (also known as the Secure Shell Daemon) listening on the standard port 22, and moved the service to listen on a different port instead.
The immediate effect in almost all cases was a much quieter authentication log, because the password guessers weren't yet trying to bother you on non-standard ports.
Those of us who could for one reason or other not be bothered to change the config to have our sshds run on other ports and using OpenBSD on anything Internet-facing soon found ourselves using the state-tracking features of OpenBSD (introduced in the May 2005 3.7 release). I wrote about it in the PF tutorial, specifically the Turning away the brutes part, and of course a slightly embellished version made it into The Book of PF. Please note the sentence somewhere down the page,
"You might consider moving the service to some other port, but then again, the ones flooding you on port 22 would probably be able to scan their way to port 22222 for a repeat performance."The news is that this is now happening. The only surprise here to my mind is that signs of that exact activity has not been reported earlier. I put that down to mainly two factors. One is that the number of sites running their ssh service only on non-standard ports is probably quite small, another is that people who run services in general pay way too little attention to their logs.
To those of us who run our packet filtering with a default deny policy and drop anything that's not relevant to services we actually run, those attempts to access ports where we do not run anything are simply part of the background noise, so we drop and ignore unless we're looking for patterns in the connection attempts we drop. So we wouldn't necessarily notice the attempts either.
And for all I know (I really can't resist a dig at the high port refugees) it could be that for the most part, those who have moved their main login service to a non-standard port pay less attention to their logs than the rest of us, now that they have obscured the entrance to the kingdom to their own satisfaction if not so many else's.
As I've argued elsewhere, obscuring your login service via non-standard ports or even a requirement to try several ports in sequence really only buys you security equivalent to lengthening your password by two characters per port. After all, TCP and UDP port numbers are limited to 65536 distinct values, within the scope of a 16-bit value. Moving your service to a non-standard port means that the would-be intruder has to guess a two-byte value. Introducing a sequence of ports buys you the added obscurity of two bytes per port.
I have reason to believe that my correspondent here has done away with passwords altogether for logins to his systems, and therein lies the real solution: SSH offers out of the box the opportunity to allow only key based authentication, and with a little extra effort, you can implement more sophisticated authentication schemes. One useful reference for ssh in addition to the manual pages (starting with this one, all of the expected OpenBSD quality) is Michael Lucas' recent book SSH Mastery.
Stay safe, please move from passwords to passphrases, and apply those to your keys only, never to be transmitted in the clear over the net where you never know who could be listening in.
For anyone else seeing this kind of activity, as always I'll be very happy to hear from you, via comments or email.
Good luck and good night.
Thanks to Lars Noodén for valuable input for this column.
Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.
Posts
Fail2Ban perhaps
ReplyDeletehttp://www.the-art-of-web.com/system/fail2ban/
...and a HID(P)S like OSIRIS or OSSIM or sagan.
DeleteYes... 2222 is very unusual choice for a non-standard SSH port number ;)
ReplyDeleteexactly!
Deleteanother good thing to be doing is jacking up LogLevel to verbose so you can keep track of all attempts to connect to the port. this'll catch detection of any accesses to the (hopefully truly random and truly high) port.
I'm sure you "dorp" and ignore what you like, but increasing your password by two characters is FAR more significant than assigning a random port to SSH. That's because the password length increases the search EXPONENTIALLY. The random port increases the search space ADDITIVELY.
ReplyDeleteTaking a password of keyboard-creatable characters (let's pretend upcase, lowcase, numbers, symbols) we have a password of n characters giving us a 96^n search space. If we added those "two characters" we get 96^(n+2).
In contrast, if we choose some port other than 22, we add 96^n + 16^2. Let's see how this compares. Comparing a password of 96^(n+2) to a password of 96^n is on the order of 96^2 simpler. These we need to calculate, hash, and test. In contrast to test 2^16 ports for an SSH listener we can use one burst and we have NO encryption or hashing to do.
Anytime we can separate a key into two pieces, one of which requires no calculations... it's a weaker key than one of the total length that cannot be split.
Ehud
Eliminating passwords and moving to 2048 bit keys is significantly better than arguing about this.
DeleteNot Really.
Deleteyou are wrong. random port is:
Delete96^n * 16^2 which is same as 96^(n+2).
if you have password 'n' long and you move from port 22 to 23, you have *2 combination, not +2 combination!
Actually... no. That would be the case if every attempt were "pick port, pick password". But a sensible attacker starts by narrowing the space with a scan of no more than 64k ports to determine which ones have an SSH listener, and then tries the favored user/pass combos. They get to memorize the port and use it without re-searching on subsequent attempts.
DeleteI suppose you could restart sshd after every successful connection with a new port number, but then every legitimate user needs to keep track of it as well.
More interesting than port refuge and the more obvious block-on-too-many-attempts is that apparently this is a staged scanning system.
ReplyDeleteI use port **** and gave good luck with a ip ban after 5 try. Just setup an auto ban. Why would you get your own password wrong 5 times?
ReplyDelete5 tries? That can happen easily. You changed your password. The new password didn't take. Or maybe you think it didn't take. Or you absent mindedly try the old password a few times, then mistype the new one.
DeleteWhy do people foolishly set the try limit so low? If it was 10 tries, then you wouldn't annoy anyone, and it's probably just as secure against attackers.
I've often gotten a password wrong 5 times. Didn't remember whether I was using capitals or not, didn't remember whether it was the old or new password, typed it wrong the first time when I was testing to see if it was one one of the above, meaning I have to cycle through all the permutations 1.5 times on average.
DeleteBut it's moot really; I don't use password authentication for SSH.
Can ssh brute forcers somehow know that key-only auth is in force? That would be nice because it is in the interest of both parties that they not keep hammering away when there is no chance of getting in.
ReplyDeleteYes, the sshd will tell the client what authentication methods it will accept. If `password` or `keyboard-interactive` isn't on the list, the brute-forcer can just move on to the next host.
DeleteI moved ssh to a non-standard port merely to reduce the traffic in my logs. It permits ssh-key only. No password logins accepted.
ReplyDeleteI'm with you Dan.
DeleteI admin a fair sized hosting service and the OpenBGPd routers copped loads of attempts.
I read the logs regularly and I have yet to see ANY attempts since moving the ssh port to a higher number years ago.
Peter's dig at port elevators doesn't fit me. I moved BECAUSE I read the logs.
R/
Exactly! Before I moved, reading logs was overwhelming. Now it's managable.
Deletehttp://ingles.homeunix.net/software/ost/
ReplyDeleteAn additional way to protect SSH or any other vital service for that matter is port knocking.
Another alternative to port knocking, is a tool a friend wrote years ago called page knocking:
Deletehttp://datacentre.otterhole.ca/knock/
It seems pretty effective to delay the ssh banner for a few seconds on an unusual (for ssh) port. Not a single attempt logged in the past 8 weeks. Only a handful of attempts logged in the past 6 months and each of those look like they were not attempting to crack into ssh. (Very different from the noise previously logged when ssh was on port 22.)
ReplyDeleteI'm thinking it might be even more effective to have a dummy service that would disconnect in response to receiving any data before ssh is given control.
The problem with ssh-key only is that you cannot login from an arbitrary remote system unless you are able to install your key which is not always the case.
ReplyDeleteThe other thing is that port 22222 isn't a huge leap from 22 - SSH would probably be my first guess at what was actually there. Not particularly enlightened as far as obfuscation is concerned.
Choosing a port < 1024 offers obfuscation and misdirection. This provides a much better approach as there is a far greater chance that the service will be assumed to be the standard service on that port.
443 is an ideal candidate (if not already used). It will be assumed to be HTTPS but not vulnerable to HTTPS cracks and it can be tunnelled to through a proxy server (via HTTP/1.1 connect method) giving you access from very locked down networks.
All that said, obfuscation is not a reliable approach to security even if it does reduce the log noise. As Peter contends, the strength of your SSH configuration should be your sole gauge of SSH security.
Bring your key with you on a USB-disk / cell phone or even, as the greybeard have done for years: a laptop. Seriously, carrying your key with you is not difficult.
DeleteYou wouldn't lock your car with an easily crackable 4-digit pin code when it's so easy to carry car keys, would you? Why are people averse to carrying digital keys? (Actually, that's what newer car keys are anyway)
Use denyhosts.
ReplyDeletehttp://denyhosts.sourceforge.net/
Yes, I moved to a non-standard port as well mostly because having the probes was annoying me. Evidently I picked a good number (no twos) so they probes haven't reappear yet.
ReplyDeleteI also use key only and no root and a few other tricks of my own devising. I did deactivate DenyHosts though. I'll probably turn that back on.
while reading your article i started to think about port-knocking, a great idea indeed: http://www.marksanborn.net/linux/add-port-knocking-to-ssh-for-extra-security/
ReplyDeleteAn attacker has to get both a username and a password correct. On a properly configured system, only real users have passwords (ie. not 'back' and 'mysql' etc.)
ReplyDeleteSo they have to get my 5+ digit user name correct AND my 8 digit password, over the net. That's HIGHLY unlikely, right?
It depends... for a person yes. For a script not as unlikely as you'd hope. Allow only ssh-keys and, deny root logins. You can also configure ssh to limit to a specific user or group as well which isn't a bad idea either.
DeletePort knocking always works great...
ReplyDeleteThere are many different implementations of port knocking. I would suggest one which uses a OTP. Static Port knocks can be figured out and this just becomes an obfuscation approach.
DeleteI use similar tactics as your friend plus no password authentication on most boxes, and on those which do accept passwords you should set fail2ban. So far, script kiddies do not bother checking for the alternative port. Perhaps your friend has been handpicked for some reason? In his shoes, I would try with some really random port like 2223. :)
ReplyDeleteBottom line: Configure sshd to...
ReplyDelete1. Listen to a non-standard, non-obvious port (not 2222...)
2. Only accept key-based authentication
3. Make use of an auto-ban solution if too many failed attempts (Fail2Ban, etc.)
Folks, you're all missing the point - changing the default port means sending 64k more SYN packets, which considering a 32-bit IPv4 space, is a LOT more scanning.
ReplyDeletePlus, you'll hit other services almost always, which admittedly, you can probably break/leverage.
For for brute-force attacks, that's a lot harder with non-default ports, but do-able...
...but that's not the main benefit of moving the port - it gives you a *LOT* more protection from 0-days and worms, which putting the idea the other way, otherwise would be 64k times more successful at propagating.
Hmm am I the only anonymous coward thinking this is a blatant commercial for openbsd book and then so close to april fools.
ReplyDeleteHow to stop this stuff happening at all?
ReplyDeleteWhat about a system where the IP addresses of the attacking system was automatically forwarded to some service, which in turn alerted the relevant hosting provider to disable their account?
I guess the source IP address is spoofed, which would stop that from working.
So would it require some packet inspection system at the ISPs themselves, to see the tell-tale signs of this being done?
That was informative. This from a Steve Gibson fan with a diffidently academic interest in such matters. Only reason I wrote was to say that there is no hyphen in "nonstandard."
ReplyDeletenmap can do a rapid scan of open ports, so if you're being targeted by a person rather than a dumb script in an attack high ports aren't a big help. While it might be helpful in reducing the noise to signal ratio of failed login attempts, I would encourage you to instead focus your mental effort on analysing the successful logins. It's less noise and with process accounting, network logging, etc it's likely to be much more valuable when diagnosing a breach. Non-standard ports can be helpful with scripted 0-day exploits but, that's about it. fail2ban as some have mentioned is good at slowing down a brute force attack but, won't help with flaws in the software The problem isn't just whether someone can break your password, it's also whether the software itself has an exploit (or other software on the system)... the best box is one with the cord unplugged... but that hardly makes for a good business prospect unless you sell paperweights.
ReplyDeleteAs some have mentioned port knocking works nicely (if not a little cumbersome). But in many corporate environments the preferred method is multilayered encryption using a VPN to access the boundary of the server network and only allowing ssh access via the VPN.
I foolishly used bastion hosts (jump boxes) for a while to ssh into a protected network. I've subsequently stopped after hearing from a security analyst that one of his clients' was seriously compromised because of something similar. The attacker gained access to the bastion host due to poor patching practises on the box and the attacker was then able to login to other machines by capturing credentials on the bastion host.