The OpenSSH Bug That Wasn't

Much has been written about a purported OpenSSH vulnerability. On closer inspection, the reports actually got most of their facts wrong. Read on for the full story.

It all started with a blog post dated July 16, 2015, titled OpenSSH keyboard-interactive authentication brute force vulnerability (MaxAuthTries bypass), where the TL;DR is that it's possible to get an almost infinite number of tries at authentication -- good for bruteforce password guessing, for example -- if you only tickle the OpenSSH server just so.

Note: This piece is also available without trackers but classic formatting only here.

This sounded interesting and scary enough that I wanted to try it out myself. The blog quite helpfully supplies a one-liner that you can cut and paste to your own conmand line to check whether the systems you have within reach are indeed vulnerable.

Here's a transcript of running those tests on the machines I happened to try (Disclaimer: The recorded sessions here are from a second try, a few days after the first). First, my home gateway, running a recent OpenBSD 5.8-beta:

[Fri Jul 24 14:58:31] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` skapet.bsdly.net
Host key fingerprint is SHA256:maeVFpNMibnkcwPSmjV4QBXfz5J97XLta6e2CrzsAYQ
+---[ECDSA 256]---+
|     .o=o+..     |
|      o.X+.o     |
|      EO.+* .    |
|      o.+oo+ =  .|
|        So=.o + o|
|         B   . o.|
|        . +   . +|
|         . +  .=.|
|         .+ .o+++|
+----[SHA256]-----+

razz@skapet.bsdly.net's password: 
Permission denied, please try again.
razz@skapet.bsdly.net's password: 
Permission denied, please try again.
razz@skapet.bsdly.net's password: 
Permission denied (publickey,password,keyboard-interactive).
[Fri Jul 24 16:53:06] peter@elke:~$ 

Here, razz is a non-existent user, and as we can see we get exactly three password prompts before the connection is shut down. Now we know that an essentially untweaked SSH server configuration on a recent OpenBSD does not behave as described in the article.

But what about earlier OpenBSD releases? I have one box I should have upgraded a while ago (to my enduring shame, it's still on 5.3-stable, but don't tell anyone). So here's the same thing pointed at that box:

[Fri Jul 24 16:53:06] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` delilah         
Host key fingerprint is SHA256:AO8rn6Va9+b3+7gdVUxby5zWQFaLnkIA6wcEsOVHukA
+---[ECDSA 256]---+
|   Eoo.+..   .+.+|
|  . +o+ . .  .++B|
|   o oo+   . . O+|
|    ..+.. . . o .|
|     ...S. . o  .|
|       ..   .   .|
|    . o o      . |
|     + = .. .  o.|
|    ..+  oo. .=+o|
+----[SHA256]-----+

razz@delilah's password: 
Permission denied, please try again.
razz@delilah's password: 
Permission denied, please try again.
razz@delilah's password: 
Permission denied (publickey,password,keyboard-interactive).
[Fri Jul 24 16:59:37] peter@elke:~$ 

razz is not a valid user here either, and even the old OpenSSH version here shuts down the connection after three failures. I don't have any OpenBSD boxes with older versions than this anywhere I know about, and we can be reasonably confident that at least close to default configurations on OpenBSD are not vulnerable.

But several of the articles hint that OpenSSH on Linux is vulnerable. I do have a few CentOS boxes within reach. I'll repeat the test on one of these, then:

[Fri Jul 24 17:05:13] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` nms
Host key fingerprint is SHA256:fdFxpvSDLq3W9D1d8U6RzuYQcd0WzAmIFfJAzcIkD8I
+---[RSA 2048]----+
|    .. o+==ooo*oB|
|     E. ++++ o+X=|
|         ....oo*.|
|         .  o.+ =|
|        S ...= ++|
|           .= =o+|
|           o . ++|
|          .     .|
|                 |
+----[SHA256]-----+

razz@nms's password: 
Permission denied, please try again.
razz@nms's password: 
Permission denied, please try again.
razz@nms's password: 
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

nms is a CentOS 6.6 box, and the result is as we can see here, pretty much identical to what we saw on the OpenBSD machines. So far we haven't seen anything like what the blogger kingcope had us to expect to see.

But looking back to the original article, he seems to have tested only on FreeBSD. I have some FreeBSD boxes within reach too, and rosalita runs a recent FreeBSD 10.1, freshly upgraded via freebsd-update. Here's what our experiment looks like pointed at rosalita:

[Fri Jul 24 17:15:03] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` rosalita           
Host key fingerprint is SHA256:Ig6F8Au3f0KYNrzuc5qRrpZgY4Q/tz0bJrS0NZMxp1g
+---[ECDSA 256]---+
|.                |
| o .             |
|o + . E .        |
|.= * o *         |
|..X * B S        |
|.=o@.= +         |
|+ *oBo+          |
| =.oo=o.         |
|oo*+  .o         |
+----[SHA256]-----+

Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
^C

Bingo! We have finally seen the reported vulnerability in action on a live system. The ^C at the end is me pressing Ctrl-C to abort after ten tries, my heart wasn't quite in it for the full ten thousand. So far, it looks like this behavior is specific to FreeBSD, but of course it is conceivable that other systems ship with their sshds configured in a similar way.


After a bit of back and forth and reading articles elsehere, it seems that only OpenSSH servers that are set up to use PAM for authentication and with a very specific (non-default on OpenBSD and most other places) setup are in fact vulnerable. Even though there is a patch available which tightens up the code a bit in the PAM-specific parts, OpenBSD users don't actually need to apply it. One big reason being that OpenBSD does not use PAM for its authentication.

The question also came up in a thread on OpenBSD-misc, titled Alleged OpenSSH bug, where several OpenBSD developers commented. Do read the whole thread, but as we've already seen, it's easy to test whether your systems behave as described in the original blog post as well as this one.

And as OpenBSD developer Marc Espie says in his message,

Not surprisingly, as the patch clearly shows, the problem is right smack in the middle of USE_PAM code.

I wouldn't call that an OpenSSH bug. I would call it a systemic design flaw in PAM. As usual. LOTS of security holes in authentication systems stem from PAM. Why ? Because that stuff is over designed. Difficult to configure. Gives you MORE than you need to hang yourself several times over. It's been that way for as long as I can remember.

As they say, do read the whole thing. TL;DR this time around is: OpenBSD is not vulnerable, and on the systems that are, changing the configuration to close this particular bruteforcing opportunity is trivial. As is checking the facts before writing up a story. (And as several correspondents have reminded me already -- switching your sshd to keys only authentication will let you sleep better at night.)

There's No Protection In High Ports Anymore. If Indeed There Ever Was.

Running your service on a non-standard port buys you an equivalent of two more characters in your password. Will that save you any trouble at all?

Looking back at the past week, the most interesting communication received here was not about meteorites hitting Russia (although some of the dashcam videos were really exciting), but rather about an old and very simple prediction of mine that appears to have come true.

It came via an email message from a friend and somewhat infrequent correspondent, who wrote on February 15 2013,

"[...] Here's something mildly curious, though. Between now and Dec 26, the current start of my authlog, the below tried to get in via port 2222 on my openbsd box. I'm surprised both that there are not many yet at the same time that there are any at all."

And the log excerpt he sent me looked like this:

Jan 11 16:02:01 net5501 sshd[3170]: Invalid user admin from 66.135.32.170
Jan 11 17:58:43 net5501 sshd[2611]: Invalid user aadriano from 66.135.32.170
Feb 10 17:16:33 net5501 sshd[23679]: Invalid user alex from 66.133.188.86
Feb 10 17:18:32 net5501 sshd[20330]: Invalid user cactiuser from 66.133.188.86
Feb 10 17:20:31 net5501 sshd[19432]: Invalid user www-data from 66.133.188.86
Feb 10 17:24:28 net5501 sshd[29455]: Invalid user user from 66.133.188.86
Feb 10 17:26:27 net5501 sshd[82]: Invalid user tomcat from 66.133.188.86
Feb 10 17:28:26 net5501 sshd[31011]: Invalid user test from 66.133.188.86
Feb 10 17:30:25 net5501 sshd[20057]: Invalid user temp from 66.133.188.86
Feb 10 17:32:23 net5501 sshd[19848]: Invalid user service from 66.133.188.86
Feb 10 17:34:22 net5501 sshd[31209]: Invalid user samba from 66.133.188.86
Feb 10 17:38:20 net5501 sshd[12045]: Invalid user postgres from 66.133.188.86
Feb 10 17:40:20 net5501 sshd[1085]: Invalid user oracle from 66.133.188.86
Feb 10 17:42:19 net5501 sshd[12896]: Invalid user nagios from 66.133.188.86
Feb 10 17:44:19 net5501 sshd[10681]: Invalid user mysql from 66.133.188.86
Feb 10 17:46:19 net5501 sshd[27897]: Invalid user guest from 66.133.188.86
Feb 10 17:48:19 net5501 sshd[13400]: Invalid user tom from 66.133.188.86
Feb 10 17:50:18 net5501 sshd[5132]: Invalid user ftp from 66.133.188.86
Feb 10 17:54:15 net5501 sshd[28789]: Invalid user jboss from 66.133.188.86
Feb 10 17:56:14 net5501 sshd[20435]: Invalid user backup from 66.133.188.86


So just a sampling of likely service user IDs on a random system, right?

The reason why this is significant is that my friend, like a few others, tired of the apparently never ending password guessing attempts that were hitting the sshd (also known as the Secure Shell Daemon) listening on the standard port 22, and moved the service to listen on a different port instead.

The immediate effect in almost all cases was a much quieter authentication log, because the password guessers weren't yet trying to bother you on non-standard ports.

Those of us who could for one reason or other not be bothered to change the config to have our sshds run on other ports and using OpenBSD on anything Internet-facing soon found ourselves using the state-tracking features of OpenBSD (introduced in the May 2005 3.7 release). I wrote about it in the PF tutorial, specifically the Turning away the brutes part, and of course a slightly embellished version made it into The Book of PF. Please note the sentence somewhere down the page,

"You might consider moving the service to some other port, but then again, the ones flooding you on port 22 would probably be able to scan their way to port 22222 for a repeat performance."

The news is that this is now happening. The only surprise here to my mind is that signs of that exact activity has not been reported earlier. I put that down to mainly two factors. One is that the number of sites running their ssh service only on non-standard ports is probably quite small, another is that people who run services in general pay way too little attention to their logs.

To those of us who run our packet filtering with a default deny policy and drop anything that's not relevant to services we actually run, those attempts to access ports where we do not run anything are simply part of the background noise, so we drop and ignore unless we're looking for patterns in the connection attempts we drop. So we wouldn't necessarily notice the attempts either.

And for all I know (I really can't resist a dig at the high port refugees) it could be that for the most part, those who have moved their main login service to a non-standard port pay less attention to their logs than the rest of us, now that they have obscured the entrance to the kingdom to their own satisfaction if not so many else's.

As I've argued elsewhere, obscuring your login service via non-standard ports or even a requirement to try several ports in sequence really only buys you security equivalent to lengthening your password by two characters per port. After all, TCP and UDP port numbers are limited to 65536 distinct values, within the scope of a 16-bit value. Moving your service to a non-standard port means that the would-be intruder has to guess a two-byte value. Introducing a sequence of ports buys you the added obscurity of two bytes per port.

I have reason to believe that my correspondent here has done away with passwords altogether for logins to his systems, and therein lies the real solution: SSH offers out of the box the opportunity to allow only key based authentication, and with a little extra effort, you can implement more sophisticated authentication schemes. One useful reference for ssh in addition to the manual pages (starting with this one, all of the expected OpenBSD quality) is Michael Lucas' recent book SSH Mastery.

Stay safe, please move from passwords to passphrases, and apply those to your keys only, never to be transmitted in the clear over the net where you never know who could be listening in.

For anyone else seeing this kind of activity, as always I'll be very happy to hear from you, via comments or email.

Good luck and good night.

Thanks to Lars Noodén for valuable input for this column.

---

Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.

SSH Mastery: A Very Welcome Addition to Any Unix User's Bookshelf

The first paragraph of this book's afterword reads:

"You now know more about SSH, OpenSSH and Putty than the vast majority of IT professionals! Congratulations".

That claim will be true for any reader of SSH Mastery who has read the book up to that point and has incorporated at least some of the elements of the configurations it describes into their own environments.

"But why a book dedicated to a single command?", you might ask. Almost all Unixes and Unix-likes have incorporated OpenSSH, the free SSH that is developed as part of the OpenBSD project, and OpenSSH comes with excellent documentation in the form of several extensive man pages.

Well, that question in itself justifies this title's existence (there are in fact several programs in the OpenSSH suite), and readers familiar with Michael Lucas' work will appreciate hearing that his latest work is task-oriented and well written, covering anything from the basic secure shell access through to the peculiarities of setting up a virtual private network (VPN) using OpenSSH. An enterprising reader would be able to find all the information in this book or close equivalents using the OpenSSH man pages or other online sources, but this book provides a very concise guide to both the basics and some rather advanced concepts and provides you with the vocabulary and understanding that you will need in order to successfully navigate the man pages.

This book has several highlights, such as the very sensible and useful discussion of key based authentication and how to set things up for a passwordless existence, a number of suggestions on how to distribute and maintain both host keys and user keys as well as very readable and useful introductions to various kinds of tunneling, forwarding and proxying available using the OpenSSH tools.

In particular I enjoyed reading the description of SSH-based virtual private networks (VPNs) in Chapter 13. This is one of the most clearly written and useful treatments I've seen of that subject, and for many readers this chapter alone will be worth the price of the book or even considerably more.

The book very sensibly covers OpenSSH on OpenBSD, FreeBSD and Ubuntu Linux, and users who are compelled to use Microsoft Windows desktops will be pleased to hear that configuration and use information for Putty, the most popular and free SSH client for their environment, is included too everywhere it's relevant to the task at hand.

Before Michael W. Lucas' new title was released in January 2012, the most recent widely available book about the Secure Shell protocol (SSH) and applications that support it was an O'Reilly title dated 2005. So even with high quality documentation available via the manual pages, it was time for a new title on the subject.

This title conveniently grew out of one of Michael W. Lucas' other technical writing projects, the second edition of Absolute OpenBSD. The SSH chapter of that manuscript simply kept growing until it made sense to branch the text off to a separate book. This probably means that the treatment of SSH in the upcoming OpenBSD title will be slimmer again, but separating out the OpenSSH parts as a separate book with information for several different environments added makes sense because it makes high-quality information about important tools available to a larger audience.

I am convinced SSH Mastery is a title that Unix users and system administrators like myself will want to keep within reach on their Kindles or other ebook readers for a quick and convenient refresh of important concepts. If you're a student or learning your Unix skills, you will certainly find this to be a very handy guide that helps you grasp both the basics of SSH and several advanced concepts that are hard to find well described elsewhere.

The ebook is available in several formats via Amazon and other ebook outlets, a printed version is planned but was not yet available at the time of writing (January 22, 2012).

Title: SSH Mastery: OpenSSH, PuTTY, Tunnels and Keys
Author: Michael W. Lucas
Publisher: Tilted Windmill Press (January 18, 2012)