As will probably not surprise any of my regular readers, I've spent a bit of time recently browsing and processing SSH authentication logs for some of the systems in my care. As usual I browse logs with a view to extracting insights and hopefully at some future date I will be able to present useful material based on analyses of that material.
But sometimes something stands out as just too wrong. Today while browsing archived logs I came across this entry from July:
Jul 8 12:53:17 skapet sshd[88344]: Invalid user <!DOCTYPE from 187.50.71.54 port 57999That string is the start of an SGML-style document declaration, basically what you would expect to find at the very start of an SGML-ish file such as an HTML document.
Instruct your browser to 'Display source' for this article, and that exact string is the first thing in the HTML source display buffer. But in the context of an authentication log for an SSH service, it's distinctly odd.
And what's more a little later in the same log I found:
Jul 8 20:59:08 skapet sshd[11083]: Invalid user content="text/html; from 175.143.54.193 port 26240Again, somebody throwing HTML at my sshd, but this time from a different IP address.
This piqued my interest enough that I decided to take a look at whatever else those jokers had been up to:
[Thu Dec 22 19:40:06] peter@skapet:~$ zgrep 187.50.71.54 /var/log/authlog.23.gz
Jul 8 12:53:17 skapet sshd[88344]: Invalid user <!DOCTYPE from 187.50.71.54 port 57999
Jul 8 12:53:17 skapet sshd[88344]: Failed password for invalid user <!DOCTYPE from 187.50.71.54 port 57999 ssh2
Jul 8 12:53:17 skapet sshd[88344]: Connection closed by 187.50.71.54 port 57999 [preauth]
Jul 8 13:02:15 skapet sshd[85203]: Invalid user PUBLIC from 187.50.71.54 port 58123
Jul 8 13:02:15 skapet sshd[85203]: Failed password for invalid user PUBLIC from 187.50.71.54 port 58123 ssh2
Jul 8 13:02:15 skapet sshd[85203]: Connection closed by 187.50.71.54 port 58123 [preauth]
Jul 8 13:11:13 skapet sshd[25261]: Invalid user XHTML from 187.50.71.54 port 57227
Jul 8 13:11:13 skapet sshd[25261]: Failed password for invalid user XHTML from 187.50.71.54 port 57227 ssh2
Jul 8 13:11:13 skapet sshd[25261]: Connection closed by 187.50.71.54 port 57227 [preauth]
Jul 8 13:20:10 skapet sshd[68619]: Invalid user Strict//EN" from 187.50.71.54 port 25941
Jul 8 13:20:10 skapet sshd[68619]: Failed password for invalid user Strict//EN" from 187.50.71.54 port 25941 ssh2
Jul 8 13:20:10 skapet sshd[68619]: Connection closed by 187.50.71.54 port 25941 [preauth]
Jul 8 13:28:58 skapet sshd[96899]: Invalid user <html from 187.50.71.54 port 48462
Jul 8 13:28:58 skapet sshd[96899]: Failed password for invalid user <html from 187.50.71.54 port 48462 ssh2
Jul 8 13:28:58 skapet sshd[96899]: Connection closed by 187.50.71.54 port 48462 [preauth]
Jul 8 13:37:48 skapet sshd[59363]: Invalid user <meta from 187.50.71.54 port 46496
Jul 8 13:37:48 skapet sshd[59363]: Failed password for invalid user <meta from 187.50.71.54 port 46496 ssh2
Jul 8 13:37:48 skapet sshd[59363]: Connection closed by 187.50.71.54 port 46496 [preauth]
Jul 8 13:46:43 skapet sshd[81970]: Invalid user content="text/html; from 187.50.71.54 port 29652
Jul 8 13:46:43 skapet sshd[81970]: Failed password for invalid user content="text/html; from 187.50.71.54 port 29652 ssh2
Jul 8 13:46:43 skapet sshd[81970]: Connection closed by 187.50.71.54 port 29652 [preauth]
Jul 8 13:55:37 skapet sshd[39952]: Invalid user <title>403 from 187.50.71.54 port 45706
Jul 8 13:55:37 skapet sshd[39952]: Failed password for invalid user <title>403 from 187.50.71.54 port 45706 ssh2
Jul 8 13:55:37 skapet sshd[39952]: Connection closed by 187.50.71.54 port 45706 [preauth]
Jul 8 14:04:33 skapet sshd[68947]: Invalid user Forbidden from 187.50.71.54 port 8465
Jul 8 14:04:33 skapet sshd[68947]: Failed password for invalid user Forbidden from 187.50.71.54 port 8465 ssh2
Jul 8 14:04:34 skapet sshd[68947]: Connection closed by 187.50.71.54 port 8465 [preauth]
Jul 8 14:13:29 skapet sshd[42324]: Invalid user is from 187.50.71.54 port 54112
Jul 8 14:13:29 skapet sshd[42324]: Failed password for invalid user is from 187.50.71.54 port 54112 ssh2
Jul 8 14:13:29 skapet sshd[42324]: Connection closed by 187.50.71.54 port 54112 [preauth]
Jul 8 14:22:20 skapet sshd[83537]: Invalid user <style from 187.50.71.54 port 41269
Jul 8 14:22:20 skapet sshd[83537]: Failed password for invalid user <style from 187.50.71.54 port 41269 ssh2
Jul 8 14:22:21 skapet sshd[83537]: Connection closed by 187.50.71.54 port 41269 [preauth]
Jul 8 14:31:06 skapet sshd[53939]: Invalid user body{margin from 187.50.71.54 port 10587
Jul 8 14:31:06 skapet sshd[53939]: Failed password for invalid user body{margin from 187.50.71.54 port 10587 ssh2
Jul 8 14:31:07 skapet sshd[53939]: Connection closed by 187.50.71.54 port 10587 [preauth]
Jul 8 14:40:08 skapet sshd[24320]: Connection closed by 187.50.71.54 port 58537 [preauth]
Jul 8 14:48:57 skapet sshd[97150]: Invalid user fieldset{padding from 187.50.71.54 port 11375
Jul 8 14:48:57 skapet sshd[97150]: Failed password for invalid user fieldset{padding from 187.50.71.54 port 11375 ssh2
Jul 8 14:48:58 skapet sshd[97150]: Connection closed by 187.50.71.54 port 11375 [preauth]
Jul 8 14:57:55 skapet sshd[38951]: Invalid user 10px from 187.50.71.54 port 43776
Jul 8 14:57:55 skapet sshd[38951]: Failed password for invalid user 10px from 187.50.71.54 port 43776 ssh2
Jul 8 14:57:55 skapet sshd[38951]: Connection closed by 187.50.71.54 port 43776 [preauth]
Jul 8 15:07:53 skapet sshd[72492]: Invalid user \^M from 187.50.71.54 port 58382
Jul 8 15:07:53 skapet sshd[72492]: Failed password for invalid user \^M from 187.50.71.54 port 58382 ssh2
Jul 8 15:07:53 skapet sshd[72492]: Failed password for invalid user \^M from 187.50.71.54 port 58382 ssh2
Jul 8 15:07:54 skapet sshd[72492]: Connection closed by 187.50.71.54 port 58382 [preauth]
Jul 8 15:17:05 skapet sshd[68616]: Invalid user 0 from 187.50.71.54 port 3795
Jul 8 15:17:05 skapet sshd[68616]: Failed password for invalid user 0 from 187.50.71.54 port 3795 ssh2
Jul 8 15:17:05 skapet sshd[68616]: Connection closed by 187.50.71.54 port 3795 [preauth]
Jul 8 15:26:09 skapet sshd[14795]: Connection closed by 187.50.71.54 port 59139 [preauth]
Jul 8 15:35:04 skapet sshd[8499]: Invalid user #header{width from 187.50.71.54 port 16030
Jul 8 15:35:04 skapet sshd[8499]: Failed password for invalid user #header{width from 187.50.71.54 port 16030 ssh2
Jul 8 15:44:12 skapet sshd[17233]: Invalid user 2% from 187.50.71.54 port 2551
Jul 8 15:44:12 skapet sshd[17233]: Failed password for invalid user 2% from 187.50.71.54 port 2551 ssh2
Jul 8 15:44:13 skapet sshd[17233]: Connection closed by 187.50.71.54 port 2551 [preauth]
Jul 8 15:53:05 skapet sshd[36380]: Invalid user 2%;font-family from 187.50.71.54 port 35369
Jul 8 15:53:05 skapet sshd[36380]: Failed password for invalid user 2%;font-family from 187.50.71.54 port 35369 ssh2
Jul 8 15:53:05 skapet sshd[36380]: Connection closed by 187.50.71.54 port 35369 [preauth]
Jul 8 16:02:05 skapet sshd[5384]: Invalid user Verdana, from 187.50.71.54 port 10140
Jul 8 16:02:05 skapet sshd[5384]: Failed password for invalid user Verdana, from 187.50.71.54 port 10140 ssh2
Jul 8 16:02:06 skapet sshd[5384]: Connection closed by 187.50.71.54 port 10140 [preauth]
Jul 8 16:11:27 skapet sshd[80640]: Invalid user #content{margin from 187.50.71.54 port 27941
Jul 8 16:11:27 skapet sshd[80640]: Failed password for invalid user #content{margin from 187.50.71.54 port 27941 ssh2
Jul 8 16:20:24 skapet sshd[71772]: Invalid user <div from 187.50.71.54 port 5467
Jul 8 16:20:24 skapet sshd[71772]: Failed password for invalid user <div from 187.50.71.54 port 5467 ssh2
Jul 8 16:20:25 skapet sshd[71772]: Connection closed by 187.50.71.54 port 5467 [preauth]
Jul 8 16:29:31 skapet sshd[22288]: Invalid user Error</h1></div>\^M from 187.50.71.54 port 2932
Jul 8 16:29:31 skapet sshd[22288]: Failed password for invalid user Error</h1></div>\^M from 187.50.71.54 port 2932 ssh2
Jul 8 16:29:31 skapet sshd[22288]: Connection closed by 187.50.71.54 port 2932 [preauth]
Jul 8 16:38:32 skapet sshd[64659]: Invalid user id="content">\^M from 187.50.71.54 port 44037
Jul 8 16:38:32 skapet sshd[64659]: Failed password for invalid user id="content">\^M from 187.50.71.54 port 44037 ssh2
Jul 8 16:38:33 skapet sshd[64659]: Connection closed by 187.50.71.54 port 44037 [preauth]
Jul 8 16:47:47 skapet sshd[60396]: Invalid user class="content-container"><fieldset>\^M from 187.50.71.54 port 50741
Jul 8 16:47:47 skapet sshd[60396]: Failed password for invalid user class="content-container"><fieldset>\^M from 187.50.71.54 port 50741 ssh2
Jul 8 16:56:46 skapet sshd[84720]: Invalid user Access from 187.50.71.54 port 56868
Jul 8 16:56:46 skapet sshd[84720]: Failed password for invalid user Access from 187.50.71.54 port 56868 ssh2
Jul 8 16:56:46 skapet sshd[84720]: Connection closed by 187.50.71.54 port 56868 [preauth]
Jul 8 17:05:47 skapet sshd[39792]: Invalid user denied.</h2>\^M from 187.50.71.54 port 55262
Jul 8 17:05:47 skapet sshd[39792]: Failed password for invalid user denied.</h2>\^M from 187.50.71.54 port 55262 ssh2
Jul 8 17:05:47 skapet sshd[39792]: Connection closed by 187.50.71.54 port 55262 [preauth]
Jul 8 17:14:42 skapet sshd[2165]: Invalid user do from 187.50.71.54 port 16650
Jul 8 17:14:43 skapet sshd[2165]: Failed password for invalid user do from 187.50.71.54 port 16650 ssh2
Jul 8 17:14:43 skapet sshd[2165]: Connection closed by 187.50.71.54 port 16650 [preauth]
Jul 8 17:23:39 skapet sshd[45938]: Invalid user have from 187.50.71.54 port 15855
Jul 8 17:23:39 skapet sshd[45938]: Failed password for invalid user have from 187.50.71.54 port 15855 ssh2
Jul 8 17:23:39 skapet sshd[45938]: Connection closed by 187.50.71.54 port 15855 [preauth]
Jul 8 17:32:35 skapet sshd[64595]: Invalid user to from 187.50.71.54 port 64962
Jul 8 17:32:35 skapet sshd[64595]: Failed password for invalid user to from 187.50.71.54 port 64962 ssh2
Jul 8 17:32:35 skapet sshd[64595]: Connection closed by 187.50.71.54 port 64962 [preauth]
Jul 8 17:41:30 skapet sshd[99157]: Invalid user this from 187.50.71.54 port 63460
Jul 8 17:41:30 skapet sshd[99157]: Failed password for invalid user this from 187.50.71.54 port 63460 ssh2
Jul 8 17:41:30 skapet sshd[99157]: Connection closed by 187.50.71.54 port 63460 [preauth]
Jul 8 17:50:27 skapet sshd[60500]: Invalid user or from 187.50.71.54 port 47364
Jul 8 17:50:27 skapet sshd[60500]: Failed password for invalid user or from 187.50.71.54 port 47364 ssh2
Jul 8 17:50:27 skapet sshd[60500]: Connection closed by 187.50.71.54 port 47364 [preauth]
Jul 8 17:59:26 skapet sshd[57379]: Invalid user using from 187.50.71.54 port 60084
Jul 8 17:59:26 skapet sshd[57379]: Failed password for invalid user using from 187.50.71.54 port 60084 ssh2
Jul 8 17:59:26 skapet sshd[57379]: Connection closed by 187.50.71.54 port 60084 [preauth]
Jul 8 18:08:22 skapet sshd[64892]: Invalid user credentials from 187.50.71.54 port 18558
Jul 8 18:08:22 skapet sshd[64892]: Failed password for invalid user credentials from 187.50.71.54 port 18558 ssh2
Jul 8 18:08:22 skapet sshd[64892]: Connection closed by 187.50.71.54 port 18558 [preauth]
Jul 8 18:17:19 skapet sshd[22377]: Invalid user you from 187.50.71.54 port 46996
Jul 8 18:17:19 skapet sshd[22377]: Failed password for invalid user you from 187.50.71.54 port 46996 ssh2
Jul 8 18:17:19 skapet sshd[22377]: Connection closed by 187.50.71.54 port 46996 [preauth]
Jul 8 18:24:50 skapet sshd[98670]: Connection closed by 187.50.71.54 port 40682 [preauth]
The other IP address offered up:
[Thu Dec 22 19:39:24] peter@skapet:~$ zgrep 175.143.54.193 /var/log/authlog.23.gz
Jul 8 16:10:42 skapet sshd[79062]: Connection closed by 175.143.54.193 port 61453 [preauth]
Jul 8 17:01:03 skapet sshd[28839]: Connection closed by 175.143.54.193 port 59520 [preauth]
Jul 8 17:49:47 skapet sshd[1472]: Connection closed by 175.143.54.193 port 39552 [preauth]
Jul 8 18:34:12 skapet sshd[58208]: Connection closed by 175.143.54.193 port 59520 [preauth]
Jul 8 19:19:12 skapet sshd[93151]: Connection closed by 175.143.54.193 port 6465 [preauth]
Jul 8 20:07:33 skapet sshd[84813]: Connection closed by 175.143.54.193 port 39552 [preauth]
Jul 8 20:59:08 skapet sshd[11083]: Invalid user content="text/html; from 175.143.54.193 port 26240
Jul 8 20:59:08 skapet sshd[11083]: Failed password for invalid user content="text/html; from 175.143.54.193 port 26240 ssh2
Jul 8 20:59:08 skapet sshd[11083]: Connection closed by 175.143.54.193 port 26240 [preauth]
Jul 8 21:47:54 skapet sshd[50641]: Connection closed by 175.143.54.193 port 59520 [preauth]
Jul 8 22:38:16 skapet sshd[33990]: Invalid user Forbidden from 175.143.54.193 port 64640
Jul 8 22:38:16 skapet sshd[33990]: Failed password for invalid user Forbidden from 175.143.54.193 port 64640 ssh2
Jul 8 22:38:16 skapet sshd[33990]: Connection closed by 175.143.54.193 port 64640 [preauth]
Jul 8 23:29:47 skapet sshd[84765]: Invalid user is from 175.143.54.193 port 49280
Jul 8 23:29:47 skapet sshd[84765]: Failed password for invalid user is from 175.143.54.193 port 49280 ssh2
Jul 8 23:29:47 skapet sshd[84765]: Connection closed by 175.143.54.193 port 49280 [preauth]
Jul 9 00:18:32 skapet sshd[75290]: Connection closed by 175.143.54.193 port 13952 [preauth]
Jul 9 01:07:11 skapet sshd[15889]: Connection closed by 175.143.54.193 port 16000 [preauth]
Jul 9 01:54:19 skapet sshd[3570]: Connection closed by 175.143.54.193 port 22144 [preauth]
Jul 9 02:38:44 skapet sshd[212]: Connection closed by 175.143.54.193 port 57472 [preauth]
Jul 9 03:24:00 skapet sshd[38938]: Connection closed by 175.143.54.193 port 50304 [preauth]
Jul 9 04:08:22 skapet sshd[60530]: Connection closed by 175.143.54.193 port 21481 [preauth]
Jul 9 04:55:14 skapet sshd[77880]: Connection closed by 175.143.54.193 port 40064 [preauth]
Jul 9 05:45:26 skapet sshd[65360]: Invalid user 0;color from 175.143.54.193 port 20096
Jul 9 05:45:26 skapet sshd[65360]: Failed password for invalid user 0;color from 175.143.54.193 port 20096 ssh2
Jul 9 05:45:26 skapet sshd[65360]: Connection closed by 175.143.54.193 port 20096 [preauth]
Jul 9 06:35:50 skapet sshd[49775]: Invalid user #header{width from 175.143.54.193 port 8320
Jul 9 06:35:50 skapet sshd[49775]: Failed password for invalid user #header{width from 175.143.54.193 port 8320 ssh2
Jul 9 07:24:21 skapet sshd[88261]: Invalid user 2% from 175.143.54.193 port 57472
Jul 9 07:24:21 skapet sshd[88261]: Failed password for invalid user 2% from 175.143.54.193 port 57472 ssh2
Jul 9 07:24:22 skapet sshd[88261]: Connection closed by 175.143.54.193 port 57472 [preauth]
Jul 9 08:16:55 skapet sshd[79482]: Invalid user 2%;font-family from 175.143.54.193 port 57984
Jul 9 08:16:55 skapet sshd[79482]: Failed password for invalid user 2%;font-family from 175.143.54.193 port 57984 ssh2
Jul 9 08:16:55 skapet sshd[79482]: Connection closed by 175.143.54.193 port 57984 [preauth]
Jul 9 09:05:58 skapet sshd[67909]: Connection closed by 175.143.54.193 port 38016 [preauth]
Jul 9 09:57:24 skapet sshd[51227]: Connection closed by 175.143.54.193 port 22144 [preauth]
Jul 9 10:47:35 skapet sshd[89081]: Invalid user The sequences become a little easier to read if we extract the user field from the
"Invalid user ..." messages:[Thu Dec 22 20:23:23] peter@skapet:~$ zgrep 187.50.71.54 /var/log/authlog.23.gz | grep Invalid | awk '{print $8}'
<!DOCTYPE
PUBLIC
XHTML
Strict//EN"
<html
<meta
content="text/html;
<title>403
Forbidden
is
<style
body{margin
fieldset{padding
10px
\^M
0
#header{width
2%
2%;font-family
Verdana,
#content{margin
<div
Error</h1></div>\^M
id="content">\^M
class="content-container"><fieldset>\^M
Access
denied.</h2>\^M
do
have
to
this
or
using
credentials
you
Now looking at what came from the the other IP address we get
[Fri Dec 23 00:51:28] peter@skapet:~$ zgrep 175.143.54.193 /var/log/authlog.23.gz | grep Invalid | awk '{print $8}'
content="text/html;
Forbidden
is
0;color
#header{width
2%
2%;font-family
<div
Error</h1></div>\^M
id="content">\^M
Access
have
using
you
Well, definitely HTML-ish, but no proper document start. Distinctly odd.
The two machines involved are apparently far apart geographically (one in Brazil and the other in Malaysia if the data from whois is anything to go by). It is of course possible that they're still connected somehow, perhaps compromised by the same group of cybercriminals and run by the same operators.
These operators then fatfingered some command or other, and their charges started pushing bizarre streams of HTML at unsuspecting SSH servers in the great elsewhere of the Internet. What they were actually trying to achieve I suspect we'll never know, but HTML was involved.
HTML is also part of the problem in one of the other bizarre phenomena I find at semi-random intervals in the SSH authentication logs.
Here are some samples from the same preserved log file:
[Thu Dec 22 20:16:36] peter@skapet:~$ zgrep "Bad protocol" /var/log/authlog.23.gz
Jul 6 19:03:17 skapet sshd[28549]: Bad protocol version identification 'GET / HTTP/1.1' from 107.179.242.130 port 36147
Jul 8 16:14:07 skapet sshd[89181]: Bad protocol version identification 'GET / HTTP/1.1' from 204.9.214.98 port 49385
Jul 8 20:15:15 skapet sshd[28469]: Bad protocol version identification 'GET / HTTP/1.1' from 189.90.20.100 port 55039
Jul 9 10:56:40 skapet sshd[67430]: Bad protocol version identification 'GET / HTTP/1.1' from 195.88.41.10 port 52504Again, it's not clear what these operations were attempting to do, but to me this looks like they were expecting to find either a web server or perhaps a web proxy listening on port 22.
Just like the first kind of web-to-ssh stupidity this won't actually get the requester anything and you can safely ignore both kinds of activity if you see traces of them in your own logs.
That is, if you have seen something similar in your own logs and you would care to share, I would like to hear from you, via email or in the comments below.
Even more so if you have any input on the question 'what were these clowns trying to achieve?'.
If the log analyses or related activities turn up any useful insights, you won't need to go far from here to check the results.
Good night and good luck.
Update 2016-12-23: Among the various comments that the initial version of this piece generated, two stand out as particularly useful.
The first came in a Facebook comment on my post about the story there, from my former colleague Egil Mõller, who wrote:
"Maybe they read password guesses to try from a central REST service, but that service has somehow broken, and is serving a default error message (which, since it's REST, is most likely in html)?"
The other came from OpenBSD developer Stuart Henderson, who tweeted:
@pitrh the GETs aren't all that odd - could easily be scanning via proxy. I see similar on SMTP too. Not sure about the html though.— Stuart Henderson (@sthen_) December 23, 2016
If the link Twitter gave me doesn't work, here's the plaintext of Stuart's tweet:
"@pitrh the GETs aren't all that odd - could easily be scanning via proxy. I see similar on SMTP too. Not sure about the html though.
1:00 PM - 23 Dec 2016 "
I must admit I had not noticed any GETs in any SMTP related logs on my systems, but now I an honor bound to check. Stuart has given me a task, and I must finish it separately.
Also, I really like Egil's input here, because it fits so well with the data we have. In the meantime I discovered more data from a second host. Unfortunately the actual logs had been rotated out of existence, but it was still possible to piece together data on failed logins from the summaries logsentry sends me.
It appears that one machine apparently located in Hong Kong that had been trying a few logins earlier that month, with no apparent succcess, started spitting HTML at roughly the same time the other two did.
Here is all the activity in early July 2016 from that host:
Jul 4 01:08:19 delilah sshd[13635]: Failed password for invalid user bob from 218.188.213.5 port 38728 ssh2
Jul 4 01:18:28 delilah sshd[26798]: Failed password for root from 218.188.213.5 port 9224 ssh2
Jul 4 01:18:29 delilah sshd[26798]: Failed password for root from 218.188.213.5 port 9224 ssh2
Jul 4 01:27:13 delilah sshd[24543]: Failed password for invalid user ts from 218.188.213.5 port 9224 ssh2
Jul 4 01:35:54 delilah sshd[16906]: Failed password for invalid user pi from 218.188.213.5 port 9224 ssh2
Jul 7 18:30:48 skapet sshd[89165]: Failed password for invalid user admin from 218.188.213.5 port 43498 ssh2
Jul 7 18:40:33 skapet sshd[35698]: Failed password for invalid user lp from 218.188.213.5 port 9224 ssh2
Jul 7 18:50:21 skapet sshd[21112]: Failed password for root from 218.188.213.5 port 9224 ssh2
Jul 9 11:33:25 delilah sshd[10234]: Failed password for invalid user <!DOCTYPE from 218.188.213.5 port 46959 ssh2
Jul 9 11:42:28 delilah sshd[10230]: Failed password for invalid user PUBLIC from 218.188.213.5 port 9224 ssh2
Jul 9 11:51:36 delilah sshd[25489]: Failed password for invalid user XHTML from 218.188.213.5 port 9224 ssh2
Jul 9 12:09:47 delilah sshd[28023]: Failed password for invalid user <html from 218.188.213.5 port 9224 ssh2
Jul 9 12:18:51 delilah sshd[9873]: Failed password for invalid user <meta from 218.188.213.5 port 9224 ssh2
Jul 9 12:28:01 delilah sshd[13890]: Failed password for invalid user content="text/html; from 218.188.213.5 port 9224 ssh2
Jul 9 12:37:01 delilah sshd[10856]: Failed password for invalid user <title>403 from 218.188.213.5 port 9224 ssh2
Jul 9 12:46:04 delilah sshd[19947]: Failed password for invalid user Forbidden from 218.188.213.5 port 9224 ssh2
Jul 9 13:04:32 delilah sshd[22444]: Failed password for invalid user <style from 218.188.213.5 port 9224 ssh2
Jul 9 13:13:14 delilah sshd[15268]: Failed password for invalid user body{margin from 218.188.213.5 port 9224 ssh2
Jul 9 13:22:31 delilah sshd[19611]: Failed password for invalid user Helvetica, from 218.188.213.5 port 9224 ssh2
Jul 9 13:31:35 delilah sshd[15652]: Failed password for invalid user fieldset{padding from 218.188.213.5 port 59101 ssh2
Jul 9 13:40:39 delilah sshd[15607]: Failed password for invalid user 10px from 218.188.213.5 port 9224 ssh2
Jul 9 13:51:00 delilah sshd[18900]: Failed password for invalid user \^M from 218.188.213.5 port 9224 ssh2
Jul 9 13:51:00 delilah sshd[18900]: Failed password for invalid user \^M from 218.188.213.5 port 9224 ssh2
Jul 9 14:00:09 delilah sshd[24794]: Failed password for invalid user 0 from 218.188.213.5 port 9224 ssh2
Jul 9 14:18:23 delilah sshd[22897]: Failed password for invalid user #header{width from 218.188.213.5 port 9224 ssh2
Jul 9 14:27:11 delilah sshd[12713]: Failed password for invalid user 2% from 218.188.213.5 port 9224 ssh2
Jul 9 14:35:56 delilah sshd[32320]: Failed password for invalid user 2%;font-family from 218.188.213.5 port 9224 ssh2
Jul 9 14:44:46 delilah sshd[30676]: Failed password for invalid user Verdana, from 218.188.213.5 port 9224 ssh2
Jul 9 14:53:46 delilah sshd[12799]: Failed password for invalid user #content{margin from 218.188.213.5 port 9224 ssh2
Jul 9 15:02:29 delilah sshd[19535]: Failed password for invalid user <div from 218.188.213.5 port 9224 ssh2
Jul 9 15:11:17 delilah sshd[6404]: Failed password for invalid user Error</h1></div>\^M from 218.188.213.5 port 9224 ssh2
Jul 9 15:20:08 delilah sshd[2837]: Failed password for invalid user id="content">\^M from 218.188.213.5 port 9224 ssh2
Jul 9 15:29:13 delilah sshd[7831]: Failed password for invalid user class="content-container"><fieldset>\^M from 218.188.213.5 port 9224 ssh2
Jul 9 15:38:05 delilah sshd[12172]: Failed password for invalid user Access from 218.188.213.5 port 9224 ssh2
Jul 9 15:46:56 delilah sshd[23460]: Failed password for invalid user denied.</h2>\^M from 218.188.213.5 port 9224 ssh2
Jul 9 15:55:48 delilah sshd[19891]: Failed password for invalid user do from 218.188.213.5 port 9224 ssh2
Jul 9 16:13:29 delilah sshd[5999]: Failed password for invalid user to from 218.188.213.5 port 9224 ssh2
Jul 9 16:22:20 delilah sshd[17535]: Failed password for invalid user this from 218.188.213.5 port 9224 ssh2
Jul 9 16:31:11 delilah sshd[8428]: Failed password for invalid user or from 218.188.213.5 port 9224 ssh2
Jul 9 16:40:00 delilah sshd[2522]: Failed password for invalid user using from 218.188.213.5 port 9224 ssh2
Jul 9 16:48:52 delilah sshd[15034]: Failed password for invalid user credentials from 218.188.213.5 port 9224 ssh2
Jul 9 16:57:43 delilah sshd[14515]: Failed password for invalid user you from 218.188.213.5 port 53073 ssh2If we again extract only the user name field, we get:
bob
ts
pi
admin
lp
<!DOCTYPE
PUBLIC
XHTML
<html
<meta
content="text/html;
<title>403
Forbidden
<style
body{margin
Helvetica,
fieldset{padding
10px
\^M
\^M
0
#header{width
2%
2%;font-family
Verdana,
#content{margin
<div
Error</h1></div>\^M
id="content">\^M
class="content-container"><fieldset>\^M
Access
denied.</h2>\^M
do
to
this
or
using
credentials
youFrom this we see that this host was already busy poking us at long intervals with 'likely' user names, then suddenly started spewing HTML instead of likely user names in roughly the same time frame as the other two.
Seen from my perch here, this serves to validate Egil's suggestion that a common back end system started misbehaving is what caused the odd activity we're seeing.
And the apparent coordination brings to mind the Hail Mary Cloud incidents that we reported on earlier.
I suppose further digging in the logs is warranted.
If you would like to join me in the hunt for more of this, please let me know.
Posts
Very odd.
ReplyDeleteI've been seeing weird being sent to TCP/587 (SMTP Submission port):
Dec 24 12:45:41 postfix/submission/smtpd[27730]: warning: non-SMTP command from unknown[113.240.250.156]: GET / HTTP/1.0
Dec 24 12:46:29 postfix/submission/smtpd[6157]: warning: non-SMTP command from unknown[113.240.250.156]: Via: SIP/2.0/TCP nm;branch=foo
Dec 24 12:46:29 postfix/submission/smtpd[92587]: warning: non-SMTP command from unknown[113.240.250.156]: GET /nice%20ports%2C/Tri%6Eity.txt%2ebak HTTP/1.0
We have a lot of these on two our machines for more than a half year. Almost every day.
ReplyDelete# zcat /var/log/auth.log.*.bz2 | grep 'GET / HTTP'
Dec 19 07:20:11 MyHost sshd[14793]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 59817
Dec 19 17:25:12 MyHost sshd[76582]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 37817
Dec 20 07:15:12 MyHost sshd[61303]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 36558
Dec 20 17:15:14 MyHost sshd[23506]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 52326
Dec 21 07:20:12 MyHost sshd[10057]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 33720
Dec 21 17:25:12 MyHost sshd[76284]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 32966
Dec 13 07:25:17 MyHost sshd[26646]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 62313
Dec 13 17:30:13 MyHost sshd[88136]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 60471
Dec 14 07:15:11 MyHost sshd[71605]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 60037
Dec 14 17:20:11 MyHost sshd[37482]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 37506
Dec 15 07:20:11 MyHost sshd[20664]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 58248
Dec 15 17:25:11 MyHost sshd[84836]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 43397
Dec 16 07:50:11 MyHost sshd[86760]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 49417
Dec 16 17:55:12 MyHost sshd[47666]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 51202
Dec 8 17:55:12 MyHost sshd[60806]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 58734
Dec 9 07:15:12 MyHost sshd[42073]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 58881
Dec 9 17:20:11 MyHost sshd[3626]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 54401
Dec 12 07:45:11 MyHost sshd[82174]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 58818
Dec 12 17:45:13 MyHost sshd[43618]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 33835
Dec 5 07:20:12 MyHost sshd[51490]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 50872
Dec 5 17:25:12 MyHost sshd[12979]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 37062
Dec 6 07:15:11 MyHost sshd[96566]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 44851
Dec 6 17:20:11 MyHost sshd[58017]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 34036
Dec 7 08:10:11 MyHost sshd[47704]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 53449
Dec 7 18:15:11 MyHost sshd[14013]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 55395
Dec 8 07:50:12 MyHost sshd[96518]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 61339
Nov 30 07:05:14 MyHost sshd[13793]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 54288
Nov 30 17:10:12 MyHost sshd[79548]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 57325
Dec 2 07:20:12 MyHost sshd[15070]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 61535
Dec 2 17:25:12 MyHost sshd[76530]: Bad protocol version identification 'GET / HTTP/1.1' from 46.235.158.196 port 61352
It's simpler than you think :-)
ReplyDeleteSSH mass scans are more than widespread. On one of my Internet facing machines, there are 20k bruteforce attempts each month.
In your case it looks like an attacker used some static logins plus a login list, which was downloaded from location, where it's already gone (and never checked for contents).
Another way is that some lists were concatenated, where last one was 403'd. Such list given as a parameter to some bruteforce bot can cause identical behaviour.
Yes, it does look like their backend was broken for a while, and these traces show coordination of several hosts.
DeleteIt would of course be useful to see auth logs data for roughly the same period from elsewhere, in order to identify more hosts in the coordinated set, but I suspect most people do not keep their logs around that long.