Sunday, April 3, 2022

Spammers in the Public Cloud, Protected by SPF; Intensified Password Groping Still Ongoing; Spamware Hawked to Spamtraps

This past week was truly one for the blooper reel. A public cloud service provider let the great unwashed into the address ranges published as safe mailers via their SPF records, with hilarious if rather predictable results. Next up, we find an intensive advertising campaign for spamware aimed at our imaginary friends. And the password guessing aimed at an ever-expanding dictionary of non-existing users continues.

To the rest of the world, is known variously as a honeypot, a source of various kinds of blocklists or a frequent target of domain joejobs that contribute to the ever-expanding list of imaginary friends also know as spamtraps.

To me and a very small set of other people, it's home on the net, providing a set of services we need fairly painlessly on an OpenBSD platform that rarely requires much work besides the odd pkg_add -u -D snap followed by sysupgrade -s (yes, we jump from snapshot to snapshot on this one).

Then the past week served us with three separate events that, while actually harmless to our side, together serve to show that a certain subset of humans would perhaps be better diverted to activities that do not involve computers.

Note: This piece is also available, with more basic formatting but with no trackers, here.

A Public Cloud Provider Let Spammers Set Up Within Their Published SPF Ranges

The first event started on Tuesday. While looking for something else entirely in my mail server logs, I noticed an unusually high number of delivery attempts to what definitely looked like spamtraps reaching the actual mail server. The earliest entry was from that morning:

2022-03-29 06:28:31 [] X=TLS1.2:ADH-AES256-GCM-SHA384:256 CV=no F=<> rejected RCPT <>: Unknown user

There were several thousand entries of that type (full log here, extracted source IP addresses here). My initial impulse was of course to check the logs to see how they got past spamd in the first place. Oddly, I found no trace of any activity involving spamd and a random sampling of those IP addresses. That would in turn indicate that they had been pass-listed, most likely by being included in the permanent nospamd pass list, which we generate here mainly based on the published SPF records of domains we need to communicate with.

Again taking a random subset of the extracted IP addresses and using whois identified the IP address range owner as a large cloud services provider that among other things also provide hosted or hybrid on premise plus hosted email service. This in turn means that domains that use those services also include large segments of that provider's IP address ranges in their SPF records. Not quite knowing what to do I tweeted,

In addition to tweeting and looking for feedback (which was not huge but dominated by Y answers), I notified the relevant abuse@ address by mail, including links to the log data and the IP addresses.

I also tweaked the log reader hinted at in this earlier piece so any attempt at delivering mail from that domain in the future will put the sending IP address safely away both in the spamd blocklist and in the safety of a table that is subject to block drop quick from for six weeks after the activity stops, and exported to downloadable blocklists as described in the article I referenced earlier.

The abuse@ handlers at the company I am not naming explicitly here (yes, there's a unix command you can use to find out who if it matters to you) were quite responsive and said that the activity seemed to be coming from their public cloud section, and yes they were forwarding my data to their internal CERT. As a followup I suggested to them that using our slowly expanding list of spamtraps in their outbound filtering might be a good idea if they intend to offer SMTP-for-hire in the future.

What seems to have happened is that the miscreants here set up using a range of the provider's services including domain registration, DNS hosting, and judging from the consistent use of root@ as the sender addess, set up some number of Linux virtual machines to do the spamming.

Before the activity stopped later in the week, we identified two more campaigns that fit the pattern. The data can be found here: log entries and IP addresses for the second wave, log entries and IP addresses for the third. Each of the campaigns appear to have stopped shortly after their domains were de-registered. I never saw the contents of the messages since not a single one appears to have inboxed here.

The episode has a few teachable items: First, that some subset of our list of spamtraps is indeed incorporated in the address lists used by gullible spammers and their customers, and second, that if you run a public cloud service, you need to pay attention to what your customers do and be wary of letting them use IP address ranges that have been announced as being really safe to accept mail from.

I notified the cloud provider that I would be writing an article about the events and asked them for any and all useful input they could provide. No such information has surfaced by the time of writing. If any useful information turns up from them after publication, I will of course update this piece accordingly.

A Separate Set of Spammers Market Their Spamming Software, Intensively Targeting Our Imaginary Friends

While the public cloud spammers thing was developing, I noticed another campaign that was actively targeting our spamtraps.

Sent from as far as I could tell only three IP addresses and with a total of 58 different subject lines all about spamming tools. It is possible that the campaigns did not target our spamtraps exclusively in our domains, but the log archive (1.5M compressed but expands to 40M raw) serves as testament that our imaginary friends are definitely targeted by some subset of the online marketing community, and they are pouring resources into doing that, one byte per second.

And once again, I have not seen the actual contents of the messages beyond what turns up in the logs after greytrapping kicks in. Not one of those messages found its way to an actual mailbox here.

SSH Password Groping Still High, But Bursty

As noted in a previous article, SSH password guessing activity went up significantly in the days leading up to the Russian invasion of Ukraine i February.

In addition to the data referenced directly in the article, the archived logs and the summaries of numbers of attempts per day (March and April so far) the number of attempts per day are consistently on a higher level than before the Ukraine war started, with a new higher intensity episode ongoing as I type.

One interesting feature of the password guessing attempts during the last few days is that they feature a much larger number of new user names attempted than usual. This means that the list of spamtraps here is now growing at the highest rate since the episode involving what was likely one or more phishing campaigns targeting Chinese users during early 2019 as mentioned in my 2019 summed up piece published at the end of that year.

If you have further data on these or similar incidents that you are able to share or if you want to look further into these and similar incidents, please let me know.

If you find any errors in the material I publish or disagree with my sentiments, or if you find this article interesting, useful or annoying, please let me know, either in comments or via email.

Sunday, February 27, 2022

Predicting developments in real world conflict from patterns of failed logins

Anecdotal evidence indicates that it may be possible to predict developments in real world conflicts from certain indicators in cybercrime traffic.

Is it possible to glean useful information about international developments or even predict real world attacks from the activity that we record in the logs of Internet-facing systems? 

Note: A trackerless (other than my webserver log) version of the article is now available on my web site.

Looking at data I collect for other, quite pragmatic, reasons I see a clear correlation between the run-up to the Russian invasion of Ukraine earlier this month and the password guessing activity targeting non-classified systems in my care.

I'll be backing up that assertion with data later, but first, a bit of background. 

As returning readers already know, I have been running Internet facing systems for a select group of friends and family for decades. In the late noughties I noticed a pattern of slow, distributed password guessing that I dubbed The Hail Mary Cloud, summed up in the summary article linked here and links therein. The data I collect from those failed logins make it into a set of blocklists, along with data from a few other sources. And yes, this is also one source of new spamtraps, as noted in the blocklists article.

A few years after the original Hail Mary Cloud events, in January 2016, I started seeing Hail Mary-like activity again, and started collecting data (available in the raw here), but failing to see any new patterns worth writing about, never started a new article based on the data. Until now, that is.

The table here has the totals for number of attempts per month since then (the table as a .csv file is available here):

Failed SSH login attempts per month, 2016 - 2022









































































































It is worth noting that these attempts in almost all cases lead to getting the host blocked and published in the blocklist, some immediately if they come in very quickly, others after a short time, such as if they try logging on as root, admin or a few other obvious ones.

Anyway, the data shows that the typical number of attempts per month is at least in five digit numbers, with seasonal variations. But at the end of 2021 we saw a marked decrease in activity from October 2021 onwards, only to see a sharp rise again in February of 2022. 

The development becomes even more striking if we look at the per day data for February, complete up to about mid-day CET on the 27th:

Failed SSH login attempts February 2022

Feb  1: 66
Feb  2: 13
Feb  3: 50
Feb  4: 31
Feb  5: 35
Feb  6: 85
Feb  7: 13
Feb  8: 70
Feb  9: 28
Feb 10: 13
Feb 11: 32
Feb 12: 13
Feb 13: 48
Feb 14: 28
Feb 15: 30
Feb 16: 337
Feb 17: 2006
Feb 18: 1906
Feb 19: 1608
Feb 20: 2113
Feb 21: 2207
Feb 22: 2424
Feb 23: 1978
Feb 24: 2976
Feb 25: 3044
Feb 26: 2071
Feb 27: 992

The developments stand out even clearer when presented as a graph:

Again, it is worth noting that the data is complete only up to mid-day on February 27th. 

But we see clearly that the number of attempts (effectively attempts by new hosts, quickly blocked) increased slightly on the 15th, only to essentially explode into levels equal to or slightly higher than the previous normal roughly a week before the Russian invasion of Ukraine started in the early hours of Febuary 24.

On Febuary 15th, activity that was obviously preparations for an attack on Ukraine had been underway for weeks, and it was obvious to me that the activity was part of a campaign to acquire new nodes for botnets to be used for any kind of cybercrime or disinformation bot that can be run from things that run a SSH service, typically Unix-ish systems. 

Even before the current Ukraine episode I had a vague feeling that similar correlations would be visible in the data collected around earlier international events, but by Febuary 17th I was fairly confident that the attack was imminent.

Then on the day of the attack, I tweeted:
I will of course be monitoring the situation for developments and may follow up with further analysis.

If you do not want to wait for those write-ups, you are welcome to go ahead and analyse the data (as long as any reuse or resulting analysis credits the source) -- the archive is here while the data for the month so far can be found here (note: that link always points to freshest collection for the current month).

If you too collect logs on similar activity and you are able to share data or analysis, I would like to hear from you.

Update 2022-03-01: The month ended, and the final data for February 2022 are in. The counts for February 27 and 28 were 1509 and 1138 respectively, continuing the decline in new hosts attempting that we had been seeing over the previous few days. The total number of failed attempts for that month ended up at 25873. 

The final month graph looks like this:

The full data for February can be found in this file as well as in the current year's .zip archive in the archive directory.

Update 2022-03-05 11:50 CET: We may be seeing another uptick in expectation of a propaganda-needing event. 

The numbers for failed ssh logon attempts in March so far are:

Mar  595
Mar  1003
Mar  699
Mar  657
Mar  727

Notice that the the number of attempts before noon today already exceeds the total for the previous day. We may be seeing the preparation for another offensive or other event in need of propaganda or other cybercrime support. Stay tuned for updates.

Update 2022-03-09: The month to date daily numbers for failed attempts are now available, updated at quasi-random intervals a few times a day as ssh_attempts_month_to_date.txt

Update 2022-04-02: As can be seen from the month to date numbers (now covering April 2022) we are now seeing another busy day on the password guessing front with almost 2000 attempts already by only a little past noon CEST. For reference, the numbers for March can be found along with previous data in the archive directory.