“Name and Shame”, or socially responsible use of your log data

Your logs contain an ever-growing mass of data on spammers. How about making an effort to make that data useful to others?

Those of us who run email services know, from sometimes painful experience, what it takes to ensure that the minimum possible amount of unwanted advertising and scams that may turn out to be security hazards reaches our users' inboxes.

Email: This should have been very simple
Handling email should really be quite simple: The server is configured to know what domains it receives mail for and what users actually exist in those domains. When a machine makes contact and indicates that it intends to deliver email, the server check if the recipient is a valid user. If the recipient is valid, the message is received and put in the relevant user's mailbox. Otherwise, a message about a failed delivery and optionally the reason for the failure is sent to the user specified as the sender.

If they were all honest people
In each part of the process, the underlying premise is that the communicating parners offer each other correct information. Frequently that is the case, and we have legitimate communications between partners with a valid reason for contacting each other. Unfortunately there are other cases where the implicit trust is abused, such as when email messages are sent with a sender address other than the real one, quite likely a made-up one in a domain that belongs to other people. Some of us occasionally receive delivery failure messages for messages we verfiably did not send[1]. If we take the time to study the contents of those messages, in almost all cases we will find that the messages are spam, sometimes the scamming kind and perhaps part of an attempt to take control of the recipient's computer or steal sensitive data.

What do the ones in charge do, then?
If you ask a typical system administrator what measures are in effect to thwart attempts at delivering unwanted or malicious messages to their users, you will most likely get a description that says, essentially, the messages are filtered through systems that inspect message contents. If the message does not contain anything known to be bad (known spam or malware) or something sufficiently similar to a known bad, the message is delivered to the user's mailbox. If the system determines that the message contents indicates it should not be delivered, the messages is thrown away undelivered, and some system administrators will tell you that the system also sends a message about the decision not to deliver the message to the stated sender address.

Large parts or this is likely part of moderately educated users' passive knowledge, and most of us are likely to accept that content filtering is all we can do to keep dubious or downright criminal elements out of our working environment. For the individual end user, only minor adjustments to this are likely to be possible.

Measures based on observed behavior
But those of us who actually run the service also have the opportunity to study the automatically generated log data from our systems and use spammers' (that is, senders of all types of unwanted mail, including malware) behavior patterns to remove most of the unwanted traffic before actual message content is known. In order to do that, it is necessary to go to a more basic level of network traffic and study sender behavior on the network level.

One of the simpler forms of behavior based measures emerged in the form of a technique called greylisting in 2003. The technique is based on a slightly pedantic and rather creative interpretation of established standards. The Internet protocol for email transfer, SMTP (the Simple Mail Transfer Protocol) allows servers that experience temporary problems that make it impossible to receive mail to report a specific 'temporary local problem' status code to correspondents trying to deliver mail. Correctly configured senders will interpret and act on the status code and delay delivery for a short time. In most circumstances, the delivery will succeed within a short time. It is worth noting that this part of the standard was formulated to help the mail service's reliability. At most times, the retries happen without alerting the person who wrote and sent the message. The messages generally reach their destination eventually.

Lists of grey and black, little white lies
Greylisting works like this: the server reports a temporary local problem to all attempts at delivery from machines the server has not exchanged mail with earlier. Experience shows that the pre-experiment hypothesis was mainly correct: Essentially all machines that try to deliver valid email are configured to check return codes and act on them, while almost all spam senders dump as many messages as possible, and never check any return codes. This means that somewhere in the eighty to high nineties percentage of all spam volume is discarded at the first delivery attempt (before any content filtering), while legitimate email reaches its intended recipients, occasionally with delayed delivery of the initial message from a new correspondent.

One other behavior based technique that predates greylisting is the use of 'blacklists' - lists of machines that have been classified as spam senders - and rejecting mail from machines on such lists. Some groups eventually started experimenting with 'tarpits', a technique that essentially means your end of the communication moves along very slowly. A much cited example is the spamd program, released as a part of the free operating system OpenBSD in May of 2003. The program's main purpose at the time was to answer email traffic from blacklisted hosts one byte per second, never leaving a blacklisted host any real chance of delivering messages.

The combination of blacklists and greylisting proved to work very well, but the quest for even more effective measures continued. Yet again, the next logical step grew out of observing spammer behavior. We saw earlier that spammers do not bother to check whether individual messages are in fact delivered.

Laying traps and bait
By early 2005, these observations lead to a theory that was soon proved useful: If we have one or more addresses in our own domains the are certain to never receive any valid mail, we can be almost a hundred percent certain that any mail addressed to those addresses is spam. The addresses are spamtraps. Any machines that try to deliver spam to those addresses are placed in a local blacklist, and we keep them busy by answering their traffic at a rate of one byte per second. The machines stay on the blacklist for 24 hours unless otherwise specified.

The new technique, dubbed greytrapping was launched as part of the improved spamd in OpenBSD 3.8, released May 2005. In early 2006, Bob Beck, one of the main spamd developers announced that his greytrapping hosts at the University of Alberta generates a downloadable blacklist based on the greyptrap data, updated once per hour, ready for inclusion in spamd setups elsewhere. This is obviously useful. Machines that try to deliver mail to addresses that were never deliverable most likely do not have any valid mail to deliver, and it we are doing society at large a favor by delaying their deliveries and wasting their time to the maximum extent possible.

It is worth mentioning that during the period we have used the University of Alberta blacklist at our site, it has contained a minimum of twenty-some thousand IP addresses, and during some busy periods have reached almost two hundred thousand.

You can help, too
Fortunately you do not need to be a core developer to be able to contribute. The exact same tools Bob Beck uses to generate his blacklist is available to everybody else as part of OpenBSD, and they are actually not very hard to use productively.

Here at BSDdly.net and associated domains we saw during the (Northern hemisphere) summer of 2007 a marked increase in email sent to addresses that have never actually existed in our domains. This was clearly a case of somebody, one or more groups, making up or generating sender addresses to avoid seeing any reactions to the spam they were sending. This in turn lead to us starting an experiment that is still ongoing. We record invalid addresses in our own domains as they turn up in our logs. From these addresses we pick the really improbable ones, put them in our local spamtrap list and publish the list on a specific web page on our server[2].

Experience shows that it it takes a very short time for the addresses we put on the web page to turn up as target addresses for spam. This means that we have succeeded in feeding the spammers data that makes it easier for us to stop their attempts, and frequently we make spam senders use significant amounts of time communicating with our machines with no chance of actually achieving anything. The number of spamtrap addresses has reached fifteen thousand, and we have at times observed groups of machines that spend weeks working through the whole list, with average time spent per unsuccessful delivery attempt clocked at roughly seven minutes.

As a byproduct of the active spammer trapping we started exporting our own list of machines that had been trapped via the spamtrap addresses during the last 24 hours and making the list available for download. This list's existence has only been announced via the spamtrap addresses web page and a few blog posts, but we see that it's retrieved, most likely automatically, at intervals and is apparently used by other sites in their systems.

At this point we have established that it is possible to create a system that makes it very unlikely that spam actually makes it through to users, while at the same time it is quite unlikely that legitimate mail is adversely affected. In other words, we have the cyberspace equivalent of good fences around our property, but spammers are still out there and may create serious probles for those who are without adequate protection.

Collecting evidence, or at least seek clarity
We would have loved to see law enforcement take the spammer problem seriously. This is not just because the spam that reaches its targets is irritating, but rather because almost all spam is sent via equipment that spammers use without the legal owners' consent. We would have liked to see resources allocated in proportion to the criminal activity the spam represents. We would have liked to help, but it might seem that we would not have usable evidence available due to the fact that we do not actually receive the messages the spammers try to deliver. On the other hand, we have at all times a list of machines that have tried to deliver spam, identified with an almost hundred percent certainty based on the spammer trapping addresses. In addition, our systems routinely produce logs of all activity, with the level of detail we set ourselves. This means that it is possible to search our logs for the IP addresses that have tried to deliver spam to our systems during the last 24 hours, and get a summary of what those machines have done.

A search of this kind typically yields a result like this:

Aug 10 02:34:29 skapet spamd[13548]: 190.20.132.16: connected (4/3)
Aug 10 02:34:41 skapet spamd[13548]: (GREY) 190.20.132.16: <kristie@iland.net> -> <asasaskosmicki@bsdly.net>
Aug 10 02:34:41 skapet spamd[13548]: 190.20.132.16: disconnected after 12 seconds.
Aug 10 03:41:42 skapet spamd[13548]: 190.20.132.16: connected (14/13), lists: spamd-greytrap
Aug 10 03:42:23 skapet spamd[13548]: 190.20.132.16: disconnected after 41 seconds. lists: spamd-greytrap
Aug 10 06:30:35 skapet spamd[13548]: 190.20.132.16: connected (23/22), lists: spamd-greytrap becks
Aug 10 06:31:16 skapet spamd[13548]: 190.20.132.16: disconnected after 41 seconds. lists: spamd-greytrap becks

The first line here states that 190.20.132.16 contacts our system at 02:34:29 AM on August tenth, as the fourth active SMTP connection, three blacklisted. A few seconds later it appears that this is an attempt at delivering a message to the address asasaskosmicki@bsdly.net. That address was already one of our spamtraps, most likely one that was harvested from our logs and was originally made up somewhere else. After 12 seconds, the machine disconnects. The attempted delivery to a spamtrap address means that the machine is added to our local spamd-greytrap blacklist, as indicated in the entry for the next attempt about one hour later. This second attempt lasts for 41 seconds. The third try in our log material happens just after 06:30, and the addition of the list name becks indicates that in the meantime has tried to deliver to one of Bob Beck's spammer trap addresses and has entered that blacklist, too.

Unfortunately, it is unlikely that logs of this kind are sufficient as evidence for criminal prosecution purposes, but the data may be of some use to those who have an interest in keeping machines in their care from sending spam.

“Name And Shame“, or just being neighborly?
After some discussions with colleagues I decided in early August 2008 to generate daily reports of the activities of machines that had made it into the local blacklist on bsdly.net and publish the results. If all we have is the fact that a machine has entered a blacklist as an IP address (such as 24.165.4.190), and there is no supporting material, it is fairly easy for whoever is in charge of that address range to just ignore the entry as an unsupported allegation. We hope that when whoever is responsible for the network containing 24.165.4.190 sees a sequence like this,

Host 24.165.4.190:
Aug 10 02:57:40 skapet spamd[13548]: 24.165.4.190: connected (9/8)
Aug 10 02:57:54 skapet spamd[13548]: (GREY) 24.165.4.190: <hand@itnmiami.com> -> <kimberlee.ledet@ehtrib.org>
Aug 10 02:57:55 skapet spamd[13548]: (GREY) 24.165.4.190: <hand@itnmiami.com> -> <kimberliereffett@ehtrib.org>
Aug 10 02:57:56 skapet spamd[13548]: 24.165.4.190: disconnected after 16 seconds.
Aug 10 02:58:16 skapet spamd[13548]: 24.165.4.190: connected (8/6)
Aug 10 02:58:30 skapet spamd[13548]: (GREY) 24.165.4.190: <brunson@jebconet.com> -> <kimberlee.ledet@ehtrib.org>
Aug 10 02:58:31 skapet spamd[13548]: (GREY) 24.165.4.190: <brunson@jebconet.com> -> <kimberliereffett@ehtrib.org>
Aug 10 02:58:32 skapet spamd[13548]: 24.165.4.190: disconnected after 16 seconds.
Aug 10 02:58:39 skapet spamd[13548]: 24.165.4.190: connected (7/6), lists: spamd-greytrap
Aug 10 03:02:24 skapet spamd[13548]: (BLACK) 24.165.4.190: <aarnq@abtinc.com> -> <kimberlee.ledet@ehtrib.org>
Aug 10 03:03:17 skapet spamd[13548]: (BLACK) 24.165.4.190: <aarnq@abtinc.com> -> <kimberliereffett@ehtrib.org>
Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: From: "Preston Amos" <aarnq@abtinc.com>
Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: To: kimberlee.ledet@ehtrib.org
Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: Subject: Wonderful enhancing effect on your manhood.
Aug 10 03:06:04 skapet spamd[13548]: 24.165.4.190: disconnected after 445 seconds. lists: spamd-greytrap
they will find that to be a sufficient for action of some kind. The material we generate is available via the “The Name And Shame Robot” web page. The latest complete report of log excerpts is available via links at that page. Previous versions are archived offline, but will be made available on request to parties with valid reasons to request the data.

“The Name And Shame Robot”" is rather new, and it is too early to say what effect, if any, the publication has had. We hope that others will do similar things based on their local log data or even synchronize their data with ours. If you are interested in participating, please make contact.

Regardless of other factors, we hope that the data can be useful as indicators of potential for improvement in the networks that appear regularly in the reports as well as material for studies that will produce even better techniques for spam avoidance.

A shorter version of this article in Norwegian was published in Computerworld's Norwegian edition on August 22, 2008; the longer Norwegian version is available as an earlier blog post.

[1] A collection of such failure messages collected earlier this year is available at http://www.bsdly.net/~peter/joejob-archive.2008-07-28.txt.

[2] See http://www.bsdly.net/~peter/traplist.shtml, references at that page lead to my blog, which consists of public field notes, as well as other relevant material.

About the author
Peter N. M. Hansteen (peter@bsdly.net) is a consultant, system administrator and writer, based in Bergen, Norway. In October 2008, he joined FreeCode, the Norwegian free software consultancy. He has written various articles as well as "The Book of PF", published by No Starch Press in 2007, and lectures on Unix- and network-related topics. He is a main organizer of BLUG (Bergen (BSD and) Linux User Group), vice president of NUUG (Norwegian Unix User Group) and an occasional activist for EFF's Norwegian sister organization EFN (Elektronisk Forpost Norge).

[.NO] “Name and Shame” eller samfunnsnyttig bruk av loggdata om spammere

Today's post is in Norwegian - I'll be back in English later

Vi sitter med stadig voksende mengder med data om spammere. Kan vi bruke dette på en måte som er nyttig for andre?

Vi som selv står for driften av eposttjenester vet av tidvis smertelig erfaring hva som skal til for at minimalt med uønsket reklame og lureri som kan være direkte trusler mot sikkerheten faktisk havner i innboksene til brukerne våre.

Epost: Dette burde vært enkelt
I utgangspunktet burde det være en enkel sak å håndtere epost: Serveren er satt opp slik at den vet hvilke domener den skal ta imot post for, og hvilke brukere som eksisterer i domenene. Når en maskin tar kontakt og signaliserer at den ønsker å levere epost, sjekker serveren om meldingen er adressert til en gyldig bruker. Er det snakk om en gyldig bruker, blir meldingen mottatt og lagt i innboksen til den aktuelle brukeren, i motsatt fall får den som er oppgitt som avsender beskjed om at meldingen ikke lot seg levere, og hvorfor.

Hvis bare alle var ærlige
I hvert enkelt ledd av denne prosessen er det en underliggende forutsetning at kommunikasjonspartnerne oppgir korrekt informasjon. I mange tilfeller er det slik, og det dreier seg om legitim kommunikasjon mellom parter som har grunn til å ønske kontakt. Dessverre finnes det også tilfeller der denne grunnleggende tilliten blir brutt eller misbrukt, for eksempel når epostmeldinger blir sendt med annen avsenderadresse enn den reelle, gjerne noe oppdiktet i et domene som tilhører andre. En del av oss har også opplevd å få returmeldinger om at levering ikke var mulig på grunnlag av meldinger som vi vitterlig ikke har sendt[1]. Når vi ser nøyere på innholdet i disse meldingene, vil vi i nesten alle tilfeller se at dette er søppelpost, tidvis med innslag av svindel og kanskje ledd i førsøk på å overta mottakerens maskin eller stjele sensitive data.

Hva gjør de ansvarlige?
Hvis du spør en typisk systemadministrator om hvilke tiltak som er satt i verk for å hindre at uønskede eller skadelige meldinger faktisk kommer frem til brukerne, vil du antakelig få høre en beskrivelse som stort sett kan kokes ned til at posten blir filtrert gjennom systemer som studerer innholdet i meldingene. Hvis meldingen ikke inneholder noe som er kjent som uønsket (kjent spam eller skadelig programvare) eller noe som likner på noe som kunne være det, blir meldingen levert til brukeren den er adressert til. Hvis systemet avgjør at meldingen har innhold som gjør at den ikke skal leveres til mottaker, blir meldingen i mange tilfeller kastet uten å bli levert, og noen systemadministratorer vil nok også fortelle deg at systemet sender melding om avgjørelsen tilbake til adressen som er oppgitt som avsender.

Mye av dette hører antakelig til den passive kunnskapen hos mange, og de fleste slår seg til ro med at innholdsfiltrering er det eneste vi kan gjøre for å holde tvilsomme eller direkte kriminelle elementer unna arbeidsmiljøet. For en enkelt sluttbruker er det sannsynligvis bare mindre justeringer ut fra dette som er mulig.

Tiltak basert på observert adferd
Men for oss som håndterer driften av selve tjenesten er det mulig å studere dataene som registreres automatisk i loggene våre og bruke spammernes (her brukt om avsendere av alle typer uønsket epost, inkludert skadevare) adferd til å fjerne mesteparten av den uønskede trafikken før innholdet i meldingene er kjent. Det er nødvendig å gå ned på et noe mer grunnleggende nivå i nettverkstrafikken og studere adferden på nettverksnivå.

En av de enkleste formene av slike adferdsbaserte tiltak dukket opp i form av en teknikk som fikk navnet grålisting (greylisting) i 2003. Teknikken bygger på en litt pedantisk og ganske kreativ tolkning av allerede vedtatte standarder. Protokollen som brukes for epost-overføring på Internett, SMTP (Simple Mail Transfer Protocol) inneholder en mulighet for at en server som har antatt forbigående problemer med å ta imot epost kan rapportere om tilstanden ved å svare med en spesiell feilkode når andre maskiner prøver å levere. Hvis avsendermaskinen er korrekt konfigurert, vil den vente en viss tid før den gjør et nytt forsøk på levering, og etter all sannsynlighet vil den lykkes etter kort tid. Det er verd å merke seg at dette er en del av standarden som først og fremst skulle sørge for at eposttjenesten skulle være så pålitelig som mulig, og i de fleste tilfeller skjer alt dette uten at personen som skrev og sendte meldingen merker noe til det. Meldingene kommer frem, og alle er fornøyde.

Grå og svarte lister, små hvite løgner
Grålisting går ut på at serveren gir melding om midlertidig feil til alle forsøk på epostlevering fra maskiner den ikke har hatt kontakt med før. Erfaringene viste at antakelsene fra før eksperimentet i hovedsak var korrekte: De aller fleste maskiner som sender legitim epost er satt opp til å sjekke returkoder og handle etter dem, mens de aller fleste avsendere av spam bare pøser på så mange meldinger som mulig, uten å sjekke returkoder. Resultatet er at et sted mellom åtti og noenognitti prosent av spam-mengden blir stoppet på første forsøk (før eventuell innholdsfiltrering), mens legitim post kommer frem, i noen tilfeller med en viss forsinkelse ved første kontakt.

En annen adferdsbasert teknikk, som faktisk var i bruk før grålisting ble utbredt, er å bruke såkalte svartlister - lister over maskiner som er blitt klassifisert som spam-avsendere - og avvise post fra maskiner som var med i listen. Noen grupper begynte etterhvert å eksperimentere med såkalte tjærehull (tarpit), der teknikken går ut på å forsinke trafikk fra maskiner som er med i en svartliste ved å la sin del av kommunikasjonen gå svært sakte. Et eksempel som ofte nevnes er programmet spamd, som det frie operativsystemet OpenBSD lanserte i mai 2003. Programmet hadde da som sin hovedoppgave å svare på eposttrafikk fra svartlistede maskiner med ett tegn i sekundet, uten at avsendere som var med i en svartliste hadde noen reell mulighet til å få levert meldingene.

Kombinasjonen av svartlister og grålisting har vist seg å fungere utmerket. Likevel fortsetter praktikere og utviklere forsøkene på å få til enda mer effektive teknikker. Enda en gang kom neste logiske steg som resultat av observert adferd. Vi så tidligere at spammere ikke bryr seg om å sjekke om hver enkelt melding faktisk kommer frem.

Legge ut snarer og agn
Tidlig i 2005 førte dette til at det ble det formulert en teori som det viste seg å være hold i: Hvis vi så lager en eller flere adresser i vårt eget domene som vi vet aldri vil ha noen grunn til å få legitim post, så kan vi med nesten hundre prosent sikkerhet vite at post som blir forsøkt levert til disse adressene er spam. Adressene er spammerfeller. Maskiner som prøver å levere spam til disse adressene, kan vi legge i en lokal svartliste som vi så oppholder med ett tegn i sekundet. Maskinene blir liggende i svartlisten i 24 timer dersom ikke annet tilsier det.

Den nye teknikken fikk navnet greytrapping, og ble lansert som del av en forbedret spamd i OpenBSD 3.8 i mai 2005. Bob Beck, som var sentral deltaker i utviklingen av spamd, annonserte tidlig i 2006 at han brukte greytrapping på sentralt plasserte maskiner ved University of Alberta, og gjorde svartlistene som er resultat av fangsten, med oppdateringer hver time, tilgjengelig for nedlasting slik at andre kan bruke listene i sine oppsett. Dette er åpenbart nyttig, maskiner som sender til adresser som aldri har vært leverbare, har etter all sannsynlighet ikke legitim epost å levere, og vi gjør samfunnet en tjeneste ved å hindre dem i leveringen og å få dem til å kaste bort så mye tid som mulig.

Det er kanskje verd å nevne at svartlisten fra University of Alberta så lenge undertegnede har fulgt den har inneholdt minst noenogtyvetusen IP-adresser, og har i travle perioder vært oppe i nesten tohundretusen maskiner.

Også du kan bidra
Likevel er det ikke nødvendig å være sentralt plassert programvareutvikler for å kunne bidra positivt. De samme verktøyene som Beck bruker til å generere sin svartliste er tilgjengelige for alle som del av OpenBSD, og er ikke så vanskelig å bruke som man kunne frykte.

Her på bsdly.net og beslektede domener observerte vi sommeren 2007 en markert økning i meldinger om ikke leverbar epost til epostadresser som aldri har eksistert i noen av våre domener. Her var det helt klart snakk om at noen, en eller flere grupper, genererte eller fant på avsenderadresser for å unngå å få reaksjoner på spammen sin tilbake til seg. Dette førte i sin tur til et eksperiment som vi fortsatt har gående. Vi registrerer ugyldige adresser i våre egne domener som dukker opp i loggene våre. Av disse adressene velger vi ut de helt usannsynlige, legger dem inn i vår lokale spammerfelle-liste og legger ut listen på en egen side på webserveren vår[2].

Erfaringene viser at det tar svært kort tid før adressene vi fører opp på denne siden dukker opp som mottakeradresser. Kort og godt har vi klart å fore spammere med data som gjør det enklere for oss å stoppe dem, og i mange tilfeller får vi i tillegg spamsenderne til å bruke betydelige mengder tid på å kommunisere med våre maskiner uten å oppnå noe som helst. Antallet spammerfelle-adresser i vår liste er nå oppe i rundt femtentusen, og vi har tidvis observert grupper av maskiner som bruker noen uker på å arbeide seg gjennom hele listen, med gjennomsnittlig noe under syv minutter brukt per mislykkede leveringsforsøk.

Som et biprodukt av denne aktive spammerfangingen begynte vi å eksportere vår egen liste over maskiner som har blitt fanget via spammerfelle-adressene de siste 24 timer og legge den ut tilgjengelig for nedlasting. At denne listen finnes, er kun annonsert via siden med spammerfelle-adressene og noen bloggposter, men vi ser at den blir hentet regelmessig og antakelig automatisk av andre som bruker den i sine systemer.

Så langt har vi etablert at det er mulig å lage et system som gjør at sannsynligheten for at spam kommer gjennom til brukere er svært liten, samtidig som det er nesten helt usannsynlig at legitim post blir merkbart hindret. Dermed har vi det som tilsvarer gode gjerder rundt egen eiendom, men spammerne finnes fortsatt der ute og er et potensielt alvorlig problem for de som ikke har tilstrekkelig beskyttelse mot dem.

Samle bevis, eller i det minste skape klarhet
Aller helst hadde vi ønsket at politi og påtalemyndigheter hadde tatt spammerproblemet alvorlig. Dette ikke bare fordi den spammen som kommer frem er irriterende å se, men fordi nesten all spam sendes via utstyr som spammerne bruker uten eiernes samtykke. Kort og godt ønsker vi at det blir satt inn ressurser som står i forhold til den kriminelle virksomheten spammen representerer. Vi ville gjerne hjelpe til, men i utgangspunktet kan det virke som vi kan ha problem med å skaffe til veie brukbart bevismateriale siden vi ikke mottar meldingene som spammerne forsøker å levere. På den annen side har vi til enhver tid en liste over maskiner som har prøvd å levere spam, noe nær hundre prosent sikkert identifisert på grunnlag av spammerfelle-adressene. I tillegg produserer systemene våre rutinemessig logger over all aktivitet, med det detaljnivået vi selv velger. Dermed går det an å søke i loggene etter IP-adressene som har forsøkt å levere spam til oss siste 24 timer, og få oversikt over hva maskinene har foretatt seg.

Resultatet av et typisk søk av denne typen ser slik ut:

Aug 10 02:34:29 skapet spamd[13548]: 190.20.132.16: connected (4/3)
Aug 10 02:34:41 skapet spamd[13548]: (GREY) 190.20.132.16: <kristie@iland.net> -> <asasaskosmicki@bsdly.net>
Aug 10 02:34:41 skapet spamd[13548]: 190.20.132.16: disconnected after 12 seconds.
Aug 10 03:41:42 skapet spamd[13548]: 190.20.132.16: connected (14/13), lists: spamd-greytrap
Aug 10 03:42:23 skapet spamd[13548]: 190.20.132.16: disconnected after 41 seconds. lists: spamd-greytrap
Aug 10 06:30:35 skapet spamd[13548]: 190.20.132.16: connected (23/22), lists: spamd-greytrap becks
Aug 10 06:31:16 skapet spamd[13548]: 190.20.132.16: disconnected after 41 seconds. lists: spamd-greytrap becks
Den første linjen angir at 190.20.132.16 tar kontakt med vårt system klokken 02.34.29 om morgenen tiende august, som fjerde aktive forbindelse, derav tre svartlistede. Noen sekunder senere blir det klart at dette er et forsøk på å levere en melding til adressen asasaskosmicki@bsdly.net, som er blant de vi har som spammerfelle, sannsynligvis høstet fra logger og diktet opp et annet sted i verden. Etter 12 sekunder kobler denne maskinen fra. Forsøket på å levere til en spammerfelle gjør at maskinen blir oppført i vår lokale svartliste, spamd-greytrap, noe som vises klart når maskinen prøver igjen litt mer enn en time senere. På dette forsøket blir den oppholdt i 41 sekunder. Det tredje forsøket i vårt loggmateriale skjer like etter 06.30, og at listenavnet becks har kommet til, viser at maskinen i mellomtiden har forsøkt å levere til en av Bob Becks spammerfelle-adresser og nå er med i også den svartlisten.

Det er dessverre lite sannsynlig at slike logger er tilstrekkelige som bevismateriale i straffesaker, men for de som har interesse av enten å sørge for at maskinene de administrerer i så liten grad som mulig brukes til spamutsendelse eller de som har interesse av spammeradferd er dette nyttige data.

“Name And Shame”, eller kanskje bare godt naboskap?
Etter noen diskusjoner med kolleger bestemte jeg meg tidlig i august 2008 for å generere daglige oversikter over aktivitetene til maskiner som har kommet inn i vår lokale svartliste på bsdly.net og legge dem ut offentlig. Når en maskin er oppført i en svartliste med bare IP-adresse (for eksempel 24.165.4.190), uten annet materiale om oppføringen, er oppføringen mest en påstand som de ansvarlige godt kan velge å ikke tro på. Vårt håp er at om den som er ansvarlig for nettverket der 24.165.4.190 hører hjemme ser en sekvens som denne,

Host 24.165.4.190:
Aug 10 02:57:40 skapet spamd[13548]: 24.165.4.190: connected (9/8)
Aug 10 02:57:54 skapet spamd[13548]: (GREY) 24.165.4.190: <hand@itnmiami.com> -> <kimberlee.ledet@ehtrib.org>
Aug 10 02:57:55 skapet spamd[13548]: (GREY) 24.165.4.190: <hand@itnmiami.com> -> <kimberliereffett@ehtrib.org>
Aug 10 02:57:56 skapet spamd[13548]: 24.165.4.190: disconnected after 16 seconds.
Aug 10 02:58:16 skapet spamd[13548]: 24.165.4.190: connected (8/6)
Aug 10 02:58:30 skapet spamd[13548]: (GREY) 24.165.4.190: <brunson@jebconet.com> -> <kimberlee.ledet@ehtrib.org>
Aug 10 02:58:31 skapet spamd[13548]: (GREY) 24.165.4.190: <brunson@jebconet.com> -> <kimberliereffett@ehtrib.org>
Aug 10 02:58:32 skapet spamd[13548]: 24.165.4.190: disconnected after 16 seconds.
Aug 10 02:58:39 skapet spamd[13548]: 24.165.4.190: connected (7/6), lists: spamd-greytrap
Aug 10 03:02:24 skapet spamd[13548]: (BLACK) 24.165.4.190: <aarnq@abtinc.com> -> <kimberlee.ledet@ehtrib.org>
Aug 10 03:03:17 skapet spamd[13548]: (BLACK) 24.165.4.190: <aarnq@abtinc.com> -> <kimberliereffett@ehtrib.org>
Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: From: "Preston Amos" <aarnq@abtinc.com>
Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: To: kimberlee.ledet@ehtrib.org
Aug 10 03:05:01 skapet spamd[13548]: 24.165.4.190: Subject: Wonderful enhancing effect on your manhood.
Aug 10 03:06:04 skapet spamd[13548]: 24.165.4.190: disconnected after 445 seconds. lists: spamd-greytrap

så er det tilstrekkelig grunnlag for å gjøre noe aktivt. Materialet er tilgjengelig via The Name And Shame Robot-siden på http://www.bsdly.net/~peter/nameandshame.html. Siste genererte loggoversikt er tilgjengelig via referanser på den siden, tidligere utgaver blir arkivert, men vil være tilgjengelige ved godt begrunnet forespørsel.

The Name and Shame Robot er såpass ny at vi ikke kan si spesielt mye om effekten av offentliggjøringen. Det er lov å håpe på at andre vil gjøre noe tilsvarende ut fra sine lokale loggdata eller kanskje til og med synkronisere sine data med våre. Ta gjerne kontakt hvis du er interessert i dette arbeidet.

Uavhengig av alt annet håper vi at dataene kan være nyttige, både som påpeking av forbedringspotensiale for de nettverkene som opptrer jevnlig i oversiktene og som materiale for studier som kan gi oss enda bedre spambekjempelse.

Noter

[1] En samling av slike returmeldinger fra tidligere i år kan beskues på http://www.bsdly.net/~peter/joejob-archive.2008-07-28.txt


[2] http://www.bsdly.net/~peter/traplist.shtml, referanser på den siden fører videre til bloggen min som jeg bruker til offentlige notater, og annet relevant materiale.

En forkortet utgave av denne artikkelen ble trykt i Computerworld Norge 22. august 2008.

Spam is a solved problem

Executive summary: Spam is a solved problem, email works again. There are a few knuckle draggers out there who haven't noticed yet, but we'll get around to dealing with those shortly.

I've been looking over my log summaries again. My regular logs get rotated out of existence after seven days, but from the summaries I do keep around, it looks like various made up @datadok.no addresses have been used as spammers' fake From: addresses for about a month. I was too busy with other Very! Urgent! Things! to notice at first, but it finally dawned on me when I searched my mail server logs for "Unknown" as in "Unknown user" and saw from the results that somebody, somewhere, was using that domain for generating sender addresses.

Note: This piece is also available without trackers but classic formatting only here.

After about two weeks of observation and collecting made up or generated addresses for my traplist, my conclusion is what the title of this post says. Spam is not a problem anymore. I know, of course, that "how to cope with email and spam" self help guides are best sellers, and a recent Salon.com piece even went so far as saying about email,

Now, it seems, we're drowning. There's simply too much e-mail. The tide of spam buries valuable messages.

That kind of surprises me, because it's not what we're seeing here at all. Of course we know that there's a lot of junk being sent, but ask any of the people on the sites I run on any given day how much spam they've received recently and they have to look up the date of the last one in their "Junk mail" folders.

I do see some spam myself, mainly because I still fetch and read mail I receive at an ISP address I've used a lot for USENET and mailing lists over the years. And since unfortunately no method ever has a zero error rate, occasionally a spam message or two trickles through that shouldn't have on the systems I run myself. But if the tide of spam buries valuable messages, you haven't kept up with the technology, plain and simple.

By and large, from the perspective of somebody who has been the purported sender of an unknown portion of the tide which drowns out the Salon.com writer's messages, it looks like spam is treated correctly or at least in ways that do not annoy others unnecessarily at most sites. (In all fairness that piece is more about email versus other types of writing than technical matters, and certainly worth reading for that reason. The same writer has written a number of other articles which are worth your time too.)

At the last count, our main spamd running gateway had all of 316 addresses in the local spamd-greytrap table, meaning that only that many hosts have actually tried to send mail to one or more of the addresses listed at our spamtrap page during the last 24 hours. Some of the trapped machines would have been active spam senders, and most of the rest seem to have been sites which were configured to receive spam and bounce back to the From: address when the spam was not deliverable.

That is an important point to note. If your system sends a 'message undeliverable' bounce message for spam sent to a non-existent user, it is configured to deliver spam to the users you do have, and there are certainly ways to avoid that. I've decided not to plug any of my other writing directly in this post, but you should be able to find the references easily enough if you're interested.

Reading the spamd logs is sometimes quite entertaining if you're that kind of guy or girl. Here is one example of a site with clearly deficient spam and/or malware filtering, possibly their own homebrew:

Jul 14 09:13:28 delilah spamd[29851]: 66.35.252.70: From postmaster@trendmicro.com
Jul 14 09:13:28 delilah spamd[29851]: 66.35.252.70: To: jonson3846@datadok.no
Jul 14 09:13:28 delilah spamd[29851]: 66.35.252.70: Subject: Delivery Status Notification (Failure)

It does not matter much to us, but they'll be unable to get mail through to us for the next 24 hours.

The next one is clearly problematic, since whoever set up the system appears to have left back in the time when there still was a chance that spammers used real addresses. Or maybe the poor wretch stayed on and now suffers from delusions, incompentence or both:

Jul 13 14:36:50 delilah spamd[29851]: 212.154.213.228: Subject: Considered UNSOLICITED BULK EMAIL, apparently from you
Jul 13 14:36:50 delilah spamd[29851]: 212.154.213.228: From: "Content-filter at srv77.kit.kz" <postmaster@srv77.kit.kz>
Jul 13 14:36:50 delilah spamd[29851]: 212.154.213.228: To:
<skulkedq58@datadok.no>


Would SPF have helped? Possibly. We have our records set up, but clearly these guys are not using it in any meaningful way, and after -- what is it -- five years it's still not clear which of the competing RFCs with varying degrees of proprietary content is going to come out on top.

Staying out of our traplist would have saved some resources on their side. On our side, well, we have a working system. Email works. Mail from us has to go through our mail server. Incoming mail needs to clear spamd's greylisting (and really needs to come from IP addresses which are not in any of the blocklists we use) and pass content filtering inspection by spamassassin and clamav, all of it conveniently within reach on any freely available BSD system. The content filtering packages are available on your favorite Linux as well, but on our sites, we use OpenBSD and FreeBSD.

Living spam free and unworried by malware is possible. If you make a few right choices it's actually easy, it doesn't cost much and just imagine how much of your time you stop wasting.

Hey, spammer! Here's a list for you!

Last week I started noticing from my log summaries that my mail servers had seen a lot more mail to non-existent users than usual. This usually happens when somebody has picked one of our domains as the home of their made-up return addresses for their spam run. This time, from the looks of it, the spam runs were mainly targeted at Russian and Ukrainian users. At least that's where most of the backscatter appears to have come from.

As I've written before in the PF tutorial and the malware paper (updated version available as this blog post -- that's the end of today's plugging, I promise), I've used the "Unknown user" messages as a valuable data source for my spamtrap list, just quitely adding addresses that looked really unlikely to ever become valid. After a brief airing on the OpenBSD-misc mailing list and running it by my colleagues at Datadok and Dataped, I've decided to take it a bit further.

Now that I've got a list of addresses which will never receive any legitimate mail, I really want spammers to try to send mail to those addresses. After all, if they send anything to an address which consists of a random string with one of our domains stuck on after the '@', we know it's all spam from there on.

We don't care about the rest, for the next 24 hours. Your SMTP dialogue with us (actually our spamd) will be all a-stutter, receiving answers one byte at the time until you give up. For the record, that usually takes about 400 seconds, with the really imbecile ones taking a lot longer. See the paper or the tutorial for some numbers.

The other possibility is of course that your system is set up in a way which makes it actually receive and try to deliver spam. Some of the spam will be addressed to non-existent users in your domain, so if your users receive spam, you will be trying to send bounce messages back to the purported sender for spam to non-existent users. That's tough, kid. If you're set up that way, your machine will be treated to the tarpit here for the next 24 hours. All a-stutter and all that. Repeat offenders stay there longer.

Now for the spamtrap list, I've checked that my colleagues and associates have never actually wanted to use those addresses for anything, and I made this page which wraps it all in a bit of explanation. For some reason, the list keeps growing each time I look at my log summaries.

When I get around to it and find a visually not-too-horrible way to do it, I'll include links to that page where they fit naturally on our web sites. In the meantime, here's hoping that the spammers' address harvesting robots find this list and put it to good use.

The chapter, it's improving. More later.

UPDATE 12-jul-2007: The softer side of me ponders the possibility of sending email form letters to the various postmaster@s with the URL to this blog post. On the other hand, I'm not sure I'm ready for another round of finding out that postmaster@ is in fact not deliverable at a surprising number of sites around the world.

One other thing I've noticed since I published the traplist is that bounces to addresses like mixt.apex.dp.ua-1184227575-testing@datadok.no have started appearing in the logs. I don't see how messages like these could be useful by themselves, but the addresses are of course obvious traplist material.

13-jul-2007: Oddly enough, there's still a stream of backscatter, and my logs tell me a few new addresses turn up every day. This morning's fresh ones were careersogt2083@datadok.no, phalanxesxb88@datadok.no and retryingvtt@datadok.no. Another few bytes to help weed out the bad ones early, thanks to the robots out there.