DNSSEC Mastery, Or How To Make Your Name Service Verifiable And Trustworthy

A DNSSEC book for the working sysadmin, likely to put you ahead of the pack in securing an essential Internet service.

I have a confession to make. Michael W. Lucas is a long time favorite of mine among tech authors. When Michael descends on a topic and produces a book, you can expect the result to contain loads of useful information, presented along with humor and real-life anecdotes so you will want to explore the topic in depth on your own systems.

In DNSSEC Mastery (apparently the second installment in what could become an extensive Mastery series -- the first title was SSH Mastery, reviewed here -- from Michael's own Tilted Windmill Press), the topic is how to make your own contribution to making the Internet name service more reliable by having your own systems present verifiable, trustworthy information.

Before addressing the book itself, I'll spend some time explaining why this topic is important. The Domain Name System (usually referred to as DNS or simply 'the name service' even if nitpickers would be right that there is more than one) is one of the old-style Internet services that was created to solve a particluar set of problems (humans are a lot better at remembering names a than strings of numbers) in the early days of networking when security was not really a concern.

Old-fashioned DNS moves data via UDP, the connectionless no-guarantees-ever protocol mainly because the low protocol overhead in most cases means the answer arrives faster than it would have otherwise. Reliable delivery was sacrificed for speed, and in general, the thing just works. DNS is one of those things that makes the Internet usable for techies and non-techies alike.

The other thing that was sacrificed, or more likely never even considered important enough to care about at the time, was any hope of reliably verifying that the information received via the DNS service was in fact authentic and correct.

When you ask an application to look up a name, say you want to see if anything's new at bsdly.blogspot.com or if you want to send me mail to be delivered at bsdly.net, the answer comes back, not necessarily from the host that answers authoritatively for the domain, but more likely from the cache of a name server near you, and serves mainly one or more IP addresses, with no guarantee other than it is, indeed a record type that contains one or more IP addresses that appear to match your application's query.

Or to put it more bluntly, with traditional DNS, it's possible for a well positioned attacker to feed you falsfied information (ie leading your packets to somewhere they don't belong or to somewhere you never intended, potentially along with your confidential data), even if the original DNS designers appear to have considered the scenario rather unlikely back then in the nineteen-eighties.

With the realization that the Internet was becoming mainstream during the 1990s and that non-techies would rely on it for such things as banking services came support cryptographically enhanced versions of several of the protocols that take care of the bulk of Internet traffic payloads, and even the essential and mostly ignored (at least by non-techies) DNS protocol was enhanced several times over the years. Around the turn of the century came the RFCs that describe cryptographic signatures as part of the enhanced name service, and finally in 2005 the trio of RFCs (4033, 4034 and 4035) that form the core of the modern DNSSEC specification were issued.

But up until quite recently, most if not all DNSSEC implementations were either incomplete or considered experimental, and getting a working DNSSEC setup in place has been an admirable if rarely fulfilled ambition among already overworked sysadmins.

Then at what seems to be the exactly right moment, Michael W. Lucas publishes DNSSEC Mastery, which is a compact and and extremely useful guide to creating your own DNSSEC setup, avoiding the many pitfalls and scary manouvres you will find described in the HOWTO-style DNSSEC guides you're likely to encounter after a web search on the topic.

The book is aimed at the working sysadmin who already has at least basic operational knowledge of running a name service. Starting with one DNSSEC implementation that is known to be complete and functional (ISC BIND 9.9 -- Michael warns early on very clearly that earlier versions will not work -- if your favorite system doesn't have that packaged yet, you can build your own or start bribing or yelling at the relevant package maintainer), this book takes a very practical, hands on approach to its topic in a way that I think is well matched to the intended audience.

Keeping in mind that the one thing a working sysadmin is always short on is time, it is likely a strong advantage that this book is so compact. With 12 chapters, it comes in at just short of 100 pages in the PDF version I used for most of this review. With the stated requirement that the reader needs to be reasonably familiar with running a DNS service, the introductory chapters fairly quickly move on to give an overview of public key cryptography as it applies to DNSSEC, with pointers to wordier sources for those who would want to delve into details, before starting the steps involved in setting up secure name service using ISC BIND 9.9 or newer.

Always taking a practical approach, DNSSEC Mastery covers essentially all aspects of setting up and running a working service, including such topics as key management, configuring and debugging both authoritative and recursive resolvers, various hints for working with or around strengths or deficiencies in various client operating systems, how the new world of DNSSEC influences how you manage your zones and delegations, and did I mention debugging your setup? DNSSEC is a lot less forgiving of errors than your traditional DNS, and Michael includes both some entertaining examples and pointers to several useful resources for testing your work before putting it all into production. And for good measure, the final chapter demonstrates how to distribute data you would not trust to old fashioned DNS: ssh host key fingerprints and SSL certificates.

As I mentioned earlier, this title comes along at what seems to be the perfect time. DNSSEC use is not yet as widespread as it perhaps should be, in part due to incomplete implementations or lack of support in several widely used systems. The free software world is ahead of the pack, and just as the world is getting to realize the importance of a trustworthy Internet name service, this book comes along, aimed perfectly at the group of people who will need an accessible-to-techies book like this one. And it comes at a reasonable price, too. If you're in this book's target group, it's a recommended buy.

The ebook is available in several formats from Tilted Windmill Press, Amazon and other places. A printed version is in the works, but was not available at the time this review was written (May 11, 2013).

Note: Michael W. Lucas gives tutorials, too, like this one at BSDCan in Ottawa, May 15 2003.

Title: DNSSEC Mastery: Securing The Domain Name System With BIND
Author: Michael W. Lucas
Publisher: Tilted Windmill Press (April 2012)

---

Michael W. Lucas has another, somewhat chunkier book out this year too, Absolute OpenBSD, 2nd edition, a very good book about my favorite operating system. It would have been reasonable to expect a review here of that title too, except that I served as the book's technical editor, and as such a review would be somewhat biased.

But if you're interested in OpenBSD and haven't got your copy of that book yet, you're in for a real treat. If a firewall or other networking is closer to your heart, you could give my own The Book of PF and the PF tutorial (or here) it grew out of. You can even support the OpenBSD project by buying the books from them at the same time you buy your CD set, see the OpenBSD Orders page for more information.

Upcoming talks: I'll be speaking at BSDCan 2013, on The Hail Mary Cloud And The Lessons Learned. There will be no PF tutorial at this year's BSDCan, fortunately my staple tutorial item was crowded out by new initiatives from some truly excellent people. (I will, however, be bringing a few copies of The Book of PF and if things work out in time, some other items you may enjoy.)

Keeping Your OpenBSD System In Trim: A Works For Me Guide

Keeping your OpenBSD systems in trim is dead easy. Occasional reboots are inevitable, but then that's where our famous redundancy features really shine, right? Read on for notes on upgrading your system. (Most of the steps here are relevant for new installs too, but do visit the Install Guide if you're new to OpenBSD.)

Note: If you're looking for a one-stop article, you will be better off with the 2024 update You Have Installed OpenBSD. Now For The Daily Tasks. (also available tracked, prettified), which covers the main care and feeding tasks.

Unless you have very specific requirements such as omitting sets, the base system upgrade can be performed painlessly with the sysupgrade(8) command followed by the sysmerge(8) and pkg_add(8) commands as described in this article. See the man pages or the OpenBSD FAQ for more information. (Updated 2021-07-02).

My upgrades always start with the same couple of commands:

$ cd ~/upgrade
$ ls -l
$ ncftp eu-openbsd
ncftp ...penBSD/snapshots/amd64 > dir

The first two commands of this sequence show me the date of the latest snapshot I downloaded, and the next two give me the date on what is on the OpenBSD European mirror site.

The ncftp bookmark eu-openbsd in this case expands to

ftp://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/amd64/

which is appropriate for a North European like myself with an AMD64 based system on hand. If you're not in Europe, it's likely you're better off with some other choice of mirror. Check the Getting Releases page on the OpenBSD site for a mirror near you. And of course, do pick the correct architecture for your system.

If the files on the mirror server are newer than the ones I have locally, I'll download them with

ncftp ...penBSD/snapshots/amd64 > get *

If there are no updates, well, that means I'll just check back later.

And of course, if you do not have the ncftp package installed, this command will work on any OpenBSD box with a full base system installed:

$ ftp -ia ftp://ftp.eu.openbsd.org/pub/OpenBSD/snapshots/`uname -m`/{index.txt,*tgz,bsd*,INS*,BOOT*,BUILDINFO,SHA*}

(thanks, Pedro Caetano!)

One of the things that makes doing OpenBSD upgrades so amazingly easy is the sysmerge(8) program.  Much inspired by FreeBSD's classic mergemaster(8), this program is, as we shall see by executing the following commands,


$ which sysmerge
/usr/sbin/sysmerge
$ file `which sysmerge`
/usr/sbin/sysmerge: Korn shell script text executable

actually a shell script that makes extensive use of sdiff(1) to highlight the differences between your installed version of a configuration file and the one from the source tree or your install sets, and offers to merge (hence the name) your current customizations into the newer version files and install them on the spot if you like.

Do take a peek with at what the script does:

$ less /usr/sbin/sysmerge

possibly complemented by looking up the sdiff(1) man page if you want. But more about that later.

If you run regular upgrades like I tend to, with snapshots only days or at most a few weeks apart, the differences sysmerge(8) detects are likely few and trivial (but a word of caution: it usually pays to check the Following -current page from time to time).

If you do longer jumps, such as between releases, you will almost certainly find more differences, and in rare cases the changes in the configuration file baselines will be large enough that you will need to take some time out to edit the files by hand. In those cases, studying the upgrade notes relevant to your versions (and all intermediate ones if you do a long jump -- go to http://www.openbsd.org/faq/ and choose the Upgrade Guide link at the top of the left column) will be well worth your time.

The next step is to copy the fresh bsd.rd to the root directory of my boot disk:

$ doas cp bsd.rd /

With the bsd.rd in place, the next step is to reboot your system. Either choose your window manager's reboot option or simply from a shell prompt type

$ doas reboot

When the boot> prompt appears, type b bsd.rd and press Enter.

The familiar bsd.rd boot messages appear (much like the ones illustrated in my earlier installation piece, The Goodness Of Men And Machinery), and the first question you need to answer is:


(I)nstall, (U)pgrade or (S)hell?

Since we're upgrading, we type u (upper or lower case doesn't matter) and press Enter.

The next question up is the choice of keyboard layout:


Choose your keyboard layout ('?' or 'L' for list)

Here I type no for my Norwegian keyboard layout, if you want a different one you can get a list of available choices by typing L instead, and pressing Enter.

Once you've chosen your keyboard layout, the upgrade script prompts you to specify where your root file system is located. The upgrader's guess is only very rarely wrong, so more likely than not you'll just press Enter here.

The upgrader performs a full file system check on the designated root file system, and proceeds to configure any network interfaces it finds configuration files for in the designated root file systems etc directory. You will see the output of DHCP negotiations or similar.

When the network configuration is done, the upgrader prompts you to decide whether you want to perform a full file system check on the other partitions it finds.  The default choice here is no, so if you press enter here, the upgrade script simply runs a fsck -p on each of the file system listed in the system's fstab. You'll see the output for each one of these relatively lightweight checks.

Next, the upgrade script asks where to find the install sets:

Location of sets? (disk http or 'done') [disk]

Here, since I already downloaded the install set files to a local directory, the natural choice is disk. The upgrade script has already mounted all partitions from the system's fstab under /mnt/, so the files I downloaded to /home/peter/upgrade are now available under /mnt/home/peter/upgrade, and that's what I type in response to the prompt for where the sets are to be found.

Unless you have prepared a site.tgz for your site there is normally no reason to add or subtract sets from the default list, so pressing Enter will do nicely. The sets are copied and extracted, and when the extraction is done, you confirm that the upgrade is [done], and when the prompt appears, choose reboot (the default)

to let the system boot into the upgraded operating system.

Watch the system boot, and if you look closely, you will notice that on first boot the updated sysmerge(8) program runs and does the obvious things that do not require manual intervention, and if there are non-obvious things left, a message and an email to root alerts you of the need to do a manual sysmerge(8) run.

I tend to do a sysmerge run anyway after upgrade, if only to see it complete silently:

$ doas sysmerge

That's it! There is a chance that there have been updates to packages you have installed (such as ncftp in my case), so I tend to do a package upgrade pretty soon after rebooting into a freshly upgraded system. The basic update command is pkg_add -u, but I tend to want more verbose output and prompts and generally choose this variant:

$ doas pkg_add -vVuUmr

This command will most likely just work, fetching packages from the site and directory specified in /etc/installurl which is automagically created and maintained by the install program. If the installer somehow guessed wrong, you need to adjust the contents of that file and restart pkg_add (fiddling with the environment variable PKG_PATH as was the procedure in previous releases will not do any good). During the relatively short time when snapshots identify as the release but actual -release packages are not yet available (as in the recent 7.5 release cycle) you may have to adjust the pkg_add command to include -D snap, like this:

$ doas pkg_add -D snap -vuV

For upgrading from one snapshot to the next, there is really not much more to the process. You will occasionally want to run

$ doas pkg_delete -a

to remove packages that were installed as dependencies (mainly libraries) but are no longer needed. In addition, you may want to install and run sysclean to help you indentify other files such as obsolete configuration files that are no longer needed in your current configuration.

If you're upgrading from one release to the next, it makes sense to check the Errata page for any patches you need to apply to the release you're running.

An if you've read this far and found this interesting or useful, please head over to the OpenBSD Donations page to make a donation to the project.

Update 2017-04-17: Minor updates to adjust to the post-6.1 world, such as no more CDs. Thanks to Marc Espie for (as always) valuable input.

Update 2018-06-12: Modern man.openbsd.org links are much simpler now than when this was originally written, links updated. Also small tweaks to make the text reflect the user visible changes to the upgrade program.

Update 2019-04-26 + 2024-09-01: Now that you've read this piece to the end and have probably gone through more than a few manual steps, you will probably be pleased to hear that the process is now about to become significantly simpler in almost all cases.

Only hours after the news of the OpenBSD 6.5 release hit, the new sysupgrade(8) command was added to OpenBSD-current, and so will most likely be part of the OpenBSD 6.6 release, expected in about six months. In its present form, sysupgrade(8)  only performs the base system upgrade, so the pkg_add(8) parts mentioned in this piece will still be needed as a separate step. Further details will be available as development proceeds, but now you know about at least one thing to look forward to in OpenBSD 7.6 (due some time in late October to early November of 2024, with the release page starting to emerge some weeks ahead of that date.

A Third Time, Uncharmed

Spamwashers hijacked, a wake-up call for lazy sysadmins everywhere. The slow bruteforcers are back for another round.

A new round of slow, distributed bruteforce attacks is in progress. Just like the other times we know about (see references later), the initial target is root. This time around I see only one of my ssh-contactable machines targeted, and the dribble started on September 30th.

I've put the raw data so far up for study here (a total of 6067 attempts), and a list of hosts sorted by number of attempts (the first column) can be found here (770 hosts, with up to 32 attempts each). Quite likely I'll be collecting more data and publishing updates when I have a few free moments.

A number of people were kind enough to contact me in the followup of the earlier articles, and from one of my correspendents (who asked not to be named) I learned that the likely culprit is a piece of Linux malware known as dt_ssh5. If you type dt_ssh5 into your favorite search engine, it will turn up a few hits, but significantly fewer than the number of hosts in my sample. A couple of those documents have some analysis of how a badly secured web application let the miscreants in.

We should not be surprised that whoever is behind this has more than one takeover method in their arsenal. Unfortunately it should not surprise anybody either that we're now seeing a third round of those attacks. Most likely the perpetrators keep going because they occasionally succeed, and when they do, it's because every now and then they luck on a Linux machine with either

• a maintenance regime that's disorganized enough that software with known and exploitable bugs is left in place for long enough to open the doors to undesirables, or


• at least one user (whoever is manning root or any of the other user IDs we know they will be sniffing out later) with a guessable password and a system administration regime that lets weak passwords exist in the first place.

You see where this is heading? Over the last few years we have seen Linux and other free systems take over ever more niches and even mission critical application areas in businesses and organizations all over the world. I for one think this move to free systems with source code accessible is a good thing.

But there is a downside: In our efforts to entice the suits into the wonderful new world of free software, we likely oversold the security part.

Yes, bugs are easier to find and fix when the source code is available, yes, the Unix security model or some variant thereof is vastly preferable to that disorienting hellhole system marketed from somewhere up Seattle way, and yes, there are a few hundred more good and valid reasons (technical and otherwise) why basing your business on free license software is generally a good idea. Security-wise you have a better starting point, and with a little effort you will gain control and stay in control. But it does take some degree of effort by one or more qualified persons who are willing and able to take charge and make sensible decisions.

Looking at the data earlier I can see that one of the hosts now trying to gain illegitimate access to one of my systems was originally set up as a "spam washer" (grep for spamvask in the data). Setting up a Linux box to filter spam is likely a good idea in many contexts, but please keep this in mind: The fact that your rig runs Linux does not mean you're home free. You need to keep paying attention. When your spam washer has been hijacked and tries to break into other people's systems, you urgently need to get your act together, right now.

We do not have self-propagating worms on Linux just yet (and in all fairness the likelihood of that ever happening is rather slim), but we certainly do not have a system that will stay secure if you ignore the basics of useful system administration. This recurring episode of dt_ssh5 and the slow bruteforcers should serve as a wakeup call for system administrators everywhere: Your system stays secure only as long as you pay proper attention to maintenance. Out there in the real world there are at least 770 machines where the maintenance regime slipped, either through incompetence or work overload or a combination of the two with a dash of bad planning thrown in.

The only way to make a fourth round of slow bruteforcers not happen is to make the people responsible for the 770 hosts listed here do the right thing. I suppose we will find out soon enough how many of those domains actually have the RFC2142-mandated mailboxes in place.

References: Previous posts on this topic include A low intensity, distributed bruteforce attempt (December 2, 2008), A Small Update About The Slow Brutes (December 6, 2008), Into a new year, slowly pounding the gates (December 21, 2008), The slow brutes, a final roundup (January 22, 2009) and The slow brute zombies are back (April 12, 2009).

---

If you found this article useful, enjoyable or irritating, please drop me a line. Material related to this article is available free via links from my web space. Some additional material will be made available for reasonable research purposes. If you want more extensive assistance, please contact me (via email or other means) to make arrangements.

---

The reason for my rather long silence in the blogosphere can be summed up in two words: Client Confidentiality. Some of the projects I've been working on might have been material for interesting articles, but separating the generally useful bits from what the customer could reasonably expect to be kept confidential has proved difficult, at least in the short run.

Update 2009-10-05 just before 10:00 CEST: Slightly newer log extracts uploaded, total attempts now 6590, from 775 hosts. I will likely upload fresher versions at intervals.

Update 2009-10-14 19:30ish: After a couple of days with literally none to three or four attempts per hour, my afternoon log check turned up activity almost at start of run levels, so total numbers are now 9405 attempts from a total of 946 unique hosts. I never bothered to change the file names, but the data have been updated a couple of times each day since I started publishing them.

Update 2009-10-29 00:10: Because a file transfer took a little longer than expected, I was awake to see what may actually be the start of the alphabetic phase -


Oct 28 23:58:35 rosalita sshd[61664]: Invalid user anthony from 62.225.63.99
Oct 28 23:58:35 rosalita sshd[61664]: error: PAM: authentication error for illegal user anthony from 62.225.63.99
Oct 28 23:58:35 rosalita sshd[61664]: Failed keyboard-interactive/pam for invalid user anthony from 62.225.63.99 port 58683 ssh2
Oct 28 23:59:43 rosalita sshd[61668]: Invalid user anthony from 217.136.253.239
Oct 28 23:59:43 rosalita sshd[61668]: error: PAM: authentication error for illegal user anthony from 239.253-136-217.adsl-static.isp.belgacom.be
Oct 28 23:59:43 rosalita sshd[61668]: Failed keyboard-interactive/pam for invalid user anthony from 217.136.253.239 port 54728 ssh2
Oct 29 00:00:26 rosalita sshd[61695]: Invalid user barbara from 112.78.124.159
Oct 29 00:00:27 rosalita sshd[61695]: error: PAM: authentication error for illegal user barbara from 112.78.124.159
Oct 29 00:00:27 rosalita sshd[61695]: Failed keyboard-interactive/pam for invalid user barbara from 112.78.124.159 port 56990 ssh2
Oct 29 00:03:42 rosalita sshd[61722]: Invalid user brian from 122.224.128.197
Oct 29 00:03:42 rosalita sshd[61722]: error: PAM: authentication error for illegal user brian from 122.224.128.197
Oct 29 00:03:42 rosalita sshd[61722]: Failed keyboard-interactive/pam for invalid user brian from 122.224.128.197 port 47058 ssh2
Oct 29 00:04:21 rosalita sshd[61725]: Invalid user brian from 218.69.27.138
Oct 29 00:04:21 rosalita sshd[61725]: error: PAM: authentication error for illegal user brian from 218.69.27.138
Oct 29 00:04:21 rosalita sshd[61725]: Failed keyboard-interactive/pam for invalid user brian from 218.69.27.138 port 41622 ssh2
Oct 29 00:05:00 rosalita sshd[61738]: Invalid user carol from 58.60.106.121
Oct 29 00:05:01 rosalita sshd[61738]: error: PAM: authentication error for illegal user carol from 58.60.106.121
Oct 29 00:05:01 rosalita sshd[61738]: Failed keyboard-interactive/pam for invalid user carol from 58.60.106.121 port 55916 ssh2
Oct 29 00:05:51 rosalita sshd[61744]: Invalid user carol from 200.248.242.218
Oct 29 00:05:52 rosalita sshd[61744]: error: PAM: authentication error for illegal user carol from 200.248.242.218
Oct 29 00:05:52 rosalita sshd[61744]: Failed keyboard-interactive/pam for invalid user carol from 200.248.242.218 port 54547 ssh2
Oct 29 00:07:24 rosalita sshd[61748]: Invalid user charles from 222.128.36.60
Oct 29 00:07:24 rosalita sshd[61748]: error: PAM: authentication error for illegal user charles from 222.128.36.60
Oct 29 00:07:24 rosalita sshd[61748]: Failed keyboard-interactive/pam for invalid user charles from 222.128.36.60 port 42322 ssh2
Oct 29 00:08:09 rosalita sshd[61755]: Invalid user christopher from 72.148.242.188
Oct 29 00:08:10 rosalita sshd[61755]: error: PAM: authentication error for illegal user christopher from adsl-072-148-242-188.sip.asm.bellsouth.net
Oct 29 00:08:10 rosalita sshd[61755]: Failed keyboard-interactive/pam for invalid user christopher from 72.148.242.188 port 37157 ssh2
Oct 29 00:09:06 rosalita sshd[61758]: Invalid user christopher from 58.223.251.232
Oct 29 00:09:06 rosalita sshd[61758]: error: PAM: authentication error for illegal user christopher from 58.223.251.232
Oct 29 00:09:06 rosalita sshd[61758]: Failed keyboard-interactive/pam for invalid user christopher from 58.223.251.232 port 46350 ssh2
Oct 29 00:09:48 rosalita sshd[61762]: Invalid user daniel from 64.69.114.115
Oct 29 00:09:49 rosalita sshd[61762]: error: PAM: authentication error for illegal user daniel from mulligan.softspike.org
Oct 29 00:09:49 rosalita sshd[61762]: Failed keyboard-interactive/pam for invalid user daniel from 64.69.114.115 port 39960 ssh2
Oct 29 00:11:29 rosalita sshd[61781]: Invalid user david from 80.255.179.150
Oct 29 00:11:29 rosalita sshd[61781]: error: PAM: authentication error for illegal user david from ppp-vpdn-80.255.179.150.yarnet.ru
Oct 29 00:11:29 rosalita sshd[61781]: Failed keyboard-interactive/pam for invalid user david from 80.255.179.150 port 49013 ssh2

More material later, for now I'd be happy to hear confirmation (reports of similar activity) if you see it in your logs.

Note: A Better Data Source Is Available
Update 2013-06-09: For a faster and more convenient way to download the data referenced here, please see my BSDCan 2013 presentation The Hail Mary Cloud And The Lessons Learned which summarizes this series of articles and provides links to all the data. The links in the presentation point to a copy stored at NUUG's server, which connects to the world through a significantly fatter pipe than BSDly.net has.