Saturday, March 14, 2009

What have the black boxes wrought

How compartmentalization turned into a security disaster. Greed, incompetence and dishonesty was involved. 

IT security or the lack of any such thing has grabbed headlines lately here in Norway. A series of high profile public institutions have seen large scale worm infections on their Microsoft based networks. 

Late last year the regional government agency responsible for essentially all health care in the western part of the country had a worm infection so bad they essentially shut down their network as a preventive measure. 

During the last few weeks, the national police force, of all thinkable organizations, has seen not one, but two large-scale incidents. Use of Microsoft products and sloppy system maintenance are both pervasive enough that similar incidents are likely happening right now elsewhere, somewhere near you too. 

The news reports about the Norwegian police force's IT problems contained one item that was particularly shocking to IT types like me: 

Apparently large parts of the bureaucracy that is responsible for the confidential and correct processing of criminal matters and all sorts of sensitive personal information associated with the crimes runs essential services on Microsoft Windows NT 4.0. 

That version of the Microsoft product is so old it is officially abandonware, and early reports of the police network problems included the oldish news that even the antiware vendors have stopped supporting the system. 

Later reports had police IT department officials claim that the worm infections were not that much of a security problem, since at this point all the worm actually did was spread. Break out the popcorn, boys and girls: In an upcoming episode, we will see how the worm infected Windows machines the Royal Norwegian Police did not find or couldn't clean well enough are used in the perpetration of some cybercrime or other. 

 It's all pretty sickening, and at this point it would be rather tempting to spend the rest of the column ranting about the general stupidity of Windows users. But a smarter approach is to see if there is a lesson to be learned. 

To do that, we need to backtrack quite a bit and look at the cult of the little black boxes. 

The cult of the little black boxes, and Microsoft the 1980s corporation 

We need to go back and take in what the world was like in the nineteen-eighties. This was back when the world was divided into real computers (from the likes of IBM, Digital, and regional quasi-monopolies like our own Norsk Data) and those annoying toys called 'personal' microcomputers, where the 'IBM PC compatibles' had emerged as the surprise leader of the pack. 

Computer networks were usually private, corporate things and rarely interconnected much, with the rare exception of those institutions that were part of the US Department of Defense science experiment that was becoming known variously as ARPANET or 'the Internet'. 

If you took your programming classes back in the nineteen-eighties, you likely know that we were taught that black boxes were good. 

Compartmentalization was the order of the day, meaning that as a developer you were supposed to create code with a well defined inteface, so anybody interacting with your code would be presented with a cleanly predictable result to any given input. 

Your code should be a black box and for any particular specificiation there could be written several interchangeable modules to fit the bill. 

So far so good, predictability is nice and with compartmentalization comes, we hope at least, clear chains of responsibility. 

But several factors combined to take the cult of the black boxes and turn it into a corporate culture at a corporation that was growing from a handful of furry hackers into a corporation at the time, namely Microsoft. 

Microsoft' early successes all came from writing software for single-user systems that were so limited that working around the limitations of the hardware became much of a lifestyle. 

At the start of the decade, networking on microcomputers in Microsoft's range was pretty much out of the question, and by the end of the eighties any sort of connectivity (even dial-up) was still an expensive optional extra that was still too hard to configure for most people. 

On the storage side, we progressed from 128 kilobyte floppies to hard drives of just over a hundred megabytes for personal systems, with the 32 megabyte partition size still a very present limiting factor.

Amazing developments all, but both the applications and the organization grew faster than the hardware could keep up with. 

The organization now had several levels of management, and each one demanded maximum productivity from their underlings. 

Keeping in mind that each programmer or team would be writing little black boxes anyway, it made perfect sense to set up the software production line so each developer only had access to those parts of the system he or she was supposed to be working on. 

That way developers would be concentrating on their main task and minimize time spent waiting for compiles to finish. At predetermined times the developers would then upload the source code for their little black boxes to a central build system. 

The only people who had all the parts of the projects were in fact the custodians of the build system. 

Source code version control systems were made part of the process, but there is anecdotal evidence that the transition from standalone hacking to a version control regime was a rough one for many early Microsoft developers. 

Only a few days ago I offered pretty much the content of the last few paragraphs to a table of youngish free software developers over beer. 

The reaction was quick an unanimous: 

"That way, nobody really knows what's going on in the software". 

That is a very valid point, and it proves how far we've come with free software. 

At the same time there is every reason to believe that the extreme compartmentalization that Microsoft established for its product development in the 1980s was the way things were done there until very recently, if indeed it has changed at all. 

By the mid-1990s when Microsoft had been dragged kicking and screaming into modern-day network environments, and the ongoing saga of internet-enabled malware started in earnest (I've written a summary in a malware paper -- an updated version is available here), with the company moving from early denial of any bugs whatsoever through a near-constant barrage of emergency hotfixes to today's monthly megapatch regime. 

With the source code still a closely guarded (if occasionally leaked) secret, there is really no way for us to know if they've learned any lessons at all. 

One indication that they still have some way to go is this Infoworld article about the state of their protocol documentation (summary: it's not to be trusted at all). As for the state of the source code, all we can do is to study the flow of urgent patches. 

Much better then to learn how it should be done - Say from Theo de Raadt's AsiaBSDCon 2009 presentation about how OpenBSD's release process works, and if you want more of the gory details, do check his classic exploit mitigation presentation. Also, most likely you could do worse than read Damien Miller's OpenSSH security presentation (full text here). 

It's all those little things we do, at FreeCode and in free software in general. If you found this column useful, entertaining or irritating, please drop me a line.


  1. It's nice to know there's a few people out here who know how much Microsoft is stuck in their bad habits from the Eighties. You can't teach an old dog new tricks :-)

  2. What a long post to just say that 1) it is quite stupid to use NT4.0 blackbox in a mission critical institution. 2) All eyes on it is better than just my vendor's eyes on it and 3) That's why OSS could be more secure than proprietary one (which is still not proven)

  3. This "black box" approach to development had actually been tried by the post-McCarthy-era U.S. military. To maintain secrecy on their projects, they subdivided them into "black box" components. The results were systems that simply didn't work, once the components were integrated. Sound familiar?

    I'm not sure what you find surprising in IBM's early dominance of the PC market. The Osbornes, Kaypros and numerous others had established a market existed, but CP/M software for one machine wouldn't work on another. With the advent of the IBM PC, a reliable standard was set, and software for the platform flourished.

    Of course, the real crux was the breaking of the PC's ROM BIOS, without copyright infringement. That is what caused the explosion of clones, the erosion of price to commodity levels, (a desktop, Z80/8085 system with a couple 8" floppies sold for around $10,000 in 1980, in 1980 dollars), and the beginning of a general adoption of the PC by the public.

  4. I remember when Bill Gates said that the Internet was a "fad".

  5. I've known plenty of old dogs to be taught new tricks, it just takes a while. What you can't teach new tricks are management staff, young or old.

  6. Old dogs very often have a problem where they urinate or defecate all over the house. Does this sound like a software maker we know?

  7. Microsoft's problems are actually quite easy to fix. And it's the same fix every game program under goes every 5 to 7 years.

    What is that fix ?

    Tear it all down and start over from scratch.

    But no , they insist on building from the already existing (bug infested security nightmare) foundation to save themselves time.

  8. Unfortunately i work for an organization that does not seem to believe in keeping up, nether do our vendors. One of our software vendors (at least the last version we got in 2006) made us install adobe 4.0 and Java 2 1.3 because the software would not run with out such old versions. The Default password on the server is "Support" and yes that is for the administrator account on a win2k3 machine with some updates installed (only when our vendor deems it ok to install updates, and only then they have to do it). While i have raised complaints there is nothing we can do as we are contracted into this mess. Also we use Oracle DB 10g. We are not allowed to install any patches (we still do (at least on the client machines) because of security concerns but have to have a light touch as any patch and it seems to be random breaks something so rolling back a patch is not uncommon) we handle a lot of information that could be used for identity theft if crooks were to get into our systems. They also use net meeting to remote nting and very frustrating to deal with thislog into the machines with no password set at all. it is just very disappointing.


Note: Comments are moderated. On-topic messages will be liberated from the holding queue at semi-random (hopefully short) intervals.

I invite comment on all aspects of the material I publish and I read all submitted comments. I occasionally respond in comments, but please do not assume that your comment will compel me to produce a public or immediate response.

Please note that comments consisting of only a single word or only a URL with no indication why that link is useful in the context will be immediately recycled so those poor electrons get another shot at a meaningful existence.

If your suggestions are useful enough to make me write on a specific topic, I will do my best to give credit where credit is due.