Buying Computers Properly
There’s been a bit of a fuss lately about a New York Times article that said that Baltimore city computers were hacked in part by an exploit known as EternalBlue that was stolen from the NSA a couple of years ago. I don’t know if the attackers targeted the vulnerability used by EternalBlue, nor if they used the actual purloined code. What’s more interesting to me is that Microsoft released a patch for the vulnerablity in March 2017, but many city government systems remained unpatched and hence vulnerable. Why? What did Baltimore (and other organizations, especially governments) do wrong? Why were these systems unpatched and hence vulnerable for so long?
(A semantic note: a vulnerablity is the actual flaw in some software package. An exploit is the code that takes advantage of it. EternalBlue is NSA’s exploit for the CVE-2017-0144 vulnerability.)
Don’t get me wrong; as I’ve written before, patching is hard and risky. I also noted that sometimes, it’s important to take the risk. (And yes, in that latter blog post I was writing about EternalBlue.) But this incident shows a more serious failing: EternalBlue and other exploits targeting that vulnerability have been a threat for more than two years. Baltimore’s systems have remained unpatched for that long.
The root of the problem is attitudinal: in many places, computers are treated as capital equipment with a fairly long lifespan, and as devices that need operation but not maintenance. These attitudes may date back to the 1950s, when the first was fairly true and the cost of maintenance was hidden in the operational cost. Neither is true today. Computers are consumables that require regular, skilled care. Skipping this care is like not changing the oil in your car: you can get away with it for a little while, but at some point you’re in trouble. In fact, and as I explain below, it’s worse than dirty engine oil: not only are you at risk for a security incident, you end up in a maintenance trap.
To paraphrase George R.R. Martin’s famous line “valar morghulis”, all code must die. In fact, software is rarely healthy even to start with; vendors constantly issue patches for their products. Eventually, though, the patches stop: there’s a new version of the product, and vendors have no interest in continuing to support ancient versions. Not only would they rather sell you something new, there’s no viable economic model to pay for continued patch development for older versions. The cost of the first few years of patching is, of course, built in to the initial price of the software. Ultimately, of course, a software package is succeeded by a newer version. Switching versions is in some sense the ultimate patch, and just as patching can be hard, upgrading can be very hard.
Worse yet, patching and upgrading are expensive. System administrators and application programmers need to test and perhaps modify their code for compatiblity with the new version. Extra hardware may be needed, partially as test machines but also because at some point, newer vendor packages (especially operating systems) simply won’t run on older computers. If money is tight, it becomes very tempting to postpone patches and upgrades. After all, the reasoning seems to go, if things are working now, why bother changing?
Government agencies are particularly vulnerable here. Year-to-year budgets aren’t that predictable; they’re at the mercy of political winds, and tax increases are never popular. (Partisan anti-tax rhetoric is another factor, of course.) If there’s a funding crunch, deferring software maintenance is an easy thing to cut, especially since it’s hard to lay off civil servants and city governments can’t easily sell off agencies. (Privatization? That’s a political question I won’t go into, and ultimately it doesn’t matter: whatever the irreducible core of governmental functions, that core will be susceptible to the same dilemma.)
So: software maintenance is deferred. What are the consequences? If the problem is a delayed security patch, your site is vulnerable until the patch is installed. But if you miss a version upgrade, you’re in big trouble. For one thing, security patch support tends to end soon thereafter. For another, upgrading to the next version is much harder: there are generally tools to help you migrate to the next version, but tools that reliably upgrade two versions are much, much harder to build, and may not even be available.
The net result is a system that is insecure, unsupported, and more expensive to maintain and upgrade than one that had been patched and upgraded properly all along.
Ultimately, this boils down to money: where will maintenance money come from? The way to solve this problem is to accept, in a tangible form, that a computer is first, a consumable item, and second, one that requires ongoing expenditures. Think of it as a leased automobile: while you have it, you have to insure it, refuel it, and change the oil and tires; when the lease is up, you have to replace it—or do without a car as a means of transportation. In other words, a leased car costs you something to own and operate, and has a relatively short lifespan.
It’s the same with computers. An enterprise computer should be regarded as having a lifespan of about four years, and an enterprise’s budget should include money for the expense of operating and maintaining each and every computer it owns. Furthermore, budgets should include money for operating system and major application upgrades. They will happen, and doing these upgrades will cost money—but these upgrades are utterly necessary. (No, I’m not saying that a corporation should instantly switch to the next release of WhateverOS as soon as it comes out—I learned “never install .0 of anything” back in 1970—but not all that long after WhateverOS X.0 comes out, release X-1 will drop out of support.)
In a well-run corporation, the CIO would put advice like that into practice. (In a corporation with a well-run IT department, they already know this and do it.) It would be an interesting exercise to try to mandate it for government agencies, either by law or by executive order. I’d be interested in suggestions on how to do it.
