Security is a System Property
There’s lots of security advice in the press: keep your systems patched, use a password manager, don’t click on links in email, etc. But there’s one thing these adages omit: an attacker who is targeting you, rather than whoever falls for the phishing email, won’t be stopped by one defensive measure. Rather, they’ll go after the weakest part of your defenses. You have to protect everything—including things you hadn’t realized were relevant. Security is a systems problem: everything matters, including the links between the components and even the people who use the system.
Passwords are a good illustration of this point. We all know the adage: "pick strong passwords". There are lots of things wrong with this and other simplistic advice with passwords, but we’ll ignore most of them to focus on the systems problem. So: what attacks do strong passwords protect against?
The original impetus for this advice came from a 1979 paper by Bob Morris and Ken Thompson. (Morris later became Chief Scientist of the NSA’s National Computer Security Center; Thompson is one of the creators of Unix.) When you read it carefully, you realize that strong passwords guard against exactly two threats: someone who tries to login as you, and someone who has hacked the remote site and is trying to guess your password. But strong passwords do nothing if your computer (in those days, computer terminal…) is hacked, or if the line is tapped, or if you’re lured to a phishing site and send your password, in the clear, to an enemy site. To really protect your password, then, you need to worry about all of those factors and more.
It’s worth noting that Morris and Thompson understood this thoroughly. Everyone focuses on the strong password part, and—if they’re at least marginally competent—on password salting and hashing, but few people remember this quote, from the first page of the paper:
Remote-access systems are peculiarly vulnerable to penetration by outsiders as there are threats at the remote terminal, along the communications link, as well as at the computer itself. Although the security of a password encryption algorithm is an interesting intellectual and mathematical problem, it is only one tiny facet of a very large problem. In practice, physical security of the computer, communications security of the communications link, and physical control of the computer itself loom as far more important issues. Perhaps most important of all is control over the actions of ex-employees, since they are not under any direct control and they may have intimate knowledge about the system, its resources, and methods of access. Good system security involves realistic evaluation of the risks not only of deliberate attacks but also of casual authorized access and accidental disclosure.(True confession: I’d forgotten that they noted the scope of the problem, perhaps because I first read that paper when it originally appeared.)
I bring this up now because of some excellent reporting about hacking and the 2016 election. Voting, too, is a system—it’s not just voting machines that are targets, but rather, the entire system. This encompasses registration, handling of the "poll books"—which may themselves be computerized—the way that poll workers sign in voters, and more. I’ll give an example, from the very first time I could vote in a presidential election: the poll workers couldn’t find my registration card. I was sent off to a bank of phones to try to call the county election board. The board had far too few phone lines, so I kept getting busy signals, all the while thinking nasty thoughts about attempts to keep Yankees (I’d just moved to North Carolina) and students (I was there for grad school) from voting.
Think of all of the system pieces in just that part of the election. There was the poll worker—was she honest? There was the election book, and whatever processes, mechanisms, software, or people had gone into compiling it. There was the phone bank I was using, the phone network, the phones at the election board, the people there, and their backend systems that had a master copy of the election roll. My story had a happy ending—the poll worker kept checking, and found that my card has been misalphabetized—but if an analogous problem happened today with an electronic poll book, it’s hard to see how the poll worker’s diligence could have resolved it. (For other interesting systems aspects of voting, including issues with poll books, see an old blog post of mine.)
The systems aspect of voting is apparent to some, of course, including the New York Times reporters who are covering the hacking story:
Michael Wines, who covers election issues for the Times, said that what stood out to him was the vulnerability of the nation’s vast Rube Goldberg election system. Elections, he explained, "are run by understaffed, underfinanced and sometimes undertrained local officials, serviced by outside contractors who may or may not be well vetted, conducted with equipment and software that may or may not be secure." [emphasis added]
Almost all security problems are system problems; beware of people who try to sell you simplistic, point solutions. It’s not that these solutions are wrong; rather, they have to be examined for their role in securing the system. Consider HTTPS—encrypted—web connections. Unless you’re being targeted by law enforcement or a major intelligence agency, the odds of your connection being tapped on the backbone are vanishingly small. However, it’s trivial to tap someone’s WiFi connection if you’re on the same net as them, e.g., in a public hotspot. So—it’s a good idea to encrypt web pages, but if the environment is strictly controlled LAN to controlled LAN, that should be far down on your list of security priorities. And remember: encrypting one link does not solve any of the many other vulnerable points in your system.
Update: vote-tallying software is also part of the system—and can be insecure.
Preliminary Thoughts on The Equifax Hack
As you’ve undoubtedly heard, the Equifax credit reporting agency was hit by a major attack, exposing the personal data of 143 million Americans and many more people in other countries. There’s been a lot of discussion of liability; as of a few days ago, at least 25 lawsuits had been filed, with the state of Massachusetts preparing its own suit. It’s certainly too soon to draw any firm conclusions about who, if anyone, is at fault—we need more information, which may not be available until discovery during a lawsuit—but there are a number of interesting things we can glean from Equifax’s latest statement.
First and foremost, the attackers exploited a known bug in the open source Apache Struts package. A patch was available on March 6. Equifax says that their "Security organization was aware of this vulnerability at that time, and took efforts to identify and to patch any vulnerable systems in the company’s IT infrastructure." The obvious question is why this particular system was not patched.
One possible answer is, of course, that patching is hard. Were they trying? What does "took efforts to identify and to patch" mean? Were the assorted development groups actively installing the patch and testing the resulting system? It turns out that this fix is difficult to install:
You then have to hope that nothing is broken. If you’re using Struts 2.3.5 then in theory Struts 2.3.32 won’t break anything. In theory it’s just bug fixes and security updates, because the major.minor version is unchanged. In theory.Were they trying hard enough, i.e., devoting enough resources to the problem?In practice, I think any developer going from 2.3.5 to 2.3.32 without a QA cycle is very brave, or very foolhardy, or some combination of the two. Sure, you’ll have your unit tests (maybe), but you’ll probably need to deploy into your QA environment and do some kind of integration testing too. That’s assuming, of course, that you have a compatible QA environment within which you can deploy your old, possibly abandoned application.
Ascertaining liability here—moral and/or legal—can’t be done without seeing the email traffic between the security organization and the relevant development groups; you’d also have to see the activity logs (code changes, test runs, etc.) of these groups. Furthermore, if problems were found during testing, it might take quite a while to correct the code, especially if there were many Struts apps that needed to be fixed.
As hard as patching and testing are, though, when there are active exploitations going on you have to take the risk and patch immediately. That was the case with this vulnerability. Did the Security group know about the active attacks or not? If they didn’t, they probably aren’t paying enough attention to important information sources. Again, this is information we’re only likely to learn through discovery. If they did know, why didn’t they order a flash-patch? Did they even know which systems were vulnerable? Put another way, did they have access to a comprehensive database of hardware and software systems in the company? They need one—there are all sorts of other things you can’t do easily without such a database. Companies that don’t invest up front in their IT infrastructure will hurt in many other ways, too. Equifax has a market capitalization of more than $17 billion; they don’t really have an excuse for not running a good IT shop.
It may be, of course, that Equifax knew all of that and still chose to leave the vulnerable servers up. Why? Apparently, the vulnerable machine was their "U.S. online dispute portal". I’m pretty certain that they’re required by law to have a dispute mechanism, and while it probably doesn’t have to be a web site (and some people suggest that complainants shouldn’t use it anyway), it’s almost certainly a much cheaper way to receive disputes than is paper mail. That opens the possibility that there was a conscious decision that taking the risk was worthwhile. Besides, if many applications needed patching and they had limited development resources, they’d have had to set priorities on whic web servers were more at risk. Again, we need more internal documents to know.
Some text in the announcement does suggest either ignorance or a conscious decision to delay patching—the timeline from Equifax implies that they were able to patch Struts very quickly after observing anomalous network traffic to that server. That is, once they knew that there was a specific problem, rather than a potential one, they were able to respond very quickly. Alternatively, this server was on the "must be patched" list, but was too low down on the priority list until the actual incident was discovered.
We thus have several possible scenarios: difficulty in patching a large number of Struts applications, ignorance of the true threat, inadequate IT infastructure, or a conscious decision to wait, possibly for priority reasons. The first and perhaps last would seem to be exculpatory; the others would seem to leave the company in a bad moral position. But without more data we can’t distinguish among these cases.
A more interesting question is why it took Equifax so long to detect the breach. They did notice anomalous network traffic, but not until July 29. Their statement says that data was exposed starting May 13. Did they have inadequate intrusion detection? That might be more serious from a liability standpoint—unlike patching, running an IDS doesn’t risk breaking things. You need to tune your IDS correctly to avoid too many false positives, and you need to pay attention to alerts, but beyond dispute an enterprise of Equifax’s scale should have such deployed. It is instructive to read what Judge Learned Hand wrote in 1932 in a liability case when some barges sank because the tugboat did not have a weather radio:
Indeed in most cases reasonable prudence is in fact common prudence; but strictly it is never its measure; a whole calling may have unduly lagged in the adoption of new and available devices. It may never set its own tests, however persuasive be its usages. Courts must in the end say what is required; there are precautions so imperative that even their universal disregard will not excuse their omission… But here there was no custom at all as to receiving sets; some had them, some did not; the most that can be urged is that they had not yet become general. Certainly in such a case we need not pause; when some have thought a device necessary, at least we may say that they were right, and the others too slack… We hold [against] the tugs therefore because [if] they had been properly equipped, they would have got the Arlington [weather] reports. The injury was a direct consequence of this unseaworthiness.It strikes me as entirely possible that Equifax’s exposure is greater on this issue than on patching.
This is a big case, affecting a lot of people. The outcome is likely to change the norms of how corporations world-wide protect their infrastructure. I hope the change will be in the right direction.
Update on Equifax
A
news report
today claims that Equifax was hacked twice, once in March
(which is very soon after the Struts vulnerability was disclosed) and once
in mid-May. The news article does not say if the same vulnerability
was exploited; it does, however, say that their sources claim that
"the breaches involve the same intruders".
If it was the same exploit, it suggests to me one of the possibilities
I mentioned
two days ago: that the company lacked an comprehensive software
inventory. After all, if you know there’s a hole in some package and
you know that you’re being targeted by attackers who know of it and have
used it against you, you have very strong incentive to fix all instances
immediately. That Equifax did not do so would seem to indicate that
they were unaware that they were still vulnerable. In fact, the real
question might be why it took the attackers so long to return.
Maybe they couldn’t believe that that door would still be open…
On another note, several people have sent me notes pointing out that Susan Mauldin, the former CSO at Equifax, graduated with degrees in music, not computer science. I was aware of that and regard it as quite irrelevant. As I and others have pointed out, gender bias seems to be a more likely explanation for the complaints. And remember that being a CSO is a thankless job.
Update: based on later information, disregard the first two paragrphs of this.
Yet Another Update on Equifax
The other day, I noted that Equifax had been breached in March, and quoted the article as saying that the attackers had been "the same intruders" as in the May breach. In a newer news report, Equifax has denied that:
"The March event reported by Bloomberg is not related to the criminal hacking that was discovered on 29 July," Equifax’s statement continues. "Mandiant has investigated both events and found no evidence that these two separate events or the attackers were related. The criminal hacking that was discovered on 29 July did not affect the customer databases hosted by the Equifax business unit that was the subject of the March event."So: I’ll withdraw the speculation I posted about this incident confirming one of my hypotheses and wait for further, authoritative information. I repeat my call for public investigations of incidents of this scale.
Also worth noting: Brian Krebs was one of the very few to report the March incident.
