Preliminary Thoughts on The Equifax Hack
As you’ve undoubtedly heard, the Equifax credit reporting agency was hit by a major attack, exposing the personal data of 143 million Americans and many more people in other countries. There’s been a lot of discussion of liability; as of a few days ago, at least 25 lawsuits had been filed, with the state of Massachusetts preparing its own suit. It’s certainly too soon to draw any firm conclusions about who, if anyone, is at fault—we need more information, which may not be available until discovery during a lawsuit—but there are a number of interesting things we can glean from Equifax’s latest statement.
First and foremost, the attackers exploited a known bug in the open source Apache Struts package. A patch was available on March 6. Equifax says that their "Security organization was aware of this vulnerability at that time, and took efforts to identify and to patch any vulnerable systems in the company’s IT infrastructure." The obvious question is why this particular system was not patched.
One possible answer is, of course, that patching is hard. Were they trying? What does "took efforts to identify and to patch" mean? Were the assorted development groups actively installing the patch and testing the resulting system? It turns out that this fix is difficult to install:
You then have to hope that nothing is broken. If you’re using Struts 2.3.5 then in theory Struts 2.3.32 won’t break anything. In theory it’s just bug fixes and security updates, because the major.minor version is unchanged. In theory.Were they trying hard enough, i.e., devoting enough resources to the problem?In practice, I think any developer going from 2.3.5 to 2.3.32 without a QA cycle is very brave, or very foolhardy, or some combination of the two. Sure, you’ll have your unit tests (maybe), but you’ll probably need to deploy into your QA environment and do some kind of integration testing too. That’s assuming, of course, that you have a compatible QA environment within which you can deploy your old, possibly abandoned application.
Ascertaining liability here—moral and/or legal—can’t be done without seeing the email traffic between the security organization and the relevant development groups; you’d also have to see the activity logs (code changes, test runs, etc.) of these groups. Furthermore, if problems were found during testing, it might take quite a while to correct the code, especially if there were many Struts apps that needed to be fixed.
As hard as patching and testing are, though, when there are active exploitations going on you have to take the risk and patch immediately. That was the case with this vulnerability. Did the Security group know about the active attacks or not? If they didn’t, they probably aren’t paying enough attention to important information sources. Again, this is information we’re only likely to learn through discovery. If they did know, why didn’t they order a flash-patch? Did they even know which systems were vulnerable? Put another way, did they have access to a comprehensive database of hardware and software systems in the company? They need one—there are all sorts of other things you can’t do easily without such a database. Companies that don’t invest up front in their IT infrastructure will hurt in many other ways, too. Equifax has a market capitalization of more than $17 billion; they don’t really have an excuse for not running a good IT shop.
It may be, of course, that Equifax knew all of that and still chose to leave the vulnerable servers up. Why? Apparently, the vulnerable machine was their "U.S. online dispute portal". I’m pretty certain that they’re required by law to have a dispute mechanism, and while it probably doesn’t have to be a web site (and some people suggest that complainants shouldn’t use it anyway), it’s almost certainly a much cheaper way to receive disputes than is paper mail. That opens the possibility that there was a conscious decision that taking the risk was worthwhile. Besides, if many applications needed patching and they had limited development resources, they’d have had to set priorities on whic web servers were more at risk. Again, we need more internal documents to know.
Some text in the announcement does suggest either ignorance or a conscious decision to delay patching—the timeline from Equifax implies that they were able to patch Struts very quickly after observing anomalous network traffic to that server. That is, once they knew that there was a specific problem, rather than a potential one, they were able to respond very quickly. Alternatively, this server was on the "must be patched" list, but was too low down on the priority list until the actual incident was discovered.
We thus have several possible scenarios: difficulty in patching a large number of Struts applications, ignorance of the true threat, inadequate IT infastructure, or a conscious decision to wait, possibly for priority reasons. The first and perhaps last would seem to be exculpatory; the others would seem to leave the company in a bad moral position. But without more data we can’t distinguish among these cases.
A more interesting question is why it took Equifax so long to detect the breach. They did notice anomalous network traffic, but not until July 29. Their statement says that data was exposed starting May 13. Did they have inadequate intrusion detection? That might be more serious from a liability standpoint—unlike patching, running an IDS doesn’t risk breaking things. You need to tune your IDS correctly to avoid too many false positives, and you need to pay attention to alerts, but beyond dispute an enterprise of Equifax’s scale should have such deployed. It is instructive to read what Judge Learned Hand wrote in 1932 in a liability case when some barges sank because the tugboat did not have a weather radio:
Indeed in most cases reasonable prudence is in fact common prudence; but strictly it is never its measure; a whole calling may have unduly lagged in the adoption of new and available devices. It may never set its own tests, however persuasive be its usages. Courts must in the end say what is required; there are precautions so imperative that even their universal disregard will not excuse their omission… But here there was no custom at all as to receiving sets; some had them, some did not; the most that can be urged is that they had not yet become general. Certainly in such a case we need not pause; when some have thought a device necessary, at least we may say that they were right, and the others too slack… We hold [against] the tugs therefore because [if] they had been properly equipped, they would have got the Arlington [weather] reports. The injury was a direct consequence of this unseaworthiness.It strikes me as entirely possible that Equifax’s exposure is greater on this issue than on patching.
This is a big case, affecting a lot of people. The outcome is likely to change the norms of how corporations world-wide protect their infrastructure. I hope the change will be in the right direction.
