The Open Source Quality Challenge
I realized this morning that I had to upgrade Firefox — again. It seems that 3.0.9 had a security problem. It’s less than a week since the last time I had to upgrade: 3.0.8 had security problems, too. In fact, in the year or so that Firefox 3 has been out, there have been about 50 official security advisories, 30 rated critical or high severity. What’s going on?
We have known for a long time that most security holes are simply a particular form of bug. The corollary, of course, is that reducing bugs in general is a good way to reduce the incidence of security problems. Is Firefox too buggy, and hence too insecure?
We’ve also known for a long time that good, structured development process do work. They may be expensive in the short run, but they do pay off. This is the challenge for the open source movement: can it impose such discipline?
Microsoft committed publicly to security improvements several years ago. From where I sit, the effort has been working. Windows is neither bug-free nor security hole-free, and probably never will be; that said, it’s a lot better today than it would have been had Bill Gates not gotten religion about security. I’ve heard a number of theories about why that happened, but those aren’t important; what counts is the end result.
I’ve also heard the claim that Firefox has had fewer days of vulnerability. That sounds great, until you realize that one way to achieve that is by shipping patches quickly, without adequate testing. Consider this security advisory:
One of the security fixes in Firefox 3.0.9 introduced a regression that caused some users to experience frequent crashes. Users of the HTML Validator add-on were particularly affected, but other users also experienced this crash in some situations. In analyzing this crash we discovered that it was due to memory corruption similar to cases that have been identified as security vulnerabilities in the past.Was 3.0.9 released too quickly, necessitating the very rapid release of 3.0.10?
It has been said that "given enough eyeballs, all bugs are shallow". That may be true. What we need now is a way for the many eyeballs to prevent the bugs in the first place. It won’t be easy. Submitting to discipline is difficult for many, and fun for very few. Programmers like to write code, not requirements, design documents, test scripts, and the like. I fear, though, that there are no other choices. Just as there is no royal road to geometry, I fear there is no royal road to correct software.
I’m a fan of open source software. Indeed, I’m not just writing this on a machine running an open source operating system, NetBSD, I’m a NetBSD developer (albeit not a very active one). However, if the open source movement is to fulfill its promise, it needs to solve its buggy code problem. We have several decades of experience that teach us there are no magic solutions or tools that will solve that problem. We’re going to have to do it the hard way.
