BSD project management

BSD project management My last article aroused more interest than I've had for a while. Gratifyingly, almost all people who commented agreed that we need to be open to the GNU and Linux people. I did have one strong objection, though: Brett Glass complained that I was trying to defame him.

I would like to make it very clear that I have had no intention of defamation. I am criticizing his approach, not his person. In fact, I find him quite an agreeable person when he's not in GPL-bashing mode. On a previous occasion I referred to his approaches without mentioning his name, and he became very upset about that, so I agreed to mention his name if I referred to him again. And of course he is welcome to submit an article explaining his point of view.

The BSDCon

In February we had the third US BSDCon, this time in San Francisco. For the first time it was run by USENIX.

I found that the event was not as successful as previous BSDCons. It would be unfair to blame USENIX for this: things have changed a lot in the last year, and we've seen much less activity at other conferences as well. In addition, of course, the novelty is beginning to wear off. This is the fourth BSDCon I've been to, and people are getting used to them, so for example there were fewer photos taken and published on the web than on previous occasions.

From my point of view, I found the subsequent FreeBSD kernel summit to be the most interesting part of the conference: increasingly, we're getting a lot of FreeBSD hackers together to discuss the direction of the project. This time we had about 30 people crammed into a room which had been intended for far fewer. It was well organized, and we got through our material on time. It did, however, highlight a problem that is becoming increasingly important: how do we organize the project?

Free software project management

Last month marked the tenth anniversary of the first public release of 386BSD. That release was not even what we would call alpha quality today: to give a feeling, the FAQ which Karl Lehenbauer published on 29 March 1992 contained the following warning:

WARNING * WARNING * WARNING
===========================

Do not even test-boot 386BSD from floppy with a hard disk present on
the system if you do not want to risk trashing your hard disk!

If 386BSD crashes, it may write a core dump on your hard disk!

At that time, there was plenty of work for all volunteers to do just to make it into a system reliable enough to be an alternative to commercial software. There are a lot of stories to tell about how that happened, and how the NetBSD and FreeBSD projects arose as a result, but I would guess that the systems passed the Microsoft ``Windows'' reliability threshold in 1994 or 1995.

NetBSD, FreeBSD or BSD/386

I was in a different position. At almost exactly the same time, I bought my first copy of BSDI's BSD/386, beta version 0.3.2 on tape. By comparison with 386BSD, it was rock solid. By comparison with Microsoft, it was rock solid. In fact, by comparison with the Interactive UNIX/386 that I was running on another machine, it was rock solid. I had originally bought BSD/386 because I had just left a company where I had access to the UNIX (System V) source code, and I felt lost without it. In comparison with the cost of a System V license, $1000 seemed to be a small price to pay, and I hadn't expected to get very much for it. After a couple of weeks, though, I was hooked. This was a system that really worked, and which didn't hang or panic the way the Interactive system did. It also ``felt'' better than System V: things worked without having to be worked around.

At that time, I didn't have very much to do with the free versions of BSD. I had paid my money, I had a product which I thought was very good, and which definitely didn't overwrite my disks on the rare occasions when it panicked. I came to FreeBSD by another channel: I was porting software to Novell's UnixWare, and I needed a US publisher. I finally came to Walnut Creek CDROM, who had supported FreeBSD right from the beginning. That's how I became involved with the FreeBSD project. I installed FreeBSD and played around with it, and I installed it on some customer sites where they didn't want the expense of BSD/386, but I didn't really use it much myself.

That had to change in 1995, when Walnut Creek asked me to write a book about FreeBSD. To my surprise, by that time it was no longer clear that FreeBSD was any worse than BSD/OS, as BSD/386 was now called. In the course of time, I stopped using BSD/OS altogether--I didn't even install the last update I received. FreeBSD had made it.

What next?

That was six years ago, more than half the life of the project. Once the system was stable and showed reasonable performance, development targets changed. There was still plenty to do:

Device drivers were sadly lacking. As new hardware came out, new drivers were needed. This is still an ongoing issue.
The system needed to be adapted to new buses. In 1992, about the only buses a computer were the ISA bus and the Microchannel (MCA). EISA was also there, but it was pretty compatible with ISA. MCA wasn't important enough to worry about, and FreeBSD has never supported it. In the years that followed, though, a number of new buses appeared, notably the PCI bus and the laptop PCMCIA standard. In FreeBSD, the first support for both was pretty rudimentary, and to get things right required significant restructuring of the kernel, some of which is still going on. Similar issues apply to interfaces like USB, FireWire and infrared (IrDA) and system management features like APM and APCI.
Multiple processor machines appeared and became relatively commonplace. FreeBSD had to be adapted to handle multiple processors. The first attempt was simple but not very effective: UNIX was never designed for multiple processors, and the compromise chosen performed very badly under some circumstances, notably those chosen by the Mindcraft Benchmark. As a result of this, we decided to reimplement the SMP code in a more intelligent way, about which I reported in August 2000. That work is still ongoing, and there's no evidence that it's nearing completion.

And the problem?

There is a difference in approach between these three categories. If you decide to write a driver, say, for a new sound card or Ethernet adaptor, you can go and do it. Depending mainly on how much support you get from the manufacturer, it may be easy, difficult or impossible. But when you're finished (if you finish), the rest is simple: you commit it to the source tree and wait for people to tell you about bugs, then you fix them. The majority of users who don't have that specific hardware hardly even know you've committed the software, and they are certainly not inconvenienced.

Now look at the new bus code. That looks pretty simple, really: if you only have an ISA bus, what do you care if somebody commits software for PCMCIA?

To a certain extent, you don't. But rewriting the bus code to support multiple buses also affects the ISA code. There could be a number of cases where code introduced to support PCMCIA will also affect the ISA code. Even if you, as a user, don't have any problems, the device driver writers will. The interfaces change, and typically people like writing code more than documenting it, so it can be a frustrating experience just to find out how to modify your existing driver, or how to write a new driver.

Rewriting the SMP code is the most invasive of all. It affects everything in the system. You can't write kernel code without being affected. In August 2000 I wrote:

For some time, I have had a theory that the open source model works well for small projects, but it is not optimal for really big undertakings. Even before the Mindcraft incident I had decided that getting good SMP support for BSD would be a proof of this theory.

As I went on to describe, help came from an unexpected quarter: BSDi gave us alpha-quality SMP code, and we got it working pretty quickly. That didn't solve the problem, of course, it just gave us the framework in which we could develop the new software.

And the problem, again?

The real problem we're seeing here is the issue of project management. In the old days, people could go off and do their own thing. Nobody worried, because most things didn't affect anybody else, and it was pretty much left to the individual to decide what to do and how to do it.

All that changes radically with projects like the SMPng project. Just about anything anybody does will have an effect on the other developers. Under these circumstances, the ``go it alone'' approach doesn't work well. We've seen the results already in the project: the initial set of locking primitives (which the project tenaciously insists on calling mutexes) were not adequate for the desires of some of the developers, so they introduced their own favourite constructs. It didn't take at all long before rules (not just documentation) had to be introduced to describe how synchronization should be done.

And the solution?

So we've identified the problem. But how do we solve it? Commercial software development is quite clear: there's a project manager, and his word is law. But that means more planning than the average free software hacker wants to do, and more compromise than many are prepared to accept. We still have people in the project who take the attitude ``I want to work on this particular issue, and it's my God-given right to do so''. That worked once upon a time, but it really doesn't work any more. BSD may be free, but even volunteer projects need some management.

I don't know the answer to this one. As I said over two years ago, this issue may be the one which changes the shape of our projects for ever. My best bet is that some commercial entity developing BSD takes over the project and supplies the management; but that would probably involve an independent development which would be put back into the project on completion.

I'd be interested in feedback on what other people think about these issues. In a certain sense, we're treading new ground. As always, the time ahead will be interesting.