Are we having fun yet?

Are we having fun yet? I'm obviously not the only person interested in the aspects of project management that I discussed two months ago: I had quite a bit of feedback. Ville-Pertti Keinonen writes:

In a commercial environment it's easy to designate someone as the lead developer in a project and this person can be guaranteed to work on the project full-time. Even though FreeBSD does have an ``authority'' in place (the core team), it remains problematic to assign someone with the responsibility for something that they are ultimately working on in their spare time and utilizing their limited spare energy.
A mere arbitrator/coordinator (a typical ``project management'' role) is not helpful unless that person is actually capable of significant technical decisions and even a significant part of the implementation. IMO one good, motivated, full-time developer is a significant step towards a project that will produce results, no matter how large the project. For free software, it is difficult to find people for such roles; motivated people with time and energy most likely prefer to work on ``build-from-scratch'' projects, since they are much more exciting than incremental, even if significant, improvements to existing projects.

David Beck wrote:

To my opinion the need for the project management is a direct consequence of the FreeBSD kernel architecture. I believe that a different design where each part of the system would be more independent could be of great help. One example is the microkernel architecture. I also believe that this could be even better with a more revolutionary approach.

That's an interesting idea, but it's obviously not practical at this stage. Any project manager must work within the constraints of the project. I also think that, while different source structure might simplify the problem, it won't remove it.

Mark Pickersgill wrote:

I'd agree with your comment regarding developers no longer being able to work on whatever and however they like. I have seen many open-source projects fall-over in an un-maintained heap simply because to fix things that someone else had done in an incorrect or bizzare way would require a fair amount of effort.
I think the way in which FreeBSD (and many other open-source projects) needs to approach its development is to architect and aim the project in a direction that allows people to work on items of the project more separately.
There's no point in attempting to implement a new scheduler, if the resource of skills is high in the department of driver developers, and there's only a handful of people with the know-how of implementing a new scheduler.
So design and project goals would be picked, based on:

agreement between interested parties

the resources available

If there are agreed goals for which there is a shortage of skills, then perhaps:

scouting for help by making FreeBSD better known, or

working on decoupling that section of the project from the rest (via re-architecting) to allow other goals to be carried out relatively un-hindered,

is in order.
In short, I think FreeBSD may need to re-architect some aspects of the code/project, so that things are not as coupled. For example, if I have a piece of hardware, for which I want to write a driver for, I should be able to do so without needing to worry about the changes that joe-blow may make to the scheduler.

There's a lot of interesting stuff in this message, which I have already trimmed significantly for publication. Much of it, of course, is unattainable: one of the issues he didn't mention was that the core team and the project as a whole needs to buy off on these ideas, and I don't see that happening. Still, it's possible that some of these ideas could help.

FreeBSD Core team again

The really interesting thing that happened in the last two months was something else, though, which I hadn't expected: two members resigned from the FreeBSD Core team. Since one had left last year, we were down to six members, and that meant that we needed an early election.

As this article goes to press, voting has just started for the new elections. Twenty-four candidates are standing for nine positions. At least nobody can complain that there's a lack of interest: last time round there were only seventeen. Let's take a look at the issues surrounding the election.

How to have fun

The big problem seems to be the conception that FreeBSD is no fun any more. To quote Jordan Hubbard's resignation message, written on 29 April:

Finally, it also bears noting that while being part of the FreeBSD project is many things, it should always be "fun" to at least some degree for its participants or there's really not much point in being involved.

A little over a week later, Mike Smith also resigned from the core team. While his resignation message was not made available outside the FreeBSD project, the subject line summed it up: ``It's not fun anymore''.

What does this mean? Should being on a core team be fun? Two years ago I observed that the BSDs are anarchies: nobody's in control. At the time, I stated that they seem to be working. Maybe that's changing.

An anarchy works when everybody is in agreement: you don't need an arbiter to make decisions. That's easy in a small group. You shouldn't need anybody in charge of a group of five people. By the time they get to be fifty, things aren't quite as easy, but with a bit of goodwill and mutual respect it can still work.

The trouble is that the FreeBSD team now has over 300 developers. Not everybody knows everybody, and there's a tendency to subgroup. That's what I see happening in the FreeBSD project at the moment. That can lead to tensions, of course, if there's nobody in charge.

Who makes the decisions?

When the projects were first formed, the role of the Core Team was pretty clear: provide a direction for the project. As a result, the members of the team were primarily the most active developers. As time went on, and the subject matter became more complicated (see the BSD Project Management article), it became more difficult for the core teams to maintain an architectural overview. More and more it was left to individual developers to decide on features.

There are obvious problems with this approach:

Different people decide on different features. The result looks inconsistent. I personally think that one of the outstanding features of BSD is the uniform ``feel'' of the system. Compare this to the System Vs which are available for the Intel architecture and you'll see the difference. To maintain this look and feel, we need a top-down approach.

There's a danger that the system will become more difficult to use. For example, FreeBSD currently comes with a tool called UserConfig to tailor the kernel to specific ISA board settings. FreeBSD 5 will have a significantly improved boot process, and UserConfig doesn't interface to it. People are left with the choice: modify UserConfig to use the new interface, or discard it. Currently, the choice has been to discard it.
Why? Well, there are good and bad reasons. A good reason is ``ISA is evil. There are almost no ISA boards left worth using. Why burden yourself with it?''. A bad reason is ``I don't have any ISA boards any more, and I don't care about people who do. Let them write the code if they want it''.
Certainly I can relate to both of these reasons. The real issue, though, is: who should make the decision? It's my opinion that any such decision should be made by a group dedicated to the continuity of the project.

Real hackers don't need documentation. If a developer introduces a feature, he may not feel obliged to document it. It's true that we have plenty of willing documentation helpers, but first they need to find out about it. There are certainly too many features which haven't been documented because they ``slipped through the cracks''.

What is the FreeBSD project, anyway?

A more fundamental question is: what is the FreeBSD project? Currently the core team is elected by the developers. That's only part of the picture, though:

The project has somewhat over 300 committers (developers). About half of them have access to the main source tree. The others maintain documentation and the Ports Collection. In a commercial company, these people would be engineers and manual writers.

There's also a relatively small release engineering team. All its members are committers, since they need that access to perform their job. Nevertheless, the duties are separate: developers develop new code, the release engineering team produces properly packaged releases. In a commercial company, they'd be the production people.

In addition, a number of people are involved in FreeBSD advocacy, spreading the word and making FreeBSD better known. The Daemon News people are an example, though they're not tied to a particular project. In a commercial company, they would be the marketing people.

Then there are the people who answer questions on the FreeBSD-questions and FreeBSD-newbies mailing lists. In a commercial company, they would be the support people.

Finally there are the users. In a commercial company, they're the people who pay the money the company needs, and who are ultimately responsible for its survival. Is that so different for FreeBSD? Gone are the times when the project got by without money from outside. A number of companies are pleased enough with FreeBSD that they make donations or fund development. Without them, the project might not fold, but it would certainly suffer, and a number of the project members might wander off in search of more interesting things.

Imagine a company run only by the engineers. I've seen them. They didn't last very long. So if the FreeBSD project can survive in that manner, what's the big difference from a commercial company?

There are a number of obvious differences, of course, but I can't see any that makes it a good idea to have the engineers run the project. That's a historical carry-over from the days when the engineers were the project. But engineers don't always make good managers, and once management enters the scene, the rules change. Commercial interest isn't the key: size is. Volunteer associations face the same issues. I'm currently the secretary of the Australian UNIX User's Group, and we have as much of a formalized management as any company. Without it we couldn't survive. Even with it things aren't easy.

Given this background, I can see three basic future scenarios for the FreeBSD project:

The project becomes a managed entity, like a volunteer association. The core team becomes a management body and makes policy decisions for the project as a whole. In order to do so, it needs to recognize that management functions are different from development functions. This realization has only been partially successful in the current core team.

The project stops being as commercially active as it is currently, and goes back to the way it was five years or so ago. This would have to include scaling back the number of committers to the values they were at the time, where everybody knew everybody else.

Both could happen: the dreaded code fork. It's possible, but I think it's unlikely. Like the majority of people I know, I also think it's a very bad idea. The commercial side would lose good developers, and possibly critical mass, while the old-style project would probably fade into oblivion. If it didn't, we'd just have YaBSD to contend with.

Which is it going to be? I have no better idea than anybody else. More than ever before, I don't know where the project is going. We'll see in about a month's time.

But is it fun?

Getting back to the issue of having fun: yes, I've had a lot of fun in the FreeBSD project, and I agree that it's been more trying lately. But why? I've had lots of fun in large commercial organizations as well, so that's not the real issue. My take is that the problem is simply a matter of expectations: we're facing problems we haven't seen before, and we don't want to. It's not that we can't deal with the problems, it's that we don't want to have to deal with them. Yes, a larger project will require more rules, but that doesn't mean that it'll be less fun as long as people adhere to them. The problem I'm seeing at the moment is that we don't have the rules. One of the first things that the next core team should do is to ensure that enough sensible and intelligible rules are put in place.

What does this have to do with NetBSD and OpenBSD?

The last couple of months I've been talking specifically about FreeBSD. When I started these articles nearly four years ago, I promised I wouldn't do that too often. Obviously I think the issue is important, but these articles are supposed to be about all kinds of BSD.

I think that the issues we're seeing here have a direct relevance to the other BSDs. It happened in FreeBSD first because of the size of the developer community; we can expect to see similar issues in the NetBSD community when it reaches a similar size. OpenBSD may be an exception, however: the project is much more directly controlled by Theo de Raadt. Last time I checked, he had been responsible for over 30% of all commits to the OpenBSD tree, so he's also the obvious choice for management. But Theo's dominance of the project also limits the size it can become. Maybe OpenBSD will succeed as an example of the second alternative I mention above.