For the times, they are a'changing

For the times, they are a'changing
by Greg Lehey

Things have been pretty active lately. The big news of the month (or is that of the year?) is that Berkeley Software Design, Inc. (BSDI) have acquired Walnut Creek CDROM, the main sponsor of the FreeBSD project. We'll look at that below; first, a couple of other observations.

``Not invented here'' department

Only the most die-hard of UNIX hackers will have missed the fact that Microsoft has released a new operating environment, called Windows 2000 (and thus ending our joking about whether the next Microsoft system would be called Windows 0). Understandably, Microsoft has put up a lot of publicity on its web site, and recently a number of people stumbled over the description of the Single Instance Store, which is intended to reduce the storage of redundant copies of files. The description reminded us so much of symbolic links that a number of jokes went around about it, along the lines of Henry Spencer's famous quote, to be found in fortune:

Those who do not understand Unix are condemned to reinvent it, poorly.

In fact, the SIS is not the same thing as a symbolic link. The discussions were obviously significant enough that Microsoft sat up, took notice, and modified the page to describe the differences between SIS and symbolic links. I'll summarize their version in my own words:

It's not a real symbolic link, because a modification to one instance doesn't change the other one. Instead, a ``copy on write'' function makes a second copy. The web page doesn't say whether it makes a copy only of a section of the file (which would be quite clever) or the entire file (which could pose a significant performance problem).

There's a reference count, like for real (``hard'') links. Microsoft doesn't mention hard links, and it's not clear that they know about them.

SIS does things ``automatically'' by scanning the system for identical files, a job which places a significant load on the system. I once wrote a program to do this, and it's not something that you would want to run automatically. Microsoft seems to think this is a feature, not a bug.

To quote: ``SIS exposes special backup APIs that allow a backup application to backup SIS files so that only one copy of the data is placed on the backup media''. Again, this sounds like hard links. UNIX backup programs only back up a single copy of files with multiple links. When they encounter the other links, they simply back up a reference to the first file.

In summary, SIS seems to do the same things that links and symbolic links do, just in a slightly different manner. It also seems to require a lot of resources. It may under certain circumstances have the advantage of automatically creating a copy of files if a modification is made to one of the copies, but it's unlikely that there would be a significant number of copies of frequently modified files. Microsoft obviously expects massive replication:

The result is a feature that frees up as much as 80 to 90 percent of the space on a server, allowing users to store as much as five to 10 times the information as they could before. ``The bottom line is that it saves the administrator time, which is why it?s part of Zero Administration for Windows,'' Bolosky said. ``It?s designed to ease the lives of the technical support staff.''

The ? marks in the text above represent non-standard additions to the specified character set in the original. They appear to be intended to represent an apostrophe, but since they're not part of the ISO 8859-1 character set, which this document claims to be written in, Netscape doesn't recognize them and represents them as ?.

Cultural differences

Lately I've had a fair amount of contact with Linux developers. It's been an interesting experience for a number of reasons, and it's given me a new perspective on how people ``in the know'' view the BSD community (or communities, as they tend to see us). Some of this is reflected in a news article about the BSDI merger, which unfortunately is factually so inaccurate that it's not worth reading. It does make claims that BSD has suffered because of a recently ended lawsuit with AT&T (that was April 1994, for those of you who want to check), and that the BSD communities are continually fighting each other.

The Linux people I have met seem to share some of these views. I can't say, ``no, we're all one big happy family'', because it's not true, but I suspect that the incidents that they're thinking about also lie years back. We have had some spectacular flame wars, and without them there would probably be fewer BSD communities.

But what about Linux, I asked my friends. They all agree that serious disagreements between Linux hackers are very rare. This reminds me somehow of the respective mascots, a daemon for BSD and a particularly placid looking penguin for Linux (never mind that the penguin mascot came into being after Linus was bitten by a penguin, nor that I can't recall any cases of daemons biting).

I can't believe that Linux hackers are such fundamentally different people that they are all sweetness and light where we are continually flaming each other. I'm sure there's some other reason, and I discussed it with my friends. We spoke specifically about kernel modifications, including FreeBSD's famous ``Danish axes''. About the only thing we could come up with is that Linus' word is final: if there's a difference of opinion, Linus decides. In the FreeBSD development environment, in which I am located, things aren't so clear cut: we need a consensus, and it's not always easy to get one. The ensuing flames^H^H^H^H^H^Hdiscussions can become somewhat heated. I assume that the situation in NetBSD and OpenBSD is similar.

I don't think any of us particularly value flaming, so it might seem as if the Linux approach is better. If you value peace and quiet above everything else, this is a reasonable viewpoint. But I don't think it is conducive to the best possible code. Linus is definitely an exceptional hacker, but he's only one person, and he can't always be right. The BSD way may be more strenuous, but it has the potential to create a better overall code base.

Looking at the issue in social terms, it would seem as if Linux is a monarchy, and BSD is a collection of anarchies. Nowadays the term ``anarchy'' has a somewhat negative flavour about it, but that's mainly because in real life an anarchy doesn't work. In BSD development, it seems that it is working, at least up to a point. I'm thinking about the comparison, and I'd welcome feedback about it.

Finally, the merger

Unless you have spent the last month hiding under a rock without a network connection, you'll know about the merger between Walnut Creek CDROM and Berkeley Software Design, Inc, usually known as BSDI. At this point, I should present my rant on the name of the latter company: for reasons I don't understand, a number of people write it BSDi, with a lower case i. I have even heard rumours from people who should know better that the new, merged company will spell its abbreviation BSDi. This is a long-standing tradition, but I can't see any evidence that there is any truth in the spelling. Check out the BSDI web site for further details.

What's in it for me?

The news of the merger brought a flood of mail messages with subjects such as What result would *you* like from the merger? and Is FreeBSD dead ?. I've heard some FUD (Fear, Uncertainty and Doubt) on the FreeBSD lists (``FreeBSD will no longer be free'') and on the NetBSD lists (``How come we missed out on this?'').

So what's really going on? I think the real issue is that some of the details haven't been finalized, but here's my take:

BSDI and Walnut Creek will become one company. The new company will be called BSDI.

Walnut Creek CDROM will stop selling Linux products, particularly Slackware. Instead, it will spin off a company to handle these products. Personally, I think this is bad news for Slackware, which for some time has been losing in popularity against other more recent Linux distributions. Without the Walnut Creek name behind it, things could become even more difficult.

BSDI and the FreeBSD Project will not merge. This means that there is no way for the new BSDI to directly influence the FreeBSD project. On the other hand, this doesn't mean that they won't be able to influence FreeBSD indirectly. Walnut Creek CDROM has been one of the main sources of funding for the FreeBSD project, and the intention is that BSDI will continue this funding. In theory BSDI could influence FreeBSD by controlling the funding.

BSDI and the FreeBSD Project will cooperate to merge BSDI's BSD/OS product and FreeBSD into a single product. The various statements about the way this will happen are less than clear, and this has given rise to some FUD. I think it's easy enough to take a more positive viewpoint: merging two operating systems is a very difficult task, one that to my knowledge has never been done successfully.
A good example of how not to do it is UNIX System V.4, which was supposed to be formed by merging the previous System V code with 4.3BSD. In fact, if you look at the source trees, it seems to be more of an addition than a merge: base System V.4 can do most things either the System V way or the BSD way, which goes quite some way to explain its enormous size. I wouldn't give BSD/OS and FreeBSD much chance, either, except that the systems really are very similar.
Nevertheless, it's a daunting task. At a meeting of the BAFUG in Berkeley on the 23rd March, Bob Bruce, the president of Walnut Creek, described some of the difficulties. It now looks as if the merge will take longer than originally hoped. Given the magnitude of the task, that's not really surprising.

Significant parts of the BSD/OS kernel are better than the corresponding FreeBSD implementation. This applies particularly to SMP (symmetric multiprocessor) support and kernel threads. These two items, in particular, would greatly benefit FreeBSD. On the other hand, merging the code base would also benefit BSD/OS: some (unstated) aspects of FreeBSD are better than the corresponding parts of BSD/OS. Sure, it would have been possible for BSDI to incorporate these improvements into BSD/OS, and indeed some things have been imported, but the main issue is the time that it takes.

Some parts of BSD/OS can't be merged into FreeBSD: there are a number of drivers developed under non-disclosure agreements (NDA), for which BSDI is not allowed to release the source. BSDI will continue to market these in a value-added distribution, which will not be free.

What does this mean for NetBSD and OpenBSD? They're not part of the deal. But that doesn't mean they're left out in the cold. The press announcements specifically state that BSDI intends to maintain close contacts with NetBSD and OpenBSD. It's too early to know what form these contacts might take. I'll discuss this point further in the following section.

OmniBSD

So how do the remaining BSDs relate to each other? It's clearly a good idea to stress the communality of the operating systems, as we have been doing for a year and a half now in Daemon News. As my Linux friends observe, there's very much a feeling that the BSD projects each sit in their own corner and don't communicate. While announcing the merger, the Wall Street Journal spoke of ``Balkanization'' of the BSD landscape. I don't think that's fair.

Looking at Linux, you'll see that there are a large number of different Linux distributions. Each of these has their own version of some programs, their own directory layout, but they are all called Linux. By contrast, the BSD systems don't (yet) call themselves BSD. This nurtures the general perception that the BSDs are very dissimilar operating systems.

There's more to this issue than the name. Each BSD has a different kernel, and the interface isn't completely uniform. Where it's possible at all, you may need an emulation module to run an xBSD program on yBSD. Device names are different, especially since FreeBSD changed the names for disks and tapes.

The differences aren't big; I'm currently writing a book on Systems Administration which I hope will be able to cover all three BSDs. But they're big enough to confuse users who aren't interested in the nitty-gritty and who only want to get their work done. In a word, we need (dare I say it?) standards.

In some circles, the term ``standards'' is treated as a dirty word. They hinder progress and bind people to the lowest common denominator. On the other hand, too much innovation can be confusing. Where do we draw the line? We already try to adhere to many standards, such as POSIX.1 and POSIX.2. It would make a lot of sense to have a single standard for other things, such as the system call interface.

Probably the biggest challenge for BSD in the next 12 months will to be to address this kind of issue and show the world that the individual projects are not groups of enemy gangs spending all their efforts fighting each other. The BSDI/WC merger is a good step in this direction. Let's hope that they're successful.