This is a tidied-up version of an article I wrote for Daemon News, published in February 2000. I have removed obvious errors, but it remains a historical document. This also explains the first section, which refers to the December 1999 article.

Feedback

I had intended December's article to be provocative, but in fact most of the feedback was fully in agreement. As a result, I've decided to talk about a similar kind of problem this month.

A number of people reported that they also used methods like those I described in my spreadsheet example, and for the same reason. The details differ, of course, but the general feeling was that they would rather expend a little more effort initially and understand what they were doing, because it would give them more confidence later on.

In addition, Marko Schütz pointed out that Emacs does, indeed, perform spreadsheet calculations. There's an Emacs macro package called dismal (“DIS Mode Aint Lotus”). I suppose I should have expected that. In the meantime, I also discovered that a WYSIWYG spreadsheet called Wingz is also available for free from Sunsite.

I suppose it's a sign of the times that modern versions of firefox have difficulty with FTP links.

Social contact, 1999

Some people claim that I live in the middle of nowhere. While that's not quite true, even people who live in Adelaide sometimes get the impression that I live a long way from anywhere (in fact, about 43 km or 27 miles from the centre of Adelaide). For many people, that's literally the end of the world. Still, I'm active all over the world: I get about 600 mail messages every day, and send between 40 and 60 messages. On the other hand, in the last 3 months I have only physically visited anybody once, and that was a family reunion on Christmas Day. In many senses, mail has taken the place of conventional social interaction.

Electronic mail isn't new. Even the sendmail program that we know and love is nearly 20 years old, and there were mail programs before that. I first used mail in 1978, and I've been using it for my daily business since 1982. In that time, a lot of conventions have grown up.

The obvious model for electronic mail was real, typewritten mail, the sort of thing we learnt at school: your address at the top right, address of the person you're writing to slightly lower on the left, then “Dear Mr. Bloggs”, message, “Yours sincerely” and a signature. It didn't take long to streamline that somewhat: mail had headers, so you didn't need to write down the addresses again. But the rest stayed pretty much the same, and up to a point it has hardly changed even today. Even though the name of the recipient is in the header, many people still write “Dear Fred” at the top, and most still write their name at the bottom in lieu of a signature.

A number of things ensure that it didn't change very much: in the early days, we used terminals, which almost invariably had 24 or 25 lines of 80 characters, somewhat wider than the average typewritten page, and that strongly influenced the format of the mail message. Also, the software itself was primitive. Usually it contained a bare-bones “editor” which allowed you to type in your text, but not much more, not even to back up to a previous line. UNIX's mail program is a typical example, and for some things (shell scripts, for example) it's still useful. But for sending mail, it was painful, so people stuck to what they could do easily.

In reply to your message

Probably the biggest difference between paper mail and electronic mail is in the way you reply. In the days before computers, when you answered a letter, you'd say something like:

Dear Mr. Bloggs,

Thank you for your letter of the 15th of December.  I'm not completely in
agreement with your opinion...

That would be the only reference that Mr. Bloggs would have to indicate what the reply was about. If he hadn't kept a copy of the letter, he might have difficulty understanding or remembering what you were talking about. Computers made things easier, of course: it became trivial to include the original message with your reply, normally after your own reply. It didn't take system administrators long to discover that this significantly increased network traffic, and clever programmers started caching messages at remote sites, so the reply only needed to include the message ID.

This was a great improvement over not having the previous message available at all, but it was still a problem. The screens only had 25 lines, and it was difficult to display both the original message and the reply on the screen at the same time. Take the following example, which I have broken up into the 25 line screens that we were using at the time:

In this example, it's easy enough to see the message. It starts on the third screen and works its way back. Now you're probably going to say “yes, but it's easy enough to follow, at least if your mail reader could scroll backwards”.

Well, our mailer couldn't scroll backwards. But it still wasn't too difficult. There's a good reason for this: when replying, people didn't have access to the original message. They had the reply editor on the screen, so they had to rely on their memory of the original. This had its effect on the accuracy of the reply. In this respect it was worse than paper mail, where you would normally have a copy of the original on hand when writing the reply. People frequently left things out, especially if the original message was long and contained multiple topics. If you look at this example more carefully, you'll notice that the replies weren't too related to the original message.

By comparison, take a look at this one:

This example contains parts of seven different messages, each a reply to an earlier one. The messages are distinguished by putting a > character to the left of the text, gradually moving the text itself to the right. There's also a line indicating who wrote the thread, and when he did it.

It's no coincidence that this message is in one part: it's from a modern BSD system running X, and it shows the advantage of being able to see the whole message at one time, though this is less important than with the old-style message. Most importantly, there's a strong connection between the texts: they're short and to the point, and there's no confusion because they're exactly where they are needed. The “quoting” with > characters makes it clear who has said what, and so it's relatively easy to find your way through seven levels of nesting. This message would be completely unintelligible if it used the old way:

This is an important advantage which only electronic mail can offer: it allows us to communicate in a much more exact manner than has previously been practical. Even verbal communication doesn't measure up to it, because Email gives you the exact context.

What's wrong with this picture?

This method of quoting grew up in the 1980s in a UNIX environment; it was already well in place when the first example was written on a proprietary system. In addition, a very similar method had evolved in the DOS-based bulletin-board systems which existed before Microsoft invented the Internet. So why isn't everybody using it? There are two reasons: the way the system software works, and the wish to earn money.

UNIX relies strongly on the interaction between programs to perform any particular task. The second example requires a certain amount of editing:

• First, we need to copy the original message and add the > characters in front of it. In practice, it makes it much more legible if the > character is followed by a space.
• Then we need to add our own text.
• Finally, it's particularly important to remove superfluous text. For example, each message usually has a .sig, a file containing signature information, usually 3 or 4 lines long. In this example, if it hadn't been removed, there would have been about 25 lines of superfluous .sig, which doesn't exactly help the legibility, about half the total length of this message.

To do all this, we need a Real Editor. Most mail packages include an editor, but they are relatively simple. A real editor is a big piece of software, and much more importantly, it's a religious thing. As we've seen in previous Dæmon News articles, most experienced users have their favourite editor, and some spend considerable effort trying to convince others to convert to it. The idea of using somebody else's idea of an editor is not overly attractive. For example, the Emacs command “Move to End of Line” is c-e (an abbreviation meaning “control-E”: hold down the Ctrl key and press the e key). Wingz uses this input to stop the program.

UNIX mailers take a different way: instead of requiring you to use their built-in editor, they are willing to invoke your favourite editor to edit the text. Typically you set the EDITOR environment variable or some setup file for the mailer, and when you reply to a message, the mailer will start your editor.

Meanwhile, in the commercial world...

It's easier to understand the other reason why not everybody does it this way: commercial vendors want to make money out of Email software. Sure, they want to innovate as well–after all, that's a good way to ensure that you make money. But most commercial packages appear to have been designed by people who have little experience using electronic mail, and for people who have little experience using computers. When replying to a message, it's just plain easier to produce a window in which the user can type his text. Editing facilities are minimal, but you can go back to previous lines to make corrections, and even a beginner can master the technique quickly.

The first mailers which appeared for Microsoft platforms used this method, but of course they quickly started adding features. The most far-reaching was that they showed various views in the same window. For example, looking at the second example with Netscape, we might see:

This is on a display reduced to 800x600 pixels, which I consider too small, but even at this resolution many fields can only be displayed partially. Things could be worse: many users still use a 640x480 display. We'll look at what that does below.

This image has at least four parts of importance:

1. A row of toolbars over the top. To reduce the space they use, and to make them more intelligible, I've chosen text representation.
2. A visual display of the folder hierarchy on the left. The space is too small, so the names have been truncated in the middle. You need to know that name of the highlighted folder (sa...il) is really savedmail.
3. An index in the folder at middle right. This folder contains 539 messages, not an unusual number. There's a tiny little scroll bar about three-quarters of the way down. It's barely usable. Again because of space, the subject lines have been truncated, although the priority field (which I think is hardly needed) is far too wide. These appear to be the standard Netscape defaults.
4. Finally, at bottom left is a version of the same message we have seen above. It's truncated in two directions and given scroll bars.

The real problem with this display is that it's trying to show too much. If you double click on an entry in the middle right section, it'll show you the message in a separate window. At this resolution, it's not too bad.

But what about the 640x480 case?

Remember not everybody uses “high” resolution graphics? Let's see what happens if we reply to this message in 640x480 mode. We select Reply from the second line of the toolbar, and get this display after scrolling down to where we want to go:

Now the horizontal scroll bar is gone, and the lines have been broken instead. I specified that I wanted quoted text, but the broken lines are not quoted, and the resultant text has alternate long lines with quotes and short lines without quotes.

This wrapping comes because, by default, Netscape limits line lengths to 72 characters. You can change that; I tried changing it to 0, which apparently means “ignore input”: it accepted it, but next time I looked, it was still set to 72. Never mind, you can set 999 characters and it won't change it back.

Well, that's what it seems like. When you return to the compose screen, nothing appears to have changed. The lines are still wrapped, and you have no way of knowing whether the recipient will get them that way or not. In fact, he doesn't, but this uncertainty means that a very large proportion of mail messages goes out incorrectly formatted.

The other problem is just as bad: now the quoted lines don't get wrapped, but neither do the ones you enter, unless you remember to press ENTER at the end of every line. Have you done so? There doesn't seem to be any way to tell, apart from reading the message after it has been sent.

For a long time I thought that this was an incompatibility problem between UNIX and the new generation of full-screen mailers, but that's only part of the problem. Full-screen mailers seem to handle unbroken lines better, but they don't do any better with the broken quote lines.

People really do get mail like this, and they put up with it. I have even had flames (usually badly formatted) telling me I'm stupid to stand in the way of progress.

The lemmings

The word progress originally meant “moving forward”. It didn't get the meaning “improvement” until the early 17th century. Well, yes, if we use the old meaning of the word, the current situation with full-screen mailers represents progress: we've certainly moved forward from what we had. In this sense, of course, lemmings rushing to jump into the sea are also progressing. I'm sorry, I don't buy this description of “progress”. There is nothing in this “simplification” which brings any mentionable improvement.

What about the rest of the message?

If you look carefully at the original message, you'll note that the quotes are nearly all joined together: instead of } > > > > > > >, we have } >>>>>>>. The originator of this message prefers to use the } character rather than the > character, and he doesn't remove the initial space.

Removing the spaces saves a lot of space and makes things more readable, but it's not a standard function on any mailer that I know. I use a custom macro under Emacs to perform this function: I press F11, and it tidies things up for me. It also changes the } characters into > characters. The other correspondent doesn't use Emacs and doesn't have the corresponding function.

How to I do this under Netscape? The simple answer is: I don't. It's just too difficult. About the only way the Netscape editor allows it is to manually select each > and remove the space in front of it. That's the tradeoff of an “easy to use” mailer. Even removing excess text is difficult enough that you tend to find messages with dozens of dead .sig files in the quoted text.

But what about Microsoft Outlook?

This example shows Netscape, simply because I have it running on my machine. Microsoft Outlook wouldn't be that primitive, would it? As an outsider, it's difficult for me to tell. It certainly has enough bells and whistles, but I don't use Microsoft, so I can only judge by the mail that I receive from different mail programs.

Sometimes I get well-formatted mail with Netscape, though it frequently shows that the user couldn't be bothered to grapple with the clumsy, primitive editor. On the other hand, I have had many mail messages sent with Microsoft Outlook. With few exceptions, texts containing parts of prior mail messages were mutilated in a manner which certainly sets new standards. Large professional computer companies use Outlook to send messages which look like they have been written by people who had only that day learnt to use a computer.

The funny thing is that until the last few years, nobody ever had trouble with this problem. Well, hardly ever. UNIX and BBS users knew how to use their tools, and the results were more or less legible. Others used simpler mailers which didn't quote, and which may or may not have appended the original message. Either way, what you saw on the screen was what went out to the recipient, and it was very rare to see a really badly formatted mail message.

What to do?

It's easy for me to stand on my soapbox and complain. The right thing is to make some constructive suggestions about how to send mail properly. Well, I've done my best at http://www.lemis.com/email.html, but the bad news is that I just don't know how to use any of these mailers well. From what I can gather from people who use them, nobody else does either. They just accept it as an unchangeable situation, and are more or less happy with it.

Of course, with BSD you don't need to have this problem. There are plenty of mailers out there; the most popular are elm, pine and mutt. elm (“electronic mail”) was the original “full-screen” mailer. pine (“Pine is not elm”) was an early successor, and mutt (the mongrel) is a relatively recent successor to elm. If you're using elm, you probably should move to something more modern. I found it difficult to use pine, so I didn't change until mutt came along.

Is that all?

There's a lot more to say about mail. I don't want to give the impression that full-screen mailers are a bad idea. They have some significant advantages over the elm, pine and mutt triumvirate. I'll talk about these matters in another article.