From: utzoo!decvax!harpo!eagle!mhuxt!mhuxa!mhb5b!smb
Newsgroups: net.news
Title: Re: increasing mail/netnews efficiency
Article-I.D.: mhb5b.201
Posted: Tue Mar 15 11:30:14 1983
Received: Tue Mar 22 08:25:17 1983
References: <32@vortex.UUCP>

Lauren raises several good points.  However, I don't completely agree
with his proposed solutions.

First, I agree that the packet driver needs to be kept for dial-up use.
The MMDF packet mechanism, though apparently simpler, gets far worse
throughput and (at least in some versions) is rather susceptible to
catatonia.  Uucp's driver gets fairly good throughput on 1200 baud lines
(though it starts falling off badly at higher speeds), and it seems to
be quite reliable.

The issue is different, though, if you're using an underlying
transmission path that is itself flow-controlled and error-corrected,
such as a TCP/IP channel.  Performance improvements of at least a factor
of 10 can be obtained by replacing the packet driver with some other
protocol, as outlined by Lauren.

Which brings up my second point -- the alternate protocol mechanism
isn't geared towards higher-level interchanges like "here's some mail";
it's intended for lower-level functions.  The primitives a new protocol
must provide are "open", "close", "send/receive message", and
"send/receive file".  Decisions like what file should be sent, and what
the contents of it mean, are handled at a higher level, and are not
as easily negotiated.

It is also unclear that changing file formats will really help netnews,
especially for sites that feed more than one other site.  The 2.10 code
can make use of the '-c' option (via code changes) to uux; this causes
the text of the article to be transmitted directly from the news spool
area, and hence avoids the creation of the second D. file in the
outbound uucp spool area.  This change, plus Truscott's subdirectory mod
to uucp (separate subdirectories for C., D., and X. files) should yield
a large performance improvement.  (To be sure, they're not the whole
problem; a lot of the overhead with uucp seems to be the per-file
handshaking that goes on.  Much of this is directory-search time, but I
don't have a feel for just how mcuh.)

The real problem with mail transfer is that uucp is *too* general; it
can't do the sorts of special-purpose mail handling that one might like.
MMDF and SMTP (the ARPAnet's "simple mail transfer protocol" -- not to
be confused with the message format standard), on the other hand, allow
a site to validate each address individually before sending the body of
the message.  They also make it much easier to deal with temporary
resource problems, such as no space or no i-nodes -- by the time uuxqt
learns of such a problem, it's too late to reject the message with a
request for retry, and it may not even be possible to mail it back.
With SMTP, the sending site *knows* whether or not the next relay
received it correctly.  (Uucp also has problems with the stupidity of
uuclean; if my mail can't be delivered, I would really like the letter
back, rather than being told the uucp filenames....)

Where does this leave us?  One idea is to change the C. file mechanism
locally to some more efficient scheme.  The outbound X. file could
probably be generated dynamically, especially for the simple cases,
i.e., mail and news.  The same might be done for inbound X. files,
though there are timing considerations to worry about -- it isn't
feasible to attempt delivery immediately upon receipt of an X. file,
especially if the delivery attempt involves expensive operations like
alias-list expansion.  If you want to experiment with alternate
protocols, use uucp (rather than uux) to create Q. files (or some such)
in the receiving site's spool directory, and have some variant of uuxqt
interpret them.  To be sure, that can't be negotiated at transfer time,
but it can easily be controlled for news hops via the 'sys' file.

Finally, let me make one appeal to anyone implementing a new queuing
mechanism:  implement a "requeue" counter.  That is, any time a
transmission fails and is requeued, a counter should be bumped.  If it
reaches a certain limit, that particular job should be abandoned;
otherwise, one failing job can wedge the whole queuing system.  A good
example is an attempt to transmit a gigantic file via uucp.  Even if
it's received properly -- by no means certain -- if the receiving site
has to copy it to another file system from the TM. file, the sending
site will time out waiting for a response.  And the next time the two
systems connect, the file will be sent again....


		--Steve