From: utzoo!decvax!harpo!eagle!mhuxt!mhuxa!mhb5b!smb Newsgroups: net.news Title: Re: increasing mail/netnews efficiency Article-I.D.: mhb5b.201 Posted: Tue Mar 15 11:30:14 1983 Received: Tue Mar 22 08:25:17 1983 References: <32@vortex.UUCP> Lauren raises several good points. However, I don't completely agree with his proposed solutions. First, I agree that the packet driver needs to be kept for dial-up use. The MMDF packet mechanism, though apparently simpler, gets far worse throughput and (at least in some versions) is rather susceptible to catatonia. Uucp's driver gets fairly good throughput on 1200 baud lines (though it starts falling off badly at higher speeds), and it seems to be quite reliable. The issue is different, though, if you're using an underlying transmission path that is itself flow-controlled and error-corrected, such as a TCP/IP channel. Performance improvements of at least a factor of 10 can be obtained by replacing the packet driver with some other protocol, as outlined by Lauren. Which brings up my second point -- the alternate protocol mechanism isn't geared towards higher-level interchanges like "here's some mail"; it's intended for lower-level functions. The primitives a new protocol must provide are "open", "close", "send/receive message", and "send/receive file". Decisions like what file should be sent, and what the contents of it mean, are handled at a higher level, and are not as easily negotiated. It is also unclear that changing file formats will really help netnews, especially for sites that feed more than one other site. The 2.10 code can make use of the '-c' option (via code changes) to uux; this causes the text of the article to be transmitted directly from the news spool area, and hence avoids the creation of the second D. file in the outbound uucp spool area. This change, plus Truscott's subdirectory mod to uucp (separate subdirectories for C., D., and X. files) should yield a large performance improvement. (To be sure, they're not the whole problem; a lot of the overhead with uucp seems to be the per-file handshaking that goes on. Much of this is directory-search time, but I don't have a feel for just how mcuh.) The real problem with mail transfer is that uucp is *too* general; it can't do the sorts of special-purpose mail handling that one might like. MMDF and SMTP (the ARPAnet's "simple mail transfer protocol" -- not to be confused with the message format standard), on the other hand, allow a site to validate each address individually before sending the body of the message. They also make it much easier to deal with temporary resource problems, such as no space or no i-nodes -- by the time uuxqt learns of such a problem, it's too late to reject the message with a request for retry, and it may not even be possible to mail it back. With SMTP, the sending site *knows* whether or not the next relay received it correctly. (Uucp also has problems with the stupidity of uuclean; if my mail can't be delivered, I would really like the letter back, rather than being told the uucp filenames....) Where does this leave us? One idea is to change the C. file mechanism locally to some more efficient scheme. The outbound X. file could probably be generated dynamically, especially for the simple cases, i.e., mail and news. The same might be done for inbound X. files, though there are timing considerations to worry about -- it isn't feasible to attempt delivery immediately upon receipt of an X. file, especially if the delivery attempt involves expensive operations like alias-list expansion. If you want to experiment with alternate protocols, use uucp (rather than uux) to create Q. files (or some such) in the receiving site's spool directory, and have some variant of uuxqt interpret them. To be sure, that can't be negotiated at transfer time, but it can easily be controlled for news hops via the 'sys' file. Finally, let me make one appeal to anyone implementing a new queuing mechanism: implement a "requeue" counter. That is, any time a transmission fails and is requeued, a counter should be bumped. If it reaches a certain limit, that particular job should be abandoned; otherwise, one failing job can wedge the whole queuing system. A good example is an attempt to transmit a gigantic file via uucp. Even if it's received properly -- by no means certain -- if the receiving site has to copy it to another file system from the TM. file, the sending site will time out waiting for a response. And the next time the two systems connect, the file will be sent again.... --Steve