Newsgroups: news.software.b Path: utzoo!utstat!geoff From: geoff@utstat.uucp (Geoff Collyer) Subject: Re: NNTP vs Cnews (was: Re: Cnews is not for me) Message-ID: <1989Aug14.025821.1887@utstat.uucp> Organization: Statistics, U. of Toronto References: <2828@ndsuvax.UUCP> <1989Aug12.221624.12153@utstat.uucp> <1894@ucsd.EDU> <1989Aug13.071802.5187@utzoo.uucp> <46555@oliveb.olivetti.com> Date: Mon, 14 Aug 89 02:58:21 GMT Jerry Aguirre: >Is there a reason that NNTP can't insert the batch headers and pass the >news to the processing portion (recnews?) via a pipe instead of a file? >In that way a single setup overhead would cover the entire NNTP session. The C news version of nntpd generates the #! rnews batch headers and copies the incoming articles into a spool file in the incoming-batch queue. Every so many bytes or seconds, and at the end of each session, the current spool file is handed to newsrun and thus relaynews. This is fast enough that recently, after jarvis.csri.toronto.edu had been down for a long weekend, I watched nntpd-newsrun-relaynews race with expire: free i-nodes were low on jarvis's news partition and even with the expire niced to -1 (that's preferred niceness for you csh users), nntpd-newsrun-relaynews was creating files faster than expire was removing them, with the difference being roughly one file every few seconds (yes, the partition ran out of i-nodes eventually). This is fast enough for me. There just doesn't appear to be enough setup overhead in the nntpd-newsrun-relaynews processing to matter, at least on jarvis, which does have a non-vanilla news configuration. Sending incoming articles directly into relaynews via a pipe from nntpd would save a small amount of setup overhead but would introduce some new problems (or features, depending on your point of view): since only one relaynews can run at a time, only one nntpd could be actively receiving at a time, unless you introduce substantial buffering in nntpd's data segment, which introduces the risk of dropping articles in the case of a crash. Having at most one nntpd actively conversing at any time would prevent nntpd from accepting duplicates, though, which might well be the best solution to minimise duplicates. >By using a pipe instead of a temporary file the delay between >transmission and appearing in the history file could be minimized. By adjusting the age and size batch-file thresholds in the C nntpd, one can already have fine control over the size of that window (our defaults are 300k bytes or 5 minutes, since we'd prefer to trade off larger window size for better performance, and we usually hit the size limit (or end of session) well before the age limit). Henry and I talked about some sort of hook to let nntpd directly tell relaynews or the history file itself "put this message-id in history as a partially-completed entry and the entry will be completed later when the article arrives", but we haven't done anything more than talk about it. We would really like to sell our changes back to the owners of NNTP so that we can concentrate on the non-NNTP news software, and eventually (one hopes soon!) on other matters altogether, such as the work we are paid to do. -- Geoff Collyer utzoo!utstat!geoff, geoff@utstat.toronto.edu