Newsgroups: news.software.b
Path: utzoo!utstat!geoff
From: geoff@utstat.uucp (Geoff Collyer)
Subject: Re: NNTP vs Cnews (was: Re: Cnews is not for me)
Message-ID: <1989Aug14.025821.1887@utstat.uucp>
Organization: Statistics, U. of Toronto
References: <2828@ndsuvax.UUCP> <1989Aug12.221624.12153@utstat.uucp> <1894@ucsd.EDU> <1989Aug13.071802.5187@utzoo.uucp> <46555@oliveb.olivetti.com>
Date: Mon, 14 Aug 89 02:58:21 GMT

Jerry Aguirre:
>Is there a reason that NNTP can't insert the batch headers and pass the
>news to the processing portion (recnews?) via a pipe instead of a file?
>In that way a single setup overhead would cover the entire NNTP session.

The C news version of nntpd generates the #! rnews batch headers and
copies the incoming articles into a spool file in the incoming-batch
queue.  Every so many bytes or seconds, and at the end of each session,
the current spool file is handed to newsrun and thus relaynews.  This
is fast enough that recently, after jarvis.csri.toronto.edu had been
down for a long weekend, I watched nntpd-newsrun-relaynews race with
expire: free i-nodes were low on jarvis's news partition and even with
the expire niced to -1 (that's preferred niceness for you csh users),
nntpd-newsrun-relaynews was creating files faster than expire was
removing them, with the difference being roughly one file every few
seconds (yes, the partition ran out of i-nodes eventually).  This is
fast enough for me.  There just doesn't appear to be enough setup
overhead in the nntpd-newsrun-relaynews processing to matter, at least
on jarvis, which does have a non-vanilla news configuration.

Sending incoming articles directly into relaynews via a pipe from nntpd
would save a small amount of setup overhead but would introduce some
new problems (or features, depending on your point of view): since only
one relaynews can run at a time, only one nntpd could be actively
receiving at a time, unless you introduce substantial buffering in
nntpd's data segment, which introduces the risk of dropping articles in
the case of a crash.  Having at most one nntpd actively conversing at
any time would prevent nntpd from accepting duplicates, though, which
might well be the best solution to minimise duplicates.

>By using a pipe instead of a temporary file the delay between
>transmission and appearing in the history file could be minimized.

By adjusting the age and size batch-file thresholds in the C nntpd, one
can already have fine control over the size of that window (our
defaults are 300k bytes or 5 minutes, since we'd prefer to trade off
larger window size for better performance, and we usually hit the size
limit (or end of session) well before the age limit).  Henry and I
talked about some sort of hook to let nntpd directly tell relaynews or
the history file itself "put this message-id in history as a
partially-completed entry and the entry will be completed later when
the article arrives", but we haven't done anything more than talk about
it.

We would really like to sell our changes back to the owners of NNTP so
that we can concentrate on the non-NNTP news software, and eventually
(one hopes soon!) on other matters altogether, such as the work we are
paid to do.
-- 
Geoff Collyer		utzoo!utstat!geoff, geoff@utstat.toronto.edu