Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Posting-Version: version B 2.10.2 (Tek) 9/28/84 based on 9/17/84; site tekchips.UUCP
Path: utzoo!watmath!clyde!cbosgd!ihnp4!houxm!mtuxo!mtunh!vax135!cornell!uw-beaver!tektronix!tekcrl!tekchips!stevev
From: stevev@tekchips.UUCP (Steve Vegdahl)
Newsgroups: net.lang
Subject: Re: Efficiency of Languages (and complexity)
Message-ID: <345@tekchips.UUCP>
Date: Wed, 6-Nov-85 13:37:44 EST
Article-I.D.: tekchips.345
Posted: Wed Nov  6 13:37:44 1985
Date-Received: Fri, 8-Nov-85 21:57:56 EST
References: <15100004@ada-uts.UUCP> <15100007@ada-uts.UUCP> <189@opus.UUCP> <12860@rochester.UUCP> <196@opus.UUCP>
Organization: Tektronix, Beaverton OR
Lines: 105

This message is in response to a number of articles regarding the above-
named subject.  My intent is to eliminate some of the misunderstanding and
miscommumincation that has been taking place.  (And probably replace
it with new misunderstanding ... :-))

First, the O(n log n) result for sorting applies when
  1.  The architecture has a single processor with a "large" amount of memory
  2.  Sorting is done using comparisons
 and
  3.  Probablistic algorithms are not used

1-->
 Arguments crying "foul, you can't use an infinite amount of hardware", do
not hold water.  The point of the original posting (as I read it anyway)
was that a problem's complexity depends on the architecture.  Comparison
sorting is O(n^2) on a Turing Machine, and is O(n log n) on a von Neumann
architecture.  It is perfectly reasonable to assume an architecture
in which each memory element has a dedicated processor.  The O-notation
allows one to assume that there is an infinite amount of memory (or at
least an amount of memory that is proportional to problem size).  Thus,
on a processor-per-memory-element architecture, the assumption of
n-processors for a problem of size n is not only reasonable, but
manditory.

2-->
 There are, of course, sorting algorithms such as radix-sort
(see Knuth, Vol. 3), that do not use comparisons.

3-->
 It has been shown (e.g., Bruce Weide's thesis, Carnegie-Mellon U. (1977?)),
that probablistic techniques can be used to sort in linear time.  (This
must be qualified by defining what "worst-case" means for a probablistic
algorithm, but I'll leave that as an exercise ...)

With that as a preface, I will now respond to some of the statements
that have been made regarding this subject.

>NO!  You cannot throw "n different processors" at the array!  N is
>(potentially) larger than the number of processors you have.  Actually,
>there is an assumption in analyzing algorithms that one does not have an
>infinite number of computational elements (whatever they may be).  If you
>throw out that assumption, you're nowhere--because (1) you can't build, or
>even emulate, the hardware implied and mostly (2) all the algorithms
>you're going to find interesting will take constant time!  (If you have
>unlimited hardware, you just keep replicating and working in parallel.)

Responding to (1): certain computation models allow an infinite number
of processors.  Even disallowing these as impractical, assumming N
processors is no more impractical than assuming N memory elements.

Responding to (2): there are professors that make careers studying the
complexity of algorithms assuming an infinite number of processors.
There are lots of interesting algorithms do not work in constant time
even with that assumption.

>OK, now I am SURE that the folks who are responding to this don't
>understand O() notation.  This notation is explicitly designed to express
>the ASYMPTOTIC COMPLEXITY of an algorithm.  If that's not what you mean,
>use some other notation.  The usefulness of this sort of complexity measure
>is that it tells how the required processing time (or space) increases in
>relation to the "size" of the problem being solved.  Again, O() notation is
>an asymptotic measure.  This makes it quite explicit that you cannot have
>an "...amount of hardware...proportional to size of the problem..." unless
>you consider an infinite amount of hardware.
>
>The "order-of" notation is not the only measure that can be applied to an
>algorithm--but it DOES have a specific meaning and a specific use, and it's
>a very basic part of analysis of algorithms.  Try to understand it before
>you use it.

The O-notation makes no such assumption.  In fact, it assumes just the
opposite: that is enough memory (i.e., hardware) to hold the problem.
The important thing is that the be O-notation used with respect to a
computation model, and that the everyone agree (for a given discussion)
on what the model is.  The writer of the above quote seems to insist that
the von Neumann model (or something very close to it) is the only model
of computation that can be used.  Allowing the number of processors to
increase in proportion to the number of memory elements simply trades
space for time.  It is still appropriate to use the O-notation.

>If I remember right, even though the time complexity of sort is O(log n)
>on a single processor, the median of a set can be found in O(n).
>It is hard to imagine how this might be done, but...

The median of an array of elements (or more generally, the kth-smallest
element) can definitely be found in linear time.  See Aho, Hopcroft and
Ullman (1974), page 97.

>There is inherent sequencing in many
>problems.  For example, it is extremely difficult to sort in O(log n)
>time using n processors.  Noone knows (and indeed I think it is
>impossible) to sort in O(1) time with n log n processors.

If pipelining is allowed, sorting can be done in "constant time" on a
network of size O(n * log^2(n)) (see Knuth, vol. 3).  By this I mean
that you present n elements to the network at each time unit, and
you get n sorted elements out of the network at each time unit, with
a latency of log^2(n).  I seem to remember reading recently that
someone discovered an O(n * log(n)) sorting network that would do the
same thing.  Ullman, maybe?

		Steve Vegdahl
		Computer Research Lab.
		Tektronix, Inc.
		Beaverton, Oregon