Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Posting-Version: version B 2.10.3 4.3bsd-beta 6/6/85; site ucbvax.ARPA
Path: utzoo!watmath!clyde!burl!ulysses!mhuxr!mhuxt!houxm!ihnp4!ucbvax!anton
From: anton@ucbvax.ARPA (Jeff Anton)
Newsgroups: net.database
Subject: Re: UNIX + database (long quick overview)
Message-ID: <10185@ucbvax.ARPA>
Date: Sat, 24-Aug-85 16:46:04 EDT
Article-I.D.: ucbvax.10185
Posted: Sat Aug 24 16:46:04 1985
Date-Received: Sun, 25-Aug-85 13:22:31 EDT
References: <164@3comvax.UUCP>
Reply-To: anton@ucbvax.UUCP (Jeff Anton)
Organization: University of California at Berkeley
Lines: 126

Follow up-To: 
Distribution: net
Organization: University of California at Berkeley
Keywords: 

UNIX & Databases: A quick overview.

UNIX provides a great environment for program development, but
while building a powerful and efficient DBMS for it you are going
to run into a fair number of problems.

Small address space: This problem is quickly disappearing.
PDP's are going out of style and 32bit micros which can page
are eliminating it.  However, physical memory is not growing
on most machines, and this will affect performance.

Relational model dominant: Most DMBS's on UNIX are relational.
The relational model allows a nonprocedural language to build
queries.  It's very easy to write a query that will not have
an easy solution, but since the databases are in general small
on UNIX this is ok.  Network and Hierarchical DBMS's have strict
methods of accessing the data and will be faster and stored
more efficiently in consequence.  Relational systems will
not just scale up to gigabyte sizes and keep their performance.
(Try sorting a billion names sometime, n log n is still the rule and
sort merge requires as much free space as what you are sorting)

Locking (concurrency): Locking, what locking?
UNIX file locking schemas are, in general, bad.
v7: klugey file locking by making links.
BSD: file locking with 2 levels.
ATT: /usr/group flock call that can lock ranges of files but is only 2 levels.
A DBMS would be well off with three (3) levels of locks,
(operating system people LISTEN!)  We need shared, update, and
exclusive locks. (This may be advisory since the data is accessed
by special routines)
	Conflict table:
		S U E
	      S + + -
	      U + - -
	      E - - -
Reasoning: If you think you might have to change some data
in a file you get an update lock.  Read only routines can have
shared locks.  If you find you don't have to change data you
remove the lock or lower it to shared, allowing someone else to
get an update lock.  If you do have to change the data you
raise the lock to exclusive and wait until all the shared locks go away.
Know what else goes away?  Deadlock.
(Well, not completely but the problem is easier.)
Since no UNIX provides adequate locking you see things like
INGRES's lock driver kernel mods.  I'd like to know how
any UNIX DMBS can be concurrent without such a beast.

Filesystem (recovery):
The UNIX filesystem (new or old) can be treated as blocks
instead of bytes by starting and moving data at multiples of
512 bytes.  This is more efficient for UNIX and good for DBMS's.
The Berkeley fast filesystem does a good job of putting
blocks of the same file together.  However, there are no
synchronous disk writes.  The system may crash with data
supposably written to disk that is not there when the system comes
up again.  Also, the system might drop inodes (files) on the floor
during reboot disk auditing.  INGRES uses a system of batching
updates and transactions can be completed intact if the batch
file is complete.  If it is not then what is there can be used
to back out of partially complete transactions.  There is a
system call, fsync, which will cause a file to be flushed out
of the system buffers and onto the disk, but it's very slow.
Random access of large files on UNIX implies an increase in
disk I/O in order to fetch indirect blocks which point
to indirect blocks which point to data.  And again, when
you cause a new block to be allocated to a file and change
the indirect block, does the system write that indirect block
before or after it writes the data block.  The solution to these
problems, "Don't Panic."  UNIX DBMS's are reliable up to the
last dump of the system and from then on it's unlikely you'll lose.
But for the banks and the purests, we're working on what may be
a way around the problems.  I'm not sure enough right now to
publicize yet.

Security (IPC): Microcomputers can't really worry about
such things.  Any program can reach into MSDOS and convince
it that things are fine.  On UNIX we must be more careful.
If you are going to allow programs to be written that
interact with the DBMS you will have to ether trust that
program not to touch things it should not or put IPC
between that program and a DBMS.  Pipes can do the job, but
they are simple and do not lend themselves to sequenced
packets which would be ideal.  The early days of INGRES used
pipes as the center piece of a complex system allowing
the DBMS to be broken into many pieces connected by pipes.
Seven (7) processes were connected at one time on a PDP
that did not have separate text and data space and only
64k bytes in a proc.  Today, remote procedure calls are a
fad and are likely to handle the job fine.  But, they are
only in BSD.  A commercial DBMS would be limiting its
market too much.  Shared memory could also work with
some coercion.  Privileged sections of processes, programs
which the effective uid could change while the program counter
is on certain memory pages, would be best since it would
allow real procedure calls and security that unauthorized
data will not be examined.

Devices: The number of physical devices on a UNIX system
is a constraint. 4.3 BSD has increased the number of filesystems
from 16 to 256, but files can not extend beyond a filesystem and
a filesystem can only be 4 gigabytes (2^32)?  (something makes
me think 1G.)  Beyond these number you don't have many UNIX machines.
Few 'C' compilers support integers > 32 bits, hardware dependent.

The future: I believe UNIX can grow to meet these problems.  However,
It may take lots of argument to get the operating systems people
to move on these matters.  They are 'dull' database stuff.
Databases run the modern world.  Windowing and networking are the
current fads, and perhaps the database community should move
into the OS world with it's own ideas.  We've have to work around the OS
for a long time, perhaps the effort of working around could change into
working with and then our jobs would be much easier for a long time to come.
And the OS world might learn something too.
-- 
C knows no bounds.
					Jeff Anton
					U.C.Berkeley
					Ingres Group
					ucbvax!anton
					anton@BERKELEY.EDU