Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP Posting-Version: version B 2.10.3 4.3bsd-beta 6/6/85; site ucbvax.ARPA Path: utzoo!watmath!clyde!burl!ulysses!mhuxr!mhuxt!houxm!ihnp4!ucbvax!anton From: anton@ucbvax.ARPA (Jeff Anton) Newsgroups: net.database Subject: Re: UNIX + database (long quick overview) Message-ID: <10185@ucbvax.ARPA> Date: Sat, 24-Aug-85 16:46:04 EDT Article-I.D.: ucbvax.10185 Posted: Sat Aug 24 16:46:04 1985 Date-Received: Sun, 25-Aug-85 13:22:31 EDT References: <164@3comvax.UUCP> Reply-To: anton@ucbvax.UUCP (Jeff Anton) Organization: University of California at Berkeley Lines: 126 Follow up-To: Distribution: net Organization: University of California at Berkeley Keywords: UNIX & Databases: A quick overview. UNIX provides a great environment for program development, but while building a powerful and efficient DBMS for it you are going to run into a fair number of problems. Small address space: This problem is quickly disappearing. PDP's are going out of style and 32bit micros which can page are eliminating it. However, physical memory is not growing on most machines, and this will affect performance. Relational model dominant: Most DMBS's on UNIX are relational. The relational model allows a nonprocedural language to build queries. It's very easy to write a query that will not have an easy solution, but since the databases are in general small on UNIX this is ok. Network and Hierarchical DBMS's have strict methods of accessing the data and will be faster and stored more efficiently in consequence. Relational systems will not just scale up to gigabyte sizes and keep their performance. (Try sorting a billion names sometime, n log n is still the rule and sort merge requires as much free space as what you are sorting) Locking (concurrency): Locking, what locking? UNIX file locking schemas are, in general, bad. v7: klugey file locking by making links. BSD: file locking with 2 levels. ATT: /usr/group flock call that can lock ranges of files but is only 2 levels. A DBMS would be well off with three (3) levels of locks, (operating system people LISTEN!) We need shared, update, and exclusive locks. (This may be advisory since the data is accessed by special routines) Conflict table: S U E S + + - U + - - E - - - Reasoning: If you think you might have to change some data in a file you get an update lock. Read only routines can have shared locks. If you find you don't have to change data you remove the lock or lower it to shared, allowing someone else to get an update lock. If you do have to change the data you raise the lock to exclusive and wait until all the shared locks go away. Know what else goes away? Deadlock. (Well, not completely but the problem is easier.) Since no UNIX provides adequate locking you see things like INGRES's lock driver kernel mods. I'd like to know how any UNIX DMBS can be concurrent without such a beast. Filesystem (recovery): The UNIX filesystem (new or old) can be treated as blocks instead of bytes by starting and moving data at multiples of 512 bytes. This is more efficient for UNIX and good for DBMS's. The Berkeley fast filesystem does a good job of putting blocks of the same file together. However, there are no synchronous disk writes. The system may crash with data supposably written to disk that is not there when the system comes up again. Also, the system might drop inodes (files) on the floor during reboot disk auditing. INGRES uses a system of batching updates and transactions can be completed intact if the batch file is complete. If it is not then what is there can be used to back out of partially complete transactions. There is a system call, fsync, which will cause a file to be flushed out of the system buffers and onto the disk, but it's very slow. Random access of large files on UNIX implies an increase in disk I/O in order to fetch indirect blocks which point to indirect blocks which point to data. And again, when you cause a new block to be allocated to a file and change the indirect block, does the system write that indirect block before or after it writes the data block. The solution to these problems, "Don't Panic." UNIX DBMS's are reliable up to the last dump of the system and from then on it's unlikely you'll lose. But for the banks and the purests, we're working on what may be a way around the problems. I'm not sure enough right now to publicize yet. Security (IPC): Microcomputers can't really worry about such things. Any program can reach into MSDOS and convince it that things are fine. On UNIX we must be more careful. If you are going to allow programs to be written that interact with the DBMS you will have to ether trust that program not to touch things it should not or put IPC between that program and a DBMS. Pipes can do the job, but they are simple and do not lend themselves to sequenced packets which would be ideal. The early days of INGRES used pipes as the center piece of a complex system allowing the DBMS to be broken into many pieces connected by pipes. Seven (7) processes were connected at one time on a PDP that did not have separate text and data space and only 64k bytes in a proc. Today, remote procedure calls are a fad and are likely to handle the job fine. But, they are only in BSD. A commercial DBMS would be limiting its market too much. Shared memory could also work with some coercion. Privileged sections of processes, programs which the effective uid could change while the program counter is on certain memory pages, would be best since it would allow real procedure calls and security that unauthorized data will not be examined. Devices: The number of physical devices on a UNIX system is a constraint. 4.3 BSD has increased the number of filesystems from 16 to 256, but files can not extend beyond a filesystem and a filesystem can only be 4 gigabytes (2^32)? (something makes me think 1G.) Beyond these number you don't have many UNIX machines. Few 'C' compilers support integers > 32 bits, hardware dependent. The future: I believe UNIX can grow to meet these problems. However, It may take lots of argument to get the operating systems people to move on these matters. They are 'dull' database stuff. Databases run the modern world. Windowing and networking are the current fads, and perhaps the database community should move into the OS world with it's own ideas. We've have to work around the OS for a long time, perhaps the effort of working around could change into working with and then our jobs would be much easier for a long time to come. And the OS world might learn something too. -- C knows no bounds. Jeff Anton U.C.Berkeley Ingres Group ucbvax!anton anton@BERKELEY.EDU