Path: utzoo!attcan!uunet!seismo!sundc!pitstop!sun!amdcad!ames!lll-tis!helios.ee.lbl.gov!nosc!ucsd!ucsdhub!jack!crash!pnet01!haitex
From: haitex@pnet01.cts.com (Wade Bickel)
Newsgroups: comp.sys.amiga.tech
Subject: Re: New IFF format details (long).
Message-ID: <3470@crash.cts.com>
Date: 26 Sep 88 11:56:28 GMT
Sender: news@crash.cts.com
Organization: People-Net [pnet01], El Cajon CA
Lines: 244


	Thanks Stu, for the explanation of IFF '85.  I had not looked at the
sub specifier in that light, and now see that it is indeed a good idea.
I still think it would be nice to incorporate the individual BAD chunk
handling and Dirty/Modified chunk bits (please remember this was to be a
user/program option), but not sufficeint to warrent re-doing IFF.  A MAP
chunk is of course one alternate solution, but seems to me to be clumsy
and inefficient (what happens when parallel tracks of audio data are being
stored?).
 
>But the crux of Wade's proposal is this:
>
>|         What I really want to do is create a purely Data driven mechanism, as
>| opposed to the Code driven one in the current IFF.  Rather than having to 
>| write code to handle each type of occurance, a structure would be initialized
>| at run time, and this would be passed to the Reader or Writer parser to be
>| handled.  In this way it would never be necessary to update the Library(s).
>
>He's primarily interested in the mechanism for reading/writing such
>files, and writes about it to great length in his article.  If this
>mechanism were useful for reading and writing "IFF 88" files, then it
>would be equally applicable to existing IFF files just by changing the
>file grammar slightly, as I did above.
>
	Quite so.  I had originally intended to apply this to the IFF '85
standard.  Then, in my telephone discussion with you I had mistakenly come
to the conclusion that nesting control was limited in IFF '85, and thus
the first half of my proposal.

>Wade has used the phrase "data-driven" several times, but I don't see
>what he's talking about. 

	Simple.  In the current IFF system the knowledge to read/write IFF
files is contained in the IFF code.  Thus it is "code-driven".  To extend
the systems knowledge you must modify the code.  In the  "data-driven" system
I propose the knowledge is contained in data (which may include code).  Thus
the data, not the code, is modified to extend the systems knowledge.

>|                         The Writer Mechanism
>|                       ------------------------
>What Wade describes here is basically a tree-like data structure to
>control writing an IFF file.  The tree would presumably be traversed by
>the writer library code and user functions would be called to write the
>actual bytes of data.  Wade refers to this as "data driven."
>
>| Whereas IFF '85 reader/writers' require re-compilation of the
>| source to accomodate format updates, IFF '88 will not.
>
>But I don't get it.  Sure, the data structure controls the chunk
>nesting, but the actual business of writing bytes gets handled by user
>code, so where's the extendability?  I still have to have the code to
>write the chunks in my program which means re-compilation when something
>changes.
	
	Not so.  Only if you use some format which has not yet become part
of the standard library.  Under normal circumstances you would call routines
found in the standard support librarys.  Thus, a developer of a new format
includes a library of private support routines to be used with the
system.  It would also be possible (though a bit cludgy) to include routines
that are part of your active code (ie: not in a library).  An AddFunc() type
routine to add these calls to a table and assign an index might be desirable.

>However, since the structure of the file and the code to write the
>actual bytes are both provided by the client program, I fail to see how
>creating this structure and passing it to a generic writer is any
>different from just having the following piece of code in the client
>program: 
>
>	/* WriteILBM: bitmap, colormap */
>
>	PushChunk (iff, ID_FORM, ID_ILBM);
>
>		PushChunk (iff, ID_BMHD, 0);
>		WriteBMHD (bitmap);
>		PopChunk (iff);
>
>		PushChunk (iff, ID_CMAP, 0);
>		WriteCMAP (colormap);
>		PopChunk (iff);
>
>		PushChunk (iff, ID_BODY, 0);
>		WriteBODY (bitmap);
>		PopChunk (iff);
>
>	PopChunk (iff);
>
>(This is an actual example of the use of the iff.library.  RSN!)  It
>seems that for either method I need to have the writing code in my
>program, and I need to know the structure of the file I want.  If
>anything, I would think that constructing a large tree data structure
>would be more difficult than just having code to write the file
>directly.  What's the advantage?
>
	Think about the control structure from the point of view of the
file reader.  It contains the sum of what the library and your code know
about IFF files.

        Also, how about the following minor change in the way
you handle file writing:

	PushChunk (iff, ID_FORM, ID_ILBM);

		PushChunk (iff, ID_BMHD, 0);
		WriteBMHD (bitmap);
		PopChunk (iff);

		PushChunk (iff, ID_CMAP, 0);
		WriteBytes (colortable,size);  (* colortable is a ptr *)
		PopChunk (iff);

		PushChunk (iff, ID_BODY, 0);
		WriteBODY (bitmap);
		PopChunk (iff);

	PopChunk (iff);
	
Of course, these are supported types.  Lets pretend they're not.  In this
case it would look something like this:


	PushChunk (iff, ID_FORM, ID_ILBM);

		PushChunk (iff, ID_BMHD, 0);
		bmhd := MakeBMHD(bitmap);
		WriteBytes (bmhd, sizeofBMHD); 
		PopChunk (iff);

		PushChunk (iff, ID_CMAP, 0);
		WriteBytes (colortable,sizeofCT);  (* colortable is a ptr *)
		PopChunk (iff);

		PushChunk (iff, ID_BODY, 0);
		WriteCustom (rtn, bitmap);     (* rtn is a ptr to code *)
		PopChunk (iff);

	PopChunk (iff);
	
	
The chunks are now all generic.  The library supports them even though they
may not exist at the time of libary compilation.  "rtn" is the address of
a routine expecting one parameter on the stack, in this case a BitMap pointer.
In this instance, it would be a routine in the library, but it could be any
routine.

I have not decided on the best way to handle custom routines, and would like
suggestions.  One parameter on the stack, push everything but D0,
seems good to me.  In practice it would probably be best to reference such
routines from a jumptable.

>I'm genuinly interested in this *mechanism* for reading and writing
>files since it should work equally well for real IFF files.  If there
>are real advantages to this, Wade, I missed them.  Could you provide an
>example of a file format changing and user programs not needing to be 
>recompiled?

I had not pictured the file format changing.  Rather, I had been thinking
of how to handle new IFF data types.  With EA's code, I found it somewhat
frustrating to have to go in, modify parts of the code, and re-compile
it to add new chunks.  What I want to be able to do is inform the system
of a new data type (and the rules for reading it), without touching the
original source.

Why?  Well, for one thing, having myriads of modified IFF reader code
all based on the same originals, but containing different mods, is a mess.
More importantly, library management, should your IFF library be made available,
will be a mess if the library is constantly being updated to support new
data types.

Your example writer code looks superior to the writer structure method
I described.  The structure really comes into use for the reading of 
files, but since describing the writer was simpler than describing the
reader, and we have to have a format before we can discuss how to read
it, I chose to describe the writer.  Please take a look at the writer
system I described, and imagine using the same method for reading the
files.  Some method of creating "groups" of items on the same level
would be desireable, which complicates the format, but is irrelevant
as far a theory goes.

Basically, a system of nodes is created which describes readable files, and
how to read them.  These nodes reference *knowledge*, which can be in the
IFF library(s) or elsewhere.  In this way, programs using new IFF data
types can describe only the new knowledge to the system. 

For instance, lets suppose the library understands ILBMs but not ANIM's.
The program defines ANIM by first calling library routines to generate a 
structure which reads ILBMs, then adds nodes which describe ANIM's and
how to read them at the appropriate points.  Now the system understands
and can read ANIM. 

Periodically new types would be added to the library, as circumstances warrant.

Standard calls to write/read standard types would also be in the library,
to make things easy for the novice, and act as jumping off points for the
designer of new IFF data types.

----------------------------------------

>On separate issue, Wade talks about "dirty" chunks.

>While this is an interesting and valid idea, it really makes life 
>miserable for programmers.  They have to retain chunks they don't need, 
>don't understand and can't use, just so they can write them out again
>trying to preserve the original IFF file as much as possible only to
>fail much of the time.  It also means that all programs need to fully 
>support standard chunks so that standard chunks will never be marked as 
>dirty.  It also means that programs that use "non-standard" chunks need
>to make some intelligent decisions about whether a chunk marked as 
>dirty" is good within the context of a specific file.  It might be
>possible, but it could also be a real headache.  I'm just not convinced 
>that the advantages are great enough to want to provide such a
>mechanism.

	Simple, just turn it off.  It should be a switch in the library,
If the switch is on when the library reads a file, it tracks dirty chunks.
If a file containing dirty chunks is read, it looks at the reader control
structure (inited by the useing program) and determines which chunks are
bad.  If the switch is off unrecognized chunks or dirty chunks are ignored
on reads.

>This facility can be provided for any new IFF formtypes, however, by 
>equiping them with a "MAP" chunk (or some such, but it should be
>consistent across FORMs) which contains a list of the chunks in the
>file and their status.  It is not possible or even desireable to retro-
>fit this capability into existing formats. 

True.  However, I think it was a mistake in the format not to leave a WORD for
this.  Not leaving a word for the future is shortsightedness.  Using a Map
chunk will be messy, making dirtychunk tracking harder than it would otherwise
be.

Also, what about data correction for chunks?  It won't be that much longer
and musicians will be using IFF to store CD quality sound.  We can't be
throughing away a file just because one bit has gone bad in a chunk.


						Thanks,
						
							Wade.


UUCP: {cbosgd, hplabs!hp-sdd, sdcsvax, nosc}!crash!pnet01!haitex
ARPA: crash!pnet01!haitex@nosc.mil
INET: haitex@pnet01.CTS.COM
Opionions expressed are mine, and not necessarily those of my employer.