Path: utzoo!attcan!uunet!aecom!werner From: werner@aecom.yu.edu (Craig Werner) Newsgroups: comp.lang.postscript Subject: Re: Bug in PostScript math Summary: A restatement and a fix Keywords: 200/3 Message-ID: <2398@aecom.yu.edu> Date: 13 Aug 89 02:23:02 GMT References: <2395@aecom.yu.edu> Distribution: na Organization: Albert Einstein Coll. of Med., NY Lines: 36 In article <2395@aecom.yu.edu>, werner@aecom.yu.edu (Craig Werner) writes: > > I recently encountered a problem that seems to be do to a bug in > the PostScript arithmetic routines built into an Apple LaserWriter IINT. > > Consider the following: > 1) divide 200 by 5, save the 40 in a variable > 2) add 200, later subtract 200 > 3) divide the variable byt the current value, the answer is 1.0 > 4) truncate the 1.0, the answer should still be 1.0. > > Now divide 200 by 3, saving 66.6667. > When you truncate the 1.0 in step 4, the result is 0.0 > Let me restate the problem. I referred to the problem as a bug when the real problem is that it has been twelve years since I've encountered such an egregious arithmetic precision error. (The last time was on a TRS-80 Model I, Level I, system complete with 4K RAM.) Just out of curiousity, what is the precision of PostScript's floating point routines? It's obviously a lot less than I had expected. I'm still confused as to why cvs would report the number as 1.0 instead of what it really was. However, I've used this to my advantage, and a fix for the particular problem seems to be replacing truncate with 20 string cvs cvr truncate (11 string gives me a cvs rangecheck error, and at this point I'd rather not ask why.) -- Craig Werner (future MD/PhD, 4.5 years down, 2.5 to go) werner@aecom.YU.EDU -- Albert Einstein College of Medicine (1935-14E Eastchester Rd., Bronx NY 10461, 212-931-2517) "Sometimes you have to run as fast as you can just to stay in the same place."