Path: utzoo!attcan!uunet!cs.utexas.edu!csd4.milw.wisc.edu!uakari.primate.wisc.edu!indri!uflorida!haven!umd5!zben From: zben@umd5.umd.edu (Ben Cranston) Newsgroups: comp.lang.postscript Subject: Re: Bug in PostScript math Summary: Floating point error strikes again! Keywords: 200/3 Message-ID: <5199@umd5.umd.edu> Date: 10 Aug 89 19:18:02 GMT References: <2395@aecom.yu.edu> Reply-To: zben@umd5.umd.edu (Ben Cranston) Distribution: na Organization: University of Maryland, College Park Lines: 28 In article <2395@aecom.yu.edu> werner@aecom.yu.edu (Craig Werner) writes: > 1) divide 200 by 5, save the 40 in a variable > 2) add 200, later subtract 200 > 3) divide the variable byt the current value, the answer is 1.0 > 4) truncate the 1.0, the answer should still be 1.0. > When you truncate the 1.0 in step 4, the result is 0.0 Yeah, I was bit by something similar two weeks ago trying to implement a PostScript routine to expand tab stops. It took the current X position, subtracted off the left margin, and divided by the width of a character in the current (constant-width) font. Only trouble was that at the very beginning of the line the result was not zero but -1E-6 (a very small negative number). I certainly hope you take a course in floating point numbers someday, cause this sort of thing happens all the time. What you generate is just a whisker below 1.0, and the floating point to character routine actually TRANSLATES it as the string "1.0" but when you truncate it you get zero instead of one. Your best bet is to round the number up like this: whatever .5 add cvi fixes a lot of these errors. -- Sig DS.L ('ZBen') ; Ben Cranston* Computer Science Center Network Infrastructures Group * University of Maryland at College Park