Path: utzoo!attcan!uunet!cs.utexas.edu!csd4.milw.wisc.edu!uakari.primate.wisc.edu!indri!uflorida!haven!umd5!zben
From: zben@umd5.umd.edu (Ben Cranston)
Newsgroups: comp.lang.postscript
Subject: Re: Bug in PostScript math
Summary: Floating point error strikes again!
Keywords: 200/3
Message-ID: <5199@umd5.umd.edu>
Date: 10 Aug 89 19:18:02 GMT
References: <2395@aecom.yu.edu>
Reply-To: zben@umd5.umd.edu (Ben Cranston)
Distribution: na
Organization: University of Maryland, College Park
Lines: 28

In article <2395@aecom.yu.edu> werner@aecom.yu.edu (Craig Werner) writes:
>	1) divide 200 by 5, save the 40 in a variable
>	2) add 200, later subtract 200
>	3) divide the variable byt the current value, the answer is 1.0
>	4) truncate the 1.0, the answer should still be 1.0.
>	When you truncate the 1.0 in step 4, the result is 0.0

Yeah, I was bit by something similar two weeks ago trying to implement a
PostScript routine to expand tab stops.  It took the current X position,
subtracted off the left margin, and divided by the width of a character in
the current (constant-width) font.  Only trouble was that at the very
beginning of the line the result was not zero but -1E-6 
(a very small negative number).

I certainly hope you take a course in floating point numbers someday, cause
this sort of thing happens all the time.  What you generate is just a whisker
below 1.0, and the floating point to character routine actually TRANSLATES it
as the string "1.0" but when you truncate it you get zero instead of one.

Your best bet is to round the number up like this:

whatever .5 add cvi

fixes a lot of these errors.
-- 
Sig     DS.L    ('ZBen')       ; Ben Cranston 
* Computer Science Center Network Infrastructures Group
* University of Maryland at College Park