Path: utzoo!utgpu!watmath!att!tut.cis.ohio-state.edu!cica!ctrsol!uakari.primate.wisc.edu!indri!caesar!blake!uw-beaver!ubc-cs!van-bc!mdivax1!hiebert
From: hiebert@mdivax1.uucp (Graeme Hiebert)
Newsgroups: comp.sys.mac
Subject: Re: Bug in EXCEL v. 2.2, and a flame
Message-ID: <1989Aug1.193850.4999@mdivax1.uucp>
Date: 1 Aug 89 19:38:50 GMT
References: 
Reply-To: hiebert@mdivax1.uucp (Graeme Hiebert)
Organization: Mobile Data International, Richmond, B.C., Canada
Lines: 45

In article  jk3t+@andrew.cmu.edu (Jonathan King) writes:
> 
> Interesting.  My calculator (a Sharp EL-512 II) gives virtually the same
> erroneous square root as Excel!  That is, the square root of .999975 is
> given as .999987499.  It looks like this problem is not unique to Excel;
> to make its way into the entirely silicon brain of my calculator this error
> must be the result of an ancient and honorable (although badly conditioned)
> numerics routine...
> 
> jking

Very interesting indeed.  My Sun gives
        sqrt(0.9999750000000000000000000) = 0.9999874999218740234222409
and
        0.9999749992187 * 0.9999749992187 = 0.9999499990624

Now, let's see, if I do the multiplication by hand, I get

                0.9999749992187
                0.9999749992187
                ---------------
                 69998249945309
                799979999374960
                999974999218700
              19999499984374000
             899977499296830000
            8999774992968300000
           89997749929683000000
          399989999687480000000
         6999824994530900000000
        89997749929683000000000
       899977499296830000000000
      8999774992968300000000000
     89997749929683000000000000
     --------------------------
   0.99994999906243906561042969
   ^^^^^^^^^^^^^^^
   Well look at that.  My Sun multiplies the same way I do.  Shame on us.

      -g
-- 
"I didn't sleep well last night.  This girl kept knocking on my hotel room
door.  After awhile, I had to get up and let her out."
						-Henny Youngman
Graeme Hiebert (hiebert@mdivax1.uucp, ...{uunet,ubc-cs}!van-bc!mdivax1!hiebert)