Path: utzoo!utgpu!watmath!att!tut.cis.ohio-state.edu!cica!ctrsol!uakari.primate.wisc.edu!indri!caesar!blake!uw-beaver!ubc-cs!van-bc!mdivax1!hiebert From: hiebert@mdivax1.uucp (Graeme Hiebert) Newsgroups: comp.sys.mac Subject: Re: Bug in EXCEL v. 2.2, and a flame Message-ID: <1989Aug1.193850.4999@mdivax1.uucp> Date: 1 Aug 89 19:38:50 GMT References:Reply-To: hiebert@mdivax1.uucp (Graeme Hiebert) Organization: Mobile Data International, Richmond, B.C., Canada Lines: 45 In article jk3t+@andrew.cmu.edu (Jonathan King) writes: > > Interesting. My calculator (a Sharp EL-512 II) gives virtually the same > erroneous square root as Excel! That is, the square root of .999975 is > given as .999987499. It looks like this problem is not unique to Excel; > to make its way into the entirely silicon brain of my calculator this error > must be the result of an ancient and honorable (although badly conditioned) > numerics routine... > > jking Very interesting indeed. My Sun gives sqrt(0.9999750000000000000000000) = 0.9999874999218740234222409 and 0.9999749992187 * 0.9999749992187 = 0.9999499990624 Now, let's see, if I do the multiplication by hand, I get 0.9999749992187 0.9999749992187 --------------- 69998249945309 799979999374960 999974999218700 19999499984374000 899977499296830000 8999774992968300000 89997749929683000000 399989999687480000000 6999824994530900000000 89997749929683000000000 899977499296830000000000 8999774992968300000000000 89997749929683000000000000 -------------------------- 0.99994999906243906561042969 ^^^^^^^^^^^^^^^ Well look at that. My Sun multiplies the same way I do. Shame on us. -g -- "I didn't sleep well last night. This girl kept knocking on my hotel room door. After awhile, I had to get up and let her out." -Henny Youngman Graeme Hiebert (hiebert@mdivax1.uucp, ...{uunet,ubc-cs}!van-bc!mdivax1!hiebert)