Number accuracy

[Dutchman]

George said in viewtopic.php?f=20&t=1732

Our manual is still wrong. We are able to do 2^53, not 2^52.

To be sure I made the following test:

Code: Select all

'Number accuracy
n=2^53
n$="2^53"
print n$&"=";"#":n,"= HEX"&"""0"&HEX$(n)&""""
print "-"&n$&"=";"#":-n
print n$&"-1=";"#":n-1,"= HEX"&"""0"&HEX$(n-1)&""""
PRINT "Wrong result:"
'The following gives wrong result
print n$&"+1=";"#":n+1,"= HEX"&"""0"&HEX$(n+1)&""""

with the following output:

output.PNG (66.66 KiB) Viewed 1939 times

I corrected the manual. Thanks George

[GeorgeMcGinn]

Thanks.

It was so long ago I wasn't sure where we left off on this, until it was brought up again by sarossell.

After digging through my old paper long hand and redid the testing including multpying and division (I never did the +1 or -1 tests) my initial thought that I put in my programs were also wrong. I was using the 2^52 instead of the 2^53.

I could only find half of the posts, but I saved the document that Mr. K. told me about that would make interesting reading and I did read it and came to the conclusion that it should be 2^53.

Sorry io bring this up again, but I too wanted to make sure when I put it in my programmer's guide that it not only matches what you had, but that we both agreed on which was right.

Thanks for looking into this again and I'm glad it's settled or solved.

Now if we can get the string format conversion to binary (which is why I started looking at one of my favorite resources, http://www.simotime.com/asc2ebc1.htm#AscEbcTables, maybe we can have a function to give us the kind of precision needed for larger numbers.

Thanks Ton,

George.

George said in viewtopic.php?f=20&t=1732

Our manual is still wrong. We are able to do 2^53, not 2^52.

To be sure I made the following test:

Code: Select all

'Number accuracy
n=2^53
n$="2^53"
print n$&"=";"#":n,"= HEX"&"""0"&HEX$(n)&""""
print "-"&n$&"=";"#":-n
print n$&"-1=";"#":n-1,"= HEX"&"""0"&HEX$(n-1)&""""
PRINT "Wrong result:"
'The following gives wrong result
print n$&"+1=";"#":n+1,"= HEX"&"""0"&HEX$(n+1)&""""

with the following output:
output.PNG

I corrected the manual. Thanks George

[Mr. Kibernetik]

viewtopic.php?f=20&t=1314&start=20