Subject: Re: NumberFormat
Date: Mon, 28 Feb 2000 17:50:22 +1100
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
References: [EMAIL PROTECTED]
Raymond E. Griffith wrote:
I set the numberFormat to 0.## (10 places to the right of
the decimal point) for calculation, but all answers are returned as
integers.
"Steven D'Aprano" [EMAIL PROTECTED] graciously replied:
As example calculation would be useful. You didn't state whether the
answer actually is an integer.
Eg: you want 10.00 but only get 10 with no decimal point.
OK, I am taking the standard deviation of a set of 50 numbers. I am trying
to recreate the precision of my TI-83 calculator. Under ten-decimal place
accuracy I get it.
I then set the numberFormat to 0.00 for calculation. The
calculations were stored correctly. I then set the numberFormat to
0.##, added 0 to the calculated amounts for display. Again,
the displayed amounts were integers.
Hashes in the NumberFormat specify that only non-zero trailing figures
are kept (eg 2.001000 - 2.001), so again we need to know the actual
result of the calculation before we can tell if its a bug.
Right: here goes.
Calculations with numberFormat set to 0.00
µ = 74.364000
x = 3718.20
x^2 = 279382.46
Sx = 7.6694932382
x = 7.5924109478
n = 50.00
minX = 56.90
Q1 = 68.80
Med = 74.55
Q3 = 79.60
maxX = 88.60
As you can see, everything except for Std deviation and population deviation
possess extra zeros. Definitely unneeded and unwanted. But I want the extra
digits on Sx and x.
So I use the following script to filter out the extra zeros
set numberformat to 0.##
repeat with i = 1 to 11
put item i of
"pmean,psumx,psumxsq,pstddev,ppopdev,pn,pminX,pQ1,pmed,pQ3,pmaxX" into
thatprop
put line i of the StatSymb of me 0+the thatprop of me into line i of
ans
end repeat
put ans into fld "Stats"
Here is the result.
µ = 74
x = 3718
x^2 = 279382
Sx = 8
x = 8
n = 50
minX = 57
Q1 = 69
Med = 75
Q3 = 80
maxX = 89
What I want -- and can't seem to get -- is
µ = 74.364
x = 3718.2
x^2 = 279382.46
Sx = 7.6694932382
x = 7.5924109478
n = 50
minX = 56.9
Q1 = 68.8
Med = 74.55
Q3 = 79.6
maxX = 88.6
My apologies if you already knew this, but its been my experience that
its always good to check the simple things first before firing off bug
reports. There's nothing worse than paying for a service call on a
television because it wasn't plugged in :-)
I agree. Sorry for not being more specific, including calculations, etc. And
since I tend to think that a service call may be needed here, I really
should have provided more detail.
This is a pretty important issue for me.
Do you mind if I ask, what are you doing that requires 10 decimal places
of accuracy? That's a lot.
It is. But it is what I need for this project. Actually the TI-83 stores 14
digits with a 2-digit mantissa. I like the accuracy (after all, I am a math
teacher), but again, it is needed if I am to imitate the Ti-83.
I notice that for me, numberFormat always ignores the trailing #'s. If I
use,
set numberFormat to 0.00
the ending four digits are completely ignored, and everything winds up with
six decimal places after zero.
Of course, I could be doing something completely dumb. But this use of
numberFormat works in Hypercard.
--
Steven D'Aprano
Thanks, Steve. I hope this clears up what is happening. Am I just missing or
misusing something?
This is the MetaCard mailing list.
Archives: http://www.mail-archive.com/metacard%40lists.best.com/
Info: http://www.xworlds.com/metacard/mailinglist.htm
