>
> Subject: Re: NumberFormat
> Date: Mon, 28 Feb 2000 17:50:22 +1100
> MIME-Version: 1.0
> Content-Type: text/plain; charset=us-ascii
> Content-Transfer-Encoding: 7bit
> References: <[EMAIL PROTECTED]>
>
> Raymond E. Griffith wrote:
>
>> I set the numberFormat to 0.########## (10 places to the right of
>> the decimal point) for calculation, but all answers are returned as
>> integers.
> "Steven D'Aprano" <[EMAIL PROTECTED]> graciously replied:
> As example calculation would be useful. You didn't state whether the
> answer actually is an integer.
>
> Eg: you want 10.0000000000 but only get 10 with no decimal point.
OK, I am taking the standard deviation of a set of 50 numbers. I am trying
to recreate the precision of my TI-83 calculator. Under ten-decimal place
accuracy I get it.
>
>> I then set the numberFormat to 0.0000000000 for calculation. The
>> calculations were stored correctly. I then set the numberFormat to
>> 0.##########, added 0 to the calculated amounts for display. Again,
>> the displayed amounts were integers.
>
> Hashes in the NumberFormat specify that only non-zero trailing figures
> are kept (eg 2.001000 -> 2.001), so again we need to know the actual
> result of the calculation before we can tell if its a bug.
>
Right: here goes.
Calculations with numberFormat set to 0.0000000000
= 74.3640000000
x = 3718.2000000000
x^2 = 279382.4600000000
Sx = 7.6694932382
x = 7.5924109478
n = 50.0000000000
minX = 56.9000000000
Q1 = 68.8000000000
Med = 74.5500000000
Q3 = 79.6000000000
maxX = 88.6000000000
As you can see, everything except for Std deviation and population deviation
possess extra zeros. Definitely unneeded and unwanted. But I want the extra
digits on Sx and x.
So I use the following script to filter out the extra zeros
set numberformat to 0.##########
repeat with i = 1 to 11
put item i of
"pmean,psumx,psumxsq,pstddev,ppopdev,pn,pminX,pQ1,pmed,pQ3,pmaxX" into
thatprop
put line i of the StatSymb of me && 0+the thatprop of me into line i of
ans
end repeat
put ans into fld "Stats"
Here is the result.
= 74
x = 3718
x^2 = 279382
Sx = 8
x = 8
n = 50
minX = 57
Q1 = 69
Med = 75
Q3 = 80
maxX = 89
What I want -- and can't seem to get -- is
= 74.364
x = 3718.2
x^2 = 279382.46
Sx = 7.6694932382
x = 7.5924109478
n = 50
minX = 56.9
Q1 = 68.8
Med = 74.55
Q3 = 79.6
maxX = 88.6
> My apologies if you already knew this, but its been my experience that
> its always good to check the simple things first before firing off bug
> reports. There's nothing worse than paying for a service call on a
> television because it wasn't plugged in :-)
>
I agree. Sorry for not being more specific, including calculations, etc. And
since I tend to think that a service call may be needed here, I really
should have provided more detail.
>> This is a pretty important issue for me.
>
> Do you mind if I ask, what are you doing that requires 10 decimal places
> of accuracy? That's a lot.
>
It is. But it is what I need for this project. Actually the TI-83 stores 14
digits with a 2-digit mantissa. I like the accuracy (after all, I am a math
teacher), but again, it is needed if I am to imitate the Ti-83.
I notice that for me, numberFormat always ignores the trailing #'s. If I
use,
set numberFormat to 0.000000####
the ending four digits are completely ignored, and everything winds up with
six decimal places after zero.
Of course, I could be doing something completely dumb. But this use of
numberFormat works in Hypercard.
> --
> Steven D'Aprano
Thanks, Steve. I hope this clears up what is happening. Am I just missing or
misusing something?
This is the MetaCard mailing list.
Archives: http://www.mail-archive.com/metacard%40lists.best.com/
Info: http://www.xworlds.com/metacard/mailinglist.htm
