Also, if you don’t care about display, just set the *precision* to LC max.
Numberformat is as others have posted, really a display function. After you are
done, set the numberFormat back to default or “0”. I bit myself in the butt
because I had a counter that I was using to append a number to a variable name,
and it hosed my names.
Bob S
On Jun 16, 2014, at 11:13 , <dfepst...@comcast.net> <dfepst...@comcast.net>
wrote:
>
>
> I am trying to set the numberFormat so that calculation is precise enough for
> the situation. Since I don't know ahead of time how many decimal places will
> be used, I wrote a function that I hoped would adjust things as necessary.
> But when this function is called with parameters m = 1.09131 and n = .0000001
> and k = 1, it returns 1.09131 rather than 1.0913101.
> The last couple of lines were added for testing, and the debugger shows that
> the numberFormat is being set correctly but that the truncated value is put
> into hold.
> Can anybody see what is going wrong?
>
>
>
> Many thanks.
>
>
>
> David Epstein
>
>
>
> function preciseEnough m,n,k
> -- return the value m + k*n
> -- default numberFormat shows up to 6 decimal places
> -- If m, n, or k has more than that precision, this function sets the
> numberFormat with a margin of safety
> -- before returning the answer
> put length(m) - offset(".",m) into aPlaces
> put length(n) - offset(".",n) into bPlaces
> put length(k) - offset(".",k) into cPlaces
> put the numberFormat into myString
> if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then
> get myString & "####"
> set the numberFormat to it
> end if
> put the numberFormat into nf
> put m + k*n into hold
> return hold
> end preciseEnough
> _______________________________________________
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
_______________________________________________
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
