Here is a helpful link, that explains the theory:
http://www2.hursley.ibm.com/decimal/
tom jackson
On Sunday 04 May 2008 10:20, Bernhard van Woerden wrote:
> The only way to get what you expect is to do the rounding using integer
> arithmetic.
> The number 18.005 must be multiplied by 1000 to ke
The only way to get what you expect is to do the rounding using integer
arithmetic.
The number 18.005 must be multiplied by 1000 to keep the implied precision.
Then do the rounding and reformat as a decimal.
TCL 8.5 has libtomath which promises arbitary precision integer arithetic
but 8.4 has wide
You have to remember that floating point math is done in base 2, but your are
inputing your numbers in base 10.
Another weired thing to keep in mind is that the default precision in Tcl has
changed in Tcl 8.5, so some answers are now different than they were in 8.4.
Another thing to keep in mi
On 2008.05.03, William Scott Jordan <[EMAIL PROTECTED]> wrote:
> Yuck. Okay, so is there any practical work-around for getting X.XX5 to
> consistently round up? I suppose I could do something like add
> 0.001 to any number that I'm rounding, but that seems pretty sloppy.
> Is there a
Yuck. Okay, so is there any practical work-around for getting X.XX5 to
consistently round up? I suppose I could do something like add
0.001 to any number that I'm rounding, but that seems pretty sloppy.
Is there a best practice for dealing with this?
-William
Dossy Shiobara wrote:
On
It does explain it, but still results are not obvious :)
For example, this code:
===
set bn [set rn 0]
for {set i 0} {$i < 1000} {incr i} {
set f "${i}.005"
set r [format %.2f $f]
set d [expr $r - $i]
if {$d > 0.0} {
incr bn
} else {
incr rn
}
}
puts
The plot thickens:
% format %.2f 18.0051
18.01
No ideas, though.
Bas.
On 04/05/2008, at 8:01 AM, William Scott Jordan wrote:
Hey all!
This is really more of a tcl question, but I'm hoping that someone
on the list might have an explanation. Why does [format %.2f
18.005] round down to "1
William Scott Jordan wrote:
Hey all!
This is really more of a tcl question, but I'm hoping that someone on
the list might have an explanation. Why does [format %.2f 18.005] round
down to "18.00" and [format %.2f 1.415] round up to "1.42"? Any
guesses? Am I missing something obvious here?
On Sat, 2008-05-03 at 15:31 -0700, William Scott Jordan wrote:
> Hey all!
>
> This is really more of a tcl question, but I'm hoping that someone on
> the list might have an explanation. Why does [format %.2f 18.005] round
> down to "18.00" and [format %.2f 1.415] round up to "1.42"? Any
> gue
On 2008.05.03, William Scott Jordan <[EMAIL PROTECTED]> wrote:
> This is really more of a tcl question, but I'm hoping that someone on
> the list might have an explanation. Why does [format %.2f 18.005] round
> down to "18.00" and [format %.2f 1.415] round up to "1.42"? Any
> guesses? Am I
10 matches
Mail list logo