Re: [Flightgear-devel] rounding and truncation
Improved code. Previous version went against the POSIX convention.
# Following the POSIX convention,
# halfway cases are rounded away from zero,
# to the extent this is possible given the inherent
# inexactitude of the floating-point representation.
round = func(arg, quantum=1){
if (quantum == 0) {
return arg;
}
if (quantum < 0) {
quantum = -quantum;
}
sign = 1;
if (arg < 0) {
arg = -arg;
sign = -sign;
}
return sign * quantum * int(0.5 + arg/quantum);
}
print(0.5, " -> ", round(0.5), " should be 1");
print(-0.5, " -> ", round(-0.5), " should be -1");
print(120.37, " -> ", round(120.37, 0.025), " should be 120.375");
Result:
0.5 -> 1 should be 1
-0.5 -> -1 should be -1
120.36999 -> 120.375 should be 120.375
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
[Flightgear-devel] rounding and truncation
On 02/07/2007 02:14 PM, Andy Ross wrote:
> > The patch itself looks sane and easy. But I think I agree with
> > John, this is a workaround for a design flaw in the step animation
> > that we should just fix.
:-)
> So you want to pass in the floating point value 29.92 and a step of 10
> and get back 2, and *not* round up to 3. You want truncation for
> everything but the final digit, where rounding is more appropriate and
> the bias value is a useful workaround.
>
> So if there's a design flaw here, it's at a higher level. Maybe
> what's really needed is a "digit extractor" animation instead.
Ah, that leads us to a point I previously sorta mentioned in
passing, but didn't really do justice to.
Life is only moderately complicated if we have the exact value
stored internally, and we can then round and truncate to form
external representation, without trashing the internal representation.
This is the only case I previously discussed.
Life is even more complicated if some comedian passes in an inexact
value such as 120.37, perhaps via something like --nav1=120.37
on the command line.
In such a case, it is necessary to channelize the number before
using it. In the case of comm channels on 25 kHz spacing,
that means rounding to the nearest multiple of 25 kHz.
Below you can find the code to implement the required rounding.
It has some complexity beyond what is needed for the present
application, but is needed in order to properly round negative
quantities.
As before, we have a two-step process:
a) Channelize the frequency, by rounding (not truncating)
to the nearest exact channel value.
b) Some (not all) instruments may choose to truncate the
representation; for what is internally 120.375 may be
displayed as 120.37 externally.
# If arg is a multiple of half the quantum,
# it will be rounded *up* toward +infinity.
# This tries to preserve the property that rounding
# commutes with adding any integer multiple
# of the quantum, i.e.
# round(arg+N*quantum, quantum) = N*quantum+round(arg,quantum)
# for all integer N
# However this may occasionally be violated because of the
# inherent inexactitude of the floating-point representation.
round = func(arg, quantum=1){
if (quantum == 0) {
return arg;
}
if (quantum < 0) {
quantum = -quantum;
}
fudge = 0;
if (arg < 0) {
fudge = 1 + int(-arg/quantum);
# Fudging is necessary because int() truncates toward zero
# not toward + or - infinity.
}
return quantum * (int(0.5 + fudge + arg/quantum) - fudge);
}
print(0.5, " -> ", round(0.5), " should be 1");
print(-0.5, " -> ", round(-0.5), " should be 0");
print(120.37, " -> ", round(120.37, 0.025), " should be 120.375");
Result:
0.5 -> 1 should be 1
-0.5 -> 0 should be 0
120.36999 -> 120.375 should be 120.375
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
[Flightgear-devel] rounding and truncation (was: Textranslate Step and Scroll)
On 02/07/2007 12:26 PM, Ron Jensen wrote:
> How would the step function know to round to the proper number of
> decimal places? In the 2D instruments a printf-style statement is used,
> i.e. %02.2f. However, we're not exactly printing in
> .
>
> I thought about a "" tag where we could tell
> how to round. So we could say:
> 2
> instead of:
> 0.005
> But the code for 2 would end up
> just adding 0.005 to the property anyway, and having the finer grain
> control in may prove useful in as-yet unthought of ways.
1) I repeat that is a reasonable thing. I'm not arguing
against it, not at all.
2) The correct behavior for instrument readouts is a _two step_
process, as follows:
2a) Round (not truncate, round) to the nearest exact channel.
For example, assuming 25 kHz channel spacing, 120.374999 MHz
rounds to 120.375, which is the *exact* center frequency of
the channel. [See note.]
The following code works fine for this, without the need
for biasing anything, since printf is documented to round,
not truncate:
chan = sprintf("%7.3f", 120.37499);
There are of course other perfectly fine ways of accomplishing
the same goal, including dividing by the channel-spacing and
then rounding to the nearest integer.
2b) On some (not all) instruments, it may be desirable to
truncate (not round, truncate) this to two decimal places.
In our example, 120.375 may get truncated to 120.37, which is
not an exact frequency, but a truncated shorthand expression.
The following code works fine for this:
chan = sprintf("%7.3f", 120.37499);
chanx = substr(chan, 0, 6);
It must be emphasized that some instruments display the
frequency out to the last kHz, i.e. 120.375 in our example.
3) It is tempting to search for schemes to collapse the two-step
process into a one-step process, but all such schemes I've seen
are ugly, risky, or both.
In the existing flightgear instruments, one can find schemes
involving fudge factors that are /nearly/ but not quite equal
to unity, and offsets that are /nearly/ but not quite equal to
zero.
Schemes of that ilk are always less robust than simply saying
what you mean, and meaning what you say. If you mean to round
off, round off (as in item 2a) ... and if you mean to truncate,
truncate (as in item 2b).
Note: In a previous msg it was suggested, presumably in jest, that
"digital readouts are always +/- the last digit."
In case anyone missed the point, this jest calls attention to a profound
absurdity. A frequency specified as 121.5 does *not* mean 121.5 ± 0.1
or anything like that. Similarly, if you read that there are 2.54
centimeters to the inch, that does *not* mean 2.54 ± 0.01 or anything
like that. In fact there are exactly 2.54 centimeters to the inch,
i.e. 2.54 ± 0.
It is a widespread misconception that any number quoted to N
decimal places has been rounded off, and that the roundoff error
is the dominant contribution to the uncertainty. Actually that's
two misconceptions in one. You may have been /required/ to
learn these misconceptions in school, but they remain absurd
as the previous examples illustrate. For details on all of this,
see
http://www.av8n.com/physics/uncertainty.htm
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
