Ricky Robertson wrote: > stupid question alert.... > > has anyone run across this kind of behavior before in r.mapcalc? > > i have these maps with absurdly large values and i am trying to pull out > the pixels whose values are below some threshold. the original map > has floating point values. i was/am using integer values for the theshold. > for small thresholds (e.g., less than 2,000,000,000, i.e., 2 billion) > everything seemed hunky-dory. however, when i moved up to 3 billion, it > wouldn't pull any out. for some thresholds above that, it would generate > strange results, others would just do the no-pixels thing.
The maximum value of a signed 32-bit integer is 2^31-1 = 2,147,483,647. If you try to convert any FP value larger than that to an integer, the result is undefined (and likely platform-dependent; on Linux/x86, I get nulls). > i have written a little script to show and reproduce in excruciating > detail what the strange behavior is. here is the basic summary. when > i do a "less than" comparison with the large integer, i get no pixels > even though my threshold is well below the maximum value. when i do > a "greater than" comparison (which should give me the complement), > i get some pixels. actually, i get all of the non-null pixels regardless of > whether they are above or below the threshold. > > when i do the "less than" using 3 billion point zero (3000000000.0) > instead of 3 billion (3000000000), it pulls out some pixels. when i do > the "greater than" using the float, it pulls out some other pixels. > using the floating point versions seems to add up in that the number of > greater-than pixels plus the number of less-than pixels adds up to the > total number of non-null pixels. Integer literals are parsed using the system's atoi() function. For integers which exceed the 32-bit range, the result may be platform-dependent. On Linux/x86, oversize values are clamped to +/-2147483647. -- Glynn Clements <[EMAIL PROTECTED]> _______________________________________________ grass-user mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/grass-user
