James Cloos <cl...@jhcloos.com> writes: >>>>>> "CW" == Carl Worth <cwo...@cworth.org> writes: > > CW> But then you simply assert that the conversion from integer to > CW> floating-point must be f(i) = i/65535.0. What's the justification for > CW> this? > > When I looked into this many moons ago, I concluded that the best option > is to multiply (with saturation) by IMAX+1 and divide by IMAX. Ie, what > cairo currently does.
There are two functions being talked about here: f: uint16_t -> double and g: double -> uint16_t When you say "multiply (with saturation) by IMAX+1 and divide by IMAX" I assume you mean f(i) = i / 65535.0 and g(x) = MAX (x * 65536, 65535.0) which is indeed what cairo does now. I think everyone agrees that f(i) = i / 65535.0 is the right choice for f, but for g, I am proposing a different formula: g(x) = floor (x * 65535.0 + 0.5) that as argued elsewhere has less error. > Dividing by IMAX generates an infinitely repeating value which best > places the integer value w/in the interval which rounds to said integer. > > This is akin to mapping {0,...,9} to {0/9,...,9/9} generating the > decimal expansions 0, .111..., .222..., .333,... et cetera. > > It is widely accepted that when expanding an 8bit value to 16 bit, one > should duplicate the bits, eg by multiplying by 257. Dividing by IMAX > generates a floating point value which accomplishes that same goal. It sounds like you are arguing here that f should indeed be defined as above (as a division by IMAX). But as far as I can tell you are not arguing that g should necessarily stay the same. For the conversion of 8 to 16 bits, both discussed choices of g will just duplicate the bits (this will be the case for conversions between all integers whose width is a power of two), though with the proposed g, converting back will no longer just be a bit shift. Søren _______________________________________________ Pixman mailing list Pixman@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/pixman