On 2013-01-27 4:41 PM, Derek Buitenhuis wrote: > On 2013-01-27 4:38 PM, Ronald S. Bultje wrote: >>> - dst[i] = ((RY * r + GY * g + BY * b + (33 << (RGB2YUV_SHIFT + bpc >>> - 9))) >> RGB2YUV_SHIFT); >>> + dst[i] = ((RY * r + GY * g + BY * b + (33 << (RGB2YUV_SHIFT + bpc >>> - 9))) >> (RGB2YUV_SHIFT + bpc - 14)); >> >> This looks wrong. The original code also btw. We're doing >> (cast_to_fixed_point)(r*ry + g*gy + b*by + 16 + 0.5). I don't think >> this does the correct thing now. > > I'll look into it. > >>> - dstU[i] = (RU * r + GU * g + BU * b + (257 << (RGB2YUV_SHIFT + bpc >>> - 9))) >> RGB2YUV_SHIFT; >>> - dstV[i] = (RV * r + GV * g + BV * b + (257 << (RGB2YUV_SHIFT + bpc >>> - 9))) >> RGB2YUV_SHIFT; >>> + dstU[i] = (RU * r + GU * g + BU * b + (257 << (RGB2YUV_SHIFT + bpc >>> - 9))) >> (RGB2YUV_SHIFT + bpc - 14); >>> + dstV[i] = (RV * r + GV * g + BV * b + (257 << (RGB2YUV_SHIFT + bpc >>> - 9))) >> (RGB2YUV_SHIFT + bpc - 14); >> >> Same. The round and shift should correspond to each other in both >> cases, and now they don't. > > Ditto.
So I don't really know what to do with this patch, since according to Michael, he was aware of the error, but it worked, so there must be errors elsewhere in the code... yeah. Personally, I'm even confused why isAnyRGB affects this at all (and in what way), so I am certainly not the person to look. I can drop the patch or spend lots of time trying to hunt sws code. Opinions... ? - Derek _______________________________________________ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
