On 1/31/13 6:21 PM, Derek Buitenhuis wrote: > On 2013-01-27 4:41 PM, Derek Buitenhuis wrote: >> On 2013-01-27 4:38 PM, Ronald S. Bultje wrote: >>>> - dst[i] = ((RY * r + GY * g + BY * b + (33 << (RGB2YUV_SHIFT + bpc >>>> - 9))) >> RGB2YUV_SHIFT); >>>> + dst[i] = ((RY * r + GY * g + BY * b + (33 << (RGB2YUV_SHIFT + bpc >>>> - 9))) >> (RGB2YUV_SHIFT + bpc - 14)); >>> >>> This looks wrong. The original code also btw. We're doing >>> (cast_to_fixed_point)(r*ry + g*gy + b*by + 16 + 0.5). I don't think >>> this does the correct thing now. >> >> I'll look into it. >> >>>> - dstU[i] = (RU * r + GU * g + BU * b + (257 << (RGB2YUV_SHIFT + >>>> bpc - 9))) >> RGB2YUV_SHIFT; >>>> - dstV[i] = (RV * r + GV * g + BV * b + (257 << (RGB2YUV_SHIFT + >>>> bpc - 9))) >> RGB2YUV_SHIFT; >>>> + dstU[i] = (RU * r + GU * g + BU * b + (257 << (RGB2YUV_SHIFT + >>>> bpc - 9))) >> (RGB2YUV_SHIFT + bpc - 14); >>>> + dstV[i] = (RV * r + GV * g + BV * b + (257 << (RGB2YUV_SHIFT + >>>> bpc - 9))) >> (RGB2YUV_SHIFT + bpc - 14); >>> >>> Same. The round and shift should correspond to each other in both >>> cases, and now they don't. >> >> Ditto. > > So I don't really know what to do with this patch, since according to Michael, > he was aware of the error, but it worked, so there must be errors elsewhere > in the > code... yeah.
Meh > > Personally, I'm even confused why isAnyRGB affects this at all (and in what > way), > so I am certainly not the person to look. > > I can drop the patch or spend lots of time trying to hunt sws code. > Opinions... ? > I'm heading to Fosdem (and I had been busy preparing stuff), let me look at it after it (or have Kostya have a look by promising more bottles of $beverage) lu - hoping to not have his luggage break during the transport... _______________________________________________ libav-devel mailing list libav-devel@libav.org https://lists.libav.org/mailman/listinfo/libav-devel
