On Thu, Jun 19, 2014 at 05:04:23PM +0200, Miroslav Lichvar wrote: > Yes, it's the same check. Assuming residual can be at most > FLAC__MAX_EXTRA_RESIDUAL_BPS bits wider than subframe_bps, I think it > should be: > > if(subframe_bps + qlp_coeff_precision + FLAC__bitmath_ilog2(order) + > FLAC__MAX_EXTRA_RESIDUAL_BPS - 1 <= 32) > if(subframe_bps + FLAC__MAX_EXTRA_RESIDUAL_BPS <= 16 && > qlp_coeff_precision <= 16)
On second thought, this is probably not necessary. The residual seems to be always treated as 32-bit in the LPC encoding and decoding functions, even in the MMX assembly if I'm reading it correctly. It doesn't matter if it's wider than the original signal. -- Miroslav Lichvar _______________________________________________ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev