If the trig table is in the range -1<=t<1 , then I think the overflow must be coming from the input not being scaled by 1/N or something like that..
On 12 Jul 2010 11:13, "Mohamed Tarek" <mtare...@gmail.com> wrote: On Mon, Jul 12, 2010 at 12:41 PM, Dave Hooper <d...@beermex.com> wrote: > > I'd be surprised if cod... There's no problem with the codeclib's imdct itself. wmapro's mdct is almost just like codeclib's, but uses a different trig table and currently scales the input and trig tables with different factors to produce a correct output and avoid overflows. I'll just use that mdct for now and see if I can use codeclib's. For now, I tried scaling the input coefficients before sending them to codeclib's imdct_half, instead of after that, but it still didn't work properly. -- MT
