Hello! I just noticed how the LPC prediction is quantized by browsing the source code of the java port jFlac:
:
int sum = 0;
for (int j = 0; j < order; j++)
sum += qlpCoeff[j] * data[startAt+i-j-1];
data[startAt + i] = residual[i] + (sum >> lpQuantization);
:
I checked the C version (libFLAC) and it matches. However, it was a
surprise because this way of quantizing the prediction introduces a bias
towards negative infinity which results in residual samples that have a
bias towards positive infinity and therefore a shifted probability
distribution: residual of +1 is as common as 0 and more common than -1
etc...
This is not too bad I guess. But it CAN be accounted for -- for example
via an alternate entropy coding which uses a mapping from all possible
residual values to non-negative numbers by this interleaving:
0 -> 0
+1 -> 1
-1 -> 2
+2 -> 3
-2 -> 4
+3 -> 5
etc
This way the symbols 0, 1, 2, ... have non-increasing probabilities.
Anyhow, this quantization issue is not mentioned in the format
specification. Naturally, I would have assumed to quantize by properly
rounding like this:
data[startAt + i] = residual[i] + ((sum + a) >> lpQuantization);
where
a = (1 << lpQuantization) / 2;
but I guess now it's too late to change that behaviour of the FIR predictor.
Cheers!
SG
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Flac mailing list Flac@xiph.org http://lists.xiph.org/mailman/listinfo/flac
