ljpeg_encode_yuv_mb() reads the source samples of each macroblock at
ptr = data[i] + linesize * (v * mb_y + y) + (h * mb_x + x);
The macroblock counts are rounded up (mb_width = ceil(width / hsample[0]),
mb_height = ceil(height / vsample[0])), so when a plane dimension is not a
multiple of the sampling factor the last macroblock addresses one sample
past the right and/or bottom edge of the plane. For yuvj420p this triggers
for any odd width or height; encoding a 999x999 frame reads 1999 samples
out of bounds.
Clamp the read coordinates to the last valid sample (edge replication).
For dimensions that are a multiple of the sampling factor the clamp never
triggers, so the encoded output is bit-exact and there is no functional
change; the over-read only touched padding samples that the decoder crops
away, which is why it is not visible in the decoded output and escapes
valgrind (the read stays inside the pooled frame allocation).
Fixes: ticket #10052
Signed-off-by: Bogdan Lisman <[email protected]>
---
libavcodec/ljpegenc.c | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/libavcodec/ljpegenc.c b/libavcodec/ljpegenc.c
index ab37ab0d5c..a9bd2fd5a7 100644
--- a/libavcodec/ljpegenc.c
+++ b/libavcodec/ljpegenc.c
@@ -127,16 +127,18 @@ static inline void ljpeg_encode_yuv_mb(LJpegEncContext
*s, PutBitContext *pb,
if (mb_x == 0 || mb_y == 0) {
for (i = 0; i < 3; i++) {
const uint8_t *ptr;
- int x, y, h, v, linesize;
+ int x, y, h, v, linesize, last_x, last_y;
h = s->hsample[i];
v = s->vsample[i];
linesize = frame->linesize[i];
+ last_x = (frame->width * h - 1) / s->hsample[0];
+ last_y = (frame->height * v - 1) / s->vsample[0];
for (y = 0; y < v; y++) {
for (x = 0; x < h; x++) {
int pred;
- ptr = frame->data[i] + (linesize * (v * mb_y + y)) + (h *
mb_x + x); //FIXME optimize this crap
+ ptr = frame->data[i] + linesize * FFMIN(v * mb_y + y,
last_y) + FFMIN(h * mb_x + x, last_x); //FIXME optimize this crap
if (y == 0 && mb_y == 0) {
if (x == 0 && mb_x == 0)
pred = 128;
@@ -161,16 +163,18 @@ static inline void ljpeg_encode_yuv_mb(LJpegEncContext
*s, PutBitContext *pb,
} else {
for (i = 0; i < 3; i++) {
const uint8_t *ptr;
- int x, y, h, v, linesize;
+ int x, y, h, v, linesize, last_x, last_y;
h = s->hsample[i];
v = s->vsample[i];
linesize = frame->linesize[i];
+ last_x = (frame->width * h - 1) / s->hsample[0];
+ last_y = (frame->height * v - 1) / s->vsample[0];
for (y = 0; y < v; y++) {
for (x = 0; x < h; x++) {
int pred;
- ptr = frame->data[i] + (linesize * (v * mb_y + y)) + (h *
mb_x + x); //FIXME optimize this crap
+ ptr = frame->data[i] + linesize * FFMIN(v * mb_y + y,
last_y) + FFMIN(h * mb_x + x, last_x); //FIXME optimize this crap
PREDICT(pred, ptr[-linesize - 1], ptr[-linesize], ptr[-1],
predictor);
if (i == 0)
--
2.43.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
