PR #25909 opened by Kilian (lusoris)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25909
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25909.patch

This fixes an out-of-bounds read in the libx264 wrapper: with interlaced or 
fake-interlaced encoding, an odd macroblock row count and ROI side data, x264 
reads one row of `quant_offsets` past the map that `setup_roi()` allocates.

## Cause

`setup_roi()` allocates `mbx * mby` offsets, with `mby` the frame's own 
macroblock row count. x264 rounds its row count up to an even number when 
interlaced or fake-interlaced encoding is on (`x264_sps_init()` in 
`encoder/set.c` sets `i_mb_height` that way when `b_frame_mbs_only` is off) and 
reads `quant_offsets` over that grid in `x264_adaptive_quant_frame()` 
(`encoder/ratecontrol.c`). With an odd count it reads one row past the 
allocation.

## Change

`setup_roi()` now allocates the map for an even row count. The clamps on the 
ROI rectangles stay on the frame's own row count, so the extra row keeps offset 
0.

## Testing

- AddressSanitizer build of master with libx264 and the `addroi` filter: 
`testsrc2=s=64x48` with `addroi=0:0:iw/2:ih/2:-0.2` and `-x264-params 
fake-interlaced=1` or `interlaced=1` reports a heap-buffer-overflow read in 
`x264_adaptive_quant_frame()`, 0 bytes after the 48-byte region allocated in 
`setup_roi()`. 64x64, and both sizes without those parameters, run clean.
- The same build with this change: all six runs exit 0 without an 
AddressSanitizer report.
- The H.264 output is bit-identical between master and this change for the four 
cases whose grid does not round: 64x48 and 64x64 without those parameters, and 
64x64 with `interlaced=1` or `fake-interlaced=1`.

The read was found in review of the libx264 qpfile pull request 25875, which 
touches the same file; whichever lands second will be rebased.



>From bc7751bd8edaf798609f25d2dc38757637540d64 Mon Sep 17 00:00:00 2001
From: lusoris <[email protected]>
Date: Tue, 10 Nov 2026 05:28:26 +0100
Subject: [PATCH] avcodec/libx264: size the ROI offset map for interlaced
 encoding

x264 rounds its macroblock row count up to an even number for
interlaced and fake-interlaced encoding and reads quant_offsets over
that grid. setup_roi() allocated the map for the frame's own row
count, so with an odd count x264 read one row past the allocation:
at 64x48 with -x264-params fake-interlaced=1 or interlaced=1 and ROI
side data, AddressSanitizer reports a heap-buffer-overflow in
x264_adaptive_quant_frame().

Allocate the map for an even row count. The extra row keeps offset 0,
and the output for sizes and modes that do not round is unchanged.

Found in review of pull request 25875 (qpfile option).

Signed-off-by: lusoris <[email protected]>
---
 libavcodec/libx264.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/libavcodec/libx264.c b/libavcodec/libx264.c
index a432c0b50e5..a1a6a2e76f8 100644
--- a/libavcodec/libx264.c
+++ b/libavcodec/libx264.c
@@ -407,6 +407,10 @@ static int setup_roi(AVCodecContext *ctx, x264_picture_t 
*pic,
 
     int mbx = (frame->width + MB_SIZE - 1) / MB_SIZE;
     int mby = (frame->height + MB_SIZE - 1) / MB_SIZE;
+    /* x264 rounds the macroblock row count up to an even number for
+     * interlaced and fake-interlaced encoding and reads quant_offsets
+     * over that grid, so size the map for it. */
+    int mby_alloc = FFALIGN(mby, 2);
     int qp_range = 51 + 6 * (x4->params.i_bitdepth - 8);
     int nb_rois;
     const AVRegionOfInterest *roi;
@@ -435,7 +439,7 @@ static int setup_roi(AVCodecContext *ctx, x264_picture_t 
*pic,
     }
     nb_rois = size / roi_size;
 
-    qoffsets = av_calloc(mbx * mby, sizeof(*qoffsets));
+    qoffsets = av_calloc(mbx * mby_alloc, sizeof(*qoffsets));
     if (!qoffsets)
         return AVERROR(ENOMEM);
 
-- 
2.54.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]

Reply via email to