PR #25909 opened by Kilian (lusoris) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25909 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25909.patch
This fixes an out-of-bounds read in the libx264 wrapper: with interlaced or fake-interlaced encoding, an odd macroblock row count and ROI side data, x264 reads one row of `quant_offsets` past the map that `setup_roi()` allocates. ## Cause `setup_roi()` allocates `mbx * mby` offsets, with `mby` the frame's own macroblock row count. x264 rounds its row count up to an even number when interlaced or fake-interlaced encoding is on (`x264_sps_init()` in `encoder/set.c` sets `i_mb_height` that way when `b_frame_mbs_only` is off) and reads `quant_offsets` over that grid in `x264_adaptive_quant_frame()` (`encoder/ratecontrol.c`). With an odd count it reads one row past the allocation. ## Change `setup_roi()` now allocates the map for an even row count. The clamps on the ROI rectangles stay on the frame's own row count, so the extra row keeps offset 0. ## Testing - AddressSanitizer build of master with libx264 and the `addroi` filter: `testsrc2=s=64x48` with `addroi=0:0:iw/2:ih/2:-0.2` and `-x264-params fake-interlaced=1` or `interlaced=1` reports a heap-buffer-overflow read in `x264_adaptive_quant_frame()`, 0 bytes after the 48-byte region allocated in `setup_roi()`. 64x64, and both sizes without those parameters, run clean. - The same build with this change: all six runs exit 0 without an AddressSanitizer report. - The H.264 output is bit-identical between master and this change for the four cases whose grid does not round: 64x48 and 64x64 without those parameters, and 64x64 with `interlaced=1` or `fake-interlaced=1`. The read was found in review of the libx264 qpfile pull request 25875, which touches the same file; whichever lands second will be rebased. >From bc7751bd8edaf798609f25d2dc38757637540d64 Mon Sep 17 00:00:00 2001 From: lusoris <[email protected]> Date: Tue, 10 Nov 2026 05:28:26 +0100 Subject: [PATCH] avcodec/libx264: size the ROI offset map for interlaced encoding x264 rounds its macroblock row count up to an even number for interlaced and fake-interlaced encoding and reads quant_offsets over that grid. setup_roi() allocated the map for the frame's own row count, so with an odd count x264 read one row past the allocation: at 64x48 with -x264-params fake-interlaced=1 or interlaced=1 and ROI side data, AddressSanitizer reports a heap-buffer-overflow in x264_adaptive_quant_frame(). Allocate the map for an even row count. The extra row keeps offset 0, and the output for sizes and modes that do not round is unchanged. Found in review of pull request 25875 (qpfile option). Signed-off-by: lusoris <[email protected]> --- libavcodec/libx264.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/libavcodec/libx264.c b/libavcodec/libx264.c index a432c0b50e5..a1a6a2e76f8 100644 --- a/libavcodec/libx264.c +++ b/libavcodec/libx264.c @@ -407,6 +407,10 @@ static int setup_roi(AVCodecContext *ctx, x264_picture_t *pic, int mbx = (frame->width + MB_SIZE - 1) / MB_SIZE; int mby = (frame->height + MB_SIZE - 1) / MB_SIZE; + /* x264 rounds the macroblock row count up to an even number for + * interlaced and fake-interlaced encoding and reads quant_offsets + * over that grid, so size the map for it. */ + int mby_alloc = FFALIGN(mby, 2); int qp_range = 51 + 6 * (x4->params.i_bitdepth - 8); int nb_rois; const AVRegionOfInterest *roi; @@ -435,7 +439,7 @@ static int setup_roi(AVCodecContext *ctx, x264_picture_t *pic, } nb_rois = size / roi_size; - qoffsets = av_calloc(mbx * mby, sizeof(*qoffsets)); + qoffsets = av_calloc(mbx * mby_alloc, sizeof(*qoffsets)); if (!qoffsets) return AVERROR(ENOMEM); -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
