PR #25311 opened by Marti2203
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25311
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25311.patch
VP9 profile 1 allows 4:4:0 chroma subsampling (yuv440p), but the vpcC record of
the VP9 ISO-BMFF binding has no `chromaSubsampling` value for it (0 and 1 are
4:2:0, 2 is 4:2:2, 3 is 4:4:4, the rest reserved). `ff_isom_write_vpcc()`
therefore fails with "Unsupported pixel format" before writing anything. movenc
and flvenc ignore that error: movenc closes an empty 8-byte vpcC box, flvenc
writes a sequence start tag with an empty record, and ffmpeg exits 0. FFmpeg's
own mov demuxer then refuses the file:
```
ffmpeg -i vp91-2-04-yuv440.webm -c copy out.mp4
[mp4 @ ...] Unsupported pixel format (31)
(exit status 0)
ffprobe out.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ ...] Empty VP Codec Configuration box
[mov,mp4,m4a,3gp,3g2,mj2 @ ...] error reading header
out.mp4: Invalid data found when processing input
```
The input is `vp9-test-vectors/vp91-2-04-yuv440.webm` from the FATE suite; a
generated IVF with 4:4:0 keyframe headers behaves the same.
Since 4:4:0 cannot be signalled, the series makes both muxers return the error
instead of writing a broken file. The first patch does it in
`mov_write_vpcc_tag()` and its caller, the second makes
`flv_write_codec_header()` return an error code and checks it in
`flv_write_header()` and `flv_write_packet()`. After it:
```
[mp4 @ ...] Unsupported pixel format (31)
[out#0/mp4 @ ...] Error writing trailer: Invalid data found when processing
input
(exit status 183)
[flv @ ...] Unsupported pixel format (31)
[out#0/flv @ ...] Could not write header (incorrect codec parameters ?):
Invalid data found when processing input
(exit status 183)
```
The third patch adds fate-mov-mp4-vp9-yuv440 and fate-enhanced-flv-vp9-yuv440,
which mux that existing sample and expect failure. They fail without the fix
and pass with it.
Testing: the two new tests; fate-enhanced-flv-vp9,
fate-enhanced-flv-multitrack, fate-lavf-fate-{h264,hevc,av1}.mp4,
fate-lavf-fate-hevc.flv, all sample-free mov/mp4/flv/ismv tests and fate-source
pass. Remuxing the 4:2:0, 4:2:2 and 4:4:4 VP9 test vectors to mp4 and flv gives
byte-identical output before and after.
Caveat: for non-fragmented mp4 the moov is written in the trailer, so the error
shows up only there and the partial output file is left behind (as for other
trailer errors). With `-movflags frag_keyframe+empty_moov` it fails in the
header. If an early check is preferred, the pixel format could also be checked
in `mov_init()`; I kept the change to propagating the existing error.
This issue was found by an LLM-Enhanced Static Analysis Engine and verified
manually.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
From 96a1f530c29726db805afb3b5cb76334732f1ac1 Mon Sep 17 00:00:00 2001
From: Martin Mirchev <[email protected]>
Date: Wed, 21 Oct 2026 19:04:57 +0800
Subject: [PATCH 1/3] avformat/movenc: fail if the vpcC box cannot be written
ff_isom_write_vpcc() fails without writing anything when the pixel
format cannot be described in the vpcC box, e.g. for VP9 4:4:0
(yuv440p, allowed in profile 1), for which the VP9 ISO-BMFF binding
has no chromaSubsampling value. mov_write_vpcc_tag() ignored the error
and closed an empty 8-byte vpcC box, so ffmpeg exited successfully and
wrote a file that the mov demuxer then refuses to open ("error reading
header").
Return the error instead, so that muxing fails rather than producing a
broken file.
Signed-off-by: Martin Mirchev <[email protected]>
---
libavformat/movenc.c | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
diff --git a/libavformat/movenc.c b/libavformat/movenc.c
index 2c349ef663b..1d59ecb1ae3 100644
--- a/libavformat/movenc.c
+++ b/libavformat/movenc.c
@@ -1702,11 +1702,14 @@ static int mov_write_av3c_tag(AVIOContext *pb, MOVTrack
*track)
static int mov_write_vpcc_tag(AVFormatContext *s, AVIOContext *pb, MOVTrack
*track)
{
int64_t pos = avio_tell(pb);
+ int ret;
avio_wb32(pb, 0);
ffio_wfourcc(pb, "vpcC");
- ff_isom_write_vpcc(s, pb, track->extradata[track->last_stsd_index],
- track->extradata_size[track->last_stsd_index],
track->par);
+ ret = ff_isom_write_vpcc(s, pb, track->extradata[track->last_stsd_index],
+ track->extradata_size[track->last_stsd_index],
track->par);
+ if (ret < 0)
+ return ret;
return update_size(pb, pos);
}
@@ -2983,7 +2986,9 @@ static int mov_write_video_tag(AVFormatContext *s,
AVIOContext *pb, MOVMuxContex
} else if (track->par->codec_id ==AV_CODEC_ID_APV) {
mov_write_apvc_tag(mov->fc, pb, track);
} else if (track->par->codec_id == AV_CODEC_ID_VP9) {
- mov_write_vpcc_tag(mov->fc, pb, track);
+ ret = mov_write_vpcc_tag(mov->fc, pb, track);
+ if (ret < 0)
+ return ret;
} else if (track->par->codec_id == AV_CODEC_ID_AV1) {
mov_write_av1c_tag(pb, track);
} else if (track->par->codec_id == AV_CODEC_ID_VC1 &&
track->extradata_size[track->last_stsd_index] > 0)
--
2.54.0
From e899fa5e19024b4bbf5b9db0b43e84ac7a1a9a3a Mon Sep 17 00:00:00 2001
From: Martin Mirchev <[email protected]>
Date: Wed, 21 Oct 2026 19:04:57 +0800
Subject: [PATCH 2/3] avformat/flvenc: fail if the VP9 sequence header cannot
be written
As in movenc, ff_isom_write_vpcc() fails without writing anything when
the pixel format cannot be described in a vpcC record, e.g. for VP9
4:4:0. flv_write_codec_header() ignored the error and wrote a
PacketTypeSequenceStart tag with an empty configuration record.
Make flv_write_codec_header() return an error code and propagate it.
Signed-off-by: Martin Mirchev <[email protected]>
---
libavformat/flvenc.c | 20 ++++++++++++++------
1 file changed, 14 insertions(+), 6 deletions(-)
diff --git a/libavformat/flvenc.c b/libavformat/flvenc.c
index 2febb96d055..1f2a4deb2f1 100644
--- a/libavformat/flvenc.c
+++ b/libavformat/flvenc.c
@@ -804,7 +804,7 @@ static void flv_write_multichannel_header(AVFormatContext*
s, AVCodecParameters*
avio_wb32(pb, data_size + 5 + 11); // previous tag size
}
-static void flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par,
int64_t ts, int stream_index) {
+static int flv_write_codec_header(AVFormatContext* s, AVCodecParameters* par,
int64_t ts, int stream_index) {
int64_t data_size;
AVIOContext *pb = s->pb;
FLVContext *flv = s->priv_data;
@@ -891,9 +891,11 @@ static void flv_write_codec_header(AVFormatContext* s,
AVCodecParameters* par, i
ff_isom_write_vvcc(pb, par->extradata, par->extradata_size, 1);
else if (par->codec_id == AV_CODEC_ID_AV1)
ff_isom_write_av1c(pb, par->extradata, par->extradata_size, 1);
- else if (par->codec_id == AV_CODEC_ID_VP9)
- ff_isom_write_vpcc(s, pb, par->extradata, par->extradata_size,
par);
- else if (par->codec_id == AV_CODEC_ID_H264)
+ else if (par->codec_id == AV_CODEC_ID_VP9) {
+ int ret = ff_isom_write_vpcc(s, pb, par->extradata,
par->extradata_size, par);
+ if (ret < 0)
+ return ret;
+ } else if (par->codec_id == AV_CODEC_ID_H264)
ff_isom_write_avcc(pb, par->extradata, par->extradata_size);
else if (par->codec_id == AV_CODEC_ID_MPEG4)
avio_write(pb, par->extradata, par->extradata_size);
@@ -911,6 +913,8 @@ static void flv_write_codec_header(AVFormatContext* s,
AVCodecParameters* par, i
(av_channel_layout_compare(&par->ch_layout,
&(AVChannelLayout)AV_CHANNEL_LAYOUT_STEREO) == 1 &&
av_channel_layout_compare(&par->ch_layout,
&(AVChannelLayout)AV_CHANNEL_LAYOUT_MONO) == 1)))
flv_write_multichannel_header(s, par, ts, stream_index);
+
+ return 0;
}
static int flv_append_keyframe_info(AVFormatContext *s, FLVContext *flv,
double ts, int64_t pos)
@@ -1110,7 +1114,9 @@ static int flv_write_header(AVFormatContext *s)
}
for (i = 0; i < s->nb_streams; i++) {
- flv_write_codec_header(s, s->streams[i]->codecpar, 0, i);
+ int ret = flv_write_codec_header(s, s->streams[i]->codecpar, 0, i);
+ if (ret < 0)
+ return ret;
}
flv->datastart_offset = avio_tell(pb);
@@ -1259,7 +1265,9 @@ static int flv_write_packet(AVFormatContext *s, AVPacket
*pkt)
if (ret < 0)
return ret;
memcpy(par->extradata, side, side_size);
- flv_write_codec_header(s, par, pkt->dts, pkt->stream_index);
+ ret = flv_write_codec_header(s, par, pkt->dts, pkt->stream_index);
+ if (ret < 0)
+ return ret;
}
flv_write_metadata_packet(s, par, pkt->dts, pkt->stream_index);
}
--
2.54.0
From 639c86604cc051bd7baa26799daaa84dddbd1129 Mon Sep 17 00:00:00 2001
From: Martin Mirchev <[email protected]>
Date: Wed, 21 Oct 2026 19:05:55 +0800
Subject: [PATCH 3/3] fate: test that VP9 4:4:0 is rejected by the mp4 and flv
muxers
Uses an existing sample from the VP9 test vectors; vpcC has no
chromaSubsampling value for 4:4:0, so muxing it must fail.
Signed-off-by: Martin Mirchev <[email protected]>
---
tests/fate/flvenc.mak | 5 +++++
tests/fate/mov.mak | 5 +++++
2 files changed, 10 insertions(+)
diff --git a/tests/fate/flvenc.mak b/tests/fate/flvenc.mak
index 1cc8e1f1603..d1fd87f911c 100644
--- a/tests/fate/flvenc.mak
+++ b/tests/fate/flvenc.mak
@@ -13,6 +13,11 @@ FATE_ENHANCED_FLVENC_FFMPEG-$(call REMUX, FLV IVF,
FLV_DEMUXER VP9_PARSER) += fa
fate-enhanced-flv-vp9: CMD = transcode ivf
$(TARGET_SAMPLES)/vp9-test-vectors/vp90-2-05-resize.ivf\
flv "-c copy" "-c copy"
+# VP9 4:4:0 cannot be signalled in vpcC, so muxing must fail instead of
writing a broken file
+FATE_ENHANCED_FLVENC_FFMPEG-$(call ALLYES, MATROSKA_DEMUXER VP9_PARSER
FLV_MUXER FILE_PROTOCOL) += fate-enhanced-flv-vp9-yuv440
+fate-enhanced-flv-vp9-yuv440: CMP = null
+fate-enhanced-flv-vp9-yuv440: CMD =
out=tests/data/fate/enhanced-flv-vp9-yuv440.flv; ffmpeg -i
$(TARGET_SAMPLES)/vp9-test-vectors/vp91-2-04-yuv440.webm -c copy -f flv -y
$$out; ret=$$?; rm -f $$out; test $$ret -ne 0
+
FATE_ENHANCED_FLVENC_FFMPEG-$(call REMUX, FLV IVF, FLV_DEMUXER AV1_DECODER
AV1_PARSER EXTRACT_EXTRADATA_BSF) += fate-enhanced-flv-av1
fate-enhanced-flv-av1: CMD = stream_remux ivf
$(TARGET_SAMPLES)/av1/seq_hdr_op_param_info.ivf "-c:v av1" \
flv "-c copy" "-c:v av1" "-c copy"
diff --git a/tests/fate/mov.mak b/tests/fate/mov.mak
index 3550e1d7b81..02c433980ef 100644
--- a/tests/fate/mov.mak
+++ b/tests/fate/mov.mak
@@ -341,6 +341,11 @@ fate-mov-mp4-mpegh-5_1_4: CMD = stream_demux mp4
$(TARGET_SAMPLES)/mov/mpegh_lay
FATE_MOV_FFMPEG_FFPROBE_SAMPLES-$(call REMUX, MP4 MOV) +=
fate-mov-mp4-mpegh-5_1_4-copy
fate-mov-mp4-mpegh-5_1_4-copy: CMD = stream_remux mp4
$(TARGET_SAMPLES)/mov/mpegh_layout16.mp4 "" mp4 "" "" "-c:a copy"
"-show_entries stream=profile,level,codec_tag_string"
+# VP9 4:4:0 cannot be signalled in vpcC, so muxing must fail instead of
writing a broken file
+FATE_MOV_FFMPEG_SAMPLES-$(call ALLYES, MATROSKA_DEMUXER VP9_PARSER MP4_MUXER
FILE_PROTOCOL) += fate-mov-mp4-vp9-yuv440
+fate-mov-mp4-vp9-yuv440: CMP = null
+fate-mov-mp4-vp9-yuv440: CMD = out=tests/data/fate/mov-mp4-vp9-yuv440.mp4;
ffmpeg -i $(TARGET_SAMPLES)/vp9-test-vectors/vp91-2-04-yuv440.webm -c copy -f
mp4 -y $$out; ret=$$?; rm -f $$out; test $$ret -ne 0
+
FATE_SAMPLES_FFMPEG_FFPROBE += $(FATE_MOV_FFMPEG_FFPROBE_SAMPLES-yes)
FATE_SAMPLES_FFMPEG += $(FATE_MOV_FFMPEG_SAMPLES-yes)
--
2.54.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
