PR #24508 opened by anders-mjoll URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24508 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24508.patch
# Summary of changes Make sure that demuxers which implements `read_timestamp` seeks to the nearest key frame when `AVSEEK_FLAG_ANY` is not set. Without this fix seeking in the middle of a GOP in a MPEG-TS file will cause the demuxer to return a non-keyframe as the first video packet. This can easily be reproduced with the following steps: ``` # Generate input file with GOP size 25 ffmpeg -t 10 -f lavfi -i testsrc=s=1280x720:r=25 -t 10 -f lavfi -i sine -map 0:v:0 -map 1:a:0 -c:v mpeg2video -g 25 -c:a aac -f mpegts input.ts # Read from the middle of the GOP and dump the timestamps of the frames entering the filter graph ffmpeg -t 0.1 -ss 2.5 -i input.ts -filter_complex '[0:v:0]showinfo[vout]' -map [vout] -c:v rawvideo -f null dummy ``` Currently on master this is first line outputted by showinfo: ``` [Parsed_showinfo_0 @ 0x7f25dc002e40] n: 0 pts: 47090 pts_time:0.523222 duration: 3600 duration_time:0.04 fmt:yuv420p cl:left sar:1/1 s:1280x720 i:P iskey:1 type:I checksum:0D452999 plane_checksum:[4760C81A 3D0F21EA 9EA23F86] mean:[125 129 126] stdev:[71.8 70.2 71.9] ``` As indicated above the first decoded frame entering the filter graph has a pts_time of 0.5232222 which is the distance from the requested seek point to the following key frame. Combined with for instance constant frame rate output this causes a freeze frame lasting approx. 0.5 seconds. Compared to the output of the same command with this fix in place: ``` [Parsed_showinfo_0 @ 0x7fe04c002e40] n: 0 pts: 290 pts_time:0.00322222 duration: 3600 duration_time:0.04 fmt:yuv420p cl:left sar:1/1 s:1280x720 i:P iskey:0 type:P checksum:C130B1FE plane_checksum:[12547087 6BAC7318 958CCE50] mean:[125 129 126] stdev:[71.8 70.2 72.0] ``` For the first attempt at fixing this I tried to solve it inside the mpegts demuxer implementation. The problem with this approach was that from the `read_timestamp` function I don't have access to the seek flags so the only way I could implement it was to always seek to the nearest I-frame. I.e this implementation would not respect the `AVSEEK_FLAG_ANY`. Because of this I moved it to the generic seek code. One issue with this approach though is that it assumes that the anchoring or the `read_timestamp` byte offset and the byte offset set on the `AVPacket` `pos` field is the same. This turned out to not be the case for the "rm" demuxer. Because of this I had to add an escape hatch that checks this assumption. This fix covers both MPEG-TS and MPEG-PS. I also had to fix a related bug in MPEG-TS in which it would update the seek index cache with all frames being marked as key frames. Without this fix the main fix would not work since `ff_seek_frame_binary` would exit early with a false match for an I-frame on any of the frames encountered by the mpegts `read_timestamp` implementation. Performance impact: It's worth mentioning that this will add additional cost to seeking for demuxers that leverages `read_timestamp` for seeking. Seek will on average read half a GOP of frames for every probe. I did contemplate adding a flag for accurate seek in order to opt in to this due to the performance cost, but that is exactly the point of `AVSEEK_FLAG_ANY`. While it does break backwards compatibility it makes more sense that if a fast inaccurate seek is wanted the `-seek2any 1` command line argument should explicitly be requested. >From 2cbbb15c790e2444fad5583cb640ccb6af361d7b Mon Sep 17 00:00:00 2001 From: Anders Rein <[email protected]> Date: Tue, 15 Sep 2026 14:34:41 +0200 Subject: [PATCH 1/2] avformat/mpegts: do not mark every index entry as a keyframe mpegts_get_dts() adds an index entry for every packet it walks past while searching, but flags all of them AVINDEX_KEYFRAME. ff_seek_frame_binary() seeds pos_min/ts_min and pos_max/ts_max from that index before it starts probing, and a backward seek returns pos_min. As av_index_search_timestamp() only considers entries flagged as keyframes unless AVSEEK_FLAG_ANY is set, a mislabelled entry can become the seek result without the binary search running at all. Only the first seek on a context is unaffected, since nothing populates the index before it. Store the flag the packet actually carries. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> --- libavformat/mpegts.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/libavformat/mpegts.c b/libavformat/mpegts.c index a1b6065320..fead227568 100644 --- a/libavformat/mpegts.c +++ b/libavformat/mpegts.c @@ -3802,7 +3802,8 @@ static int64_t mpegts_get_dts(AVFormatContext *s, int stream_index, } if (pkt->dts != AV_NOPTS_VALUE && pkt->pos >= 0) { ff_reduce_index(s, pkt->stream_index); - av_add_index_entry(s->streams[pkt->stream_index], pkt->pos, pkt->dts, 0, 0, AVINDEX_KEYFRAME /* FIXME keyframe? */); + av_add_index_entry(s->streams[pkt->stream_index], pkt->pos, pkt->dts, 0, 0, + pkt->flags & AV_PKT_FLAG_KEY ? AVINDEX_KEYFRAME : 0); if (pkt->stream_index == stream_index && pkt->pos >= *ppos) { int64_t dts = pkt->dts; *ppos = pkt->pos; -- 2.52.0 >From 35d438520c4f3e7a9bdffe0334e0caf666313c91 Mon Sep 17 00:00:00 2001 From: Anders Rein <[email protected]> Date: Tue, 15 Sep 2026 14:35:09 +0200 Subject: [PATCH 2/2] avformat/seek: seek to keyframes in the generic binary search ff_gen_search() accepts whatever position read_timestamp() reports, without regard for whether a decoder can start there. For mpegts this makes -ss land in the middle of a GOP: the leading frames reference a picture that was never demuxed, the decoder discards them, and output only starts at the next keyframe. Seeking to 9.7s in a 29.97 fps MPEG-2 transport stream with a 15 frame GOP returned the first frame at 10.27s. Longer GOPs lose proportionally more. Keep reading packets after the oracle has reported a position and stop at the first keyframe of the stream being searched. Reporting a keyframe past the target drives the search backwards, so it converges on the last keyframe at or before it. Where there is none before EOF the last frame seen is reported, which is also past the target and has the same effect. AVSEEK_FLAG_ANY skips this, so seek2any keeps behaving as it did. Streams whose packets can never carry AV_PKT_FLAG_KEY, because they are handed out unparsed and are not all-intra, are recognised by ff_stream_has_keyframes() and left alone. The scan can only report pkt->pos, which is a usable seek target solely for demuxers that anchor it where read_timestamp() does. rm, for one, reports the packet header while pkt->pos lies past it, and seeking there skips data. Compare the two on the first packet read and leave the position to read_timestamp() when they disagree. mpegts and mpegps are the demuxers affected. Requests that used to return a non-keyframe, or a packet before a forward seek target, now return a keyframe; their seek references are updated accordingly. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> --- libavformat/demux.c | 19 ++++++++++++- libavformat/demux.h | 6 ++++ libavformat/seek.c | 61 +++++++++++++++++++++++++++++++++++++++-- tests/ref/seek/lavf-mpg | 4 +-- tests/ref/seek/lavf-ts | 6 ++-- 5 files changed, 88 insertions(+), 8 deletions(-) diff --git a/libavformat/demux.c b/libavformat/demux.c index 97eaa5f9bc..1a3e4e80c3 100644 --- a/libavformat/demux.c +++ b/libavformat/demux.c @@ -980,6 +980,12 @@ static void update_initial_durations(AVFormatContext *s, AVStream *st, sti->cur_dts = cur_dts; } +static int stream_is_all_intra(const AVStream *st) +{ + return st->codecpar->codec_type == AVMEDIA_TYPE_DATA || + ff_is_intra_only(st->codecpar->codec_id); +} + static void compute_pkt_fields(AVFormatContext *s, AVStream *st, AVCodecParserContext *pc, AVPacket *pkt, int64_t next_dts, int64_t next_pts) @@ -1165,10 +1171,21 @@ static void compute_pkt_fields(AVFormatContext *s, AVStream *st, presentation_delayed, delay, av_ts2str(pkt->pts), av_ts2str(pkt->dts), av_ts2str(sti->cur_dts), st->index, st->id); /* update flags */ - if (st->codecpar->codec_type == AVMEDIA_TYPE_DATA || ff_is_intra_only(st->codecpar->codec_id)) + if (stream_is_all_intra(st)) pkt->flags |= AV_PKT_FLAG_KEY; } +int ff_stream_has_keyframes(const AVFormatContext *s, const AVStream *st) +{ + const FFStream *const sti = cffstream(st); + + if (stream_is_all_intra(st)) + return 1; + /* not sti->parser, that is freed on flush and when drained at EOF */ + return sti->need_parsing != AVSTREAM_PARSE_NONE && + !(s->flags & AVFMT_FLAG_NOPARSE); +} + /** * Parse a packet, add all split parts to parse_queue. * diff --git a/libavformat/demux.h b/libavformat/demux.h index 73ecd494b8..0f43518481 100644 --- a/libavformat/demux.h +++ b/libavformat/demux.h @@ -234,6 +234,12 @@ int ff_read_packet(AVFormatContext *s, AVPacket *pkt); void ff_read_frame_flush(AVFormatContext *s); +/** + * Test whether packets of this stream can ever carry AV_PKT_FLAG_KEY. + * Valid once a packet of the stream has been read. + */ +int ff_stream_has_keyframes(const AVFormatContext *s, const AVStream *st); + /** * Perform a binary search using av_index_search_timestamp() and * FFInputFormat.read_timestamp(). diff --git a/libavformat/seek.c b/libavformat/seek.c index c0d94371e6..2064d1cd5e 100644 --- a/libavformat/seek.c +++ b/libavformat/seek.c @@ -287,6 +287,61 @@ static int64_t read_timestamp(AVFormatContext *s, int stream_index, int64_t *ppo return ts; } +/* Advance to the first keyframe at or after the position the timestamp was read + * from. Starting a decoder anywhere else leaves it with nothing to output until + * the next keyframe, so a non-keyframe is not a usable seek point. If there is + * none before EOF the last frame is reported, which is >= the target and pushes + * the search backwards. */ +static int64_t read_timestamp_keyframe(AVFormatContext *s, int stream_index, + int64_t *ppos, int64_t pos_limit, int flags, + int64_t (*read_timestamp_func)(struct AVFormatContext *, + int, int64_t *, int64_t)) +{ + int64_t ts = read_timestamp(s, stream_index, ppos, pos_limit, read_timestamp_func); + int64_t last_pos = -1, last_ts = AV_NOPTS_VALUE; + AVStream *st; + AVPacket *pkt; + + if (ts == AV_NOPTS_VALUE || stream_index < 0 || (flags & AVSEEK_FLAG_ANY)) + return ts; + + st = s->streams[stream_index]; + if (!ff_stream_has_keyframes(s, st)) + return ts; + + pkt = av_packet_alloc(); + if (!pkt) + return ts; + + ff_read_frame_flush(s); + if (avio_seek(s->pb, *ppos, SEEK_SET) >= 0) { + while (av_read_frame(s, pkt) >= 0) { + if (pkt->stream_index == stream_index && + pkt->dts != AV_NOPTS_VALUE && pkt->pos >= 0) { + /* pkt->pos is only a valid seek target for demuxers that + * anchor it where read_timestamp() does; rm for one reports + * the packet header while pkt->pos is past it. Bail out and + * leave the position to read_timestamp() if they disagree. */ + if (last_pos < 0 && pkt->pos != *ppos) + break; + last_pos = pkt->pos; + last_ts = pkt->dts; + if (pkt->flags & AV_PKT_FLAG_KEY) { + av_packet_unref(pkt); + break; + } + } + av_packet_unref(pkt); + } + } + av_packet_free(&pkt); + + if (last_pos < 0) + return ts; + *ppos = last_pos; + return ff_wrap_timestamp(st, last_ts); +} + int ff_seek_frame_binary(AVFormatContext *s, int stream_index, int64_t target_ts, int flags) { @@ -412,7 +467,8 @@ int64_t ff_gen_search(AVFormatContext *s, int stream_index, int64_t target_ts, if (ts_min == AV_NOPTS_VALUE) { pos_min = si->data_offset; - ts_min = read_timestamp(s, stream_index, &pos_min, INT64_MAX, read_timestamp_func); + ts_min = read_timestamp_keyframe(s, stream_index, &pos_min, INT64_MAX, flags, + read_timestamp_func); if (ts_min == AV_NOPTS_VALUE) return -1; } @@ -463,7 +519,8 @@ int64_t ff_gen_search(AVFormatContext *s, int stream_index, int64_t target_ts, start_pos = pos; // May pass pos_limit instead of -1. - ts = read_timestamp(s, stream_index, &pos, INT64_MAX, read_timestamp_func); + ts = read_timestamp_keyframe(s, stream_index, &pos, INT64_MAX, flags, + read_timestamp_func); if (pos == pos_max) no_change++; else diff --git a/tests/ref/seek/lavf-mpg b/tests/ref/seek/lavf-mpg index e804b84739..6c4897ef2c 100644 --- a/tests/ref/seek/lavf-mpg +++ b/tests/ref/seek/lavf-mpg @@ -4,7 +4,7 @@ ret: 0 st: 1 flags:1 dts: 0.529089 pts: 0.529089 pos: 2048 size: 208 ret: 0 st:-1 flags:1 ts: 1.894167 ret: 0 st: 1 flags:1 dts: 1.051544 pts: 1.051544 pos: 342028 size: 314 ret: 0 st: 0 flags:0 ts: 0.788333 -ret: 0 st: 0 flags:0 dts: 0.820000 pts: 0.860000 pos: 118784 size: 14717 +ret: 0 st: 0 flags:1 dts: 0.980000 pts: 1.020000 pos: 172032 size: 24964 ret: 0 st: 0 flags:1 ts:-0.317500 ret: 0 st: 1 flags:1 dts: 0.529089 pts: 0.529089 pos: 2048 size: 208 ret: 0 st: 1 flags:0 ts: 2.576667 @@ -40,7 +40,7 @@ ret: 0 st: 1 flags:1 dts: 0.529089 pts: 0.529089 pos: 2048 size: 208 ret: 0 st:-1 flags:1 ts: 1.989173 ret: 0 st: 1 flags:1 dts: 1.051544 pts: 1.051544 pos: 342028 size: 314 ret: 0 st: 0 flags:0 ts: 0.883344 -ret: 0 st: 0 flags:0 dts: 0.900000 pts: 0.940000 pos: 147456 size: 12755 +ret: 0 st: 0 flags:1 dts: 0.980000 pts: 1.020000 pos: 172032 size: 24964 ret: 0 st: 0 flags:1 ts:-0.222489 ret: 0 st: 1 flags:1 dts: 0.529089 pts: 0.529089 pos: 2048 size: 208 ret: 0 st: 1 flags:0 ts: 2.671678 diff --git a/tests/ref/seek/lavf-ts b/tests/ref/seek/lavf-ts index 3347b7ead5..0fddf3aac7 100644 --- a/tests/ref/seek/lavf-ts +++ b/tests/ref/seek/lavf-ts @@ -16,7 +16,7 @@ ret: 0 st: 0 flags:1 dts: 1.400000 pts: 1.440000 pos: 564 size: 24801 ret: 0 st:-1 flags:1 ts:-0.740831 ret: 0 st: 0 flags:1 dts: 1.400000 pts: 1.440000 pos: 564 size: 24801 ret: 0 st: 0 flags:0 ts: 2.153333 -ret: 0 st: 1 flags:1 dts: 1.794811 pts: 1.794811 pos: 308508 size: 209 +ret: 0 st: 1 flags:1 dts: 2.160522 pts: 2.160522 pos: 386716 size: 209 ret: 0 st: 0 flags:1 ts: 1.047500 ret: 0 st: 0 flags:1 dts: 1.400000 pts: 1.440000 pos: 564 size: 24801 ret: 0 st: 1 flags:0 ts:-0.058333 @@ -24,7 +24,7 @@ ret: 0 st: 1 flags:1 dts: 1.429089 pts: 1.429089 pos: 152844 size: 208 ret: 0 st: 1 flags:1 ts: 2.835833 ret: 0 st: 1 flags:1 dts: 2.160522 pts: 2.160522 pos: 386716 size: 209 ret: 0 st:-1 flags:0 ts: 1.730004 -ret: 0 st: 1 flags:1 dts: 1.429089 pts: 1.429089 pos: 152844 size: 208 +ret: 0 st: 0 flags:1 dts: 1.880000 pts: 1.920000 pos: 181420 size: 24786 ret: 0 st:-1 flags:1 ts: 0.624171 ret: 0 st: 0 flags:1 dts: 1.400000 pts: 1.440000 pos: 564 size: 24801 ret: 0 st: 0 flags:0 ts:-0.481667 @@ -38,7 +38,7 @@ ret: 0 st: 1 flags:1 dts: 1.429089 pts: 1.429089 pos: 152844 size: 208 ret: 0 st:-1 flags:0 ts:-0.904994 ret: 0 st: 0 flags:1 dts: 1.400000 pts: 1.440000 pos: 564 size: 24801 ret: 0 st:-1 flags:1 ts: 1.989173 -ret: 0 st: 0 flags:0 dts: 1.960000 pts: 2.000000 pos: 224848 size: 15019 +ret: 0 st: 0 flags:1 dts: 1.880000 pts: 1.920000 pos: 181420 size: 24786 ret: 0 st: 0 flags:0 ts: 0.883344 ret: 0 st: 0 flags:1 dts: 1.400000 pts: 1.440000 pos: 564 size: 24801 ret: 0 st: 0 flags:1 ts:-0.222489 -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
