PR #22813 opened by LTe
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22813
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/22813.patch
The threaded scheduler in ffmpeg_sched.c enforces cross-stream
backpressure by comparing each mux stream's last DTS against the
minimum across streams, choking sources that are more than
SCHEDULE_TOLERANCE ahead of the trailing one. This implicitly
assumes every stream feeds the muxer with timestamps on a common
timeline, which is not the case under one specific configuration.
When `-copyts` is combined with an output `-ss`, the two paths into
the muxer apply different ts_offset semantics:
- Transcoded streams: the filter chain subtracts ts_offset
(= mux->of.start_time, i.e. the output -ss value) before the
encoder produces packets, so send_to_mux() observes packets near
DTS 0.
- Stream-copied streams: of_streamcopy() runs at mux time, AFTER
send_to_mux() has already recorded last_dts, so the scheduler
observes packets with their original demuxer-side DTS (~120s for
-ss 120).
The scheduler then sees stream-copied audio at ~120s and transcoded
video at ~0s, concludes audio is far ahead, and chokes the audio
demuxer. Audio never reaches the HLS muxer in time for the first
segments to contain it.
Fix by tracking, per mux stream, the offset that the upstream
pipeline subtracted from the timestamps. The scheduler adds this back
when storing last_dts so that all streams are compared on the
common source timeline, regardless of pipeline rebasing.
For encoded streams the offset is the filter's ts_offset
(= of->start_time); for streamcopy streams it is zero (the default),
because of_streamcopy() runs after the scheduler observes the packet.
This preserves the scheduler's existing behavior for content where
streams legitimately start at different times (e.g. video from 0s and
audio whose first packet is at 120s). In that case both paths are
streamcopy-equivalent at scheduler-observation time, no rebase is
applied, the scheduler correctly sees the 120s gap and paces the
streams to converge before unblocking the audio demuxer.
Regression introduced by d119ae2fd82a (fftools/ffmpeg: convert to a
threaded architecture, July 2023), which made cross-stream DTS
comparison load-bearing for the first time.
Reproducer (fails before, passes after):
ffmpeg -ss 120 -i video.mkv -ss 120 -i audio.mka \
-ss 120 -map 0:v -map 1:a \
-c:v libx264 -c:a copy \
-copyts -avoid_negative_ts disabled \
-f hls -hls_time 2 out.m3u8
Also adds a FATE test exercising the regression scenario with
built-in encoders only (mpeg4 source, mpeg2video transcode,
mp2fixed audio, lavfi inputs).
Fixes https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/22765
From 612ab6cd483bf92778fa7ca94ef0063f868cdf01 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Piotr=20Nie=C5=82acny?= <[email protected]>
Date: Mon, 13 Apr 2026 18:50:08 +0200
Subject: [PATCH] fftools/ffmpeg_sched: track filter ts_offset for cross-stream
pacing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The threaded scheduler in ffmpeg_sched.c enforces cross-stream
backpressure by comparing each mux stream's last DTS against the
minimum across streams, choking sources that are more than
SCHEDULE_TOLERANCE ahead of the trailing one. This implicitly
assumes every stream feeds the muxer with timestamps on a common
timeline, which is not the case under one specific configuration.
When `-copyts` is combined with an output `-ss`, the two paths into
the muxer apply different ts_offset semantics:
- Transcoded streams: the filter chain subtracts ts_offset
(= mux->of.start_time, i.e. the output -ss value) before the
encoder produces packets, so send_to_mux() observes packets near
DTS 0.
- Stream-copied streams: of_streamcopy() runs at mux time, AFTER
send_to_mux() has already recorded last_dts, so the scheduler
observes packets with their original demuxer-side DTS (~120s for
-ss 120).
The scheduler then sees stream-copied audio at ~120s and transcoded
video at ~0s, concludes audio is far ahead, and chokes the audio
demuxer. Audio never reaches the HLS muxer in time for the first
segments to contain it.
Fix by tracking, per mux stream, the offset that the upstream
pipeline subtracted from the timestamps. The scheduler adds this back
when storing last_dts so that all streams are compared on the
common source timeline, regardless of pipeline rebasing.
For encoded streams the offset is the filter's ts_offset
(= of->start_time); for streamcopy streams it is zero (the default),
because of_streamcopy() runs after the scheduler observes the packet.
This preserves the scheduler's existing behavior for content where
streams legitimately start at different times (e.g. video from 0s and
audio whose first packet is at 120s). In that case both paths are
streamcopy-equivalent at scheduler-observation time, no rebase is
applied, the scheduler correctly sees the 120s gap and paces the
streams to converge before unblocking the audio demuxer.
Regression introduced by d119ae2fd82a (fftools/ffmpeg: convert to a
threaded architecture, July 2023), which made cross-stream DTS
comparison load-bearing for the first time.
Reproducer (fails before, passes after):
ffmpeg -ss 120 -i video.mkv -ss 120 -i audio.mka \
-ss 120 -map 0:v -map 1:a \
-c:v libx264 -c:a copy \
-copyts -avoid_negative_ts disabled \
-f hls -hls_time 2 out.m3u8
Also adds a FATE test exercising the regression scenario with
built-in encoders only (mpeg4 source, mpeg2video transcode,
mp2fixed audio, lavfi inputs).
Signed-off-by: Piotr Niełacny <[email protected]>
---
fftools/ffmpeg_mux_init.c | 8 +++++
fftools/ffmpeg_sched.c | 24 +++++++++++++--
fftools/ffmpeg_sched.h | 8 +++++
tests/fate/hlsenc.mak | 50 +++++++++++++++++++++++++++++++
tests/ref/fate/hls-sched-extaudio | 12 ++++++++
5 files changed, 100 insertions(+), 2 deletions(-)
create mode 100644 tests/ref/fate/hls-sched-extaudio
diff --git a/fftools/ffmpeg_mux_init.c b/fftools/ffmpeg_mux_init.c
index e0eecf78f2..7d23c13394 100644
--- a/fftools/ffmpeg_mux_init.c
+++ b/fftools/ffmpeg_mux_init.c
@@ -1548,6 +1548,14 @@ static int ost_add(Muxer *mux, const OptionsContext *o,
enum AVMediaType type,
keep_pix_fmt, autoscale, threads_manual, vs,
&src);
if (ret < 0)
goto fail;
+
+ // The filter chain subtracts ts_offset (= start_time) from
+ // encoded packets; tell the scheduler so cross-stream pacing
+ // can compare against streamcopy on a common timeline.
+ if (mux->of.start_time != AV_NOPTS_VALUE && ms->sch_idx >= 0)
+ sch_mux_stream_set_dts_pacing_offset(mux->sch, mux->sch_idx,
+ ms->sch_idx,
+ mux->of.start_time);
} else if (ost->ist) {
ret = ist_use(ost->ist, !!ost->enc, NULL, &src);
if (ret < 0) {
diff --git a/fftools/ffmpeg_sched.c b/fftools/ffmpeg_sched.c
index dc5800d094..718e7259c4 100644
--- a/fftools/ffmpeg_sched.c
+++ b/fftools/ffmpeg_sched.c
@@ -198,11 +198,16 @@ typedef struct SchMuxStream {
// an EOF was generated while flushing the pre-mux queue
int init_eof;
+ // offset (in AV_TIME_BASE_Q) subtracted by the upstream pipeline
+ // (e.g. filter ts_offset), added back to last_dts so that
+ // cross-stream pacing compares on a common source timeline
+ int64_t dts_pacing_offset;
+
////////////////////////////////////////////////////////////
// The following are protected by Scheduler.schedule_lock //
/* dts+duration of the last packet sent to this stream
- in AV_TIME_BASE_Q */
+ in AV_TIME_BASE_Q (with dts_pacing_offset applied) */
int64_t last_dts;
// this stream no longer accepts input
int source_finished;
@@ -1248,6 +1253,21 @@ void sch_mux_stream_buffering(Scheduler *sch, unsigned
mux_idx, unsigned stream_
ms->pre_mux_queue.data_threshold = data_threshold;
}
+void sch_mux_stream_set_dts_pacing_offset(Scheduler *sch, unsigned mux_idx,
+ unsigned stream_idx, int64_t offset)
+{
+ SchMux *mux;
+ SchMuxStream *ms;
+
+ av_assert0(mux_idx < sch->nb_mux);
+ mux = &sch->mux[mux_idx];
+
+ av_assert0(stream_idx < mux->nb_streams);
+ ms = &mux->streams[stream_idx];
+
+ ms->dts_pacing_offset = offset;
+}
+
int sch_mux_stream_ready(Scheduler *sch, unsigned mux_idx, unsigned stream_idx)
{
SchMux *mux;
@@ -2031,7 +2051,7 @@ update_schedule:
if (dts != AV_NOPTS_VALUE || !pkt) {
pthread_mutex_lock(&sch->schedule_lock);
- if (pkt) ms->last_dts = dts;
+ if (pkt) ms->last_dts = dts + ms->dts_pacing_offset;
else ms->source_finished = 1;
schedule_update_locked(sch);
diff --git a/fftools/ffmpeg_sched.h b/fftools/ffmpeg_sched.h
index 2cf3034437..64fb587d79 100644
--- a/fftools/ffmpeg_sched.h
+++ b/fftools/ffmpeg_sched.h
@@ -285,6 +285,14 @@ int sch_add_mux_stream(Scheduler *sch, unsigned mux_idx);
void sch_mux_stream_buffering(Scheduler *sch, unsigned mux_idx, unsigned
stream_idx,
size_t data_threshold, int max_packets);
+/**
+ * Set the timestamp offset (in AV_TIME_BASE_Q) that the upstream pipeline
+ * subtracted from this stream's packets. The scheduler adds it back when
+ * comparing DTS across streams for pacing. Defaults to 0 (streamcopy).
+ */
+void sch_mux_stream_set_dts_pacing_offset(Scheduler *sch, unsigned mux_idx,
+ unsigned stream_idx, int64_t offset);
+
/**
* Signal to the scheduler that the specified muxed stream is initialized and
* ready. Muxing is started once all the streams are ready.
diff --git a/tests/fate/hlsenc.mak b/tests/fate/hlsenc.mak
index b71fe219a6..4556f32c0c 100644
--- a/tests/fate/hlsenc.mak
+++ b/tests/fate/hlsenc.mak
@@ -224,6 +224,56 @@ fate-hls-start-number: tests/data/hls_start_number.m3u8
fate-hls-start-number: CMD = sed -n -e /^\#EXT-X-MEDIA-SEQUENCE:/p -e
/^[^\#]/p $(TARGET_PATH)/tests/data/hls_start_number.m3u8
fate-hls-start-number: CMP = diff
+# Regression test for the scheduler DTS-scale bug introduced by the threaded
+# fftools refactor (d119ae2fd82a). With input and output -ss combined with
+# -copyts on mixed transcode-video + stream-copy-audio inputs from separate
+# demuxers, the scheduler compared each stream's last DTS to decide pacing.
+# The transcoded video's filter chain subtracts ts_offset (= start_time),
+# while -copyts prevents of_streamcopy from doing the same to copied audio
+# (and even when it would, it only runs at mux time after the scheduler has
+# already recorded the unrebased DTS). The two streams end up on different
+# absolute timelines and the scheduler chokes the copied stream forever.
+tests/data/hls_sched_v.mkv: TAG = GEN
+tests/data/hls_sched_v.mkv: ffmpeg$(PROGSSUF)$(EXESUF) | tests/data
+ $(M)$(TARGET_EXEC) $(TARGET_PATH)/$< -nostdin \
+ -f lavfi -i "testsrc2=size=128x72:rate=24:d=180" \
+ -c:v mpeg4 -g 500 -flags +bitexact \
+ -an -y $(TARGET_PATH)/tests/data/hls_sched_v.mkv 2>/dev/null
+
+tests/data/hls_sched_a.mka: TAG = GEN
+tests/data/hls_sched_a.mka: ffmpeg$(PROGSSUF)$(EXESUF) | tests/data
+ $(M)$(TARGET_EXEC) $(TARGET_PATH)/$< -nostdin \
+ -f lavfi -i "aevalsrc=sin(2*PI*440*t):d=180:sample_rate=44100" \
+ -c:a mp2fixed -b:a 128k \
+ -vn -y $(TARGET_PATH)/tests/data/hls_sched_a.mka 2>/dev/null
+
+tests/data/hls_sched_extaudio.m3u8: TAG = GEN
+tests/data/hls_sched_extaudio.m3u8: ffmpeg$(PROGSSUF)$(EXESUF)
tests/data/hls_sched_v.mkv tests/data/hls_sched_a.mka | tests/data
+ $(M)$(TARGET_EXEC) $(TARGET_PATH)/$< -nostdin \
+ -ss 120 -noaccurate_seek -i $(TARGET_PATH)/tests/data/hls_sched_v.mkv \
+ -ss 120 -noaccurate_seek -i $(TARGET_PATH)/tests/data/hls_sched_a.mka \
+ -ss 120 -map 0:v -map 1:a -c:v mpeg2video -flags +bitexact \
+ -c:a copy -copyts -avoid_negative_ts disabled \
+ -f hls -hls_time 2 -hls_list_size 0 \
+ -hls_segment_filename
$(TARGET_PATH)/tests/data/hls_sched_extaudio_%d.ts \
+ $(TARGET_PATH)/tests/data/hls_sched_extaudio.m3u8 2>/dev/null
+
+FATE_HLSENC_LAVFI-$(call ALLYES, TESTSRC2_FILTER AEVALSRC_FILTER LAVFI_INDEV \
+ MPEG4_ENCODER MPEG2VIDEO_ENCODER MP2FIXED_ENCODER \
+ MATROSKA_MUXER MATROSKA_DEMUXER \
+ HLS_MUXER MPEGTS_MUXER FILE_PROTOCOL) += fate-hls-sched-extaudio
+fate-hls-sched-extaudio: tests/data/hls_sched_extaudio.m3u8
+# Probe the first HLS segment's audio stream parameters. Without the fix,
+# audio is starved during the segment's window and only sample_rate=0,
+# channels=0 are reported (the MPEG-TS PSI declares the codec but no audio
+# packets are present). With the fix, audio reaches the segment in time.
+fate-hls-sched-extaudio: CMD = run ffprobe$(PROGSSUF)$(EXESUF) -bitexact
-threads $(THREADS) \
+ -analyzeduration 10000000 -probesize 10000000 \
+ -show_entries stream=codec_type,codec_name,sample_rate,channels \
+ -print_format default=nw=1 \
+ $(TARGET_PATH)/tests/data/hls_sched_extaudio_0.ts
+fate-hls-sched-extaudio: CMP = diff
+
FATE_HLSENC_LAVFI-yes := $(if $(call FRAMECRC), $(FATE_HLSENC_LAVFI-yes))
FATE_FFMPEG += $(FATE_HLSENC_LAVFI-yes)
diff --git a/tests/ref/fate/hls-sched-extaudio
b/tests/ref/fate/hls-sched-extaudio
new file mode 100644
index 0000000000..88430ef3c7
--- /dev/null
+++ b/tests/ref/fate/hls-sched-extaudio
@@ -0,0 +1,12 @@
+codec_name=mpeg2video
+codec_type=video
+codec_name=mp2
+codec_type=audio
+sample_rate=44100
+channels=1
+codec_name=mpeg2video
+codec_type=video
+codec_name=mp2
+codec_type=audio
+sample_rate=44100
+channels=1
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
