PR #24732 opened by Jun Zhao (mypopydev) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24732 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24732.patch
convert ASS karaoke tags into WebVTT cue timestamps ASS/SSA subtitle files widely use karaoke tags (\k, \K, \kf, \ko) to time syllable boundaries in centiseconds. When converting ASS to WebVTT, these tags were previously dropped as unknown override codes, resulting in plain text cues without syllable timing (Issue #23879). This patch series converts ASS karaoke tags into WebVTT inline cue timestamps (<MM:SS.mmm> / <HH:MM:SS.mmm>): 1. avcodec/ass_split: Add a dedicated karaoke callback to ASSCodesCallbacks and parse \k, \K, \kf, \ko durations (in centiseconds). 2. avcodec/webvttenc: Synthesize WebVTT inline cue timestamps according to the W3C WebVTT specification and webvttdec compatibility. Inline timestamps are kept strictly monotonic within cue boundaries, and accidental zero/boundary timestamps are suppressed. 3. fftools/ffmpeg_enc: Ensure subtitle packet pts respects output start_time (-ss offset), fixing a historic cue header drift bug when output seeking is used. 4. tests/fate/subtitles: Add a regression test transcoding fate-suite/sub/a9-misc.ssa to WebVTT. Fixes: ticket #23879 Signed-off-by: Jun Zhao <[email protected]> From a6c44c3bc21ade23d43f514e2faf5d26238bf4c6 Mon Sep 17 00:00:00 2001 From: Jun Zhao <[email protected]> Date: Sat, 26 Sep 2026 22:45:23 +0800 Subject: [PATCH 1/4] avcodec/ass_split: add karaoke callback to ASSCodesCallbacks Parse karaoke override codes (\k, \K, \kf, \ko) during ASS dialogue splitting and dispatch them via a dedicated callback in ASSCodesCallbacks. The duration is passed in centiseconds (1 cs = 10 ms), matching ASS specification conventions. Other subtitle encoders (SRT, TTML, mov_text) leave the callback NULL and continue to ignore karaoke tags without functional regressions. Signed-off-by: Jun Zhao <[email protected]> --- libavcodec/ass_split.c | 7 ++++++- libavcodec/ass_split.h | 7 +++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/libavcodec/ass_split.c b/libavcodec/ass_split.c index 3f8f5e7cae..5eb96072eb 100644 --- a/libavcodec/ass_split.c +++ b/libavcodec/ass_split.c @@ -502,7 +502,7 @@ int ff_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, buf++; while (*buf == '\\') { char style[2], c[2], sep[2], c_num[2] = "0", tmp[128] = {0}; - unsigned int color = 0xFFFFFFFF; + unsigned int color = 0xFFFFFFFF, dur = 0; int len, size = -1, an = -1, alpha = -1; int x1, y1, x2, y2, t1 = -1, t2 = -1; if (sscanf(buf, "\\%1[bisu]%1[01\\}]%n", style, c, &len) > 1) { @@ -552,6 +552,11 @@ int ff_ass_split_override_codes(const ASSCodesCallbacks *callbacks, void *priv, } else if (sscanf(buf, "\\org(%d,%d)%1[\\}]%n", &x1, &y1, sep, &len) > 2) { if (callbacks->origin) callbacks->origin(priv, x1, y1); + } else if (sscanf(buf, "\\kf%u%1[\\}]%n", &dur, sep, &len) > 1 || + sscanf(buf, "\\ko%u%1[\\}]%n", &dur, sep, &len) > 1 || + sscanf(buf, "\\%1[kK]%u%1[\\}]%n", c, &dur, sep, &len) > 2) { + if (callbacks->karaoke) + callbacks->karaoke(priv, dur); } else { len = strcspn(buf+1, "\\}") + 2; /* skip unknown code */ } diff --git a/libavcodec/ass_split.h b/libavcodec/ass_split.h index 41191c0f44..228e16709a 100644 --- a/libavcodec/ass_split.h +++ b/libavcodec/ass_split.h @@ -159,6 +159,13 @@ typedef struct { void (*origin)(void *priv, int x, int y); /** @} */ + /** + * @defgroup ass_karaoke ASS karaoke + * @{ + */ + void (*karaoke)(void *priv, unsigned int duration); + /** @} */ + /** * @defgroup ass_end end of Dialogue Event * @{ -- 2.52.0 From c18c44a214b41afbc39439dce9e56f835b4e0925 Mon Sep 17 00:00:00 2001 From: Jun Zhao <[email protected]> Date: Sun, 27 Sep 2026 06:33:15 +0800 Subject: [PATCH 2/4] avcodec/webvttenc: convert ASS karaoke tags into WebVTT cue timestamps Synthesize WebVTT inline cue timestamps (<MM:SS.mmm> or <HH:MM:SS.mmm>) from ASS karaoke overrides (\k, \K, \kf, \ko) using the subtitle PTS and duration baseline. Per W3C WebVTT specification and webvttdec compatibility: - Timestamps represent absolute positions on the media timeline. - Inline timestamps must fall strictly within the cue boundary (cue_start < timestamp < cue_end) and remain strictly monotonic. - Consecutive syllable durations are accumulated across style and line breaks, and flushed after preceding syllable text is emitted. - Ending timestamps matching cue_end are omitted; webvttdec automatically telescopes the final duration up to cue_end on decoding. - If PTS is unavailable (AV_NOPTS_VALUE), timestamps are suppressed and plain text is emitted as fallback. Fixes: ticket #23879 Signed-off-by: Jun Zhao <[email protected]> --- libavcodec/webvttenc.c | 73 +++++++++++++++++++++++++++++++++++++++++- 1 file changed, 72 insertions(+), 1 deletion(-) diff --git a/libavcodec/webvttenc.c b/libavcodec/webvttenc.c index d5d3ddd17c..56cc6d99dc 100644 --- a/libavcodec/webvttenc.c +++ b/libavcodec/webvttenc.c @@ -34,6 +34,12 @@ typedef struct { AVBPrint buffer; unsigned timestamp_end; int count; + int64_t cue_start_ms; + int64_t cue_end_ms; + int64_t current_time_ms; + int64_t last_timestamp_ms; + int64_t pending_duration_ms; + int text_emitted_since_karaoke; char stack[WEBVTT_STACK_SIZE]; int stack_ptr; } WebVTTContext; @@ -106,15 +112,66 @@ static void webvtt_style_apply(WebVTTContext *s, const char *style) } } +static void webvtt_print_timestamp(WebVTTContext *s, int64_t millisec) +{ + int64_t sec = millisec / 1000; + millisec %= 1000; + int64_t min = sec / 60; + sec %= 60; + int64_t hour = min / 60; + min %= 60; + if (hour > 0) { + webvtt_print(s, "<%02"PRId64":%02"PRId64":%02"PRId64".%03"PRId64">", hour, min, sec, millisec); + } else { + webvtt_print(s, "<%02"PRId64":%02"PRId64".%03"PRId64">", min, sec, millisec); + } +} + +static void webvtt_flush_karaoke_timestamp(WebVTTContext *s) +{ + if (!s->text_emitted_since_karaoke || s->pending_duration_ms <= 0) + return; + + int64_t next_ts = s->current_time_ms <= INT64_MAX - s->pending_duration_ms ? + s->current_time_ms + s->pending_duration_ms : INT64_MAX; + if (next_ts > s->cue_start_ms && next_ts < s->cue_end_ms && + next_ts > s->last_timestamp_ms) { + webvtt_print_timestamp(s, next_ts); + s->last_timestamp_ms = next_ts; + } + s->current_time_ms = next_ts; + s->pending_duration_ms = 0; + s->text_emitted_since_karaoke = 0; +} + static void webvtt_text_cb(void *priv, const char *text, int len) { WebVTTContext *s = priv; av_bprint_append_data(&s->buffer, text, len); + if (len > 0) + s->text_emitted_since_karaoke = 1; } static void webvtt_new_line_cb(void *priv, int forced) { - webvtt_print(priv, "\n"); + WebVTTContext *s = priv; + webvtt_flush_karaoke_timestamp(s); + webvtt_print(s, "\n"); + s->text_emitted_since_karaoke = 1; +} + +static void webvtt_karaoke_cb(void *priv, unsigned int duration) +{ + WebVTTContext *s = priv; + int64_t dur_ms = (int64_t)duration * 10; + + webvtt_flush_karaoke_timestamp(s); + if (dur_ms <= INT64_MAX - s->pending_duration_ms) { + s->pending_duration_ms += dur_ms; + } else { + s->pending_duration_ms = INT64_MAX >> 1; + } + s->text_emitted_since_karaoke = 0; } static void webvtt_style_cb(void *priv, char style, int close) @@ -135,6 +192,8 @@ static void webvtt_cancel_overrides_cb(void *priv, const char *style) static void webvtt_end_cb(void *priv) { + WebVTTContext *s = priv; + webvtt_flush_karaoke_timestamp(s); webvtt_stack_push_pop(priv, 0, 1); } @@ -148,6 +207,7 @@ static const ASSCodesCallbacks webvtt_callbacks = { .alignment = NULL, .cancel_overrides = webvtt_cancel_overrides_cb, .move = NULL, + .karaoke = webvtt_karaoke_cb, .end = webvtt_end_cb, }; @@ -160,6 +220,17 @@ static int webvtt_encode_frame(AVCodecContext *avctx, av_bprint_init_for_buffer(&s->buffer, buf, bufsize); + if (sub->pts != AV_NOPTS_VALUE) { + s->cue_start_ms = av_rescale_q(sub->pts, AV_TIME_BASE_Q, (AVRational){ 1, 1000 }); + s->cue_end_ms = s->cue_start_ms + sub->end_display_time; + } else { + s->cue_start_ms = s->cue_end_ms = 0; + } + s->current_time_ms = s->cue_start_ms; + s->last_timestamp_ms = -1; + s->pending_duration_ms = 0; + s->text_emitted_since_karaoke = 0; + for (i=0; i<sub->num_rects; i++) { const char *ass = sub->rects[i]->ass; -- 2.52.0 From b5b29cb0a3002a285b2586db07e6f4ae51933d69 Mon Sep 17 00:00:00 2001 From: Jun Zhao <[email protected]> Date: Sun, 27 Sep 2026 06:33:15 +0800 Subject: [PATCH 3/4] fftools/ffmpeg_enc: ensure subtitle packet pts respects output start_time In do_subtitle_out(), local_sub.pts already subtracts of->start_time when output-side -ss is requested. However, pkt->pts was assigned from sub->pts rather than the adjusted pts variable, causing cue headers to drift across container muxers whenever output seeking was used. Assign pkt->pts from pts so that encoded subtitle packets stay aligned with audio/video streams on the shifted timeline. Signed-off-by: Jun Zhao <[email protected]> --- fftools/ffmpeg_enc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fftools/ffmpeg_enc.c b/fftools/ffmpeg_enc.c index 4745246410..cf145f13e0 100644 --- a/fftools/ffmpeg_enc.c +++ b/fftools/ffmpeg_enc.c @@ -525,7 +525,7 @@ static int do_subtitle_out(OutputFile *of, OutputStream *ost, const AVSubtitle * av_shrink_packet(pkt, subtitle_out_size); pkt->time_base = AV_TIME_BASE_Q; - pkt->pts = sub->pts; + pkt->pts = pts; pkt->duration = av_rescale_q(sub->end_display_time, (AVRational){ 1, 1000 }, pkt->time_base); if (enc->codec_id == AV_CODEC_ID_DVB_SUBTITLE) { /* XXX: the pts correction is handled here. Maybe handling -- 2.52.0 From d562d0b6d6069851f205772d9ba0aee245a09031 Mon Sep 17 00:00:00 2001 From: Jun Zhao <[email protected]> Date: Sun, 27 Sep 2026 06:33:15 +0800 Subject: [PATCH 4/4] tests/fate/subtitles: add fate-sub-karaoke-webvttenc test Add a regression test transcoding fate-suite/sub/a9-misc.ssa to WebVTT, covering both opening song (plain \k) and ending song (\k and \kf mixed) with styles, positioning, and punctuation boundary handling. Signed-off-by: Jun Zhao <[email protected]> --- tests/fate/subtitles.mak | 3 + tests/ref/fate/sub-karaoke-webvttenc | 109 +++++++++++++++++++++++++++ 2 files changed, 112 insertions(+) create mode 100644 tests/ref/fate/sub-karaoke-webvttenc diff --git a/tests/fate/subtitles.mak b/tests/fate/subtitles.mak index c8f94c2b5e..9d1aac8aad 100644 --- a/tests/fate/subtitles.mak +++ b/tests/fate/subtitles.mak @@ -112,6 +112,9 @@ fate-sub-webvtt2: CMD = fmtstdout ass -i $(TARGET_SAMPLES)/sub/WebVTT_extended_t FATE_SUBTITLES-$(call ALLYES, SRT_DEMUXER SUBRIP_DECODER WEBVTT_ENCODER WEBVTT_MUXER) += fate-sub-webvttenc fate-sub-webvttenc: CMD = fmtstdout webvtt -i $(TARGET_SAMPLES)/sub/SubRip_capability_tester.srt +FATE_SUBTITLES-$(call ALLYES, ASS_DEMUXER ASS_DECODER WEBVTT_ENCODER WEBVTT_MUXER) += fate-sub-karaoke-webvttenc +fate-sub-karaoke-webvttenc: CMD = fmtstdout webvtt -i $(TARGET_SAMPLES)/sub/a9-misc.ssa + FATE_SUBTITLES-$(call ALLYES, SRT_DEMUXER SUBRIP_DECODER TEXT_ENCODER SRT_MUXER) += fate-sub-textenc fate-sub-textenc: CMD = fmtstdout srt -i $(TARGET_SAMPLES)/sub/SubRip_capability_tester.srt -c:s text diff --git a/tests/ref/fate/sub-karaoke-webvttenc b/tests/ref/fate/sub-karaoke-webvttenc new file mode 100644 index 0000000000..9d1816c7f9 --- /dev/null +++ b/tests/ref/fate/sub-karaoke-webvttenc @@ -0,0 +1,109 @@ +WEBVTT + +00:02.420 --> 00:03.000 +All Japan Boys Soccer Tournament Opens! + +00:15.150 --> 00:16.300 +<b>Otani-san...</b> + +00:16.930 --> 00:18.170 +<b>Otani-san...</b> + +00:18.000 --> 00:21.180 +<b>Sixth Grade Camellia Class</b> + +00:18.580 --> 00:19.800 +<b>Otani-san...</b> + +00:20.450 --> 00:21.640 +<b>Otani-san...</b> + +00:21.380 --> 00:25.880 +<b>Alien Counter-Measure Officer</b> + +00:22.550 --> 00:23.910 +<b>Otani-san...</b> + +00:24.350 --> 00:25.460 +<b>Furukawa-san...</b> + +00:26.050 --> 00:27.230 +<b>Otani Furukawa Tatsuta</b> + +00:59.600 --> 01:00.600 +<b>Tak tady jsi...</b> + +01:00.810 --> 01:01.390 +<b>Ahoj.</b> + +01:01.520 --> 01:03.980 +<b>Pojď se mnou.</b> + +01:04.660 --> 01:05.870 +<b><i>{2c&H5D8FC3&\c&H75BADB&\2c&H5F7CC3&}Fu<01:04.810>tsu<01:05.030>u <01:05.140>ja <01:05.390>na<01:05.520>i</i></b> + +01:05.930 --> 01:07.430 +<b><i>{2c&H5D8FC3&\c&H75BADB&\2c&H5F7CC3&}A<01:06.230>sa <01:06.530>ga <01:06.700>ki<01:06.930>te</i></b> + +01:07.600 --> 01:10.600 +<b><i>{2c&H5D8FC3&\c&H75BADB&\2c&H5F7CC3&}Na<01:07.870>i<01:07.950>fu <01:08.180>no <01:08.370>sa<01:08.550>sa<01:08.870>t<01:08.960>ta <01:09.240>ha<01:09.420>a<01:09.510>to <01:09.790>ga <01:09.970>u<01:10.110>zu<01:10.380>ku</i></b> + +01:10.710 --> 01:11.990 +<b><i>{2c&H5D8FC3&\c&H75BADB&\2c&H5F7CC3&}Ne<01:10.930>bo<01:11.140>u <01:11.220>shi<01:11.510>ta<01:11.740>i</i></b> + +01:19.710 --> 01:30.260 +When you hear the bell of Chirin,... + +01:30.260 --> 01:40.460 +YASASHII MATSUGE WO HOHOEMI WO + +01:30.260 --> 01:40.460 +...recall those soft lashes and that gentle smile. + +01:40.460 --> 01:50.720 +CHIRIN NO SUZU DE OMOIDASU + +04:45.670 --> 04:48.370 +{My Eight-Six turned into an Eight-Five ? !}¡¿Mi Hachi-Roku se ha +convertido en un Hachi-Go?! + +04:48.970 --> 04:51.700 +Tomé el auto prestado, +te dejé un sustituto. + +05:01.520 --> 05:04.120 +{Takumi,Your dad must be really surprised by now. . .}Ahora mismo tu padre +debe estar en shock. + +25:19.730 --> 25:23.580 +<b>Elementary School #9</b> + +25:19.730 --> 25:23.580 +<b>Alien Counter-Measure Officers</b> + +25:19.730 --> 25:23.580 +<b>The End</b> + +25:35.060 --> 25:36.670 +<b><i>{0}To<25:35.310>ki-<25:35.740>do<25:36.060>ki</i></b> + +25:35.060 --> 25:36.670 +<b><i>Sometimes</i></b> + +25:36.680 --> 25:40.690 +<b><i>{0}Ko<25:37.140>no <25:37.410>ka<25:37.870>ra<25:38.230>da <25:38.700>ga</i></b> + +25:36.680 --> 25:40.690 +<b><i>Whether this body</i></b> + +25:45.040 --> 25:47.970 +<b><i>Yo<25:45.390>ro<25:45.670>ko<25:46.080>bi <25:46.430>na <25:46.710>no <25:46.970>ka</i></b> + +25:45.040 --> 25:47.970 +<b><i>Is happiness</i></b> + +25:47.980 --> 25:50.220 +<b><i>Ka<25:48.120>na<25:48.320>shi<25:48.580>mi <25:48.730>na <25:49.050>no <25:49.700>ka</i></b> + +25:47.980 --> 25:50.220 +<b><i>Or sadness</i></b> -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
