Soft Works (12020-02-14): > I am looking for some guidance regarding future plans about processing > subtitle streams in filter graphs. > > Please correct me where I'm wrong - this is the situation as I've > understood it so far: > > - Currently, ffmpeg filter graphs do not support processing subtitle > streams > - This is why filters like 'subtitles' and 'ass' need to open, read > and parse the media file a second time in parallel instead of just > taking the subtitle events from ffmpeg'demuxing > - For graphical subtitles, there exists the so-called 'sub2video' > workaround which is injecting the the graphical subtitle overlay > images into the video filtergraph by declaring them as video > - The 'sub2video' was meant to exist until filtering would support > subtitle streams > - A while ago, Clement Boesch submitted a patch for adding subtitle > streams to filtergraph processing > (https://patchwork.ffmpeg.org/project/ffmpeg/patch/20161102220934.26010-...@pkh.me/) > - I read through all the discussion about it, but nothing happened > afterwards and I couldn't find any indication about why it didn't get > merged
This looks accurate. > I'm asking because I'm intending to implement a subtitle filter that > operates on in-stream data rather than a separate input and that will > render text on transparent frames for later overlay. > > The possible options that I have identified so far for creating that > kind of "subtitle rendering filter" would be: > > - Create a video source filter and implement some hack to get the > subtitle data from the decoder to that filter - or... > - Jump on the sub2video implementation and extend it to render overlay > images in case of text subtitles - or... > - check out the situation with regards to adding subtitle filter > support in ffmpeg and ask about plans for this > > That's where I stand right now. Does it even make sense, the way I > summarized it?` I don't speak for the project as a whole, but I am quite confident that "some hack" would be accepted if and only if it is generic enough to be useful for many people, not just your use case. And tweaks to sub2video are "some hack" too. If you want to implement real support for subtitles, that would be greatly appreciated, but I have to warn you it is a very difficult and intensive task. Otherwise it would have already been done. I can summarize where the difficulty resides: - lavfi works with AVFrame, therefore subtitles need to be encoded into AVFrame. This is the work of Clément that you found; he only started on the rest. - lavfi is not ready to have a third media type: there are parts that strongly assume audio or video, and parts that merge the audio and video case but cannot handle a different type. - The utility filters that only work on metadata, like setpts, need to be ported ported to the new media type. We don't want too much code duplication, a more elegant approach needs to be found. Possibly make the media type part of a first round of format negotiation. - We need to decide the format negotiation for filters. Do we automatically insert text→bitmap renderer or does it need to be explicit? - Subtitles streams are sparse, lavfi is not designed for that, and it is a problem when subtitles are interleaved with audio or video. The best way to solve it is dummy heartbeat frames. They can be automatically generated by the framework with an API to connect two buffersrc: when a video frame is added on one, generate a heartbeat subtitle frame on the other if necessary. Feel free to ask details on any of these points: I do not have the courage to start working on it for now, but I have thought about it. Regards, -- Nicolas George
signature.asc
Description: PGP signature
_______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".