> -----Original Message----- > From: ffmpeg-devel <ffmpeg-devel-boun...@ffmpeg.org> On Behalf Of > Nicolas George > Sent: Monday, February 17, 2020 8:37 PM > To: FFmpeg development discussions and patches <ffmpeg- > de...@ffmpeg.org> > Subject: Re: [FFmpeg-devel] Status and Plans for Subtitle Filters > > Soft Works (12020-02-14): > > I am looking for some guidance regarding future plans about processing > > subtitle streams in filter graphs. > > > > Please correct me where I'm wrong - this is the situation as I've > > understood it so far: > > > > - Currently, ffmpeg filter graphs do not support processing subtitle > > streams > > - This is why filters like 'subtitles' and 'ass' need to open, read > > and parse the media file a second time in parallel instead of just > > taking the subtitle events from ffmpeg'demuxing > > - For graphical subtitles, there exists the so-called 'sub2video' > > workaround which is injecting the the graphical subtitle overlay > > images into the video filtergraph by declaring them as video > > - The 'sub2video' was meant to exist until filtering would support > > subtitle streams > > - A while ago, Clement Boesch submitted a patch for adding subtitle > > streams to filtergraph processing > > > (https://patchwork.ffmpeg.org/project/ffmpeg/patch/20161102220934.2601 > > 0-...@pkh.me/) > > - I read through all the discussion about it, but nothing happened > > afterwards and I couldn't find any indication about why it didn't get > > merged > > This looks accurate. > > > I'm asking because I'm intending to implement a subtitle filter that > > operates on in-stream data rather than a separate input and that will > > render text on transparent frames for later overlay. > > > > The possible options that I have identified so far for creating that > > kind of "subtitle rendering filter" would be: > > > > - Create a video source filter and implement some hack to get the > > subtitle data from the decoder to that filter - or... > > - Jump on the sub2video implementation and extend it to render overlay > > images in case of text subtitles - or... > > - check out the situation with regards to adding subtitle filter > > support in ffmpeg and ask about plans for this > > > > That's where I stand right now. Does it even make sense, the way I > > summarized it?` > > I don't speak for the project as a whole, but I am quite confident that "some > hack" would be accepted if and only if it is generic enough to be useful for > many people, not just your use case. And tweaks to sub2video are "some > hack" too. > > If you want to implement real support for subtitles, that would be greatly > appreciated, but I have to warn you it is a very difficult and intensive task. > Otherwise it would have already been done. I can summarize where the > difficulty resides: > > - lavfi works with AVFrame, therefore subtitles need to be encoded into > AVFrame. This is the work of Clément that you found; he only started > on the rest. > > - lavfi is not ready to have a third media type: there are parts that > strongly assume audio or video, and parts that merge the audio and > video case but cannot handle a different type. > > - The utility filters that only work on metadata, like setpts, need to > be ported ported to the new media type. We don't want too much code > duplication, a more elegant approach needs to be found. Possibly make > the media type part of a first round of format negotiation. > > - We need to decide the format negotiation for filters. Do we > automatically insert text→bitmap renderer or does it need to be > explicit? > > - Subtitles streams are sparse, lavfi is not designed for that, and it > is a problem when subtitles are interleaved with audio or video. The > best way to solve it is dummy heartbeat frames. They can be > automatically generated by the framework with an API to connect two > buffersrc: when a video frame is added on one, generate a heartbeat > subtitle frame on the other if necessary. > > Feel free to ask details on any of these points: I do not have the courage to > start working on it for now, but I have thought about it.
Hi Nicholas, Thanks a lot for your comments. To be honest, I pretty much feel the same about it. It was painful enough to read through the discussion about Clement's patch, and I'm neither ready for this nor do I have the time. My idea would be rather to take at look at the latest work from Clement and see whether it is in a state that is close to what I need - from your lines it rather seems that it is not. I guess my best bet will be to jump on the sub2video mechanism and extend it to work with text subtitles. Each time the text changes, I create a new transparent image using the code from the subtitles filter. Then I let sub2video process that image in the same way it does for graphical subs images. Does that sound reasonable to you? PS: Obviously, the most appealing part of this is that it would involve a rather limited amount of work.. _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".