Le primidi 21 brumaire, an CCXXV, wm4 a écrit : > OK, let's think about alternative approaches. How about not trying to > let libavfilter do synchronizing subtitles and other streams in all > cases? Why not just send nothing to the subtitle buffer sources if the > subtitles are sparse and there is no new packet? If it's sparse, make > it sparse.
Sometimes a filter needs sync. Therefore the framework must be able to provide it. Your suggestion does not allow that. > I assume the whole point of this exercise is to prevent excessive > buffering (e.g. not trying to read the next subtitle packet, which > might read most of the file, and necessitate buffering it in memory). > E.g. if you overlay video on subtitles, you'd normally require new > frames for both subtitles and video. If you'd treat subtitles like > video and audio, you'd have to try to read the next subtitle packet to > know, well, whether there's a new subtitle or whether to use the > previous one. You understand the issue. > If I understood this correctly, you want to send empty frames every > 100ms if the duration of a subtitle is unknown. Why is it 100ms? Does > it make a difference if it's 100ms or a few seconds (or until EOF) > until the subtitle is naturally terminated? Why not 1ms? This seems > like a very "approximate" solution - sure, it works, but it's akin to > using polling and sleep calls in I/O or multithreadeded code. That was just an example to help readers understand. > Maybe a heartbeat on every video frame? That is exactly the idea, you did not read the mail carefully enough before judging it "bad idea". > What if there's no video > stream? Then there is probably no need to sync. > This related idea of slaving sbuffersrc to a master vbuffersrc. This > approach works on video output, I have no idea why you start talking about output. > while sparseness is really a problem at > the source (i.e. demuxer). It's questionable how this would work with > subtitles demuxed from separate files When the subtitles are demuxed from a file separate from the stream they need to sync, the next frame is available on demand, and therefore the heartbeat frames are not needed, nor generated. > (which also might have A/V > streams). If that happens, it seems like a very strange and pathological use case. FFmpeg can not perform miracles. > It also works on the video output, while the issue of > subtitles with unknown duration is mostly a demuxing issue. What Again, I do not know why you speak about output. > happens if there's a video->sub filter, how would it send heartbeats? I do not know, please tell me what it does exactly and I will be able to answer you. > Would it require a new libavfilter graph syntax for filters that > generate subtitles within the graph, and would it require users to > explicitly specify a "companion" video source? Probably not. > The whole problem is that it's hard to do determine whether a new > subtitle frame should be available at a certain point inside > libavfilter, which in turn is hard because with how generic libavfilter > wants to be. Exactly. > It seems to me that not libavfilter should handle this, but the one > which feeds libavfilter. Its collaboration is mandatory. But the bulk of the work should be done by the library, not duplicated in all applications. > If it feeds a new video frame to it, but no > subtitle frame, it means there is no new subtitle yet due to > sparseness. Libavfilter can not know when there is "no subtitle frame", only when there is one. Hence the need to tell it explicitly: > There is actually no need for weird heartbeat frames. ... with a heartbeat frame. > The > libavfilter API user is in the position to know whether the subtitle > demuxer/decoder can produce a new packet/frame. It would be crazy if ^^^^^ Disparaging judgement without grounds. > the API user had to send heartbeat frames in these situations, and > had to care about how many heartbeats are sent when. Good thing it does not, then. > In complex cases (where audio/video/subs are connected in a > non-trivial way, possibly converting to each other at certain points), > the user would have to be careful which buffersinks to read in order > not to trigger excessive readahead. Also the user would possibly have to > disable "automagic" synchronization mechanisms in other parts of > libavfilter. > > Even then, you would need to "filter" sparse frames (update their > timestamps, produce new ones, etc). This sounds very complex. Yes, it is complex. The problem is complex, the solution has little chance of being simple. > What about filtering subtitles alone? This should be possible? Yes. > Why would libavfilter in general be responsible to sync subtitles and > video anyway? It should do that only on filters which have both > subtitle and video inputs or so. I do not understand why you are trying to make a distinction between lavfi and individual filters. The sparseness issue is not about where the code reside but about what information is available or not. > Why does this need decoder enhancements anyway? How about it just uses > the API in its current extent, which applications have handled for > years? Again, special snowflake libavfilter/ffmpeg.c. Mu. > Btw. video frames can also be sparse (think of mp4s that contain slide > shows). Are we going to get video heartbeat frames? How are all the > filters going to handle it? Currently they do not, but this is a separate issue. > Even for not-sparse video, there seem to be cases (possibly fixed now) > where libavfilter just excessively buffers when using ffmpeg.c. I'm > still fighting such cases with my own libavfilter API use. > (Interestingly, this often involves sparse video.) The case with not-sparse video involve sparse video? > (Oh, and I don't claim to have understood the problem in its whole > extent. But I do have a lot of experience with subtitles and > unfortunately also with using libavfilter in a "generic" way.) > > > And I would really appreciate if in the future you refrained from that > > kind of useless empty remark. You can raise practical concerns, ask for > > explanations or rationales, of course. But a purely negative reply that > > took you all of three minutes in answer to the result of years of design > > is just incredibly rude. > > I find your conduct incredibly rude as well. It's not nice to take > every reply as an offense, instead of e.g. starting a discussion. > It's also not nice to call my remarks "useless". > > No, pointing out that a solution is sub-optimal is not rudeness. Indeed. Next time, do that. Give arguments, ask question. Refrain from a disparaging blanked judgement. > Why are you asking me for a better solution? Because you are the one who claimed the solution I propose is bad. Unless the w in your nickname actually means Winston, claiming that a solution is bad when the is no better one is just a waste of time. > You're the one who wants > subtitles in libavfilter, not me. You could have noticed that I am far from being the only one. > Thus it's your responsibility to come > up with a good design. I did. > If there's no good design, then it's a sure sign > that it's not a good idea to have subtitles in libavfilter. Indeed, Or it is a sure sign that your aesthetics considerations are skewed. > subtitles are incredibly complex, and there are many cases that > somehow need to be handled in a generic way, and it's not necessarily a > good idea to add this complexity to libavfilter, which designed to > handle audio and video data (and even after years of work, isn't that > good at handling audio and video at the same time). It's like trying to > push a square peg through a round hole. I haven't come to the > conclusion yet that this is the case, so hold that thought. > > Please try not to reply to every paragraph separately or so. This makes > it hard to follow discussions. In fact, I won't take part in a > discussion of that kind. It wastes so much time because you can get lost > in meaningless details. This is exactly what I did because this is exactly what needed to be done for that kind of discussion. -- Nicolas George
signature.asc
Description: Digital signature
_______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel