[Libav-user] Force low-delay handling? question + feature suggestion
I have a camera supplying an h264 stream, whose SPS/PPS claims that it needs 4 reference frames, causing decoding to lag the input by 4 frames. At 5-fps, this is almost 1 second and is very noticeable (and undesired for this app, as in many cases the security person watching the stream can also see the events happening out the window, and a 1 second delay is confusing and seems "broken"). However, when I look at camera specs, and at the streams themselves, they only ever contain I pictures and P pictures (never B pictures). To my (limited) understanding of the h264 protocol, that suggests that it is possible to fully decode and display every image as it arrives, as they will never be out of order. Alex Cohn suggested in to modify ff_h264_get_profile(). I've done this as a test, and it seems to work for the streams that I have; I've also patched the SPS/PPS manually and it also solves the problem. But I would rather not have to do either of these patches (the first is ugly and requires me to rebuild ffmpeg myself all the time; the second is just plain ugly and error prone). Questions: 1. Does the (known from specs) fact that the input stream will only ever contains I and P really guarantee that it is safe to decode every picture as it arrives? Or have I just been lucky in my test streams, and reordering might still be needed at some point? 2. If (1) is correct, would it work to patch ff_h264_decode_init() and decode_postinit() to also check the AV_DISCARD mode, and if we are discarding B and/or nonref frames, would set avctx->has_b_frames=0 and low_delay=1 ? Something like that would make it possible to convert _every_ stream to a low-delay stream by dropping the "non-low-delay" frames. For me, it would solve the problem (there are no B frames, so I wouldn't even lose anything by discarding B frames), but it would also be useful for e.g. seek functionality in a media player - if fast-forwarding by showing only I-frames, you would not need to read and discard 3 more frames to show an I frame you have just read. Thanks in advance for your time and thoughts, Camera Man ___ Libav-user mailing list Libav-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/libav-user
Re: [Libav-user] Force low-delay handling? question + feature suggestion
Camera Man writes: > would it work to patch ff_h264_decode_init() and decode_postinit() to also > check the AV_DISCARD mode, and if we are discarding B and/or nonref frames, > would set avctx->has_b_frames=0 and low_delay=1 ? I don't know but I suggest that you write such a patch (optionally test it on streams with B-frames) and post it on ffmpeg-devel where it can be discussed. Carl Eugen PS: Your mail is nearly unreadable here. In case you are sending HTML-mails: Please don't do that. ___ Libav-user mailing list Libav-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/libav-user
Re: [Libav-user] Force low-delay handling? question + feature suggestion
On Thu, Nov 24, 2011 at 12:06, Carl Eugen Hoyos wrote: > Camera Man writes: > >> would it work to patch ff_h264_decode_init() and decode_postinit() to also >> check the AV_DISCARD mode, and if we are discarding B and/or nonref frames, >> would set avctx->has_b_frames=0 and low_delay=1 ? > > I don't know but I suggest that you write such a patch (optionally test it on > streams with B-frames) and post it on ffmpeg-devel where it can be discussed. Maybe I am missing something, but if you discard B frames you may have problems with the P frames that follow (and that until you get the next I frame). Also, note that the codecs check avctx->skip_loop_filter not as bitfield, but priority. Thus, AVDISCARD_BIDIR means also AVDISCARD_NONREF, which is probably not what you want. I believe that the correct way to fix is to modify SPS/PPS, because these are simply wrong in your case. BR, Alex Cohn ___ Libav-user mailing list Libav-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/libav-user
Re: [Libav-user] Force low-delay handling? question + feature suggestion
Carl Eugen Hoyos writes: > > would it work to patch ff_h264_decode_init() and decode_postinit() to also > > check the AV_DISCARD mode, and if we are discarding B and/or nonref frames, > > would set avctx->has_b_frames=0 and low_delay=1 ? You would have to discard all B-frames (because they require reordering) not only the non-ref ones, i.e. corruption would be possible. > I don't know but I suggest that you write such a patch (optionally test it on > streams with B-frames) and post it on ffmpeg-devel where it can be discussed. Did you write such a patch? Please note that your mails are nearly unreadable: http://ffmpeg.org/pipermail/libav-user/2012-March/001495.html Consider setting your mailer to "plain text". Carl Eugen ___ Libav-user mailing list Libav-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/libav-user
Re: [Libav-user] Force low-delay handling? question + feature suggestion
On 03/11/2012 11:57 AM, Carl Eugen Hoyos wrote:
You would have to discard all B-frames (because they require
reordering) not only the non-ref ones, i.e. corruption would be possible.
That's cool! I didn't know that B frames can be reference frames (The
last spec I read top-to-bottom was mpeg2, where B frames were never
references). Although it foils my feature suggestion...
Did you write such a patch?
I forced has_b_frames to 0 and low_delay to 1 with a patch in the
ugliest possible way (all over the places that these may be changed,
disregarding the discard setting), and it worked for the streams
produced by my camera (which claims max redorder frames = 4, but I've
never actually seen anything that was out of order), but I'm afraid to
use it in practice -- the camera says "num_reorder = 4", and although I
haven't been able to produce anything out-of-order from it, there might
be circumstances it does.
I have no idea how the decoder would behave in this case, and no
reasonable way to test.
Another possible solution for my use case, which might be useful for
others, is to somehow pull the latest-pts picture from the insides of
the h264; something like:
avcodec_decode_video2(context, frame1, packet, &frame1_finished);
avcodec_h264_internal_fetch_most_recent(context, frame2,
&frame2_finished); /* this is what I would need to write */
switch (2*(frame2_finished!=0)+1*(frame1_finished!=0)) {
case 0: /* none at all */ break;
case 1: /* frame1, no frame2 */ use(frame1);
case 2: /* frame2, no frame1 */ use(frame2);
case 3: /* both; pick latest */ use(frame1->pts >= frame2->pts ?
frame1 : frame2);
}
Where avcodec_h264_internal_fetch_most_recent() would get a copy of the
highest pts frame in the delay buffer, without changing or discarding
it. If this is possible (and I manage to write this ... I'm not sure I
understand all of h264 internals), then it would introduce a "lowest
delay" from bitstream to display - at the cost of not showing any
out-of-order frames (having an effectively lower frame rate - but much
better real-time latency).
Can you think of a reason off-hand this wouldn't work?
Please note that your mails are nearly unreadable:
http://ffmpeg.org/pipermail/libav-user/2012-March/001495.html
Consider setting your mailer to "plain text".
Thanks - even though my mailer was set to "Send both HTML and plain
text", for some reason to ffmpeg.org it insists on sending HTML only.
I've set it to plain-text only, in the hope that it would actually work
this time. Apologies for the previously unreadable emails.
___
Libav-user mailing list
Libav-user@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/libav-user
Re: [Libav-user] Force low-delay handling? question + feature suggestion
On Mon, Mar 12, 2012 at 10:28, Camera Man wrote: > the camera says "num_reorder = 4", and although I haven't been able to > produce anything out-of-order from it, there might be circumstances it does. Or maybe the camera manufacturer didn't care. After all, 0 reorders is less or equal 4, therefore it is formally correct to state 4 in the header if you never reorder. It may be a side effect of setting certain h264 profile (which comes with a list of supported video resolutions, bitrates, - and also B-frame and reorder restrictions). BR, Alex Cohn ___ Libav-user mailing list Libav-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/libav-user
Re: [Libav-user] Force low-delay handling? question + feature suggestion
On Mon, Mar 12, 2012 at 10:28, Camera Man wrote: > Where avcodec_h264_internal_fetch_most_recent() would get a copy of the > highest pts frame in the delay buffer, without changing or discarding it. If > this is possible (and I manage to write this ... I'm not sure I understand > all of h264 internals), then it would introduce a "lowest delay" from > bitstream to display - at the cost of not showing any out-of-order frames > (having an effectively lower frame rate - but much better real-time > latency). Your suggestion looks incorrect to me: the frames that you may skip this way could be referenced by the frames you want to display, therefore your result will be distorted. You should decode as many frames as possible at any time. The question is - which frames to display. You can safely choose to always display the latest (in terms of pts). Alex ___ Libav-user mailing list Libav-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/libav-user
Re: [Libav-user] Force low-delay handling? question + feature suggestion
On 03/12/2012 04:49 AM, Alex Cohn wrote: On Mon, Mar 12, 2012 at 10:28, Camera Man wrote: Where avcodec_h264_internal_fetch_most_recent() would get a copy of the highest pts frame in the delay buffer, without changing or discarding it. If this is possible (and I manage to write this ... I'm not sure I understand all of h264 internals), then it would introduce a "lowest delay" from bitstream to display - at the cost of not showing any out-of-order frames (having an effectively lower frame rate - but much better real-time latency). Your suggestion looks incorrect to me: the frames that you may skip this way could be referenced by the frames you want to display, therefore your result will be distorted. What I want to do in that routine (again, I am not sure it is possible - you are probably right, but I want to explain myself better): Let's say the stream is IPPP=1234, and the pts order is 1432. (I frame first, then P frames in reverse order). If I understand correctly, frame '2' can only depend on frame '1'. Now, if I look into the decoder's delay buffer, after the first decode, the highest pts in the delay buffer would be 1; after the 2nd decode, the highest pts in the delay buffer would be 2; after the 3rd decode and 4th decode it would still be 2. I would be completely ignoring frames 3 and 4, which is fine -- as soon as I've seen 2 in the stream, I want to display it. Assuming that there is actually NO reorder, Anton Khirnov suggested setting CODEC_FLAG_LOW_DELAY - I'll test that (together with your suggestion of zeroing num_reorder_frame) and update. You should decode as many frames as possible at any time. The question is - which frames to display. You can safely choose to always display the latest (in terms of pts). The reason I want to reach into the delay buffer to find the highest pts picture, though, is that I'm doing this in real time - and with a 5 fps stream, the 4-picture lag between bitstream arrival and decode_video2 resposne is about 1 second, which is too much for this application. If this was a pre-recorded stream, then I could just keep decoding until I get pictures and there wouldn't be any problem. ___ Libav-user mailing list Libav-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/libav-user
