From: Caden Chien <[email protected]> PR "avcodec/decode: new API for batch decoding #22618" was opened as the implementation of this RFC.
This RFC is made up of 5 parts: * why we need this? * how we achieve this? * what would it impact? * any other existed way to achieve this? * how can we go even further? [why we need this?] For modern hardware, if there are no workloads on the entire SoC, it can enter sleep mode to save power. When a video is decoded with dedicated hardware like VCN on GPU, for each frame, it will have a process as: CPU generates commands for a job -> submit the job -> hardware decodes a frame and interrupt CPU. To have entire SoC sleeping deep and long, rather than being waken up frequetly to decode, We would like to batch several of these processes together. >From our testing with MPV on Linux, when 16 VA-API decoding jobs are batched together, we can save 144mW for 1080p 24fps video playback. For 1080p 60fps video playback, we can save around 147mW, sometimes up to 200mW. Memory consumption could be a concern that we need more memory to hold frames in flight. On one side, there are structs like BatchFrameBuffer, DecodedFrames, and AVFrames, which require a few extra bytes to track buffers. On the other side, there are the actual frame buffers that hold the pixels. If clients use frame pool, memory consumption should be the same if batching doesn't exhaust the pool. If pool size is dynamic, then more memory consumption is expected when batch size is larger that pool size. When testing mpv with software decoding, which uses its dynamic image pool, with and without 16-frame batching consume 191.4MiB and 155.7MiB respectively on 1080p@60 video. Here we tested with software decoding because it's easier to capture the difference with valgrind massif heap tracking tool. Despite the increased memory usage when the pool is exhausted, I think it's still under control with batch size setting, either for fixed or dynamic pool size. [how we achieve this?] This is achieved by batching frame decoding in FFmpeg avcodec API. New avcodec_receive_frame() now will hint client to send more frame data to decode until a batch size. This results in a burst of VA-API calls. For example, with batch size 16, you will see 16 continuous decoding jobs, then no decoding jobs for next 15 frame time. Although we mainly test with VA-API, it can be enabled for software decoding as well. For implementation details please refer to commit message of each patch. [what would it impact?] It is expected to have exact the same video playback experience, but still some potential impact should be noted. One is first frame delay, since we need to wait the first burst finished, and another one is frame stuttering, due to potential long burst time. I didn't see these with batch size 16 for 1080p@60 vidoes, but it can appear when batch size is over 200. Slower hardware or videos with higher bitrate may have lower tolerance. I've noticed a `mLowLatency` option in Firefox. We may need to not enable batching for it. While newer hardware could have bigger batch size, our goal is to save power. More intensive bursting could lead to higher clock frequency, consuming more power. Thus, default batch size should be set with care. For reference, batch size 16 is recommended for 1080p@60 videos, and batch size 4 is recommended for 4k@60 video. I think we may need more testing and discussion when enabling batching in applications. Application develops may want to have a simple strategy such as enabling batching for single stream 1080p videos with fixed value 16, or more complex strategy such as detecting the stress/frame drops of current playback workload to adjust batch size dynamically. Or in another way, should driver decide which batch size to use, based on some hints given by applications? To enable batch decoding in MPV, simply add a line in init_avctx() in vd_lavc.c to set up avctx->batch_size. For Firefox that uses ffmpeg from system (e.g. Linux), add a line in InitHWCodecContext() in FFmpegVideoDecoder.cpp to set up avctx->batch_size, with conditional macro #if LIBAVCODEC_VERSION_MAJOR >= 61 around it (You may need to replace the number with the version on your system). [any other existed way to achieve this?] Before digging further, we would like to know more about is there any existed way to do video batch decoding? I see Intel VA-API driver implements vaCreateMFContext()/vaMFAddContext()/vaMFSubmit() for multiple contexts for multiple frames. How are they used in applications? [how can we go even further?] Typically, in the case of using VA-API, one decoding job will issue one system call to submit commands and handle one interrupt to get job done acknowledgement. To make multiple jobs share one system call and interrupt handling, we can accumulate commands in command buffer, submitting them at once as a single big job. It can potentially saving more power by less user-kernel mode switching and interrupt handling. From our testing by hacking mesa to do so, it can save 40~90mW more, when it is enabled together with batch decoding. However doing it a driver level might not be a good idea, since it's out of the scope of a driver and introducing surprising behaviors, eventually causing deadlock when applications trying to synchronize a dmabuf. So it's more natual to do command batching in Vulkan, fitting the original purpose of Vulkan that extracting surprising driver specific behaviors back to developer's hands. While we are investigating further, any comments on this are welcome. Thanks, Caden -- 2.53.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
