Hi, Before writing any patches I'd like to ask whether there is interest in, and an acceptable shape for, supporting the NVIDIA Jetson hardware decoder through the existing v4l2m2m decoders. Previous attempts were the 2020 nvv4l2dec wrapper (reviewers asked to reuse v4l2m2m instead) and the 2024 nvtegra series (reviewers considered a userspace driver out of scope). As far as I can tell no one has reported how the stock L4T V4L2 interface behaves against v4l2m2m, so I measured it on an AGX Xavier (L4T R35.6.4) with small standalone test programs mirroring what libavcodec/v4l2_*.c does.
What works - Decoding is bit-exact: H.264 High (with B-frames), 1920x1080, 591/591 frames match the software decoder (framemd5), after the layout fix described below. - A single-threaded, O_NONBLOCK, EAGAIN-driven loop (like v4l2_context.c) works. - V4L2_EVENT_SOURCE_CHANGE, G_FMT, V4L2_CID_MIN_BUFFERS_FOR_CAPTURE, VIDIOC_DECODER_CMD(STOP), the EOS event and V4L2_BUF_FLAG_LAST all work. - QUERYCAP reports V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING; capture offers NM12 (NV12M), output offers H264/H265/VP8/VP9/MPEG1/2/4/AV1/... - Throughput is roughly 85-145 fps for 1080p H.264 without file I/O (varies with system load; not yet compared against NVIDIA's own ffmpeg build). - Closing the fd while buffers are still allocated hangs inside the NVIDIA library (TegraV4L2_Close); STREAMOFF + REQBUFS(0) before close avoids it, which is what v4l2_m2m.c / v4l2_context.c already do. Where it deviates from what v4l2m2m expects 1. The device (/dev/nvhost-nvdec) only speaks V4L2 through NVIDIA's libv4l2 (libv4l2.so.0 is replaced by tegra/libnvv4l2.so); plain open()+ioctl() returns ENOTTY. The upstream v4l-utils libv4l2 does not find the NVIDIA plugin (it lives in plugins/nv/). driver/card strings are empty. 2. v4l2_mmap() on the device fd fails with ENODEV; buffers must be exported with VIDIOC_EXPBUF and the dmabuf fd mmap()ed. 3. The capture format is reported as NM12 (pitch-linear) but the memory is NVIDIA block-linear (64x8-byte GOBs, block height 16 GOBs). bytesperline and plane lengths are also misreported (e.g. 176x144: stride 176 reported, real pitch 192; 1080p luma: 2088960 reported, 2228224 actual). The real chroma offset is available via m.mem_offset, the total size via lseek() on the dmabuf. A software de-tile (pitch = ALIGN(width, 64), block height 16, byte swizzle ((x/32)&1)*256 + ((y/4)&1)*128 + ((x/16)&1)*64 + (y&3)*16 + (x&15)) reproduces the software decoder output exactly. NVIDIA's own tools use the proprietary NvBufSurfTransform (VIC) for this conversion. 4. VIDIOC_G/S_SELECTION is not supported; VIDIOC_G_CROP works. 5. poll() on the fd always returns immediately (POLLIN|POLLOUT|POLLRDNORM), so the poll-based waiting in v4l2_context.c degenerates into busy polling. NVIDIA's samples use a private control (V4L2_CID_MPEG_VIDEO_DEVICE_POLL). Questions - Would a v4l2m2m change along these lines be acceptable: optional libv4l2 indirection (as libavdevice/v4l2.c already does), dmabuf-based mapping, and a Jetson-specific quirk path (explicit opt-in, e.g. -device /dev/nvhost-nvdec -v4l2_quirks jetson) that de-tiles block-linear output and uses G_CROP? - Or is hardware-specific layout fixing of this kind considered out of scope for libavcodec, in which case a separate route (hwcontext exposing the dmabuf + a filter for de-tiling, or nothing at all) would be preferred? - Is anyone else interested in testing on Orin / other L4T releases? The test programs and full results are available if useful. I have only a Xavier; I cannot test Orin or other V4L2 m2m hardware. Thanks, guhyun9454 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
