Hi all, This patch series implements a hardware decoding backend for nvidia Tegra devices, notably the Nintendo Switch. It was primarily written for HorizonOS (Nintendo Switch OS), but also supports nvidia's Linux4Tegra distro. As for hardware, all Tegras later than the X1 (T210) should be supported, although the patch does not implement features that were added to subsequent revisions of multimedia engines (eg. 12-bit HEVC). However, since I only own T210 devices (Switch and jetson nano), I was not able to verify this.
The backend is essentially a userspace NVDEC driver, as due to the OS design of the Switch, we cannot link to nvidia's system libraries. It notably uses (sparse) hardware documentation released by nvidia here: https://github.com/NVIDIA/open-gpu-doc/tree/master/classes/video. It supports all codecs available in hardware (MPEG1/2/4, VC1, H264, HEVC, VP8, VP9 and JPEG), with dynamic frequency scaling, and hardware-accelerated frame transfer. At the moment I'm submitting the series with some nvidia headers pulled from various sources, but I do think they should rather be put in nv-codec-headers, let me know. The code was tested for memory bugs and leaks with valgrind and asan on L4T. Some quick performance testing (decoding with -f null -) showed results in line with official software, tested against the nvv4l2 backend that was posted here a while ago: https://lists.ffmpeg.org/pipermail/ffmpeg-devel/2020-June/263759.html. Note that the numbers are skewed because frame transfer cannot be disabled in nvidia's backend. - HEVC Main 10 @ 4k (~80Mbps): nvtegra 79fps, nvv4l2 66fps - HEVC Main 10 @ 1080p (~5Mbps): nvtegra 402fps, nvv4l2 229fps - H264 @ 1080p (~3Mbps): nvtegra 286fps, nvv4l2 260fps Several homebrew applications have been using this backend for some time, with no bugs reported. As far as I'm aware, this is the complete list of them: - NXMP, a media player based on mpv: https://github.com/proconsule/nxmp - WiliWili, a bilibili client: https://github.com/xfangfang/wiliwili - Switchfin, a Jellyfin client: https://github.com/dragonflylee/switchfin - Moonlight-Switch, a Moonlight client: https://github.com/XITRIX/Moonlight-Switch - chiaki: https://git.sr.ht/~kkwong/chiaki/ - My own media player, unreleased at this time Nintendo Switch support assumes a working devkitA64 homebrew environment, instructions regarding setup can be found here: https://devkitpro.org/wiki/devkitPro_pacman. The hwaccel can then be configured by eg.: ``` source /opt/devkitpro/switchvars.sh && ./configure --cross-prefix=aarch64-none-elf- --enable-cross-compile --arch=aarch64 --cpu=cortex-a57 --target-os=horizon --enable-pic --enable-gpl --enable-nvtegra ``` It should probably be noted that NVDEC usage on discrete gpus is very similar. As far as I know, the main difference is that the interfacing is done through the GPFIFO block (same block that manages the 3D engine), instead of host1x. Thank you for your consideration. averne (16): avutil/buffer: add helper to allocate aligned memory configure,avutil: add support for HorizonOS avutil: add ioctl definitions for tegra devices avutil: add hardware definitions for NVDEC, NVJPG and VIC avutil: add common code for nvtegra avutil: add nvtegra hwcontext hwcontext_nvtegra: add dynamic frequency scaling routines nvtegra: add common hardware decoding code nvtegra: add mpeg1/2 hardware decoding nvtegra: add mpeg4 hardware decoding nvtegra: add vc1 hardware decoding nvtegra: add h264 hardware decoding nvtegra: add hevc hardware decoding nvtegra: add vp8 hardware decoding nvtegra: add vp9 hardware decoding nvtegra: add mjpeg hardware decoding configure | 30 + libavcodec/Makefile | 11 + libavcodec/h263dec.c | 6 + libavcodec/h264_slice.c | 6 +- libavcodec/h264dec.c | 3 + libavcodec/hevcdec.c | 17 +- libavcodec/hevcdec.h | 2 + libavcodec/hwaccels.h | 10 + libavcodec/hwconfig.h | 2 + libavcodec/mjpegdec.c | 6 + libavcodec/mpeg12dec.c | 12 + libavcodec/mpeg4videodec.c | 3 + libavcodec/nvtegra_decode.c | 517 +++++++++ libavcodec/nvtegra_decode.h | 94 ++ libavcodec/nvtegra_h264.c | 506 +++++++++ libavcodec/nvtegra_hevc.c | 633 +++++++++++ libavcodec/nvtegra_mjpeg.c | 336 ++++++ libavcodec/nvtegra_mpeg12.c | 319 ++++++ libavcodec/nvtegra_mpeg4.c | 344 ++++++ libavcodec/nvtegra_vc1.c | 455 ++++++++ libavcodec/nvtegra_vp8.c | 334 ++++++ libavcodec/nvtegra_vp9.c | 665 ++++++++++++ libavcodec/vc1dec.c | 9 + libavcodec/vp8.c | 6 + libavcodec/vp9.c | 10 +- libavutil/Makefile | 9 + libavutil/buffer.c | 31 + libavutil/buffer.h | 7 + libavutil/clb0b6.h | 303 ++++++ libavutil/clc5b0.h | 436 ++++++++ libavutil/cle7d0.h | 129 +++ libavutil/cpu.c | 7 + libavutil/hwcontext.c | 4 + libavutil/hwcontext.h | 1 + libavutil/hwcontext_internal.h | 1 + libavutil/hwcontext_nvtegra.c | 1045 ++++++++++++++++++ libavutil/hwcontext_nvtegra.h | 92 ++ libavutil/nvdec_drv.h | 1858 ++++++++++++++++++++++++++++++++ libavutil/nvhost_ioctl.h | 511 +++++++++ libavutil/nvjpg_drv.h | 189 ++++ libavutil/nvmap_ioctl.h | 451 ++++++++ libavutil/nvtegra.c | 1035 ++++++++++++++++++ libavutil/nvtegra.h | 258 +++++ libavutil/nvtegra_host1x.h | 94 ++ libavutil/pixdesc.c | 4 + libavutil/pixfmt.h | 8 + libavutil/vic_drv.h | 279 +++++ 47 files changed, 11085 insertions(+), 3 deletions(-) create mode 100644 libavcodec/nvtegra_decode.c create mode 100644 libavcodec/nvtegra_decode.h create mode 100644 libavcodec/nvtegra_h264.c create mode 100644 libavcodec/nvtegra_hevc.c create mode 100644 libavcodec/nvtegra_mjpeg.c create mode 100644 libavcodec/nvtegra_mpeg12.c create mode 100644 libavcodec/nvtegra_mpeg4.c create mode 100644 libavcodec/nvtegra_vc1.c create mode 100644 libavcodec/nvtegra_vp8.c create mode 100644 libavcodec/nvtegra_vp9.c create mode 100644 libavutil/clb0b6.h create mode 100644 libavutil/clc5b0.h create mode 100644 libavutil/cle7d0.h create mode 100644 libavutil/hwcontext_nvtegra.c create mode 100644 libavutil/hwcontext_nvtegra.h create mode 100644 libavutil/nvdec_drv.h create mode 100644 libavutil/nvhost_ioctl.h create mode 100644 libavutil/nvjpg_drv.h create mode 100644 libavutil/nvmap_ioctl.h create mode 100644 libavutil/nvtegra.c create mode 100644 libavutil/nvtegra.h create mode 100644 libavutil/nvtegra_host1x.h create mode 100644 libavutil/vic_drv.h -- 2.45.1 _______________________________________________ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
