This series adds device derivation for D3D11VA and direct GPU frame
transfers between D3D11 and Vulkan, and between D3D11 and CUDA.

Motivation: the end goal is letting mpv users on a Vulkan VO use the
video processing features NVIDIA's driver exposes only through D3D11,
RTX Video Super Resolution and RTX Video HDR, without paying for the
transfers. mpv can already run D3D11 video processing under a Vulkan
VO, or decode with nvdec and filter with D3D11, but today every such
frame crosses through system memory twice per hop. With this series
the transfer stays on the GPU. The mpv side of this work is under review in
https://github.com/mpv-player/mpv/pull/18300 and
https://github.com/mpv-player/mpv/pull/18301, and the FFmpeg design
was discussed in that first thread from
https://github.com/mpv-player/mpv/pull/18300#issuecomment-5124804191
onward. Philip Langdale's guidance there shaped both transfer
implementations.

Patch 1 implements device_derive for D3D11VA by matching the source
device's LUID, so a derived D3D11 device lands on the same adapter as
a Vulkan or CUDA source.

Patch 2 imports single plane D3D11 textures into Vulkan as external
memory and copies on the Vulkan queue, ordered against D3D11 work by
a shared D3D11 fence imported as a timeline semaphore.

Patch 3 extends that to NV12 and P010. A two plane texture cannot be
imported whole, so each plane moves through a shared single plane
texture written and read by a compute shader on the D3D11 side.

Patch 4 adds CUDA transfers for D3D11 frames through CUDA's external
memory and external semaphore import, fully GPU chained in both
directions.

ffnvcodec currently lacks CU_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_RESOURCE
and CUDA_EXTERNAL_MEMORY_DEDICATED, so patch 4 defines them locally. I
am happy to send a nv-codec-headers patch adding them (and the D3D11
fence semaphore type) if that is preferred.

On the verification scope, stated plainly: this was developed and
tested on one machine, Windows 11 with an RTX 5090. Testing covered
byte exact transfer matrices in both directions for nv12, p010, bgra,
x2bgr10 and rgbaf16 including 4K and odd sizes, texture arrays with
nonzero slice indices, adversarial cases (mismatched formats, keyed
mutex textures, stress and coherency loops), Vulkan validation layers,
and mpv end to end route checks with screenshot comparison against the
native D3D11 path. I have no AMD or Intel results. Rebased onto
current master and re-verified before sending.

While re-verifying on master I hit an unrelated regression: since
4ec050211a ("avutil/hwcontext_vulkan: drop NVIDIA/MoltenVK host
transfer blocklist"), allocating a Vulkan frame pool for any single
plane RGB format fails on the NVIDIA proprietary driver with "No
memory type found for flags 0x1" (bgra, rgba, x2bgr10le, rgbaf16le
all fail, nv12 and p010 are unaffected). Bisected and reproduced on
unpatched master; the frame pool images now carry
VK_IMAGE_USAGE_HOST_TRANSFER_BIT and the driver then offers no device
local memory type for them. With that commit reverted locally, the
full matrix for this series passes on current master: the transfer
matrix with zero failures and the adversarial suite 15 of 15. I am
reporting that regression separately from this series; mentioning it
here so a reviewer reproducing on NVIDIA does not misattribute the
allocation failures to these patches.

Measured on this machine at 1080p, a direct hop is roughly 5x faster
than the system memory path it replaces.

Disclosure: this series was written with AI assistance (Claude Code).
The AI implemented the design following Philip Langdale's guidance
from the mpv thread, probed the driver behavior and ran the
verification on my machine. I have reviewed it, I understand what it
changes and why, I take responsibility for it, and review responses
will be written by me.

I have a lot of respect for this project and I do not claim this is
the best possible design. If reviewers see a better path I am happy
to rework the series, and I hope it is a useful starting point for
getting these transfers into FFmpeg either way.

flowreen (4):
  avutil/hwcontext_d3d11va: implement device derivation
  avutil/hwcontext_vulkan: add D3D11 to Vulkan frame transfers
  avutil/hwcontext_vulkan: add two plane D3D11 transfers
  avutil/hwcontext_d3d11va: add CUDA frame transfers

 libavutil/Makefile                     |   3 +-
 libavutil/hwcontext_d3d11va.c          | 993 ++++++++++++++++++++++++-
 libavutil/hwcontext_d3d11va_internal.h |  79 ++
 libavutil/hwcontext_vulkan.c           | 805 ++++++++++++++++++++
 libavutil/vulkan_functions.h           |   5 +-
 5 files changed, 1882 insertions(+), 3 deletions(-)
 create mode 100644 libavutil/hwcontext_d3d11va_internal.h


base-commit: ca99230f7176c589b8390b34f2d1c130b9e1881d
-- 
2.45.1.windows.1

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]

Reply via email to