PR #25013 opened by asor
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25013
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25013.patch
# Problem description
Screen recording on X11 was one command (`x11grab`). On Wayland the generic
path is `kmsgrab`, but on a 10-bit desktop it cannot capture at all — and when
it does capture, the pixels are silently scrambled.
Setup: KDE Plasma on Wayland (KWin 6.6.6), NVIDIA GeForce GT 1030 driving one
HDMI display. KWin composites the desktop into a 10-bit framebuffer with DRM
FourCC AB30 (DRM_FORMAT_ABGR2101010) and hands it to the display controller as
an NVIDIA block-linear tiled buffer (format modifier
0x03000000004fe014 = DRM_FORMAT_MOD_NVIDIA_BLOCK_LINEAR_2D).
Three independent defects block or corrupt capture of such a framebuffer:
1. kmsgrab's DRM FourCC table does not know the 2:10:10:10 formats with alpha
(AR30/AB30/XB30), so it refuses the framebuffer outright.
2. libavutil's DRM->Vulkan format table maps the 2:10:10:10 formats with R and
B exchanged, so the GPU-side copy path cannot import the buffer (hard error
on drivers that validate modifier+format, mislabeled exports everywhere).
3. libavutil's DRM hwcontext maps the buffer as if it were linear and reads it
row by row, silently ignoring the format modifier — tiled memory read as
linear rows produces horizontal stripes.
With the three fixes below, recording works exactly like x11grab did: a single
ffmpeg invocation, or any wrapper script around it.
All three defects are present in ffmpeg 8.0.1, 9.0.2 and current master
(verified 2026-10-10 against the 9.0.2 tarball and git master).
# Environment
- KDE Plasma 6.6.6 / KWin 6.6.6 on Wayland, kernel 7.0.0-38-generic
- NVIDIA GeForce GT 1030, driver 580.178.04 (open kernel module)
- Desktop composited into AB30 = DRM_FORMAT_ABGR2101010, 1920x1080@100Hz HDMI
- Framebuffer format modifier: 0x03000000004fe014
(DRM_FORMAT_MOD_NVIDIA_BLOCK_LINEAR_2D(0, 0, 0, 254, 4))
- ffmpeg 8.0.1 (Ubuntu 7:8.0.1-3ubuntu2) and master @ 16d33f1f16
Any compositor that renders into a 2:10:10:10 framebuffer is affected by bugs 1
and 2; any tiled (non-linear) framebuffer is affected by bug 3.
# Reproduction instructions
All commands use the first active plane; substitute your DRM device
(e.g. /dev/dri/card2 on hybrid graphics).
Bug 1 — kmsgrab rejects the framebuffer (any 10-bit compositor output):
ffmpeg -framerate 5 -device /dev/dri/cardN -f kmsgrab -i - \
-frames:v 1 -f null -
-> Framebuffer pixel format 30334241 is not a known supported format.
(30334241 = 'AB30')
Bug 2 — DRM->Vulkan import of a 2:10:10:10 DMA-BUF. On 8.0.1 (which lacks the
sw_format fallback added in 03be5630) this fails outright on drivers that
validate modifier+format compatibility, e.g. NVIDIA 580:
ffmpeg -framerate 5 -device /dev/dri/cardN -f kmsgrab -i - \
-vf 'hwmap=derive_device=vulkan,hwdownload,format=x2bgr10le' \
-frames:v 1 out.png
-> Cannot map DRM frame to Vulkan: VK_ERROR_FORMAT_NOT_SUPPORTED
On master the same command succeeds for single-plane buffers because
vulkan_map_from_drm_frame_desc() now takes the VkFormat from sw_format
(commit 03be5630), bypassing the buggy table — but the wrong entries are
still used for multi-plane layers and by vulkan_map_to_drm(), which stamps
exported DMA-BUFs with the R/B-swapped fourcc (ABGR2101010 memory labelled
ARGB2101010), so interop consumers see swapped channels.
Bug 3 — CPU download of a tiled buffer reads garbage (any tiled framebuffer):
ffmpeg -framerate 5 -device /dev/dri/cardN -f kmsgrab -i - \
-vf 'hwdownload,format=x2bgr10le' -frames:v 1 out.png
-> out.png is scrambled into horizontal stripes (silently; the command
exits 0). Mean abs difference vs. a compositor screenshot of the same
frame: ~30/255, versus ~1.5 for a correct capture.
# Bugs found
Bug 1 — libavdevice/kmsgrab.c: the DRM FourCC -> pixel format table covers only
XRGB2101010 of the 2:10:10:10 family. ARGB2101010 (AR30), ABGR2101010 (AB30)
and XBGR2101010 (XB30) are missing, so read_header() aborts with the error
above. AB30 is what KWin composites into on any 10-bit output, which makes
kmsgrab unusable there.
Bug 2 — libavutil/hwcontext_vulkan.c: vulkan_drm_format_map maps the
2:10:10:10 formats with R and B exchanged:
DRM_FORMAT_ARGB2101010 -> VK_FORMAT_A2B10G10R10_UNORM_PACK32 (wrong)
DRM_FORMAT_ABGR2101010 -> VK_FORMAT_A2R10G10B10_UNORM_PACK32 (wrong)
(and the same for the X variants). This contradicts the vkfmt_map table in the
same file, which pairs VK_FORMAT_A2R10G10B10 with AV_PIX_FMT_X2RGB10 — i.e. R
next to the 2-bit field — the same layout drm_fourcc.h documents for
DRM_FORMAT_ARGB2101010 ("[31:0] A:R:G:B 2:10:10:10 little endian"). Two
consequences: imports create the image with the wrong VkFormat (rejected by
drivers that validate the modifier against the format; visible on 8.0.1 and on
master for multi-plane layers), and exports via vulkan_map_to_drm() label
buffers with the swapped fourcc.
Bug 3 — libavutil/hwcontext_drm.c: drm_map_frame() mmaps each DMA-BUF object
and exposes it as a plain linear image (data/linesize), silently ignoring
AVDRMObjectDescriptor.format_modifier. For tiled buffers the linear read is
meaningless — on NVIDIA block-linear memory every frame comes out as horizontal
stripes — and nothing reports an error. This affects transfer_data_from,
transfer_data_to and map_from alike.
# The patch series
4 commits on top of master @ 16d33f1f16 (branch kmsgrab-10bit-fixes), each
self-contained and signed off; also attached as git format-patch output:
1. lavd/kmsgrab: support 2:10:10:10 formats with alpha
Adds the three missing FourCCs (and their big-endian variants), mapped to
the matching X2*10 formats — FFmpeg has no 2:10:10:10 format with alpha, so
the 2-bit alpha field lands in the unused X component, exactly as the
existing XRGB2101010 entries do.
2. avutil/hwcontext_vulkan: fix R/B swap in DRM 2:10:10:10 format map
Fixes all four entries so both tables agree.
3. avutil/hwcontext_drm: refuse non-linear modifiers in CPU mappings
drm_map_frame() now rejects objects whose modifier is neither
DRM_FORMAT_MOD_INVALID nor DRM_FORMAT_MOD_LINEAR, with an error naming the
modifier and pointing at the GPU-side copy (hwmap=derive_device=vulkan).
Buffers without modifiers map exactly as before. There is no generic CPU
detiler for vendor-tiled layouts, so failing loudly is the honest behavior —
silently returning garbage is the bug.
4. doc: describe capturing non-linear DRM framebuffers with kmsgrab
Documents the vulkan hwmap recipe, which works regardless of memory layout,
and the constraint that hwdownload can only produce the framebuffer's own
format.
# Verification
Built master + series with gcc 15 (--enable-vulkan). Checked with the project's
own tools/patcheck (clean) and `make fate` (pass). Functionally verified on the
hardware above, using a compositor screenshot as ground truth (mean absolute
RGB difference, /255):
vs ground truth
bug 1 (master, plain download) n/a — refuses the buffer (error above)
bug 2 (patch 1, table masked) 0.73 (masked by 03be5630 on master)
bug 2 (patch 1, table UNMASKED) n/a — Cannot map DRM frame to Vulkan
bug 2 (patch 2, table UNMASKED) 1.44
bug 3 (patch 1, plain download) 30.20 (striped garbage)
full series (vulkan hwmap) 1.64
(“UNMASKED” = the sw_format fallback of 03be5630 disabled with a one-line
change so the DRM->Vulkan table alone decides, reproducing the 8.0.1 situation;
that pair isolates patch 2's effect on master's code.)
End-to-end: a 30-second screencast (mic + desktop audio, H.264/AAC, 15fps)
recorded via the documented single-command chain: 450 frames captured at
realtime speed, U-plane chroma roughness 0.2 (>=10 would indicate corruption),
audio present (mean -24 dB). The single-command example from the docs
(grab + convert + encode in one process) was recorded and checked as well.
# Notes / possible follow-up
A transparent fallback in drm_transfer_data_from() — deriving a Vulkan device
internally and doing the copy on the GPU so plain hwdownload "just works" on
tiled buffers — would remove the need for the explicit hwmap=derive_device=
chain. That is a larger, cross-hwcontext design change, deliberately left out of
this series; happy to work on it if there is interest.
The mouse cursor is not captured (it is a separate DRM cursor plane,
composited at scanout), which is a separate feature/limitation and not part of
this series.
From 049d1bb5179f0522918f2703f349bd6508b3b638 Mon Sep 17 00:00:00 2001
From: Alexander Sorokin <[email protected]>
Date: Sat, 10 Oct 2026 23:55:45 +0300
Subject: [PATCH 1/4] lavd/kmsgrab: support 2:10:10:10 formats with alpha
The format table only covered XRGB2101010 of the 2:10:10:10 family, so
capturing a scanout framebuffer with FourCC AR30 (ARGB2101010), AB30
(ABGR2101010) or XB30 (XBGR2101010) failed with:
Framebuffer pixel format 30334241 is not a known supported format.
AB30 is not exotic: KWin (KDE Plasma on Wayland) composites the desktop
into DRM_FORMAT_ABGR2101010 on systems with 10-bit output, which makes
kmsgrab unusable there.
FFmpeg has no 2:10:10:10 pixel format with an alpha component, so map
these to the corresponding X2*10 formats: the 2-bit alpha field lands in
the unused X component and is dropped, exactly as the existing
XRGB2101010 entries do. The component layout was checked against
ffmpeg's own pixel output: AB30, i.e. [31:30]A [29:20]B [19:10]G [9:0]R
in the 32-bit word, is AV_PIX_FMT_X2BGR10LE.
Signed-off-by: Alexander Sorokin <[email protected]>
---
libavdevice/kmsgrab.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/libavdevice/kmsgrab.c b/libavdevice/kmsgrab.c
index 3d15754f3b..4d3859de05 100644
--- a/libavdevice/kmsgrab.c
+++ b/libavdevice/kmsgrab.c
@@ -399,6 +399,17 @@ static const struct {
// 10-bit RGB.
{ AV_PIX_FMT_X2RGB10LE, DRM_FORMAT_XRGB2101010 },
{ AV_PIX_FMT_X2RGB10BE, DRM_FORMAT_XRGB2101010 | DRM_FORMAT_BIG_ENDIAN },
+ // 10-bit BGR.
+ { AV_PIX_FMT_X2BGR10LE, DRM_FORMAT_XBGR2101010 },
+ { AV_PIX_FMT_X2BGR10BE, DRM_FORMAT_XBGR2101010 | DRM_FORMAT_BIG_ENDIAN },
+ // 10-bit RGB with a 2-bit alpha component, stored ahead of the colour
+ // channels. FFmpeg has no 2:10:10:10 format with alpha, so map these to
+ // the matching format without one - the alpha bits land in the unused X
+ // field and are dropped.
+ { AV_PIX_FMT_X2RGB10LE, DRM_FORMAT_ARGB2101010 },
+ { AV_PIX_FMT_X2RGB10BE, DRM_FORMAT_ARGB2101010 | DRM_FORMAT_BIG_ENDIAN },
+ { AV_PIX_FMT_X2BGR10LE, DRM_FORMAT_ABGR2101010 },
+ { AV_PIX_FMT_X2BGR10BE, DRM_FORMAT_ABGR2101010 | DRM_FORMAT_BIG_ENDIAN },
// 8-bit YUV 4:2:0.
{ AV_PIX_FMT_NV12, DRM_FORMAT_NV12 },
// 8-bit YUV 4:2:2.
--
2.52.0
From 03f2264b1ecf52c96906865ce68b127c46694499 Mon Sep 17 00:00:00 2001
From: Alexander Sorokin <[email protected]>
Date: Sat, 10 Oct 2026 23:56:07 +0300
Subject: [PATCH 2/4] avutil/hwcontext_vulkan: fix R/B swap in DRM 2:10:10:10
format map
The DRM->Vulkan table maps the 2:10:10:10 formats with R and B
exchanged:
DRM_FORMAT_ARGB2101010 -> VK_FORMAT_A2B10G10R10_UNORM_PACK32
DRM_FORMAT_ABGR2101010 -> VK_FORMAT_A2R10G10B10_UNORM_PACK32
and the same for XRGB2101010/XBGR2101010. This contradicts the
vkfmt_map table in this file, which pairs VK_FORMAT_A2R10G10B10 with
AV_PIX_FMT_X2RGB10 (R next to the 2-bit field) - the same layout DRM
calls ARGB2101010 ("[31:0] A:R:G:B 2:10:10:10 little endian" per
drm_fourcc.h).
Where this bites:
* vulkan_map_from_drm_frame_desc() creates the imported image with
this table's VkFormat. On drivers that validate the DRM format
modifier against the format (e.g. NVIDIA 580) the import then fails
outright with "Cannot map DRM frame to Vulkan:
VK_ERROR_FORMAT_NOT_SUPPORTED", which is what happens with ffmpeg
8.0.1 and makes kmsgrab capture of 10-bit desktops (KWin composites
into AB30) impossible. Since 03be5630 ("hwcontext_vulkan: fix
BGRA/BGR0 images") single-plane layers take the VkFormat from
sw_format instead and escape this table, so on master only
multi-plane layers hit it here.
* vulkan_map_to_drm() derives the exported fourcc with the reverse
lookup, so exported DMABUFs are stamped with the R/B-swapped fourcc
(ABGR2101010 memory labelled ARGB2101010) regardless.
Fix the four entries so both tables agree. Verified on an NVIDIA
GT 1030 (driver 580) capturing a 10-bit AB30 KWin desktop: with the
sw_format fallback disabled to let this table decide, the capture fails
before this change and produces correctly coloured frames after it.
Signed-off-by: Alexander Sorokin <[email protected]>
---
libavutil/hwcontext_vulkan.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
index 63d2ab5c43..fd204cce9f 100644
--- a/libavutil/hwcontext_vulkan.c
+++ b/libavutil/hwcontext_vulkan.c
@@ -3527,10 +3527,10 @@ static const struct {
{ DRM_FORMAT_XRGB8888, VK_FORMAT_B8G8R8A8_UNORM },
{ DRM_FORMAT_ABGR8888, VK_FORMAT_R8G8B8A8_UNORM },
{ DRM_FORMAT_XBGR8888, VK_FORMAT_R8G8B8A8_UNORM },
- { DRM_FORMAT_ARGB2101010, VK_FORMAT_A2B10G10R10_UNORM_PACK32 },
- { DRM_FORMAT_ABGR2101010, VK_FORMAT_A2R10G10B10_UNORM_PACK32 },
- { DRM_FORMAT_XRGB2101010, VK_FORMAT_A2B10G10R10_UNORM_PACK32 },
- { DRM_FORMAT_XBGR2101010, VK_FORMAT_A2R10G10B10_UNORM_PACK32 },
+ { DRM_FORMAT_ARGB2101010, VK_FORMAT_A2R10G10B10_UNORM_PACK32 },
+ { DRM_FORMAT_ABGR2101010, VK_FORMAT_A2B10G10R10_UNORM_PACK32 },
+ { DRM_FORMAT_XRGB2101010, VK_FORMAT_A2R10G10B10_UNORM_PACK32 },
+ { DRM_FORMAT_XBGR2101010, VK_FORMAT_A2B10G10R10_UNORM_PACK32 },
// All these DRM_FORMATs were added in the same libdrm commit.
#ifdef DRM_FORMAT_XYUV8888
--
2.52.0
From 0e81dc87a8df84173911db644494f46a32afe11d Mon Sep 17 00:00:00 2001
From: Alexander Sorokin <[email protected]>
Date: Sat, 10 Oct 2026 23:56:24 +0300
Subject: [PATCH 3/4] avutil/hwcontext_drm: refuse non-linear modifiers in CPU
mappings
drm_map_frame() mmaps each DMA-BUF object and exposes it as a plain
linear image, silently ignoring the object's format modifier. When the
buffer is tiled or otherwise non-linear (e.g. the NVIDIA block-linear
scanout buffers used on driver 580, modifier 0x3000000004fe014), the
linear read produces garbage - horizontal stripes - with no diagnostic
at all. Every consumer of transfer_data_from/map_from on such frames
is affected.
There is no generic way to un-tile a buffer on the CPU here, so make the
failure explicit: reject any object whose modifier is neither
DRM_FORMAT_MOD_INVALID nor DRM_FORMAT_MOD_LINEAR and tell the user to
copy through the GPU instead (hwmap=derive_device=vulkan imports the
buffer with its modifier and detiles it). Buffers without modifiers,
including legacy dumb buffers, map exactly as before.
Verified on an NVIDIA GT 1030 (driver 580) capturing a tiled AB30 KWin
desktop: plain hwdownload used to yield striped frames, and now fails
with an error naming the modifier and the workaround; the vulkan
hwmap path is unaffected.
Signed-off-by: Alexander Sorokin <[email protected]>
---
libavutil/hwcontext_drm.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/libavutil/hwcontext_drm.c b/libavutil/hwcontext_drm.c
index 565c02dead..d5905e5c11 100644
--- a/libavutil/hwcontext_drm.c
+++ b/libavutil/hwcontext_drm.c
@@ -31,6 +31,7 @@
#endif
#include <drm.h>
+#include <drm_fourcc.h>
#include <xf86drm.h>
#include "avassert.h"
@@ -150,6 +151,22 @@ static int drm_map_frame(AVHWFramesContext *hwfc,
av_assert0(desc->nb_objects <= AV_DRM_MAX_PLANES);
for (i = 0; i < desc->nb_objects; i++) {
+ uint64_t modifier = desc->objects[i].format_modifier;
+
+ /* Tiled or otherwise non-linear buffers cannot be read through a
+ * linear mapping - doing so silently produces garbled output.
+ * Point the user at a GPU-side copy instead, which imports the
+ * buffer with its modifier and can detile it. */
+ if (modifier != DRM_FORMAT_MOD_INVALID &&
+ modifier != DRM_FORMAT_MOD_LINEAR) {
+ av_log(hwfc, AV_LOG_ERROR, "Cannot map DRM object %d with "
+ "non-linear format modifier %"PRIx64" to system memory. "
+ "Use a GPU-side copy instead, e.g. "
+ "-vf hwmap=derive_device=vulkan,hwdownload.\n",
+ desc->objects[i].fd, modifier);
+ err = AVERROR(ENOSYS);
+ goto fail;
+ }
addr = mmap(NULL, desc->objects[i].size, mmap_prot, MAP_SHARED,
desc->objects[i].fd, 0);
if (addr == MAP_FAILED) {
--
2.52.0
From 03dc45ea4bb6afdf2a0aa8094a04e2540028538b Mon Sep 17 00:00:00 2001
From: Alexander Sorokin <[email protected]>
Date: Sat, 10 Oct 2026 23:56:43 +0300
Subject: [PATCH 4/4] doc: describe capturing non-linear DRM framebuffers with
kmsgrab
The kmsgrab example claimed that a non-linear framebuffer "may be
scrambled or fail to download" on the plain hwdownload path. Since
hwcontext_drm now rejects such buffers with an error, document the
behaviour and give the working recipe: map through Vulkan first, which
imports the buffer with its format modifier and copies it on the GPU.
Signed-off-by: Alexander Sorokin <[email protected]>
---
doc/indevs.texi | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/doc/indevs.texi b/doc/indevs.texi
index dc4b92a0a0..5a0abc8d47 100644
--- a/doc/indevs.texi
+++ b/doc/indevs.texi
@@ -962,12 +962,27 @@ content. Defaults to @code{30}.
@item
Capture from the first active plane, download the result to normal frames and
encode.
-This will only work if the framebuffer is both linear and mappable - if not,
the result
-may be scrambled or fail to download.
+This will only work if the framebuffer is both linear and mappable - if it has
a
+non-linear format modifier (for example a tiled layout, as used by some GPU
drivers),
+the download will fail with an error rather than producing scrambled output.
In that
+case, copy the frame through the GPU instead, as shown below.
@example
ffmpeg -f kmsgrab -i - -vf 'hwdownload,format=bgr0' output.mp4
@end example
+@item
+Capture from the first active plane and copy the frame through Vulkan before
+downloading. Mapping the buffer through Vulkan imports it with its format
modifier,
+so this works regardless of the memory layout, and is the reliable way to
capture
+compositors that render into tiled or otherwise non-linear buffers (for example
+some Wayland compositors on GPU-accelerated systems). Requires FFmpeg to be
built
+with Vulkan support. Note that @samp{hwdownload} can only produce the
framebuffer's
+own format (see the @option{format} option above - for example
@samp{x2bgr10le} for
+10-bit framebuffers), so any further conversion goes in a separate filter step.
+@example
+ffmpeg -f kmsgrab -i - -vf 'hwmap=derive_device=vulkan,hwdownload,format=bgr0'
output.mp4
+@end example
+
@item
Capture from CRTC ID 42 at 60fps, map the result to VAAPI, convert to NV12 and
encode as H.264.
@example
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
