Public bug reported:

With gnome-remote-desktop in remote-login (headless) mode, gnome-shell
crashes with SIGSEGV when the RDP client changes the virtual monitor to
a size that is not an exact multiple of the scale factor. The whole user
session is terminated, including all running applications.

The core dump shows that clutter_stage_paint_to_buffer() reads back a
5099×2033 texture into a PipeWire buffer that is only 5098×2032 (stride
20392 = 5098 × 4). The read-back writes one column and one row past the
end of the buffer; the actual fault happens inside Mesa's llvmpipe
(libgallium), but Mesa only executes the oversized read-back it is
given.

Environment
Ubuntu 26.04.1 LTS, kernel 7.0.0-43-generic, x86_64
Hyper-V guest, no GPU: hyperv_drm, rendering via llvmpipe
gnome-shell 50.1-0ubuntu1.5
mutter 50.1-0ubuntu2.5
gnome-remote-desktop 50.2-0ubuntu0.1 (system service, remote login, headless 
session)
FreeRDP 3.32.2+dfsg-0ubuntu0.26.04.1
Mesa 26.0.8-1ubuntu0.3 (crash dump below); the crash also reproduces with Mesa 
26.2.4-1ubuntu1~26.04.1 from resolute-proposed (no dump captured for that run)
RDP client:

Microsoft Windows App 2.0.1489.0 and 2.0.1601.0 (both reproduce), Windows 11 
Enterprise build 26300
Windows desktop: 5120×2160 at 150 % scaling (144 DPI)
RDPGFX capability set 10.7 accepted, H.264 AVC444/AVC420 available
Steps to reproduce
Enable remote login in gnome-remote-desktop (system mode).
Connect with Windows App in full-screen mode from a Windows desktop of 
5120×2160 at 150 %. The session gets virtual monitor Meta-0 with mode 
5120×2160, scale 1.5. This is stable.
Leave full-screen mode in Windows App. The client sends a new monitor layout 
and the virtual monitor changes to 5098×2032, scale 1.5.
gnome-shell crashes 1–15 s later with SIGSEGV; the session is logged out and 
all applications are closed.
Observed monitor states (from org.gnome.Mutter.DisplayConfig.GetCurrentState 
during the test):

Time    Mode    Logical monitor scale
before  5120×2160@60    1.5
after leaving full screen       5098×2032@60    1.5
~1 s later      (none, gnome-shell crashed)     
Expected result
The virtual monitor is resized and the session keeps running.

Analysis (from the core dump, symbols via debuginfod.ubuntu.com)
Frame 15, meta_screen_cast_stream_src_record_to_buffer: width=5098, 
height=2032, stride=20392

Frame 14, meta_screen_cast_virtual_stream_src_record_to_buffer:
view_rect = {x = 0, y = 0, width = 3399, height = 1355}, scale = 1.5

Frame 13, clutter_stage_paint_to_buffer: rect = {0, 0, 3399, 1355},
scale = 1.5, texture_width = 5099, texture_height = 2033, destination
bitmap COGL_PIXEL_FORMAT_BGRA_8888_PRE, width 5099, height 2033

So the logical view size is rounded up (5098 / 1.5 = 3398.67 → 3399,
2032 / 1.5 = 1354.67 → 1355), and scaling it back gives a texture of
ceil(3399 × 1.5) × ceil(1355 × 1.5) = 5099 × 2033, which is larger than
the 5098 × 2032 buffer it is read into.

With 5120×2160 the round trip fits (3413 × 1.5 = 5119.5 → 5120; 1440 ×
1.5 = 2160), which is why full screen works.

This looks like the same class of problem that was fixed for area
screencasts in 2020 ("clutter/stage: Make paint_to_buffer use the
correct texture/buffer size"), but for the virtual stream source used by
gnome-remote-desktop.

Kernel message for one of the crashes (same Mesa offset in all dumps):
gnome-shell[…]: segfault at … error 6 in
libgallium-26.0.8-1ubuntu0.3.so[3b47e1,…]

Full symbolized backtrace: see attached backtrace.txt.

Workaround
Keep the RDP client in full-screen mode, use a fixed resolution, or use a 
client scale factor that divides the window size exactly (100 % or 200 %).

** Affects: mutter (Ubuntu)
     Importance: Undecided
         Status: New

** Attachment added: "backtrace.txt"
   
https://bugs.launchpad.net/bugs/2175114/+attachment/6019572/+files/backtrace.txt

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2175114

Title:
  gnome-shell SIGSEGV in
  meta_screen_cast_virtual_stream_src_record_to_buffer when an RDP
  client resizes a remote-login session with fractional scale 1.5

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mutter/+bug/2175114/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to