Public bug reported:

  [ Impact ]

  On Ubuntu 26.04 (mutter 50.1-0ubuntu2.5), a screen cast of a virtual monitor 
(org.gnome.Mutter.ScreenCast.Session.RecordVirtual) makes gnome-shell write 
past the end of its PipeWire buffer on every frame when
  the negotiated height isn't a whole multiple of the monitor scale. 
gnome-shell then crashes with SIGSEGV, and the user's session is lost.

  Why: mutter 50 lays monitors out in logical pixels. A 1800x951 virtual 
monitor at scale 2 gets a logical size of 900 x roundf(951 / 2) = 900x476. 
meta_screen_cast_virtual_stream_src_record_to_buffer() ignores
  the buffer's own width and height and calls clutter_stage_paint_to_buffer(), 
which sizes both its texture and the CoglBitmap that wraps the buffer from the 
view: 476 x 2 = 952 rows. The MemFd buffer mutter
  allocated holds 951 rows (stride 7200 x 951), so 
cogl_framebuffer_read_pixels_into_bitmap() writes one row, 7200 bytes, past its 
end on every frame.

  What happens next depends on what is mapped after the buffer:
  - If the next page is unmapped, gnome-shell gets SIGSEGV in the GL driver.
  - If it is mapped (another screen-cast buffer, or a pipewire-memfd), that 
memory is corrupted silently. gnome-shell can then fail later somewhere else. 
One of our crashes was a SIGSEGV with si_code SI_KERNEL
  and no fault in the GL driver; we think it was such a later failure.

  It happens whenever height mod scale >= scale / 2: an odd height at scale 2, 
height mod 3 = 2 at scale 3, height mod 4 = 2 at scale 4. An odd width alone 
overruns by only a few bytes, which normally stays
  inside the last page.

  Who hits it: anything that creates a virtual monitor through RecordVirtual 
and sizes it to something outside its control, typically a remote-desktop 
server that follows the remote client's window size. We hit
  it with a remote-desktop server on Ubuntu 26.04. The scale usually comes from 
the global org.gnome.desktop.interface scaling-factor (HiDPI clients). Settings 
> Displays would not offer 200% for 1800x951, but
  the global scaling-factor isn't checked against the monitor's mode.

  [ Steps to reproduce ]

  The attached script, repro-virtual-odd-height.py, needs python3-gi, 
gstreamer1.0-tools and gstreamer1.0-pipewire. It runs 6 casts of 10 seconds 
each, one after another. Each cast creates a virtual monitor
  through RecordVirtual, pulls WIDTH x HEIGHT frames with gst-launch-1.0 
pipewiresrc, and moves the pointer over that monitor through 
org.gnome.Mutter.RemoteDesktop so frames keep coming. It prints PASS if
  gnome-shell survives all 6 casts, otherwise FAIL. It doesn't start 
gnome-shell itself; if none is running on the user's session bus, it prints 
ERROR and stops.

  A single cast overruns on every frame but often doesn't fault, because
the overflow lands in a mapped neighbour. That is why the script repeats
the cast. In our tests every run crashed by the 3rd cast.

  The steps run a headless gnome-shell, so no display is needed. As
root:

  1. Create a scratch user and start its user session:
       useradd -m repro-test
       loginctl enable-linger repro-test
       U=$(id -u repro-test)
       systemctl start user@$U.service
       RU="runuser -u repro-test -- env XDG_RUNTIME_DIR=/run/user/$U 
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$U/bus"
  2. Set scale 2 and start a headless gnome-shell:
       $RU gsettings set org.gnome.desktop.interface scaling-factor 2
       $RU systemd-run --user --unit=repro-shell 
--setenv=XDG_SESSION_TYPE=wayland gnome-shell --wayland --headless --no-x11 
--virtual-monitor 1280x720
       sleep 12
  3. Run the script:
       $RU python3 /path/to/repro-virtual-odd-height.py 1800 951

  Actual result: the script prints FAIL, because gnome-shell crashed. 
journalctl -k usually shows "gnome-shell[...]: segfault at ... error 6 in 
libgallium-...so" (we also saw error 7, and some crashes left no
  kernel line).
  Expected result: the script prints PASS, and there is no gnome-shell segfault.

  Other sizes and scales:
  - Scales 3 and 4 crash too: set scaling-factor to 3 or 4, restart gnome-shell 
($RU systemctl --user restart repro-shell), wait about 12 seconds, and run the 
script with 1800 950.
  - These don't crash: 1800 950 at scale 2, and 1800 951 at scale 1.

  Clean up:
       $RU gsettings reset org.gnome.desktop.interface scaling-factor
       loginctl disable-linger repro-test
       systemctl stop user@$U.service
       userdel -r repro-test

  [ Upstream fix ]

  Fixed upstream by commit 93a178ea "screen-cast/stream-src: Add 
meta_screen_cast_stream_src_paint_to_buffer" 
(https://gitlab.gnome.org/GNOME/mutter/-/commit/93a178eaf501c881994e86b1f5246b9a96d3581e,
 MR
  https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/5046), on main since 
2026-06-09. It is in 51.alpha and later, so the development series (tenacious, 
mutter 51.0-1ubuntu3) and Ubuntu 26.10 already have
  it.

  It was backported to the gnome-50 branch as da684eeb 
(https://gitlab.gnome.org/GNOME/mutter/-/commit/da684eebaac1d8a3fce1928b19829419c8b00778,
 2026-10-26), but no 50.x release contains it yet; 50.6 was tagged
  on 2026-10-13. Please consider backporting da684eeb to 26.04. It applies to 
50.1-0ubuntu2.6 with line offsets only.

  The new helper paints the view into an offscreen framebuffer of the buffer's 
own width x height, then reads exactly width x height back into the buffer. 
Rounding of the logical size can then only change which
  pixels get drawn, never how many bytes get written. The commit message 
describes it as a refactor and CPU saving, and doesn't mention the overflow.

  [ Other Info ]

  Tested on Ubuntu 26.04: a Xen VM with 2 vCPUs, no GPU (Mesa 26.0.3
llvmpipe), kernel 7.0.0-42 and gnome-shell 50.1-0ubuntu1.5, using the
steps above.

  Stock libmutter-18-0 50.1-0ubuntu2.5 — the reproducer printed FAIL in 7 of 7 
runs:
  - scale 2 at 1800x951: 3/3, in the 2nd cast;
  - scale 3 at 1800x950: 2/2, in the 2nd cast;
  - scale 4 at 1800x950: 2/2, in the 2nd and 3rd casts.

  Six of those crashes were the same fault: "segfault ... error 6/7 in 
libgallium-26.0.3-1ubuntu1.so" at offset 0x3b4b21. The seventh was the 
SI_KERNEL SIGSEGV mentioned above. A uprobe on
  clutter_stage_paint_to_buffer confirmed the overrun on every frame: 900x476 
at scale 2 into 951 rows, 600x317 at scale 3 into 950, 450x238 at scale 4 into 
950.

  With da684eeb applied to 50.1-0ubuntu2.6 (built locally, package tests not 
run), the reproducer printed PASS in 10 of 10 runs of 6 casts each: scale 2 at 
1800x951 4/4, scale 3 at 1800x950 2/2, scale 4 at
  1800x950 2/2, and the two sizes that don't crash 2/2. There were no 
gnome-shell segfaults, and the cast frames showed complete, correct pictures.

  We first saw this as gnome-shell crashes under a remote-desktop server 
driving RecordVirtual (2200x1151 and 1800x951 at scale 2, both crashing within 
about 20 s). The stack in that case was libgallium <-
  libmutter-cogl cogl_framebuffer_read_pixels_into_bitmap <- 
clutter_stage_paint_to_buffer <- 
meta_screen_cast_virtual_stream_src_record_to_buffer <- 
clutter_stage_paint_view <- clutter_frame_clock_dispatch.
  The faulting address was the exact end of the /memfd:mutter-screen-cast-memfd 
mapping.

  The change in 50.1-0ubuntu2.6 (multi-GPU scanout) doesn't touch this
path.

** Affects: mutter (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: resolute

** Attachment added: "repro-virtual-odd-height.py"
   
https://bugs.launchpad.net/bugs/2174817/+attachment/6019054/+files/repro-virtual-odd-height.py

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2174817

Title:
  gnome-shell crashes when a virtual-monitor screen cast's size isn't a
  multiple of the scale

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mutter/+bug/2174817/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to