Hello,

I've attached a debdiff for noble-security (lp2172882_noble-
security_on-1.20.debdiff).

- Based on 1:8.2.2+ds-0ubuntu1.20 (noble-proposed), version
1:8.2.2+ds-0ubuntu1.21, as the earlier qemu security uploads were built
on top of the SRU in -proposed.

1.20 carries four SRU fixes that are still verification-needed-noble
(LP: #2156307, #2158673, #2166503, #2168803).

Thanks.

** Patch added: "lp2172882_noble-security_on-1.20.debdiff"
   
https://bugs.launchpad.net/bugs/2172882/+attachment/6019602/+files/lp2172882_noble-security_on-1.20.debdiff

** Description changed:

+ [Impact]
+ 
+ On Noble, QEMU aborts when an NBD client disconnects from an export whose 
disk runs in an IOThread, and the guest is killed.
+ This happens on every libvirt pull-mode backup (virsh backup-begin) of a 
domain that uses IOThreads on virtio-blk or virtio-scsi:
+ 
+     qemu-system-x86_64: blockdev-nbd.c:62: nbd_blockdev_client_closed:
+ Assertion `qemu_in_main_thread() && nbd_server' failed.
+ 
+ It is a regression from the CVE-2024-7409 fix in 1:8.2.2+ds-0ubuntu1.10 
(USN-7744-1).
+ d/p/CVE-2024-7409-3.patch assumes the NBD client close callback runs in the 
main loop, but in 8.2.2 nbd_trip() calls it from the export's IOThread.
+ The fix backports the two upstream commits that move it to the main loop.
+ 
+ [Test Plan]
+ 
+ On a Noble host with KVM (no guest OS needed):
+ 
+     sudo apt install -y qemu-system-x86 qemu-utils libvirt-daemon-system
+     sudo qemu-img create -f qcow2 /var/lib/libvirt/images/lp2172882.qcow2 1G
+     sudo qemu-io -f qcow2 -c 'write -P 0xab 0 64M' 
/var/lib/libvirt/images/lp2172882.qcow2
+     echo "<domain type='kvm'><name>lp2172882</name><memory 
unit='MiB'>256</memory><vcpu>1</vcpu><iothreads>1</iothreads><os><type 
arch='x86_64' machine='q35'>hvm</type></os><devices><disk type='file' 
device='disk'><driver name='qemu' type='qcow2' iothread='1'/><source 
file='/var/lib/libvirt/images/lp2172882.qcow2'/><target dev='vda' 
bus='virtio'/></disk></devices></domain>" > lp2172882.xml
+     sudo virsh define lp2172882.xml
+     sudo virsh start lp2172882
+     # AppArmor only lets QEMU create the socket in its per-domain directory
+     SOCK=/var/lib/libvirt/qemu/domain-$(sudo virsh domid 
lp2172882)-lp2172882/backup.sock
+     echo "<domainbackup mode='pull'><server transport='unix' 
socket='$SOCK'/><disks><disk name='vda' backup='yes' type='file'><scratch 
file='/var/lib/libvirt/images/lp2172882-scratch.qcow2'/></disk></disks></domainbackup>"
 > backup.xml
+     sudo virsh backup-begin lp2172882 backup.xml
+     sudo qemu-img convert -f raw -O qcow2 "nbd+unix:///vda?socket=$SOCK" 
/tmp/lp2172882-out.qcow2
+     sleep 2
+     sudo virsh domstate lp2172882
+     sudo tail -2 /var/log/libvirt/qemu/lp2172882.log
+ 
+ Before the fix, the domain is "shut off" and the log shows the assertion 
above with "shutting down, reason=crashed".
+ After the fix, virsh domstate must print "running", the log must have no 
assertion, and "sudo virsh domjobabort lp2172882" must succeed with the domain 
still running.
+ 
+ With the fix, ending the backup job while a client is still connected must 
also keep the domain running.
+ This is the CVE-2024-7409 path where nbd-server-stop closes the remaining 
clients:
+ 
+     sudo virsh backup-begin lp2172882 backup.xml
+     sudo qemu-io -r -f raw -c 'sleep 8000' "nbd+unix:///vda?socket=$SOCK" &
+     sleep 2
+     sudo virsh domjobabort lp2172882
+     sudo virsh domstate lp2172882
+ 
+ [Where problems could occur]
+ 
+ A regression could show up as an abort on the main-thread assert the fix adds 
to nbd_client_put() if another caller still runs in an IOThread, or as 
nbd-server-stop or block-export-del waiting if a disconnecting client is not 
released.
+ Upstream iotests 281 and 307 (IOThread exports) fail with the current package 
and pass with the fix, and the -nbd iotests pass.
+ 
+ [Original Description]
+ 
  Since the CVE-2024-7409 security update (1:8.2.2+ds-0ubuntu1.10), QEMU
  on Noble aborts when an NBD client disconnects from an export whose
  block node runs in an IOThread. This is the normal flow of a libvirt
  pull-mode backup (virsh backup-begin / virDomainBackupBegin) of any VM
  that uses IOThreads (<iothreads> with <driver iothread=...> on virtio-
  blk, or <controller type='scsi' model='virtio-scsi'><driver
  iothread=.../>). The guest is killed:
  
    qemu-system-x86_64: blockdev-nbd.c:62: nbd_blockdev_client_closed: 
Assertion `qemu_in_main_thread() && nbd_server' failed.
    ... shutting down, reason=crashed
  
  Backtrace of the aborting thread (stock 1:8.2.2+ds-0ubuntu1.18, symbols
  from debuginfod; this is an IOThread, not the main thread: the main
  thread is LWP 7356 = PID; full backtrace attached):
  
    #5 __assert_fail_base (assertion="qemu_in_main_thread() && nbd_server", 
file="blockdev-nbd.c", line=62, function="nbd_blockdev_client_closed")
    #7 nbd_blockdev_client_closed (client=...) at blockdev-nbd.c:62
    #8 client_close (negotiated=true, client=...) at nbd/server.c:1550
    #10 nbd_trip (opaque=...) at nbd/server.c:3029
    #11 coroutine_trampoline () at util/coroutine-ucontext.c:175
  
  [Cause]
  
  debian/patches/CVE-2024-7409-3.patch (upstream 3e7ef738c8, "nbd/server:
  CVE-2024-7409: Close stray clients at server-stop") adds
  assert(qemu_in_main_thread() && nbd_server) to
  nbd_blockdev_client_closed(). Its commit message states that it "relies
  heavily on the fact that nbd/server.c guarantees that it only calls
  nbd_blockdev_client_closed() from the main loop".
  
  That guarantee was only introduced upstream in the QEMU 9.0 cycle
  (f816310d0c32 "nbd/server: only traverse NBDExport->clients from main
  loop thread", together with 7075d235114b, efade66d5872 and
  9c707525cbb1). Noble's 8.2.2 does not have it. In 8.2.2, nbd_trip()
  calls client_close() from the export's AioContext, so for an IOThread-
  bound export the close callback runs on the IOThread and the new
  assertion fails.
  
  Taking the upstream stable-8.2 versions of those commits (aee1039c,
  13fc21ae, a69a002d, as in qemu v8.2.3+) does not help. Both a Noble
  build with them added and unmodified upstream v8.2.7 abort instead in
  nbd_client_put(): Assertion `qemu_in_main_thread()' failed, and the
  client gets block-status EIO. Unmodified upstream 11.1.3 is not
  affected.
  
  Affected: 1:8.2.2+ds-0ubuntu1.10 through 1:8.2.2+ds-0ubuntu1.16 (noble-
  security), 1.18 (noble-updates) and 1.19 (noble-proposed); none of their
  changelogs touch NBD. Not affected: 1.9 and earlier, which are
  vulnerable to CVE-2024-7409.
  
  Reproduced on:
  - a clean Ubuntu 24.04.5 VM: qemu-system-x86 1:8.2.2+ds-0ubuntu1.18, 
libvirt-daemon-system 10.0.0-2ubuntu8.19, kernel 6.8.0-147-generic;
  - a 24.04 host with qemu 1:8.2.2+ds-0ubuntu1.16 and libvirt 
10.0.0-2ubuntu8.13.
  
  Both reproduce with virtio-blk + IOThread, virtio-scsi + IOThread, and a
  two-disk domain (crash when the first disk's client disconnects while
  the second export is still open).
  
  [Impact]
  
  - Any Noble host that takes NBD backups of running VMs whose disks use 
IOThreads loses the VM on every backup. That covers libvirt pull-mode backup, 
and therefore backup products built on it.
  - IOThreads are a common performance configuration for virtio-blk and 
virtio-scsi, so users must currently choose between IOThreads and being able to 
back up the VM.
  - The crash is a regression introduced by a security update.
  
  [Test Plan]
  
  The attached nbd_iothread_repro.sh automates all of this (run as root on
  a Noble libvirt host; no guest OS needed):
  
    bash nbd_iothread_repro.sh setup
    bash nbd_iothread_repro.sh start blk     # virtio-blk disk with <driver 
iothread='1'/>
    bash nbd_iothread_repro.sh repro blk     # pull backup, qemu-img reads the 
export and disconnects
    bash nbd_iothread_repro.sh start scsi; bash nbd_iothread_repro.sh repro scsi
    bash nbd_iothread_repro.sh start multi; bash nbd_iothread_repro.sh repro 
multi
  
  Manual steps:
  
  1. Create a disk:
     qemu-img create -f qcow2 disk.qcow2 2G; qemu-io -f qcow2 -c 'write -P 0x5a 
0 256M' disk.qcow2
  2. Start a transient domain with <iothreads>1</iothreads> and the disk on 
virtio with <driver name='qemu' type='qcow2' cache='none' io='native' 
iothread='1'/>.
  3. Begin a pull backup:
     <domainbackup mode='pull'><server transport='unix' 
socket='SOCK'/><disks><disk name='vda' backup='yes' type='file'><scratch 
file='SCRATCH'/></disk></disks></domainbackup>
     virsh backup-begin DOM backup.xml
     Under the default AppArmor profile, put SOCK and SCRATCH in the domain's 
/var/lib/libvirt/qemu/domain-<id>-<name>/ directory. Paths such as /var/tmp get 
EACCES at nbd-server-start; the script does this.
  4. Read the export and disconnect:
     qemu-img convert -f raw -O qcow2 'nbd+unix:///vda?socket=SOCK' out.qcow2
  5. Expected: the domain keeps running, and virsh domjobabort DOM ends the job.
     Actual: the domain is gone, and /var/log/libvirt/qemu/DOM.log shows the 
assertion and reason=crashed.
  
  Regression checks for the CVE fix, with the fixed package:
  - "bash nbd_iothread_repro.sh conns blk 150" runs 150 connect/disconnect 
cycles within one job. All must succeed, and the final read must work (no leak 
of the connection counter against the default cap of 100).
  - "bash nbd_iothread_repro.sh stray blk" opens a client that never 
negotiates, then ends the job. The stray client must be closed promptly and the 
domain must keep running.
  - "bash nbd_iothread_repro.sh repeat blk|scsi|multi 50" runs repeated full 
backup cycles.
  
  Results with the attached patch applied to 1:8.2.2+ds-0ubuntu1.16:
  - read-and-disconnect passes on virtio-blk, virtio-scsi and the two-disk 
domain;
  - repeat: 150/150 cycles;
  - conns: 150 connect/disconnect cycles, no failures;
  - stray: client closed in 0.01 s, domain still running.
  
  [Where problems could occur]
  
  - The patch changes only blockdev-nbd.c:nbd_blockdev_client_closed(), the 
close callback for NBD servers started with nbd-server-start (QMP/HMP).
  - The connection-list and counter bookkeeping added by the CVE fix still runs 
only in the main loop. It runs directly when the client closes on the main 
thread, and otherwise through a one-shot bottom half on the main AioContext.
  - nbd_server_free() already waits with AIO_WAIT_WHILE_UNLOCKED() for the 
counter to reach zero, so pending bottom halves run before the server is freed.
  - The nbd_client_put() reference drop stays on the thread that closed the 
client. That is exactly the behaviour of 1:8.2.2+ds-0ubuntu1.9 and of upstream 
8.2.2.
  - qemu-nbd uses its own close callback and is unchanged.
  - If something went wrong, the risks would be a regression in the 
CVE-2024-7409 stray-client handling (for example nbd-server-stop hanging if the 
counter were not decremented), or exhaustion of the connection limit. The conns 
and stray checks above cover both.
  
  [Other Info]
  
  - The patch is Noble-specific. Upstream master already guarantees main-loop 
close through the 9.0 rework, which is too large to backport. The upstream 
stable-8.2 line has the related failure described above.
  - Debugging note: apport ignores crashes in libvirt's private mount namespace 
("crashed in a separate mount namespace, ignoring"). To capture the .crash, set 
namespaces = [ ] and max_core = "unlimited" in /etc/libvirt/qemu.conf on the 
test host.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172882

Title:
  qemu 1:8.2.2+ds-0ubuntu1.10 and later aborts in
  nbd_blockdev_client_closed() when an NBD client of an IOThread-bound
  export disconnects (CVE-2024-7409 backport assumes main-loop close)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2172882/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to