On Tue, Apr 21, 2026, at 11:18, Peng Yang wrote:
> On 4/21/2026 3:38 PM, Arnd Bergmann wrote:
>>
>> Which host implementation do you use? The way the virtio_console
>> driver works really assumes that virtqueue_kick() consumes the
>> buffer synchronously. Even though that is not how virtio is
>> specified, this does tend to work. ;-)
>>
> We are using crosvm as the host VMM with its virtio-console backend,
> running on Android. The trigger is Android host reboot/shutdown: when
> Android initiates a reboot, the crosvm process exits and tears down
> the virtio-console backend. At that point, the TX virtqueue is no
> longer being drained by the host and will never be consumed again.
I see, so the normal behavior is likely just fine, but the error
handling is what goes wrong. Maybe there is a way for the guest
to detect the device being turn down already so it does not
actually have to wait any more?
> The crash dump from the actual failure confirms the exact deadlock
> scenario:
>
> Core 3 holds outvq_lock and spins forever in virtqueue_get_buf waiting
> for the host to consume the buffer:
>
> virtqueue_get_buf
> __send_to_port
> put_chars
> hvc_push
> hvc_write
> n_tty_write
> <- writev() syscall
This current loop here is
while (!virtqueue_get_buf(out_vq, &len)
&& !virtqueue_is_broken(out_vq))
cpu_relax();
which looks like the virtqueue_is_broken() check is meant to
catch this exact case. Do you know why this does not break
out of the loop after crosvm tears down the virtio-console
device?
> Core 0 has a watchdog bark ISR fire and attempts printk, holds the
> console lock, but spins on _raw_spin_lock_irqsave waiting to acquire
> outvq_lock:
>
> queued_spin_lock_slowpath
> _raw_spin_lock_irqsave
> __send_to_port
> put_chars
> hvc_console_print
> console_flush_all
> console_unlock
> vprintk_emit
> <- printk (watchdog bark handler)
My first thought here was that __send_to_port() should perhaps
release the lock during the while() loop, which should avoid
blocking the other threads on the spin_lock_irqsave() but
would not avoid blocking on the loop.
> The 200 ms timeout is intended as a minimal, targeted workaround to prevent
> the watchdog bite in our specific scenario. We are open to suggestions on a
> better long-term approach.
Not sure how to do it, but I think finding a way to call
virtio_break_device() at the point the host device goes away is
the best solution here. Ideally there would just be a notification
from the host, but since __send_to_port() may be called with
interrupts disabled and may be running on the only CPU, that
would still be unreliable.
Maybe there is a way for virtio_console to read a status
register in the virtio config that tells it whether the
host has turned it off? I was thinking vdev->config->get_status(vdev)
but that seems to only get updated by the guest.
Arnd
