On Tue, Dec 24, 2013 at 02:09:07PM +1100, Alexey Kardashevskiy wrote:
> On 12/24/2013 03:24 AM, Michael S. Tsirkin wrote:
> > On Mon, Dec 23, 2013 at 02:01:13AM +1100, Alexey Kardashevskiy wrote:
> >> On 12/23/2013 01:46 AM, Alexey Kardashevskiy wrote:
> >>> On 12/22/2013 09:56 PM, Michael S. Tsirkin wrote:
> >>>> On Sun, Dec 22, 2013 at 02:01:23AM +1100, Alexey Kardashevskiy wrote:
> >>>>> Hi!
> >>>>>
> >>>>> I am having a problem with virtio-net + vhost on POWER7 machine - it
> >>>>> does
> >>>>> not survive reboot of the guest.
> >>>>>
> >>>>> Steps to reproduce:
> >>>>> 1. boot the guest
> >>>>> 2. configure eth0 and do ping - everything works
> >>>>> 3. reboot the guest (i.e. type "reboot")
> >>>>> 4. when it is booted, eth0 can be configured but will not work at all.
> >>>>>
> >>>>> The test is:
> >>>>> ifconfig eth0 172.20.1.2 up
> >>>>> ping 172.20.1.23
> >>>>>
> >>>>> If to run tcpdump on the host's "tap-id3" interface, it shows no trafic
> >>>>> coming from the guest. If to compare how it works before and after
> >>>>> reboot,
> >>>>> I can see the guest doing an ARP request for 172.20.1.23 and receives
> >>>>> the
> >>>>> response and it does the same after reboot but the answer does not come.
> >>>>
> >>>> So you see the arp packet in guest but not in host?
> >>>
> >>> Yes.
> >>>
> >>>
> >>>> One thing to try is to boot debug kernel - where pr_debug is
> >>>> enabled - then you might see some errors in the kernel log.
> >>>
> >>> Tried and added lot more debug printk myself, not clear at all what is
> >>> happening there.
> >>>
> >>> One more hint - if I boot the guest and the guest does not bring eth0 up
> >>> AND wait more than 200 seconds (and less than 210 seconds), then eth0 will
> >>> not work at all. I.e. this script produces not-working-eth0:
> >>>
> >>>
> >>> ifconfig eth0 172.20.1.2 down
> >>> sleep 210
> >>> ifconfig eth0 172.20.1.2 up
> >>> ping 172.20.1.23
> >>>
> >>> s/210/200/ - and it starts working. No reboot is required to reproduce.
> >>>
> >>> No "vhost" == always works. The only difference I can see here is vhost's
> >>> thread which may get suspended if not used for a while after the start and
> >>> does not wake up but this is almost a blind guess.
> >>
> >>
> >> Yet another clue - this host kernel patch seems to help with the guest
> >> reboot but does not help with the initial 210 seconds delay:
> >>
> >> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
> >> index 69068e0..5e67650 100644
> >> --- a/drivers/vhost/vhost.c
> >> +++ b/drivers/vhost/vhost.c
> >> @@ -162,10 +162,10 @@ void vhost_work_queue(struct vhost_dev *dev, struct
> >> vhost_work *work)
> >> list_add_tail(&work->node, &dev->work_list);
> >> work->queue_seq++;
> >> spin_unlock_irqrestore(&dev->work_lock, flags);
> >> - wake_up_process(dev->worker);
> >> } else {
> >> spin_unlock_irqrestore(&dev->work_lock, flags);
> >> }
> >> + wake_up_process(dev->worker);
> >> }
> >> EXPORT_SYMBOL_GPL(vhost_work_queue);
> >>
> >>
> >
> > Interesting. Some kind of race? A missing memory barrier somewhere?
>
> I do not see how. I boot the guest and just wait 210 seconds, nothing
> happens to cause races.
>
>
> > Since it's all around startup,
> > you can try kicking the host eventfd in
> > vhost_net_start.
>
>
> How exactly? This did not help. Thanks.
>
> diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
> index 006576d..407ecf2 100644
> --- a/hw/net/vhost_net.c
> +++ b/hw/net/vhost_net.c
> @@ -229,6 +229,17 @@ int vhost_net_start(VirtIODevice *dev, NetClientState
> *ncs,
> if (r < 0) {
> goto err;
> }
> +
> + VHostNetState *vn = tap_get_vhost_net(ncs[i].peer);
> + struct vhost_vring_file file = {
> + .index = i
> + };
> + file.fd =
> event_notifier_get_fd(virtio_queue_get_host_notifier(dev->vq));
> + r = ioctl(vn->dev.control, VHOST_SET_VRING_KICK, &file);
No, this sets the notifier, it does not kick.
To kick you write 1 there:
uint6_t v = 1;
write(fd, &v, sizeof v);
> + if (r) {
> + error_report("Error notifiyng host notifier: %d", -r);
> + goto err;
> + }
> }
>
>
>
> >
> >>
> >>
> >>>>> If to remove vhost=on, it is all good. If to try Fedora19
> >>>>> (v3.10-something), it all good again - works before and after reboot.
> >>>>>
> >>>>>
> >>>>> And there 2 questions:
> >>>>>
> >>>>> 1. does anybody have any clue what might go wrong after reboot?
> >>>>>
> >>>>> 2. Is there any good material to read about what exactly and how vhost
> >>>>> accelerates?
> >>>>>
> >>>>> My understanding is that packets from the guest to the real network are
> >>>>> going as:
> >>>>> 1. guest's virtio-pci-net does ioport(VIRTIO_PCI_QUEUE_NOTIFY)
> >>>>> 2. QEMU's net/virtio-net.c calls qemu_net_queue_deliver()
> >>>>> 3. QEMU's net/tap.c calls tap_write_packet() and this is how the host
> >>>>> knows
> >>>>> that there is a new packet.
> >>>
> >>>
> >>> What about the documentation? :) or the idea?
> >>>
> >>>
> >>>>>
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>>
> >>>>> This how I run QEMU:
> >>>>> ./qemu-system-ppc64 \
> >>>>> -enable-kvm \
> >>>>> -m 2048 \
> >>>>> -machine pseries \
> >>>>> -initrd 1.cpio \
> >>>>> -kernel vml312_virtio_net_dbg \
> >>>>> -nographic \
> >>>>> -vga none \
> >>>>> -netdev
> >>>>> tap,id=id3,ifname=tap-id3,script=ifup.sh,downscript=ifdown.sh,vhost=on \
> >>>>> -device virtio-net-pci,id=id4,netdev=id3,mac=C0:41:49:4b:00:00
> >>>>>
> >>>>>
> >>>>> That is bridge config:
> >>>>> [aik@dyn232 ~]$ brctl show
> >>>>> bridge name bridge id STP enabled interfaces
> >>>>> brtest 8000.00145e992e88 no pin eth4
> >>>>>
> >>>>>
> >>>>> The ifup.sh script:
> >>>>> ifconfig $1 hw ether ee:01:02:03:04:05
> >>>>> /sbin/ifconfig $1 up
> >>>>> /usr/sbin/brctl addif brtest $1
> >>>
> >>>
> >>
> >>
> >> --
> >> Alexey
>
>
> --
> Alexey
