On 01/08/2014 12:18 AM, Alexey Kardashevskiy wrote:
> On 12/25/2013 02:43 AM, Michael S. Tsirkin wrote:
>> On Wed, Dec 25, 2013 at 01:15:29AM +1100, Alexey Kardashevskiy wrote:
>>> On 12/24/2013 08:40 PM, Michael S. Tsirkin wrote:
>>>> On Tue, Dec 24, 2013 at 02:09:07PM +1100, Alexey Kardashevskiy wrote:
>>>>> On 12/24/2013 03:24 AM, Michael S. Tsirkin wrote:
>>>>>> On Mon, Dec 23, 2013 at 02:01:13AM +1100, Alexey Kardashevskiy wrote:
>>>>>>> On 12/23/2013 01:46 AM, Alexey Kardashevskiy wrote:
>>>>>>>> On 12/22/2013 09:56 PM, Michael S. Tsirkin wrote:
>>>>>>>>> On Sun, Dec 22, 2013 at 02:01:23AM +1100, Alexey Kardashevskiy wrote:
>>>>>>>>>> Hi!
>>>>>>>>>>
>>>>>>>>>> I am having a problem with virtio-net + vhost on POWER7 machine - it
>>>>>>>>>> does
>>>>>>>>>> not survive reboot of the guest.
>>>>>>>>>>
>>>>>>>>>> Steps to reproduce:
>>>>>>>>>> 1. boot the guest
>>>>>>>>>> 2. configure eth0 and do ping - everything works
>>>>>>>>>> 3. reboot the guest (i.e. type "reboot")
>>>>>>>>>> 4. when it is booted, eth0 can be configured but will not work at
>>>>>>>>>> all.
>>>>>>>>>>
>>>>>>>>>> The test is:
>>>>>>>>>> ifconfig eth0 172.20.1.2 up
>>>>>>>>>> ping 172.20.1.23
>>>>>>>>>>
>>>>>>>>>> If to run tcpdump on the host's "tap-id3" interface, it shows no
>>>>>>>>>> trafic
>>>>>>>>>> coming from the guest. If to compare how it works before and after
>>>>>>>>>> reboot,
>>>>>>>>>> I can see the guest doing an ARP request for 172.20.1.23 and
>>>>>>>>>> receives the
>>>>>>>>>> response and it does the same after reboot but the answer does not
>>>>>>>>>> come.
>>>>>>>>>
>>>>>>>>> So you see the arp packet in guest but not in host?
>>>>>>>>
>>>>>>>> Yes.
>>>>>>>>
>>>>>>>>
>>>>>>>>> One thing to try is to boot debug kernel - where pr_debug is
>>>>>>>>> enabled - then you might see some errors in the kernel log.
>>>>>>>>
>>>>>>>> Tried and added lot more debug printk myself, not clear at all what is
>>>>>>>> happening there.
>>>>>>>>
>>>>>>>> One more hint - if I boot the guest and the guest does not bring eth0
>>>>>>>> up
>>>>>>>> AND wait more than 200 seconds (and less than 210 seconds), then eth0
>>>>>>>> will
>>>>>>>> not work at all. I.e. this script produces not-working-eth0:
>>>>>>>>
>>>>>>>>
>>>>>>>> ifconfig eth0 172.20.1.2 down
>>>>>>>> sleep 210
>>>>>>>> ifconfig eth0 172.20.1.2 up
>>>>>>>> ping 172.20.1.23
>>>>>>>>
>>>>>>>> s/210/200/ - and it starts working. No reboot is required to reproduce.
>>>>>>>>
>>>>>>>> No "vhost" == always works. The only difference I can see here is
>>>>>>>> vhost's
>>>>>>>> thread which may get suspended if not used for a while after the start
>>>>>>>> and
>>>>>>>> does not wake up but this is almost a blind guess.
>>>>>>>
>>>>>>>
>>>>>>> Yet another clue - this host kernel patch seems to help with the guest
>>>>>>> reboot but does not help with the initial 210 seconds delay:
>>>>>>>
>>>>>>> diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
>>>>>>> index 69068e0..5e67650 100644
>>>>>>> --- a/drivers/vhost/vhost.c
>>>>>>> +++ b/drivers/vhost/vhost.c
>>>>>>> @@ -162,10 +162,10 @@ void vhost_work_queue(struct vhost_dev *dev,
>>>>>>> struct
>>>>>>> vhost_work *work)
>>>>>>> list_add_tail(&work->node, &dev->work_list);
>>>>>>> work->queue_seq++;
>>>>>>> spin_unlock_irqrestore(&dev->work_lock, flags);
>>>>>>> - wake_up_process(dev->worker);
>>>>>>> } else {
>>>>>>> spin_unlock_irqrestore(&dev->work_lock, flags);
>>>>>>> }
>>>>>>> + wake_up_process(dev->worker);
>>>>>>> }
>>>>>>> EXPORT_SYMBOL_GPL(vhost_work_queue);
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Interesting. Some kind of race? A missing memory barrier somewhere?
>>>>>
>>>>> I do not see how. I boot the guest and just wait 210 seconds, nothing
>>>>> happens to cause races.
>>>>>
>>>>>
>>>>>> Since it's all around startup,
>>>>>> you can try kicking the host eventfd in
>>>>>> vhost_net_start.
>>>>>
>>>>>
>>>>> How exactly? This did not help. Thanks.
>>>>>
>>>>> diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
>>>>> index 006576d..407ecf2 100644
>>>>> --- a/hw/net/vhost_net.c
>>>>> +++ b/hw/net/vhost_net.c
>>>>> @@ -229,6 +229,17 @@ int vhost_net_start(VirtIODevice *dev, NetClientState
>>>>> *ncs,
>>>>> if (r < 0) {
>>>>> goto err;
>>>>> }
>>>>> +
>>>>> + VHostNetState *vn = tap_get_vhost_net(ncs[i].peer);
>>>>> + struct vhost_vring_file file = {
>>>>> + .index = i
>>>>> + };
>>>>> + file.fd =
>>>>> event_notifier_get_fd(virtio_queue_get_host_notifier(dev->vq));
>>>>> + r = ioctl(vn->dev.control, VHOST_SET_VRING_KICK, &file);
>>>>
>>>> No, this sets the notifier, it does not kick.
>>>> To kick you write 1 there:
>>>> uint6_t v = 1;
>>>> write(fd, &v, sizeof v);
>>>
>>>
>>> Please, be precise. How/where do I get that @fd? Is what I do correct?
>>
>> Yes.
>
> Turns out that no. The control device in the host kernel does not implement
> write() so it always fails.
>
> This works:
>
> uint64_t v = 1;
> int fd = event_notifier_get_fd(&vq->host_notifier);
> int r = write(fd, &v, sizeof v);
>
> By "works" I mean it helps to wake the whole thing up and the guest's eth0
> starts working after 3 minutes delay.
Checked if virtnet_napi_enable() is called as expected and it is. As I can
see "Receiving skb proto" in the guest's receive_buf(), I believe
host->guest channel works just fine but the guest is unable to send
anything until QEMU writes to event notifier (the code above).
I actually spotted the problem in the host kernel - KVM_IOEVENTFD is called
with a PCI bus address but kvm_io_bus_write() is called with a guest
physical address and these things are different on PPC64/spapr.
I am trying to make a patch for this and post it to some list tonight, I'll
put you in copy.
>>> What
>>> is uint6_t - uint8_t or uint16_t (neither works)?
>>
>> Sorry, should have been uint64_t.
>>
>>> May be it is a missing barrier - I rebooted machine several times and now
>>> sometime after even 240 seconds (not 210 as before) it works (but most of
>>> the time still does not)...
>>>
>>>
>>>>> + if (r) {
>>>>> + error_report("Error notifiyng host notifier: %d", -r);
>>>>> + goto err;
>>>>> + }
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>> If to remove vhost=on, it is all good. If to try Fedora19
>>>>>>>>>> (v3.10-something), it all good again - works before and after reboot.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And there 2 questions:
>>>>>>>>>>
>>>>>>>>>> 1. does anybody have any clue what might go wrong after reboot?
>>>>>>>>>>
>>>>>>>>>> 2. Is there any good material to read about what exactly and how
>>>>>>>>>> vhost
>>>>>>>>>> accelerates?
>>>>>>>>>>
>>>>>>>>>> My understanding is that packets from the guest to the real network
>>>>>>>>>> are
>>>>>>>>>> going as:
>>>>>>>>>> 1. guest's virtio-pci-net does ioport(VIRTIO_PCI_QUEUE_NOTIFY)
>>>>>>>>>> 2. QEMU's net/virtio-net.c calls qemu_net_queue_deliver()
>>>>>>>>>> 3. QEMU's net/tap.c calls tap_write_packet() and this is how the
>>>>>>>>>> host knows
>>>>>>>>>> that there is a new packet.
>>>>>>>>
>>>>>>>>
>>>>>>>> What about the documentation? :) or the idea?
>>>>>>>>
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This how I run QEMU:
>>>>>>>>>> ./qemu-system-ppc64 \
>>>>>>>>>> -enable-kvm \
>>>>>>>>>> -m 2048 \
>>>>>>>>>> -machine pseries \
>>>>>>>>>> -initrd 1.cpio \
>>>>>>>>>> -kernel vml312_virtio_net_dbg \
>>>>>>>>>> -nographic \
>>>>>>>>>> -vga none \
>>>>>>>>>> -netdev
>>>>>>>>>> tap,id=id3,ifname=tap-id3,script=ifup.sh,downscript=ifdown.sh,vhost=on
>>>>>>>>>> \
>>>>>>>>>> -device virtio-net-pci,id=id4,netdev=id3,mac=C0:41:49:4b:00:00
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> That is bridge config:
>>>>>>>>>> [aik@dyn232 ~]$ brctl show
>>>>>>>>>> bridge name bridge id STP enabled interfaces
>>>>>>>>>> brtest 8000.00145e992e88 no pin eth4
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The ifup.sh script:
>>>>>>>>>> ifconfig $1 hw ether ee:01:02:03:04:05
>>>>>>>>>> /sbin/ifconfig $1 up
>>>>>>>>>> /usr/sbin/brctl addif brtest $1
>>>
>>>
>>>
>>> --
>>> Alexey
>
>
--
Alexey
