On Wed, Nov 29, 2017 at 2:06 PM, Maxime Coquelin <maxime.coque...@redhat.com> wrote: > > > On 11/29/2017 11:42 AM, Ladi Prosek wrote: >> >> On Wed, Nov 29, 2017 at 9:57 AM, Maxime Coquelin >> <maxime.coque...@redhat.com> wrote: >>> >>> Hi Ladi, >>> >>> Sorry for the late reply. >>> >>> On 11/27/2017 05:01 PM, Ladi Prosek wrote: >>>> >>>> >>>> I think I understand what's going on. DPDK simply won't consider the >>>> interface 'ready' until after all queues have been initialized. >>>> >>>> http://dpdk.org/browse/dpdk/tree/lib/librte_vhost/vhost_user.c#n713 >>>> >>>> It looks like Maxime is the right person to bug about this. One of his >>>> recent commits appears to be somewhat related: >>>> http://dpdk.org/browse/dpdk/commit/?id=eefac9536a >>>> >>>> Maxime, iPXE has a simple virtio-net driver that never negotiates the >>>> VIRTIO_NET_F_MQ feature and never initializes more than one queue. >>>> This makes it incompatible with vhost-user configured with mq=on, as >>>> Rafael and Zoltan have discovered. >>>> >>>> Is there any chance DPDK can be made aware of the VIRTIO_NET_F_MQ >>>> feature bit being acked by the guest driver, and successfully operate >>>> with one queue in case it was not acked? There's some context below in >>>> this email. I can provide instructions on how to build iPXE and launch >>>> QEMU to test this if you're interested. >>> >>> >>> >>> I think I get your problem. I'm interested in instructions to reproduce >>> the issue. >> >> >> Here it is, let me know if you run into any issues: >> >> $ git clone git://git.ipxe.org/ipxe.git >> $ cd ipxe/src >> $ make bin/1af41000.rom DEBUG=virtio-net:2 >> $ ln -s bin/1af41000.rom efi-virtio.rom >> >> Then run QEMU without changing the current directory (i.e. should >> still be .../ipxe/src): >> >> qemu-system-x86_64 \ >> -machine pc,accel=kvm -m 128M -boot strict=on -device cirrus-vga \ >> -monitor stdio \ >> -object >> memory-backend-file,id=mem,size=128M,mem-path=/dev/hugepages,share=on \ >> -numa node,memdev=mem \ >> -chardev socket,id=char1,path=/var/run/openvswitch/vhost-user0 \ >> -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce \ >> -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,bootindex=0 >> >> You'll see a bunch of "enqueuing iobuf" debug messages on the screen, >> followed by at least "tx complete". Maybe also "rx complete" depending >> on what /var/run/openvswitch/vhost-user0 is connected to. >> >> Now if you enable multiqueue by replacing the last two lines with: >> >> -netdev type=vhost-user,id=mynet1,chardev=char1,vhostforce,queues=16 \ >> -device >> virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mq=on,vectors=34,bootindex=0 >> >> you'll see only "enqueuing iobuf" without any completion, indicating >> that the host is not processing packets placed in the tx virtqueue by >> iPXE. > > > > Thanks, just tested with DPDK v16.11 & DPDK v17.11 using testpmd instead > of OVS. > > In my case, the packets send by iPXE are well received both with and > without mq=on. > > I don't think there is an issue with the virtio_is_ready() code you > mentioned. Indeed, nr_vrings gets incremented only when receiving > vhost-user protocol requests for a new ring. This code has changed > between v16.11 and v17.11 but idea remains the same.
Thank you for looking into it. > In he case of iPXE, it only sends requests for queues 0 & 1, so > nr_vrings is two. Interesting, that's not what I see. Here's part of the log I recorded: dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 0 netdev_dpdk|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/var/run/openvswitch/vhost-user1'changed to 'enabled' dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1 dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 2 netdev_dpdk|INFO|State of queue 2 ( tx_qid 1 ) of vhost device '/var/run/openvswitch/vhost-user1'changed to 'disabled' dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 3 dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 4 netdev_dpdk|INFO|State of queue 4 ( tx_qid 2 ) of vhost device '/var/run/openvswitch/vhost-user1'changed to 'disabled' dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 5 dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE ... VHOST_USER_SET_VRING_ENABLE is sent for all queues and nr_vrings reflects that. > I will try with OVS to try to reproduce your issue, what OVS version are > you using? I had built the most recent OVS from the master branch at commit a7ce5b8. Thanks! > Thanks, > Maxime >> >> Thanks! >> Ladi >> > _______________________________________________ ipxe-devel mailing list ipxe-devel@lists.ipxe.org https://lists.ipxe.org/mailman/listinfo.cgi/ipxe-devel