Hi Ciara, yes i do see them, although i dont need to enter the ethtool command as the ubuntu image or qemu does this automatically it seems (ethtool -l eth0 always shows combined 8 even after boot) for me they look like this:
Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: set queue enable: 1 to qp idx: 0 Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: ovs|00142|dpdk(vhost_thread2)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/usr/local/var/run/openvswitch/vhost-user-0' 0 changed to 'enabled' Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE Apr 5 12:23:36 dpdk-test ovs-vswitchd[16128]: VHOST_CONFIG: set queue enable: 1 to qp idx: 1 [...] I do see the other queues as well but they dont transport packets Best regards Felix -----Ursprüngliche Nachricht----- Von: Loftus, Ciara [mailto:ciara.lof...@intel.com] Gesendet: Dienstag, 5. April 2016 12:54 An: Felix Brucker <fbruc...@xantaro.net> Cc: discuss@openvswitch.org Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not work > > Hi Ciara, > > ok so it should work! > for me the traffic also reaches the vm, but i only see two queues utilized. > > Configs used: > > Ovs: > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=30030 > //corresponds to cores 4+5 and their siblings, so 4 pmd threads > ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 //8 > queues for each vhost-user interface > ovs-vsctl set Interface vhost-user-1 options:n_rxq=8 > > libvirt: > > <vcpu placement='static' cpuset='0-3,12-15'>8</vcpu> //cores 0-3 > and their siblings, so 8 vcpu > <cputune> > <vcpupin vcpu='0' cpuset='0'/> > <vcpupin vcpu='1' cpuset='1'/> > <vcpupin vcpu='2' cpuset='2'/> > <vcpupin vcpu='3' cpuset='3'/> > <vcpupin vcpu='4' cpuset='12'/> > <vcpupin vcpu='5' cpuset='13'/> > <vcpupin vcpu='6' cpuset='14'/> > <vcpupin vcpu='7' cpuset='15'/> > </cputune> > [...] > <driver queues='8'/> > //used on both virtio nics > > > Output of ovs-appctl dpif-netdev/pmd-rxq-show: > > pmd thread numa_id 0 core_id 16: > port: vhost-user-0 queue-id: 2 6 > port: vhost-user-1 queue-id: 0 4 > pmd thread numa_id 0 core_id 17: > port: vhost-user-0 queue-id: 3 7 > port: vhost-user-1 queue-id: 1 5 > pmd thread numa_id 0 core_id 4: > port: vhost-user-0 queue-id: 0 4 > port: dpdk0 queue-id: 0 > port: vhost-user-1 queue-id: 2 6 > pmd thread numa_id 0 core_id 5: > port: vhost-user-0 queue-id: 1 5 > port: dpdk1 queue-id: 0 > port: vhost-user-1 queue-id: 3 7 > > > upon starting traffic and having cleared the stats: > ovs-appctl dpif-netdev/pmd-stats-show > main thread: > emc hits:0 > megaflow hits:0 > miss:0 > lost:0 > polling cycles:953844 (100.00%) > processing cycles:0 (0.00%) > pmd thread numa_id 0 core_id 16: > emc hits:11227924 > megaflow hits:1 > miss:231 > lost:0 > polling cycles:18074152077 (66.87%) > processing cycles:8955323625 (33.13%) > avg cycles per packet: 2407.29 (27029475702/11228156) > avg processing cycles per packet: 797.58 (8955323625/11228156) > pmd thread numa_id 0 core_id 17: > emc hits:1774 > megaflow hits:37 > miss:32 > lost:0 > polling cycles:20839820676 (99.99%) > processing cycles:1977540 (0.01%) > avg cycles per packet: 11308626.27 (20841798216/1843) > avg processing cycles per packet: 1073.00 (1977540/1843) pmd > thread numa_id 0 core_id 4: > emc hits:11541300 > megaflow hits:108 > miss:24 > lost:0 > polling cycles:503392599 (1.55%) > processing cycles:32036341425 (98.45%) > avg cycles per packet: 2819.38 (32539734024/11541432) > avg processing cycles per packet: 2775.77 > (32036341425/11541432) pmd thread numa_id 0 core_id 5: > emc hits:25307647 > megaflow hits:0 > miss:226 > lost:0 > polling cycles:2461529511 (7.57%) > processing cycles:30065889423 (92.43%) > avg cycles per packet: 1285.27 (32527418934/25307873) > avg processing cycles per packet: 1188.01 > (30065889423/25307873) > > inside the vm there are only two out of the eight cores utilized. > when manually setting the irq affinity inside the guest and disableing > irqbalance like so: > > echo 0 > /proc/irq/45/smp_affinity_list > //virtio0-input.0 > echo 0 > /proc/irq/46/smp_affinity_list > //virtio0-output.0 > echo 1 > /proc/irq/47/smp_affinity_list > //virtio0-input.1 > echo 1 > /proc/irq/48/smp_affinity_list //... > echo 2 > /proc/irq/49/smp_affinity_list echo 2 > > /proc/irq/50/smp_affinity_list echo 3 > /proc/irq/51/smp_affinity_list > echo 3 > /proc/irq/52/smp_affinity_list echo 4 > > /proc/irq/53/smp_affinity_list echo 4 > /proc/irq/54/smp_affinity_list > echo 5 > /proc/irq/55/smp_affinity_list echo 5 > > /proc/irq/56/smp_affinity_list echo 6 > /proc/irq/57/smp_affinity_list > echo 6 > /proc/irq/58/smp_affinity_list echo 7 > > /proc/irq/59/smp_affinity_list echo 7 > /proc/irq/60/smp_affinity_list > > echo 0 > /proc/irq/62/smp_affinity_list > //virtio1-input.0 > echo 0 > /proc/irq/63/smp_affinity_list //... > echo 1 > /proc/irq/64/smp_affinity_list echo 1 > > /proc/irq/65/smp_affinity_list echo 2 > /proc/irq/66/smp_affinity_list > echo 2 > /proc/irq/67/smp_affinity_list echo 3 > > /proc/irq/68/smp_affinity_list echo 3 > /proc/irq/69/smp_affinity_list > echo 4 > /proc/irq/70/smp_affinity_list echo 4 > > /proc/irq/71/smp_affinity_list echo 5 > /proc/irq/72/smp_affinity_list > echo 5 > /proc/irq/73/smp_affinity_list echo 6 > > /proc/irq/74/smp_affinity_list echo 6 > /proc/irq/75/smp_affinity_list > echo 7 > /proc/irq/76/smp_affinity_list echo 7 > > /proc/irq/77/smp_affinity_list > > im getting the same result, only two cores used inside the vm. > Inside the vm: ifconfig does not return any dropped packets, so it > seems the queues are not used. > > Cat /proc/interrupts shows only > virtio0-input.0 > virtio0-output.0 > virtio0-output.1 > virtio1-output.0 > virtio1-input.1 Hi Felix, That's strange. I can see: virtio0-input.0 virtio0-output.0 virtio0-input.1 virtio0-output.1 virtio0-input.2 virtio0-output.2 virtio0-input.3 virtio0-output.3 virtio0-input.4 virtio0-output.4 virtio0-input.5 virtio0-output.5 virtio0-input.6 virtio0-output.6 virtio0-input.7 virtio0-output.7 Do you get the following logs or similar when you set " ethtool -L eth0 combined 8" in the VM?: 2016-04-05T03:40:54Z|00010|dpdk(vhost_thread2)|INFO|State of queue 0 ( tx_qid 0 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0' 0 changed to 'enabled' 2016-04-05T03:40:54Z|00011|dpdk(vhost_thread2)|INFO|State of queue 2 ( tx_qid 1 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0' 0 changed to 'enabled' .... 2016-04-05T03:40:54Z|00017|dpdk(vhost_thread2)|INFO|State of queue 14 ( tx_qid 7 ) of vhost device '/usr/local/var/run/openvswitch/dpdkvhostuser0' 0 changed to 'enabled' Thanks, Ciara > > generating interrupts, all other queues have generated zero or one > interrupt since start. > > > Best regards > Felix > > > -----Ursprüngliche Nachricht----- > Von: Loftus, Ciara [mailto:ciara.lof...@intel.com] > Gesendet: Dienstag, 5. April 2016 12:14 > An: Felix Brucker <fbruc...@xantaro.net> > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues > does not work > > Hi Felix, > > It should work with as many pmd threads. I tested 8 queues with 1,2,4 > and 8 pmd threads and each time traffic was able to reach the VM. > > For example with 1 PMD thread > # sudo ./utilities/ovs-vsctl set Open_vSwitch . other_config:pmd-cpu- > mask=1 # sudo ./utilities/ovs-appctl dpif-netdev/pmd-rxq-show pmd > thread numa_id 0 core_id 0: > port: dpdkvhostuser0 queue-id: 0 1 2 3 4 5 6 7 > port: dpdk0 queue-id: 0 > > You will get better performance if you assign a core to each queue though. > > Thanks, > Ciara > > > -----Original Message----- > > From: Felix Brucker [mailto:fbruc...@xantaro.net] > > Sent: Tuesday, April 05, 2016 11:06 AM > > To: Loftus, Ciara <ciara.lof...@intel.com> > > Subject: AW: [ovs-discuss] [OVS-DPDK] vhost-user with multiple > > queues does not work > > > > Hi Ciara, > > > > thanks, one question though: > > > > you wrote: > > > > If you want a pmd to service each of the 8 queues, you can set the > > number of PMD threads via: > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will > > set up > > 8 pmd threads, on cores 0-7. > > > > Does this mean i have to use as many host pmd threads as i want to > > use queues for the vhost-user? > > So using 2 host pmd threads and 8 vhost-user queues does not work? > > > > Best regards > > Felix > > > > > > -----Ursprüngliche Nachricht----- > > Von: Loftus, Ciara [mailto:ciara.lof...@intel.com] > > Gesendet: Dienstag, 5. April 2016 12:00 > > An: Felix Brucker <fbruc...@xantaro.net>; Christian Ehrhardt > > <christian.ehrha...@canonical.com> > > Cc: discuss@openvswitch.org > > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple > > queues does not work > > > > > > > > Hi, > > > > > > with the branch-2.5 ovs and n-dpdk-rxqs option i was able to get > > > four queues with four pmd host threads, which currently is my > > > limit for vm queues in terms of usable cores and the command itself. > > > > > > To my understanding, with ovs post 2.5 (latest git master) i > > > should be able to use two n-dpdk-rxqs queues for the host and use > > > ovs-vsctl set Interface > > > vhost-user-0 options:n_rxq=8 to get eight queues inside the vm, is > > > this correct? > > > > Hi Felix, > > > > Hopefully my explanation below will help clear things up. > > > > Post-2.5 the 'n-dpdk-rxqs' option is not available to use any more. > > Here is a snippet from the commit message that removes this option > > (commit id > > a14b8947fd13d4c587addbffd24eedc7bb48ee2b) > > > > "dpif-netdev: Allow different numbers of rx queues for different ports. > > > > Currently, all of the PMD netdevs can only have the same number of > > rx queues, which is specified in other_config:n-dpdk-rxqs. > > > > Fix that by introducing of new option for PMD interfaces: 'n_rxq', > > which specifies the maximum number of rx queues to be created for > > this > interface. > > > > Example: > > ovs-vsctl set Interface dpdk0 options:n_rxq=8 > > > > Old 'other_config:n-dpdk-rxqs' deleted." > > > > In your case, now on latest master, if you want 8 queues assigned to > > vhost- > > user-0 in the guest you need to do the following: > > > > On the host: > > 1. ovs-vsctl set Interface vhost-user-0 options:n_rxq=8 2. QEMU: > > -chardev > > socket,id=char0,path=/path/to/vhost-user-0 -netdev type=vhost- > > user,id=mynet1,chardev=char0,vhostforce,queues=8 -device > > virtio-net- > > > pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off,mq=on,vectors=1 > > 8 > > Or if you're using libvirt I think the equivalent would be: <driver > > queues='8'/> > > > > On the VM: > > 3. Check queues available: > > [root@localhost ~]# ethtool -l eth0 > > Channel parameters for eth0: > > Pre-set maximums: > > RX: 0 > > TX: 0 > > Other: 0 > > Combined: 8 > > Current hardware settings: > > RX: 0 > > TX: 0 > > Other: 0 > > Combined: 1 > > # Enable 8 queues > > [root@localhost ~]# ethtool -l eth0 > > Channel parameters for eth0: > > Pre-set maximums: > > RX: 0 > > TX: 0 > > Other: 0 > > Combined: 8 > > Current hardware settings: > > RX: 0 > > TX: 0 > > Other: 0 > > Combined: 8 > > > > At this point your vhost-user-0 interface (eth0) on the guest can > > use > > 8 queues. > > > > If you want a pmd to service each of the 8 queues, you can set the > > number of PMD threads via: > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=FF This will > > set up > > 8 pmd threads, on cores 0-7. > > > > If you have other interfaces that you want to increase the number of > > rxqs for, you may do so like so: > > ovs-vsctl set Interface <iface> options:n_rxq=X > > > > Thanks, > > Ciara > > > > > > > > If so, im experiencing a problem were only two queues out of the > > > eight are used for traffic. > > > > > > Best regards > > > Felix > > > > > > > > > -----Ursprüngliche Nachricht----- > > > Von: discuss [mailto:discuss-boun...@openvswitch.org] Im Auftrag > > > von Felix Brucker > > > Gesendet: Dienstag, 5. April 2016 09:48 > > > An: Loftus, Ciara <ciara.lof...@intel.com>; Christian Ehrhardt > > > <christian.ehrha...@canonical.com> > > > Cc: Daniele Di Proietto <diproiet...@vmware.com>; > > > discuss@openvswitch.org > > > Betreff: [MASSMAIL] Re: [ovs-discuss] [OVS-DPDK] vhost-user with > > > multiple queues does not work > > > > > > Hi Ciara, > > > > > > thanks that clarified it, i got confused by > Also this does NOT > > > set the multiqueues the guest shall get i read the Install md from > > > here http://openvswitch.org/support/dist-docs/INSTALL.DPDK.md.txt > > > i thought this is related to the download on the same site > > > (http://openvswitch.org/releases/openvswitch-2.5.0.tar.gz), but it > > > seems not to. > > > With the n-dpdk-rxqs=2 option i was able to get 2 queues inside > > > the vm and working communication, too. > > > After testing i will try to get the latest (post 2.5) version of > > > ovs to get a more fine grained control over the queues. > > > Thanks all! > > > > > > Best regards > > > Felix > > > > > > -----Ursprüngliche Nachricht----- > > > Von: Loftus, Ciara [mailto:ciara.lof...@intel.com] > > > Gesendet: Montag, 4. April 2016 18:11 > > > An: Felix Brucker <fbruc...@xantaro.net>; Christian Ehrhardt > > > <christian.ehrha...@canonical.com> > > > Cc: Daniele Di Proietto <diproiet...@vmware.com>; > > > discuss@openvswitch.org > > > Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple > > > queues does not work > > > > > > > yes that part works, but for communication to work between the > > > > guest and host OVS has to use 2 queues as well, which currently > > > > does > not work. > > > > So how does one set multiple queues for vhostuser in OVS 2.5.0 > > > > or > > below? > > > > Im not talking about libvirt or qemu regarding the above > > > > question, but > > OVS. > > > > > > Hi Felix, > > > > > > As we've mentioned before, you need to use the following command: > > > > > > ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2 > > > > > > ... to assign two rx queues to the vhost-user ports in OVS. > > > > > > This is clearly stated in INSTALL.DPDK.md on the 2.5 branch. I > > > suspect you were previously looking at the latest INSTALL guide > > > which pointed you to use the n_rxq option, which is not available > > > on the > older branch-2.5. > > > > > > Essentially, if your bridge has two vhost-user ports eg. > > > vhost-user-0 and vhost-user-1, the effect of ' ovs-vsctl set > > > Open_vSwitch > . > > > other_config:n- dpdk-rxqs=2' > > > is the same as > > > ovs-vsctl set Interface vhost-user-0 options:n_rxq=2 ovs-vsctl set > > > Interface > > > vhost-user-1 options:n_rxq=2 > > > > > > On branch-2.5, you need to use the former command. > > > > > > Thanks, > > > Ciara > > > > > > > > > > > Grüße > > > > Felix > > > > > > > > Von: Christian Ehrhardt > > > > [mailto:christian.ehrha...@canonical.com] > > > > Gesendet: Montag, 4. April 2016 17:35 > > > > An: Felix Brucker <fbruc...@xantaro.net> > > > > Cc: Daniele Di Proietto <diproiet...@vmware.com>; Loftus, Ciara > > > > <ciara.lof...@intel.com>; discuss@openvswitch.org > > > > Betreff: Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple > > > > queues does not work > > > > > > > > Hi Felix, > > > > here you already do the right thing: > > > > > > > > <interface type='vhostuser'> [...] > > > > <driver queues='2'/> > > > > > > > > Given you have the recent libvirt and qemu versions that > > > > translates to the right qemu parameters as you have seen in my > > > > initial > posts. > > > > > > > > You can then log into the guest and check with "ethtool -l" if > > > > the guest really "sees" its multiple queues (also shown in my > > > > first mail that this fails for me) > > > > > > > > Kind Regards, > > > > Christian > > > _______________________________________________ > > > discuss mailing list > > > discuss@openvswitch.org > > > http://openvswitch.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss