Hi Christian,
so what command are you using to set the 4 queues in your example in
openvswitch?
As you described the ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=4
applies to the pmd threads only but not to vhost user, so how does one set
multiple queues for vhostuser in 2.5.0 or below?
Your problem, the queues not showing up in the guest, is not present in my vm.
Perhaps my config helps you with that:
<domain type='kvm'>
<name>vm1</name>
<uuid>43108a6f-ecea-4d58-bb28-c119411d0419</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB' nodeset='0'/>
</hugepages>
</memoryBacking>
<vcpu placement='static' cpuset='0-3'>4</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='0'/>
<vcpupin vcpu='1' cpuset='1'/>
<vcpupin vcpu='2' cpuset='2'/>
<vcpupin vcpu='3' cpuset='3'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
</features>
<cpu>
<numa>
<cell id='0' cpus='0-3' memory='2097152' unit='KiB' memAccess='shared'/>
</numa>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/libvirt/images/ubuntu-14.04.img'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/seed.iso'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
</controller>
<interface type='vhostuser'>
<mac address='52:54:00:e3:eb:b2'/>
<source type='unix' path='/usr/local/var/run/openvswitch/vhost-user-0'
mode='client'/>
<model type='virtio'/>
<driver queues='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</interface>
<interface type='vhostuser'>
<mac address='52:54:00:26:50:09'/>
<source type='unix' path='/usr/local/var/run/openvswitch/vhost-user-1'
mode='client'/>
<model type='virtio'/>
<driver queues='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<target port='1'/>
</serial>
<console type='pty'>
<target type='serial' port='1'/>
</console>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</memballoon>
</devices>
</domain>
Im using a cloudimage of 14.04 lts
Best regards
Felix
Von: Christian Ehrhardt [mailto:[email protected]]
Gesendet: Montag, 4. April 2016 17:19
An: Felix Brucker <[email protected]>
Cc: Daniele Di Proietto <[email protected]>; Loftus, Ciara
<[email protected]>; [email protected]
Betreff: Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not
work
Hi Felix,
you mean that warning?
2016-04-04T08:53:28Z|00038|netdev|WARN|vhost-user-0: arguments provided to
device that is not configurable
That command you use is for OpenVswitch post 2.5.
You would have to build and use a very recent Openvswitch from the repositories.
But IMO 2.5 is the latest supported release, quite new and you should work with
that.
As former people pointed out, just the format changed.
The "old" if a few weeks apply for older config is global (not per device)
ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=4
Also this does NOT set the multiqueues the guest shall get.
It sets the number of rx queues the openvswitch-dpdk shall use for its PMDs.
The sad part of all of this is - that I seemed to have helped you just to have
the issue of not having all queues in the guest on my system now.
So to all on the List if there are obvious other things to check please let me
know.
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd
On Mon, Apr 4, 2016 at 4:50 PM, Felix Brucker
<[email protected]<mailto:[email protected]>> wrote:
Hi Christian,
thanks, that worked like a charm, and also my vm now reports 2 queues, but no
communication is working on the interface (as ovs still warns about unsettable
options)
i will try the ubuntu packages for ovs-dpdk and dpdk and let you know if this
works (as i suppose you dont get this warning/error?)
best regards
Felix
Von: Christian Ehrhardt
[mailto:[email protected]<mailto:[email protected]>]
Gesendet: Montag, 4. April 2016 15:58
An: Felix Brucker <[email protected]<mailto:[email protected]>>
Cc: Daniele Di Proietto
<[email protected]<mailto:[email protected]>>; Loftus, Ciara
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Betreff: Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not
work
Hi Felix,
this is Ubuntu 16.04 (Xenial).
And I think several of your components need to be upgraded for multiqueue to
properly work.
IIRC at least your qemu and libvirt are too old.
Please feel free to upgrade your system and go on - you'll get all the newer
versions and can go on with your experiments.
"sudo do-release-upgrade -d" should do the job for you.
Yet regarding your initial question you see that the guest not "seeing" its
multiqueues still applies.
Here I hope for some insight from others on the List what else could/should be
checked.
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd
On Mon, Apr 4, 2016 at 3:49 PM, Felix Brucker
<[email protected]<mailto:[email protected]>> wrote:
Hi,
for my setup libvirts generated output found in /var/log/libvirt/qemu/vm1.log
does not seem to include the mq params:
-chardev socket,id=charnet0,path=/usr/local/var/run/openvswitch/vhost-user-0
-netdev type=vhost-user,id=hostnet0,chardev=charnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:e3:eb:b2,bus=pci.0,addr=0x2
-chardev socket,id=charnet1,path=/usr/local/var/run/openvswitch/vhost-user-1
-netdev type=vhost-user,id=hostnet1,chardev=charnet1 -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:26:50:09,bus=pci.0,addr=0x3
||/ Name Version Architecture
Description
+++-============================-===================-===================-==============================================================
ii libvirt-bin 1.2.16-2ubuntu11.15.10.3
amd64 programs for the libvirt library
ii libvirt0 1.2.16-2ubuntu11.15.10.3
amd64 library for interfacing with different virtualization systems
ii qemu-kvm 1:2.3+dfsg-5ubuntu9.2
amd64 QEMU Full virtualization
ii qemu-system-common 1:2.3+dfsg-5ubuntu9.2
amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 1:2.3+dfsg-5ubuntu9.2
amd64 QEMU full system emulation binaries (x86)
also it seems you have newer versions of libvirt and qemu, on which OS are you
running this?
Best regards
Felix
Von: Christian Ehrhardt
[mailto:[email protected]<mailto:[email protected]>]
Gesendet: Montag, 4. April 2016 15:25
An: Felix Brucker <[email protected]<mailto:[email protected]>>; Daniele
Di Proietto <[email protected]<mailto:[email protected]>>
Cc: Loftus, Ciara <[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Betreff: Re: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not
work
Hi,
I thought that is rather close to my setup so gave it a look as well.
And at least in the current setup I can confirm what Felix sees:
I added
<driver queues='4'/>
for the vhost_user device as in libvirt since
https://www.redhat.com/archives/libvir-list/2015-June/msg00201.html
The section libvirt generates out of that is
-chardev socket,id=charnet1,path=
/var/run/openvswitch/vhost-user-2 -netdev
type=vhost-user,id=hostnet1,chardev=charnet1,queues=4 -device
virtio-net-pci,mq=on,vectors=10,netdev=hostnet1,id=net1,mac=52:54:00
:62:47:b6,bus=pci.0,addr=0x4
So the 4 queues get here by libvirt, and also the 2*queues+2 vectors get into
the commandline.
I think it has all that was once summarized here
http://wiki.qemu.org/Features/vhost-user-ovs-dpdk#Enabling_multi-queue
Yet the device in the guest looks like this (no queues able to be combined)
ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX: 0
TX: 0
Other: 0
Combined: 1
Current hardware settings:
RX: 0
TX: 0
Other: 0
Combined: 1
Versions of involved programs:
||/ Name Version Architecture
Description
+++-=====================================-=======================-=======================-===============================================================================
ii libvirt-bin 1.3.1-1ubuntu6 amd64
programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu6 amd64
library for interfacing with different virtualization systems
ii qemu-kvm 1:2.5+dfsg-5ubuntu6 amd64
QEMU Full virtualization
ii dpdk 2.2.0-0ubuntu6 amd64
Data Plane Development Kit (runtime)
ii libdpdk0:amd64 2.2.0-0ubuntu6 amd64
Data Plane Development Kit (runtime libraries)
ii openvswitch-common 2.5.0-0ubuntu1 amd64
Open vSwitch common components
ii openvswitch-switch 2.5.0-0ubuntu1 amd64
Open vSwitch switch implementations
ii openvswitch-switch-dpdk 2.5.0-0ubuntu1 amd64
DPDK enabled Open vSwitch switch implementation
Full qemu commandline:
qemu-system-x86_64 -enable-kvm -name guest-dpdk-vhost-user-2 -S -machine
pc-i440fx-wily,accel=kvm,usb=o
ff -cpu host -m 2048 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1
-object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,share
=yes,size=2147483648<tel:2147483648> -numa
node,nodeid=0,cpus=0,memdev=ram-node0 -uuid
380e03d4-ea96-410a-b50f-f7fdbb6facdd -nographic -no-user-config -nodefaults
-chardev socket,id=charmo
nitor,path=/var/lib/libvirt/qemu/domain-guest-dpdk-vhost-user-2/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-boot
strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/uvtool/libvirt/images/guest-dpdk-vhost-user-2.qcow,format=qcow2,if=none,id=drive-virtio
-disk0,cache=unsafe -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive file=/var/lib/uvtool/libvirt/images/guest
-dpdk-vhost-user-2-ds.qcow,format=raw,if=none,id=drive-virtio-disk1,cache=unsafe
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1
-netdev tap,fd=31,id=hostnet0,vhost=on,vhostfd=33 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:10:f7:df,bus=pci.0,addr=0x3
-chardev socket,id=charnet1,path=
/var/run/openvswitch/vhost-user-2 -netdev
type=vhost-user,id=hostnet1,chardev=charnet1,queues=4 -device
virtio-net-pci,mq=on,vectors=10,netdev=hostnet1,id=net1,mac=52:54:00
:62:47:b6,bus=pci.0,addr=0x4 -chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-ba
lloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Kind Regards,
Christian
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd
On Mon, Apr 4, 2016 at 12:17 PM, Felix Brucker
<[email protected]<mailto:[email protected]>> wrote:
Hi,
i followed the Install.DPDK.md<http://Install.DPDK.md> initially, thats where i
got
ovs-vsctl set Interface vhost-user-2 options:n_rxq=<requested queues>
to set the rx queues for vhost-user interfaces. Im using libvirt, so im not
passing any cli arguments, rather my xml snippet looks like this:
<interface type='vhostuser'>
<mac address='52:54:00:e3:eb:b2'/>
<source type='unix' path='/usr/local/var/run/openvswitch/vhost-user-0'
mode='client'/>
<model type='virtio'/>
<driver queues='2'/>
</interface>
<interface type='vhostuser'>
<mac address='52:54:00:26:50:09'/>
<source type='unix' path='/usr/local/var/run/openvswitch/vhost-user-1'
mode='client'/>
<model type='virtio'/>
<driver queues='2'/>
</interface>
I also tried with an additional name parameter like so:
<driver name='vhost' queues='2'/>
as stated for "running ovs-vswitchd with DPDK backend inside a VM" (which im
not doing right now), but this also did not change anything.
Best regards
Felix
-----Ursprüngliche Nachricht-----
Von: Loftus, Ciara
[mailto:[email protected]<mailto:[email protected]>]
Gesendet: Montag, 4. April 2016 12:01
An: Felix Brucker <[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>
Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues does not
work
>
> Hi,
>
> setting
>
> ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=2
>
> did not create two queues inside the guest, from the name of the
> command i guess it sets the number of rx queues for the pmd host
> interface to 2, but not the vhost-user interface for the vm.
> Im using branch-2.5 because a bug i encountered regarding vhost-thread
> was fixed 6 days ago on that branch.
>
> Best regards
> Felix
Can you share the command line you are using to start the VM? Do you have the
necessary fields set correctly? eg.
qemu-system-x86_64 ...
-chardev socket,id=char0,path=/my/sock
-netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce,queues=2
-device
virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off,mq=on,vectors=6
There's more info in INSTALL.DPDK.md<http://INSTALL.DPDK.md> on configuring
multiqueue.
Thanks,
Ciara
>
>
> -----Ursprüngliche Nachricht-----
> Von: Loftus, Ciara
> [mailto:[email protected]<mailto:[email protected]>]
> Gesendet: Montag, 4. April 2016 11:29
> An: Felix Brucker <[email protected]<mailto:[email protected]>>;
> [email protected]<mailto:[email protected]>
> Betreff: RE: [ovs-discuss] [OVS-DPDK] vhost-user with multiple queues
> does not work
>
> >
> > Hi,
> >
> > when configuring OVS vhost-user interfaces with multiple queues like so:
> >
> > ovs-vsctl set Interface vhost-user-0 options:n_rxq=2 ovs-vsctl set
> > Interface vhost-user-1 options:n_rxq=2
> >
> > i get the following warnings when starting ovs-vswitchd:
> >
> > 2016-04-04T08:53:28Z|00038|netdev|WARN|vhost-user-0: arguments
> > provided to device that is not configurable
> > 2016-04-04T08:53:28Z|00039|netdev|WARN|vhost-user-1: arguments
> > provided to device that is not configurable
> >
> > The guest vm also does not have multiple queues enabled as they are
> > not set in ovs.
> > Im using OVS 2.5 from git -> branch-2.5 -> commit
> > e099c86ca432d1daeaf8edb66225e7edf061c0b0
>
> Hi,
>
> If you are using branch-2.5 you cannot use the n_rxq field. Instead
> you need to set multiple queues like so:
> ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=4
>
> This sets the number of rxqs for all devices to 4.
>
> Thanks,
> Ciara
>
> >
> > Someone had a similar issue or knows a fix?
> >
> > Best regards
> > Felix
_______________________________________________
discuss mailing list
[email protected]<mailto:[email protected]>
http://openvswitch.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list
[email protected]
http://openvswitch.org/mailman/listinfo/discuss
