[ovs-discuss] Fwd: [ovs-dpdk] bandwidth issue of vhostuserclient virtio ovs-dpdk
Hi,
Thanks for the reply, please see my inline comments below.
On Thu, Nov 29, 2018 at 6:00 PM Lam, Tiago wrote:
> On 29/11/2018 08:24, LIU Yulong wrote:
> > Hi,
> >
> > We recently tested ovs-dpdk, but we met some bandwidth issue. The
> bandwidth
> > from VM to VM was not close to the physical NIC, it's about 4.3Gbps on a
> > 10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3 test can easily
> > reach 9.3Gbps. We enabled the virtio multiqueue for all guest VMs. In the
> > dpdk vhostuser guest, we noticed that the interrupts are centralized to
> > only one queue. But for no dpdk VM, interrupts can hash to all queues.
> > For those dpdk vhostuser VMs, we also noticed that the PMD usages were
> > also centralized to one no matter server(tx) or client(rx). And no matter
> > one PMD or multiple PMDs, this behavior always exists.
> >
> > Furthuremore, my colleague add some systemtap hook on the openvswitch
> > function, he found something interesting. The function
> > __netdev_dpdk_vhost_send will send all the packets to one
> virtionet-queue.
> > Seems that there are some algorithm/hash table/logic does not do the hash
> > very well.
> >
>
> Hi,
>
> When you say "no dpdk VMs", you mean that within your VM you're relying
> on the Kernel to get the packets, using virtio-net. And when you say
> "dpdk vhostuser guest", you mean you're using DPDK inside the VM to get
> the packets. Is this correct?
Sorry for the inaccurate description. I'm really new to DPDK.
No DPDK inside VM, all these settings are for host only.
(`host` means the hypervisor physical machine in the perspective of
virtualization.
On the other hand `guest` means the virtual machine.)
"no dpdk VMs" means the host does not setup DPDK (ovs is working in
traditional way),
the VMs were boot on that. Maybe a new name `VMs-on-NO-DPDK-host`?
If so, could you also tell us which DPDK app you're using inside of
> those VMs? Is it testpmd? If so, how are you setting the `--rxq` and
> `--txq` args? Otherwise, how are you setting those in your app when
> initializing DPDK?
>
Inside VM, there is no DPDK app, VM kernel also
does not set any config related to DPDK. `iperf3` is the tool for bandwidth
testing.
The information below is useful in telling us how you're setting your
> configurations in OvS, but we are still missing the configurations
> inside the VM.
>
> This should help us in getting more information,
>
>
Maybe you have noticed that, we only setup one PMD in the pasted
configurations.
But VM has 8 queues. Should the pmd quantity match the queues?
Tiago.
>
> > So I'd like to find some help from the community. Maybe I'm missing some
> > configrations.
> >
> > Thanks.
> >
> >
> > Here is the list of the environment and some configrations:
> > # uname -r
> > 3.10.0-862.11.6.el7.x86_64
> > # rpm -qa|grep dpdk
> > dpdk-17.11-11.el7.x86_64
> > # rpm -qa|grep openvswitch
> > openvswitch-2.9.0-3.el7.x86_64
> > # ovs-vsctl list open_vswitch
> > _uuid : a6a3d9eb-28a8-4bf0-a8b4-94577b5ffe5e
> > bridges : [531e4bea-ce12-402a-8a07-7074c31b978e,
> > 5c1675e2-5408-4c1f-88bc-6d9c9b932d47]
> > cur_cfg : 1305
> > datapath_types : [netdev, system]
> > db_version : "7.15.1"
> > external_ids: {hostname="cq01-compute-10e112e5e140",
> > rundir="/var/run/openvswitch",
> > system-id="e2cc84fe-a3c8-455f-8c64-260741c141ee"}
> > iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient,
> > geneve, gre, internal, lisp, patch, stt, system, tap, vxlan]
> > manager_options : [43803994-272b-49cb-accc-ab672d1eefc8]
> > next_cfg: 1305
> > other_config: {dpdk-init="true", dpdk-lcore-mask="0x1",
> > dpdk-socket-mem="1024,1024", pmd-cpu-mask="0x10",
> > vhost-iommu-support="true"}
> > ovs_version : "2.9.0"
> > ssl : []
> > statistics : {}
> > system_type : centos
> > system_version : "7"
> > # lsmod |grep vfio
> > vfio_pci 41312 2
> > vfio_iommu_type1 22300 1
> > vfio 32695 7 vfio_iommu_type1,vfio_pci
> > irqbypass 13503 23 kvm,vfio_pci
> >
> > # ovs-appctl dpif/show
> > netdev@ovs-netdev: hit:759366335 missed:754283
> > br-ex:
> > bond1108 4/6: (tap)
> > br-ex 65534/3: (tap)
> > nic-10G-1 5/4: (dpdk: configured_rx_queues=8,
> > configured_rxq_descriptors=2048, configured_tx_queues=2,
> > configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
> > requested_rxq_descriptors=2048, requested_tx_queues=2,
> > requested_txq_descriptors=2048, rx_csum_offload=true)
> > nic-10G-2 6/5: (dpdk: configured_rx_queues=8,
> > configured_rxq_descriptors=2048, configured_tx_queues=2,
> > configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
> > requested_rxq_descriptors=2048, requested_tx_queues=2,
> > requested_txq_descriptors=2048, rx_csum_offload=true)
> > phy-br-ex 3/none: (patch: peer=int-br-ex)
> > br-int:
> > br-int 65534/2: (tap)
> > int-br-ex 1/none: (patch:
[ovs-discuss] Example rule sets
Hello, I would like to compare some OvS hardware accelerators. Problem is that I do not have a rule sets from real environment and I rely on classbench-ng. Do you know about any source of configurations for OvS which can be used to benchmark a packet classification algorthms? Best regards, Michal Orsak ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [ovs-dev] Geneve remote_ip as flow for OVN hosts
Thanks, Ben. Sorry for the delay. Please find attached a draft design proposal and let me know your comments etc. I did some quick prototyping to check for feasibility too; I can share that, if it helps. Note, the document is a draft and, I admit, there might be things that I haven't thought about/through, or missed. I am attaching a text doc, assuming it might be easier, but if you'd like it in a different format, please let me know. thanks! -venu On Wednesday, October 31, 2018, 10:30:23 AM PDT, Ben Pfaff wrote: Honestly the best thing to do is probably to propose a design or, if it's simple enough, to send a patch. That will probably be more effective at sparking a discussion. On Wed, Oct 31, 2018 at 03:33:48PM +, venugopal iyer wrote: > Hi: > Just wanted to check if folks had any thoughts on the use case Girish > outlined below. We do have > a real use case for this and are interested in looking at options for > supporting more than one VTEP IP.It is currently a limitation for us, wanted > to know if there are similar use cases folks are looking at/interested in > addressing. > > thanks, > -venu > > On Thursday, September 6, 2018, 9:19:01 AM PDT, venugopal iyer via dev > wrote: > > Would it be possible for the association to be >made > when the logical port is instantiated on a node? and relayed on to the SB by > the controller, e.g. assuming a mechanism to specify/determine a physical > port mapping for a > logical port for a VM. The mappings can be > specified as > configuration on the chassis. In the absence of physical port information for > a logical port/VM, I suppose we could default to an encap-ip. > > > just a thought, > -venu > On Wednesday, September 5, 2018, 2:03:35 PM PDT, Ben Pfaff > wrote: > > How would OVN know which IP to use for a given logical port on a > chassis? > > I think that the "multiple tunnel encapsulations" is meant to cover, > say, Geneve vs. STT vs. VXLAN, not the case you have in mind. > > On Wed, Sep 05, 2018 at 09:50:32AM -0700, Girish Moodalbail wrote: > > Hello all, > > > > I would like to add more context here. In the diagram below > > > > +--+ > > |ovn-host | > > | | > > | | > > | +-+| > > | | br-int || > > | ++-+--+| > > | | | | > > | +--v-+ +---v+ | > > | | geneve | | geneve | | > > | +--+-+ +---++ | > > | | | | > > | +-v+ +--v---+ | > > | | IP0 | | IP1 | | > > | +--+ +--+ | > > +--+ eth0 +-+ eth1 +---+ > > +--+ +--+ > > > > eth0 and eth are, say, in its own physical segments. The VMs that are > > instantiated in the above ovn-host will have multiple interfaces and each > > of those interface need to be on a different Geneve VTEP. > > > > I think the following entry in OVN TODOs ( > > https://github.com/openvswitch/ovs/blob/master/ovn/TODO.rst) > > > > ---8<--8<--- > > Support multiple tunnel encapsulations in Chassis. > > > > So far, both ovn-controller and ovn-controller-vtep only allow chassis to > > have one tunnel encapsulation entry. We should extend the implementation to > > support multiple tunnel encapsulations > > ---8<--8<--- > > > > captures the above requirement. Is that the case? > > > > Thanks again. > > > > Regards, > > ~Girish > > > > > > > > > > On Tue, Sep 4, 2018 at 3:00 PM Girish Moodalbail > > wrote: > > > > > Hello all, > > > > > > Is it possible to configure remote_ip as a 'flow' instead of an IP address > > > (i.e., setting ovn-encap-ip to a single IP address)? > > > > > > Today, we have one VTEP endpoint per OVN host and all the VMs that > > > connects to br-int on that OVN host are reachable behind this VTEP > > > endpoint. Is it possible to have multiple VTEP endpoints for a br-int > > > bridge and use Open Flow flows to select one of the VTEP endpoint? > > > > > > > > > +--+ > > > |ovn-host | > > > | | > > > | | > > > | +-+| > > > | | br-int || > > > | ++-+--+| > > > | | | | > > > | +--v-+ +---v+ | > > > | | geneve | | geneve | | > > > | +--+-+ +---++ | > > > | | | | > > > | +-v+ +--v---+ | > > > | | IP0 | | IP1 | | > > > | +--+ +--+ | > > > +--+ eth0 +-+ eth1 +---+ > > > +--+ +--+ > > > > > > Also, we don't want to bond
Re: [ovs-discuss] [ovs-dpdk] bandwidth issue of vhostuserclient virtio ovs-dpdk
On 29/11/2018 08:24, LIU Yulong wrote:
> Hi,
>
> We recently tested ovs-dpdk, but we met some bandwidth issue. The bandwidth
> from VM to VM was not close to the physical NIC, it's about 4.3Gbps on a
> 10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3 test can easily
> reach 9.3Gbps. We enabled the virtio multiqueue for all guest VMs. In the
> dpdk vhostuser guest, we noticed that the interrupts are centralized to
> only one queue. But for no dpdk VM, interrupts can hash to all queues.
> For those dpdk vhostuser VMs, we also noticed that the PMD usages were
> also centralized to one no matter server(tx) or client(rx). And no matter
> one PMD or multiple PMDs, this behavior always exists.
>
> Furthuremore, my colleague add some systemtap hook on the openvswitch
> function, he found something interesting. The function
> __netdev_dpdk_vhost_send will send all the packets to one virtionet-queue.
> Seems that there are some algorithm/hash table/logic does not do the hash
> very well.
>
Hi,
When you say "no dpdk VMs", you mean that within your VM you're relying
on the Kernel to get the packets, using virtio-net. And when you say
"dpdk vhostuser guest", you mean you're using DPDK inside the VM to get
the packets. Is this correct?
If so, could you also tell us which DPDK app you're using inside of
those VMs? Is it testpmd? If so, how are you setting the `--rxq` and
`--txq` args? Otherwise, how are you setting those in your app when
initializing DPDK?
The information below is useful in telling us how you're setting your
configurations in OvS, but we are still missing the configurations
inside the VM.
This should help us in getting more information,
Tiago.
> So I'd like to find some help from the community. Maybe I'm missing some
> configrations.
>
> Thanks.
>
>
> Here is the list of the environment and some configrations:
> # uname -r
> 3.10.0-862.11.6.el7.x86_64
> # rpm -qa|grep dpdk
> dpdk-17.11-11.el7.x86_64
> # rpm -qa|grep openvswitch
> openvswitch-2.9.0-3.el7.x86_64
> # ovs-vsctl list open_vswitch
> _uuid : a6a3d9eb-28a8-4bf0-a8b4-94577b5ffe5e
> bridges : [531e4bea-ce12-402a-8a07-7074c31b978e,
> 5c1675e2-5408-4c1f-88bc-6d9c9b932d47]
> cur_cfg : 1305
> datapath_types : [netdev, system]
> db_version : "7.15.1"
> external_ids : {hostname="cq01-compute-10e112e5e140",
> rundir="/var/run/openvswitch",
> system-id="e2cc84fe-a3c8-455f-8c64-260741c141ee"}
> iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient,
> geneve, gre, internal, lisp, patch, stt, system, tap, vxlan]
> manager_options : [43803994-272b-49cb-accc-ab672d1eefc8]
> next_cfg : 1305
> other_config : {dpdk-init="true", dpdk-lcore-mask="0x1",
> dpdk-socket-mem="1024,1024", pmd-cpu-mask="0x10",
> vhost-iommu-support="true"}
> ovs_version : "2.9.0"
> ssl : []
> statistics : {}
> system_type : centos
> system_version : "7"
> # lsmod |grep vfio
> vfio_pci 41312 2
> vfio_iommu_type1 22300 1
> vfio 32695 7 vfio_iommu_type1,vfio_pci
> irqbypass 13503 23 kvm,vfio_pci
>
> # ovs-appctl dpif/show
> netdev@ovs-netdev: hit:759366335 missed:754283
> br-ex:
> bond1108 4/6: (tap)
> br-ex 65534/3: (tap)
> nic-10G-1 5/4: (dpdk: configured_rx_queues=8,
> configured_rxq_descriptors=2048, configured_tx_queues=2,
> configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
> requested_rxq_descriptors=2048, requested_tx_queues=2,
> requested_txq_descriptors=2048, rx_csum_offload=true)
> nic-10G-2 6/5: (dpdk: configured_rx_queues=8,
> configured_rxq_descriptors=2048, configured_tx_queues=2,
> configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
> requested_rxq_descriptors=2048, requested_tx_queues=2,
> requested_txq_descriptors=2048, rx_csum_offload=true)
> phy-br-ex 3/none: (patch: peer=int-br-ex)
> br-int:
> br-int 65534/2: (tap)
> int-br-ex 1/none: (patch: peer=phy-br-ex)
> vhu76f9a623-9f 2/1: (dpdkvhostuserclient: configured_rx_queues=8,
> configured_tx_queues=8, mtu=1500, requested_rx_queues=8,
> requested_tx_queues=8)
>
> # ovs-appctl dpctl/show -s
> netdev@ovs-netdev:
> lookups: hit:759366335 missed:754283 lost:72
> flows: 186
> port 0: ovs-netdev (tap)
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:0 errors:0 dropped:0 aborted:0 carrier:0
> collisions:0
> RX bytes:0 TX bytes:0
> port 1: vhu76f9a623-9f (dpdkvhostuserclient: configured_rx_queues=8,
> configured_tx_queues=8, mtu=1500, requested_rx_queues=8,
> requested_tx_queues=8)
> RX packets:718391758 errors:0 dropped:0 overruns:? frame:?
> TX packets:30372410 errors:? dropped:719200 aborted:? carrier:?
> collisions:?
> RX bytes:1086995317051 (1012.3 GiB) TX bytes:2024893540 (1.9 GiB)
> port 2: br-int (tap)
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1393992 errors:0 dropped:4 aborted:0 carrier:0
> collisions:0
> RX bytes:0 TX
[ovs-discuss] [ovs-dpdk] bandwidth issue of vhostuserclient virtio ovs-dpdk
Hi,
We recently tested ovs-dpdk, but we met some bandwidth issue. The bandwidth
from VM to VM was not close to the physical NIC, it's about 4.3Gbps on a
10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3 test can easily
reach 9.3Gbps. We enabled the virtio multiqueue for all guest VMs. In the
dpdk vhostuser guest, we noticed that the interrupts are centralized to
only one queue. But for no dpdk VM, interrupts can hash to all queues.
For those dpdk vhostuser VMs, we also noticed that the PMD usages were
also centralized to one no matter server(tx) or client(rx). And no matter
one PMD or multiple PMDs, this behavior always exists.
Furthuremore, my colleague add some systemtap hook on the openvswitch
function, he found something interesting. The function
__netdev_dpdk_vhost_send will send all the packets to one virtionet-queue.
Seems that there are some algorithm/hash table/logic does not do the hash
very well.
So I'd like to find some help from the community. Maybe I'm missing some
configrations.
Thanks.
Here is the list of the environment and some configrations:
# uname -r
3.10.0-862.11.6.el7.x86_64
# rpm -qa|grep dpdk
dpdk-17.11-11.el7.x86_64
# rpm -qa|grep openvswitch
openvswitch-2.9.0-3.el7.x86_64
# ovs-vsctl list open_vswitch
_uuid : a6a3d9eb-28a8-4bf0-a8b4-94577b5ffe5e
bridges : [531e4bea-ce12-402a-8a07-7074c31b978e,
5c1675e2-5408-4c1f-88bc-6d9c9b932d47]
cur_cfg : 1305
datapath_types : [netdev, system]
db_version : "7.15.1"
external_ids: {hostname="cq01-compute-10e112e5e140",
rundir="/var/run/openvswitch",
system-id="e2cc84fe-a3c8-455f-8c64-260741c141ee"}
iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient,
geneve, gre, internal, lisp, patch, stt, system, tap, vxlan]
manager_options : [43803994-272b-49cb-accc-ab672d1eefc8]
next_cfg: 1305
other_config: {dpdk-init="true", dpdk-lcore-mask="0x1",
dpdk-socket-mem="1024,1024", pmd-cpu-mask="0x10",
vhost-iommu-support="true"}
ovs_version : "2.9.0"
ssl : []
statistics : {}
system_type : centos
system_version : "7"
# lsmod |grep vfio
vfio_pci 41312 2
vfio_iommu_type1 22300 1
vfio 32695 7 vfio_iommu_type1,vfio_pci
irqbypass 13503 23 kvm,vfio_pci
# ovs-appctl dpif/show
netdev@ovs-netdev: hit:759366335 missed:754283
br-ex:
bond1108 4/6: (tap)
br-ex 65534/3: (tap)
nic-10G-1 5/4: (dpdk: configured_rx_queues=8,
configured_rxq_descriptors=2048, configured_tx_queues=2,
configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
requested_rxq_descriptors=2048, requested_tx_queues=2,
requested_txq_descriptors=2048, rx_csum_offload=true)
nic-10G-2 6/5: (dpdk: configured_rx_queues=8,
configured_rxq_descriptors=2048, configured_tx_queues=2,
configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
requested_rxq_descriptors=2048, requested_tx_queues=2,
requested_txq_descriptors=2048, rx_csum_offload=true)
phy-br-ex 3/none: (patch: peer=int-br-ex)
br-int:
br-int 65534/2: (tap)
int-br-ex 1/none: (patch: peer=phy-br-ex)
vhu76f9a623-9f 2/1: (dpdkvhostuserclient: configured_rx_queues=8,
configured_tx_queues=8, mtu=1500, requested_rx_queues=8,
requested_tx_queues=8)
# ovs-appctl dpctl/show -s
netdev@ovs-netdev:
lookups: hit:759366335 missed:754283 lost:72
flows: 186
port 0: ovs-netdev (tap)
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 aborted:0 carrier:0
collisions:0
RX bytes:0 TX bytes:0
port 1: vhu76f9a623-9f (dpdkvhostuserclient: configured_rx_queues=8,
configured_tx_queues=8, mtu=1500, requested_rx_queues=8,
requested_tx_queues=8)
RX packets:718391758 errors:0 dropped:0 overruns:? frame:?
TX packets:30372410 errors:? dropped:719200 aborted:? carrier:?
collisions:?
RX bytes:1086995317051 (1012.3 GiB) TX bytes:2024893540 (1.9 GiB)
port 2: br-int (tap)
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:1393992 errors:0 dropped:4 aborted:0 carrier:0
collisions:0
RX bytes:0 TX bytes:2113616736 (2.0 GiB)
port 3: br-ex (tap)
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:6660091 errors:0 dropped:967 aborted:0 carrier:0
collisions:0
RX bytes:0 TX bytes:2451440870 (2.3 GiB)
port 4: nic-10G-1 (dpdk: configured_rx_queues=8,
configured_rxq_descriptors=2048, configured_tx_queues=2,
configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
requested_rxq_descriptors=2048, requested_tx_queues=2,
requested_txq_descriptors=2048, rx_csum_offload=true)
RX packets:36409466 errors:0 dropped:0 overruns:? frame:?
TX packets:718371472 errors:0 dropped:20276 aborted:? carrier:?
collisions:?
RX bytes:2541593983 (2.4 GiB) TX bytes:1089838136919 (1015.0 GiB)
port 5: nic-10G-2 (dpdk: configured_rx_queues=8,
configured_rxq_descriptors=2048, configured_tx_queues=2,
configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
requested_rxq_descriptors=2048, requested_tx_queues=2,
