Re: [ovs-discuss] Fwd: [ovs-dpdk] bandwidth issue of vhostuserclient virtio ovs-dpdk

[Lam, Tiago]

On 03/12/2018 10:18, LIU Yulong wrote:
> 
> 
> On Sat, Dec 1, 2018 at 1:17 AM LIU Yulong  <mailto:liuyulong...@gmail.com>> wrote:
> 
> 
> 
> On Fri, Nov 30, 2018 at 5:36 PM Lam, Tiago  <mailto:tiago@intel.com>> wrote:
> 
> On 30/11/2018 02:07, LIU Yulong wrote:
> > Hi,
> >
> > Thanks for the reply, please see my inline comments below.
>         >
> >
> > On Thu, Nov 29, 2018 at 6:00 PM Lam, Tiago
> mailto:tiago@intel.com>
> > <mailto:tiago@intel.com <mailto:tiago@intel.com>>> wrote:
> >
> >     On 29/11/2018 08:24, LIU Yulong wrote:
> >     > Hi,
> >     >
> >     > We recently tested ovs-dpdk, but we met some bandwidth
> issue. The
> >     bandwidth
> >     > from VM to VM was not close to the physical NIC, it's about
> >     4.3Gbps on a
> >     > 10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3
> test can easily
> >     > reach 9.3Gbps. We enabled the virtio multiqueue for all
> guest VMs.
> >     In the
> >     > dpdk vhostuser guest, we noticed that the interrupts are
> >     centralized to
> >     > only one queue. But for no dpdk VM, interrupts can hash
> to all queues.
> >     > For those dpdk vhostuser VMs, we also noticed that the
> PMD usages were
> >     > also centralized to one no matter server(tx) or
> client(rx). And no
> >     matter
> >     > one PMD or multiple PMDs, this behavior always exists.
> >     >
> >     > Furthuremore, my colleague add some systemtap hook on
> the openvswitch
> >     > function, he found something interesting. The function
> >     > __netdev_dpdk_vhost_send will send all the packets to one
> >     virtionet-queue.
> >     > Seems that there are some algorithm/hash table/logic
> does not do
> >     the hash
> >     > very well. 
> >     >
> >
> >     Hi,
> >
> >     When you say "no dpdk VMs", you mean that within your VM
> you're relying
> >     on the Kernel to get the packets, using virtio-net. And
> when you say
> >     "dpdk vhostuser guest", you mean you're using DPDK inside
> the VM to get
> >     the packets. Is this correct?
> >
> >
> > Sorry for the inaccurate description. I'm really new to DPDK. 
> > No DPDK inside VM, all these settings are for host only.
> > (`host` means the hypervisor physical machine in the
> perspective of
> > virtualization.
> > On the other hand `guest` means the virtual machine.)
> > "no dpdk VMs" means the host does not setup DPDK (ovs is
> working in
> > traditional way),
> > the VMs were boot on that. Maybe a new name `VMs-on-NO-DPDK-host`?
> 
> Got it. Your "no dpdk VMs" really is referred to as OvS-Kernel,
> while
> your "dpdk vhostuser guest" is referred to as OvS-DPDK.
> 
> >
> >     If so, could you also tell us which DPDK app you're using
> inside of
> >     those VMs? Is it testpmd? If so, how are you setting the
> `--rxq` and
> >     `--txq` args? Otherwise, how are you setting those in your
> app when
> >     initializing DPDK?
> >
> >
> > Inside VM, there is no DPDK app, VM kernel also
> > does not set any config related to DPDK. `iperf3` is the tool for
> > bandwidth testing.
> >
> >     The information below is useful in telling us how you're
> setting your
> >     configurations in OvS, but we are still missing the
> configurations
> >     inside the VM.
> >
> >     This should help us in getting more information,
> >
> >
> > Maybe you have noticed that, we only setup one PMD in the pasted
> > configurations.
> > But VM has 8 queues. Should the pmd quantity match the queues?
> 
> It shouldn't match the queues inside the VM per say. But in this
> case,
> 

Re: [ovs-discuss] Fwd: [ovs-dpdk] bandwidth issue of vhostuserclient virtio ovs-dpdk

[Lam, Tiago]

On 30/11/2018 17:17, LIU Yulong wrote:
> 
> 
> On Fri, Nov 30, 2018 at 5:36 PM Lam, Tiago  <mailto:tiago@intel.com>> wrote:
> 
> On 30/11/2018 02:07, LIU Yulong wrote:
> > Hi,
> >
> > Thanks for the reply, please see my inline comments below.
> >
> >
> > On Thu, Nov 29, 2018 at 6:00 PM Lam, Tiago  <mailto:tiago@intel.com>
> > <mailto:tiago@intel.com <mailto:tiago@intel.com>>> wrote:
> >
> >     On 29/11/2018 08:24, LIU Yulong wrote:
> >     > Hi,
> >     >
> >     > We recently tested ovs-dpdk, but we met some bandwidth
> issue. The
> >     bandwidth
> >     > from VM to VM was not close to the physical NIC, it's about
> >     4.3Gbps on a
> >     > 10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3 test
> can easily
> >     > reach 9.3Gbps. We enabled the virtio multiqueue for all
> guest VMs.
> >     In the
> >     > dpdk vhostuser guest, we noticed that the interrupts are
> >     centralized to
> >     > only one queue. But for no dpdk VM, interrupts can hash to
> all queues.
> >     > For those dpdk vhostuser VMs, we also noticed that the PMD
> usages were
> >     > also centralized to one no matter server(tx) or client(rx).
> And no
> >     matter
> >     > one PMD or multiple PMDs, this behavior always exists.
> >     >
> >     > Furthuremore, my colleague add some systemtap hook on the
> openvswitch
> >     > function, he found something interesting. The function
> >     > __netdev_dpdk_vhost_send will send all the packets to one
> >     virtionet-queue.
> >     > Seems that there are some algorithm/hash table/logic does not do
> >     the hash
> >     > very well. 
> >     >
> >
> >     Hi,
> >
> >     When you say "no dpdk VMs", you mean that within your VM
> you're relying
> >     on the Kernel to get the packets, using virtio-net. And when
> you say
> >     "dpdk vhostuser guest", you mean you're using DPDK inside the
> VM to get
> >     the packets. Is this correct?
> >
> >
> > Sorry for the inaccurate description. I'm really new to DPDK. 
> > No DPDK inside VM, all these settings are for host only.
> > (`host` means the hypervisor physical machine in the perspective of
> > virtualization.
> > On the other hand `guest` means the virtual machine.)
> > "no dpdk VMs" means the host does not setup DPDK (ovs is working in
> > traditional way),
> > the VMs were boot on that. Maybe a new name `VMs-on-NO-DPDK-host`?
> 
> Got it. Your "no dpdk VMs" really is referred to as OvS-Kernel, while
> your "dpdk vhostuser guest" is referred to as OvS-DPDK.
> 
> >
> >     If so, could you also tell us which DPDK app you're using
> inside of
> >     those VMs? Is it testpmd? If so, how are you setting the
> `--rxq` and
> >     `--txq` args? Otherwise, how are you setting those in your app
> when
> >     initializing DPDK?
> >
> >
> > Inside VM, there is no DPDK app, VM kernel also
> > does not set any config related to DPDK. `iperf3` is the tool for
> > bandwidth testing.
> >
> >     The information below is useful in telling us how you're
> setting your
> >     configurations in OvS, but we are still missing the configurations
> >     inside the VM.
> >
> >     This should help us in getting more information,
> >
> >
> > Maybe you have noticed that, we only setup one PMD in the pasted
> > configurations.
> > But VM has 8 queues. Should the pmd quantity match the queues?
> 
> It shouldn't match the queues inside the VM per say. But in this case,
> since you have configured 8 rx queues on your physical NICs as well, and
> since you're looking for higher throughputs, you should increase that
> number of PMDs and pin those rxqs - take a look at [1] on how to do
> that. Later on, increasing the size of your queues could also help.
> 
> 
> I'll test it. 
> Yes, as you noticed that the vhostuserclient  port has n_rxq="8",
> options:
> {n_rxq="8",vhost-server-path="/var/lib/vhost_sockets/vhu76f9a623-9f"}.
> And the physical NIC ha

Re: [ovs-discuss] Fwd: [ovs-dpdk] bandwidth issue of vhostuserclient virtio ovs-dpdk

[Lam, Tiago]

On 30/11/2018 02:07, LIU Yulong wrote:
> Hi,
> 
> Thanks for the reply, please see my inline comments below.
> 
> 
> On Thu, Nov 29, 2018 at 6:00 PM Lam, Tiago  <mailto:tiago@intel.com>> wrote:
> 
> On 29/11/2018 08:24, LIU Yulong wrote:
> > Hi,
> >
> > We recently tested ovs-dpdk, but we met some bandwidth issue. The
> bandwidth
> > from VM to VM was not close to the physical NIC, it's about
> 4.3Gbps on a
> > 10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3 test can easily
> > reach 9.3Gbps. We enabled the virtio multiqueue for all guest VMs.
> In the
> > dpdk vhostuser guest, we noticed that the interrupts are
> centralized to
> > only one queue. But for no dpdk VM, interrupts can hash to all queues.
> > For those dpdk vhostuser VMs, we also noticed that the PMD usages were
> > also centralized to one no matter server(tx) or client(rx). And no
> matter
> > one PMD or multiple PMDs, this behavior always exists.
> >
> > Furthuremore, my colleague add some systemtap hook on the openvswitch
> > function, he found something interesting. The function
> > __netdev_dpdk_vhost_send will send all the packets to one
> virtionet-queue.
> > Seems that there are some algorithm/hash table/logic does not do
> the hash
> > very well. 
> >
> 
> Hi,
> 
> When you say "no dpdk VMs", you mean that within your VM you're relying
> on the Kernel to get the packets, using virtio-net. And when you say
> "dpdk vhostuser guest", you mean you're using DPDK inside the VM to get
> the packets. Is this correct?
> 
> 
> Sorry for the inaccurate description. I'm really new to DPDK. 
> No DPDK inside VM, all these settings are for host only.
> (`host` means the hypervisor physical machine in the perspective of
> virtualization.
> On the other hand `guest` means the virtual machine.)
> "no dpdk VMs" means the host does not setup DPDK (ovs is working in
> traditional way),
> the VMs were boot on that. Maybe a new name `VMs-on-NO-DPDK-host`?

Got it. Your "no dpdk VMs" really is referred to as OvS-Kernel, while
your "dpdk vhostuser guest" is referred to as OvS-DPDK.

> 
> If so, could you also tell us which DPDK app you're using inside of
> those VMs? Is it testpmd? If so, how are you setting the `--rxq` and
> `--txq` args? Otherwise, how are you setting those in your app when
> initializing DPDK?
> 
> 
> Inside VM, there is no DPDK app, VM kernel also
> does not set any config related to DPDK. `iperf3` is the tool for
> bandwidth testing.
> 
> The information below is useful in telling us how you're setting your
> configurations in OvS, but we are still missing the configurations
> inside the VM.
> 
> This should help us in getting more information,
> 
> 
> Maybe you have noticed that, we only setup one PMD in the pasted
> configurations.
> But VM has 8 queues. Should the pmd quantity match the queues?

It shouldn't match the queues inside the VM per say. But in this case,
since you have configured 8 rx queues on your physical NICs as well, and
since you're looking for higher throughputs, you should increase that
number of PMDs and pin those rxqs - take a look at [1] on how to do
that. Later on, increasing the size of your queues could also help.

Just as a curiosity, I see you have a configured MTU of 1500B on the
physical interfaces. Is that the same MTU you're using inside the VM?
And are you using the same configurations (including that 1500B MTU)
when running your OvS-Kernel setup?

Hope this helps,

Tiago.

[1]
http://docs.openvswitch.org/en/latest/topics/dpdk/pmd/#port-rx-queue-assigment-to-pmd-threads

> 
> Tiago.
> 
> > So I'd like to find some help from the community. Maybe I'm
> missing some
> > configrations.
> >
> > Thanks.
> >
> >
> > Here is the list of the environment and some configrations:
> > # uname -r
> > 3.10.0-862.11.6.el7.x86_64
> > # rpm -qa|grep dpdk
> > dpdk-17.11-11.el7.x86_64
> > # rpm -qa|grep openvswitch
> > openvswitch-2.9.0-3.el7.x86_64
> > # ovs-vsctl list open_vswitch
> > _uuid               : a6a3d9eb-28a8-4bf0-a8b4-94577b5ffe5e
> > bridges             : [531e4bea-ce12-402a-8a07-7074c31b978e,
> > 5c1675e2-5408-4c1f-88bc-6d9c9b932d47]
> > cur_cfg             : 1305
> > datapath_types      : [netdev, system]
> > db_version          : "7.15.1"
> > external_ids        : {hostname="

Re: [ovs-discuss] [ovs-dpdk] bandwidth issue of vhostuserclient virtio ovs-dpdk

[Lam, Tiago]

On 29/11/2018 08:24, LIU Yulong wrote:
> Hi,
> 
> We recently tested ovs-dpdk, but we met some bandwidth issue. The bandwidth
> from VM to VM was not close to the physical NIC, it's about 4.3Gbps on a
> 10Gbps NIC. For no dpdk (virtio-net) VMs, the iperf3 test can easily
> reach 9.3Gbps. We enabled the virtio multiqueue for all guest VMs. In the
> dpdk vhostuser guest, we noticed that the interrupts are centralized to
> only one queue. But for no dpdk VM, interrupts can hash to all queues.
> For those dpdk vhostuser VMs, we also noticed that the PMD usages were
> also centralized to one no matter server(tx) or client(rx). And no matter
> one PMD or multiple PMDs, this behavior always exists.
> 
> Furthuremore, my colleague add some systemtap hook on the openvswitch
> function, he found something interesting. The function
> __netdev_dpdk_vhost_send will send all the packets to one virtionet-queue.
> Seems that there are some algorithm/hash table/logic does not do the hash
> very well. 
> 

Hi,

When you say "no dpdk VMs", you mean that within your VM you're relying
on the Kernel to get the packets, using virtio-net. And when you say
"dpdk vhostuser guest", you mean you're using DPDK inside the VM to get
the packets. Is this correct?

If so, could you also tell us which DPDK app you're using inside of
those VMs? Is it testpmd? If so, how are you setting the `--rxq` and
`--txq` args? Otherwise, how are you setting those in your app when
initializing DPDK?

The information below is useful in telling us how you're setting your
configurations in OvS, but we are still missing the configurations
inside the VM.

This should help us in getting more information,

Tiago.

> So I'd like to find some help from the community. Maybe I'm missing some
> configrations.
> 
> Thanks.
> 
> 
> Here is the list of the environment and some configrations:
> # uname -r
> 3.10.0-862.11.6.el7.x86_64
> # rpm -qa|grep dpdk
> dpdk-17.11-11.el7.x86_64
> # rpm -qa|grep openvswitch
> openvswitch-2.9.0-3.el7.x86_64
> # ovs-vsctl list open_vswitch
> _uuid               : a6a3d9eb-28a8-4bf0-a8b4-94577b5ffe5e
> bridges             : [531e4bea-ce12-402a-8a07-7074c31b978e,
> 5c1675e2-5408-4c1f-88bc-6d9c9b932d47]
> cur_cfg             : 1305
> datapath_types      : [netdev, system]
> db_version          : "7.15.1"
> external_ids        : {hostname="cq01-compute-10e112e5e140",
> rundir="/var/run/openvswitch",
> system-id="e2cc84fe-a3c8-455f-8c64-260741c141ee"}
> iface_types         : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient,
> geneve, gre, internal, lisp, patch, stt, system, tap, vxlan]
> manager_options     : [43803994-272b-49cb-accc-ab672d1eefc8]
> next_cfg            : 1305
> other_config        : {dpdk-init="true", dpdk-lcore-mask="0x1",
> dpdk-socket-mem="1024,1024", pmd-cpu-mask="0x10",
> vhost-iommu-support="true"}
> ovs_version         : "2.9.0"
> ssl                 : []
> statistics          : {}
> system_type         : centos
> system_version      : "7"
> # lsmod |grep vfio
> vfio_pci               41312  2 
> vfio_iommu_type1       22300  1 
> vfio                   32695  7 vfio_iommu_type1,vfio_pci
> irqbypass              13503  23 kvm,vfio_pci
> 
> # ovs-appctl dpif/show
> netdev@ovs-netdev: hit:759366335 missed:754283
> br-ex:
> bond1108 4/6: (tap)
> br-ex 65534/3: (tap)
> nic-10G-1 5/4: (dpdk: configured_rx_queues=8,
> configured_rxq_descriptors=2048, configured_tx_queues=2,
> configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
> requested_rxq_descriptors=2048, requested_tx_queues=2,
> requested_txq_descriptors=2048, rx_csum_offload=true)
> nic-10G-2 6/5: (dpdk: configured_rx_queues=8,
> configured_rxq_descriptors=2048, configured_tx_queues=2,
> configured_txq_descriptors=2048, mtu=1500, requested_rx_queues=8,
> requested_rxq_descriptors=2048, requested_tx_queues=2,
> requested_txq_descriptors=2048, rx_csum_offload=true)
> phy-br-ex 3/none: (patch: peer=int-br-ex)
> br-int:
> br-int 65534/2: (tap)
> int-br-ex 1/none: (patch: peer=phy-br-ex)
> vhu76f9a623-9f 2/1: (dpdkvhostuserclient: configured_rx_queues=8,
> configured_tx_queues=8, mtu=1500, requested_rx_queues=8,
> requested_tx_queues=8)
> 
> # ovs-appctl dpctl/show -s
> netdev@ovs-netdev:
> lookups: hit:759366335 missed:754283 lost:72
> flows: 186
> port 0: ovs-netdev (tap)
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:0 errors:0 dropped:0 aborted:0 carrier:0
> collisions:0
> RX bytes:0  TX bytes:0
> port 1: vhu76f9a623-9f (dpdkvhostuserclient: configured_rx_queues=8,
> configured_tx_queues=8, mtu=1500, requested_rx_queues=8,
> requested_tx_queues=8)
> RX packets:718391758 errors:0 dropped:0 overruns:? frame:?
> TX packets:30372410 errors:? dropped:719200 aborted:? carrier:?
> collisions:?
> RX bytes:1086995317051 (1012.3 GiB)  TX bytes:2024893540 (1.9 GiB)
> port 2: br-int (tap)
> RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1393992 errors:0 dropped:4 aborted:0 carrier:0
> collisions:0
> RX bytes:0  TX 

Re: [ovs-discuss] Recirculation context in dpdk-ovs

[Lam, Tiago]

Hi,

A few comments in-line.

On 27/11/2018 13:20, 张萌 wrote:
>  Hi,
> 
>    I`m using “ovs-appctl ofproto/trace “ to trace the flows in ovs-dpdk.
> 
>    When integrated with conntrack, the ovs rule ended in the
> table=10, which will record the ct as the flowing flow:
> 
>  
> 
> -
> 
> [root@zm ~]# ovs-ofctl dump-flows br0 -O openflow15 table=10
> 
> OFPST_FLOW reply (OF1.5) (xid=0x2):
> 
> cookie=0x156ad2f7efd2d389, duration=15058.242s, table=10, n_packets=0,
> n_bytes=0, priority=3000,ip,nw_frag=later actions=goto_table:20
> 
> cookie=0x156ad2f7efd2d333, duration=15058.249s, table=10, n_packets=737,
> n_bytes=72226, priority=2000,icmp
> actions=ct(table=15,zone=NXM_NX_REG6[0..15])
> 
> cookie=0x156ad2f7efd2d337, duration=15058.249s, table=10,
> n_packets=4992, n_bytes=380540, priority=2000,udp
> actions=ct(table=15,zone=NXM_NX_REG6[0..15])
> 
> cookie=0x156ad2f7efd2d367, duration=15058.245s, table=10,
> n_packets=2028037440, n_bytes=183176086711, priority=2000,tcp
> actions=ct(table=15,zone=NXM_NX_REG6[0..15])
> 
> -
> 
>  
> 
>  
> 
>  
> 
>    And when I mock a packet using ofproto/trace, ovs recorded the
> contrack, and prints:
> 
>  
> 
> -
>   
> 
> 
> [root@ zm ~]# ovs-appctl ofproto/trace br0
> tcp,in_port=25,nw_dst=172.19.11.6,tp_dst=320,dl_dst=fa:16:3e:03:39:5f,dl_src=fa:16:3e:e5:cb:2c
>   
>    
> 
> Flow:
> tcp,in_port=25,vlan_tci=0x,dl_src=fa:16:3e:e5:cb:2c,dl_dst=fa:16:3e:03:39:5f,nw_src=0.0.0.0,nw_dst=172.19.11.6,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=320,tcp_flags=0
> 
>  
> 
> bridge("br0")
> 
> -
> 
>  0. in_port=25, priority 100, cookie 0x156ad2f7efd2d4fb
> 
>     set_field:0x29->reg5
> 
>     set_field:0x19->reg6
> 
>     write_metadata:0x290001
> 
>     goto_table:5
> 
>  5. ip,in_port=25,dl_src=fa:16:3e:e5:cb:2c, priority 100, cookie
> 0x156ad2f7efd2d51f
> 
>     goto_table:10
> 
> 10. tcp, priority 2000, cookie 0x156ad2f7efd2d367
> 
>     ct(table=15,zone=NXM_NX_REG6[0..15])
> 
>     drop
> 
>  
> 
> Final flow:
> tcp,reg5=0x29,reg6=0x19,metadata=0x290001,in_port=25,vlan_tci=0x,dl_src=fa:16:3e:e5:cb:2c,dl_dst=fa:16:3e:03:39:5f,nw_src=0.0.0.0,nw_dst=172.19.11.6,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=320,tcp_flags=0
> 
> Megaflow:
> recirc_id=0,tcp,in_port=25,dl_src=fa:16:3e:e5:cb:2c,nw_dst=172.0.0.0/6,nw_frag=no
> 
> Datapath actions: ct(zone=25),recirc(0x4123)   
> 
> -
> 
>  
> 
>    Bug when I set the recirc_id in the flow, ovs puts:
> 
> -
> 
> [root@zm ~]# ovs-appctl ofproto/trace br0
> recirc_id=0x4123,ct_state=new,tcp,in_port=25,nw_dst=172.19.11.6,tp_dst=320,dl_dst=fa:16:3e:03:39:5f,dl_src=fa:16:3e:e5:cb:2c
> 
> Flow:
> recirc_id=0x4123,ct_state=new,tcp,in_port=25,vlan_tci=0x,dl_src=fa:16:3e:e5:cb:2c,dl_dst=fa:16:3e:03:39:5f,nw_src=0.0.0.0,nw_dst=172.19.11.6,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=320,tcp_flags=0
> 
>  
> 
> bridge("br0")
> 
> -
> 
>   Recirculation context not found for ID 4123 
> 
>  
> 
> Final flow: unchanged
> 
> Megaflow: recirc_id=0x4123,ip,in_port=25,nw_frag=no
> 
> Datapath actions: drop
> 
> Translation failed (No recirculation context), packet is dropped.
> 

I believe you're getting the above message because by the time you issue
the command the re-circulation context is already gone.

>  
> 
> -
> 
>  
> 
>    And when dump the contracks in ovs:
> 
> -
> 
>  
> 
> [root@A04-R08-I137-204-9320C72 ~]# ovs-dpctl dump-conntrack ovs-netdev 
> 
> 2018-11-27T05:01:30Z|1|dpif_netlink|WARN|Generic Netlink family
> 'ovs_datapath' does not exist. The Open vSwitch kernel module is
> probably not loaded.
> 
> ovs-dpctl: opening datapath (No such file or directory)
> 

Use the one below instead. That should give you more information

$ovs-appctl dpctl/dump-conntrack

> -
> 
>  
> 
>    Can anyone tells how to mock a packet can pass the ct in dpdk-ovs
> 

What are you 

Re: [ovs-discuss] poor ovs-dpdk performance vs ovs

[Lam, Tiago]

Hi,

I have a couple of questions, but at a first glance this seems to be a
result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are
in the works of addressing that.

The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar
to the ones we have been getting with and without our TSO series [1]
applied.

On the description below, is this a virtualized environment by any
change? I.e. are you running the iperf tests on top of VMs running on
those Centos hosts, or directly on them?

Can you also confirm you are using TSO when using OvS with the Kernel by
reporting what you get with $ethtool -k $iface | grep
"tcp-segmentation-offload:". Is it reported as "on"?

Regards,
Tiago.

[1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html

On 15/11/2018 22:17, Robert Brooks wrote:
> I have a pair of Centos 7 hosts on 10gig switch.
> 
> Using OVS without dpdk enabled I can get 9.4Gb/s with a simple iperf test.
> 
> After switching the receiving host to ovs-dpdk following guidance here:
> http://docs.openvswitch.org/en/latest/intro/install/dpdk/
> 
> I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.
> 
> Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus, 256GB
> RAM, Intel 92599ES NIC.
> 
> I have confirmed the doucmented kernel boot options are set and 1GB
> hugepages are in use.
> 
> # cat /proc/cmdline 
> BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64
> root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto
> rd.lvm.lv =vg01/swap rd.lvm.lv
> =vg01/usr console=ttyS0,115200 iommu=pt intel_iommu=on
> default_hugepagesz=1G hugepagesz=1G hugepages=8
> 
> Software is openvswitch 2.10.1 built using the provided rpm spec file. I
> have used both the Centos provided dpdk 17.11 and a re-compile using
> latest LTS of 17.11.4.
> 
> I have tried various performance tweaks, including cpu pinning and
> isolated cpus.
> 
> Switching back to regular ovs returns the performance to close to wire
> speed.
> 
> Under load the following is logged:
> 
> 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to [POLLIN] on
> fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98%
> CPU usage)
> 
> Config dump:
> 
> # ovsdb-client dump Open_vSwitch
> 
> AutoAttach table
> 
> _uuid mappings system_description system_name
> 
> -  -- ---
> 
> 
> Bridge table
> 
> _uuid                                auto_attach controller datapath_id 
>       datapath_type datapath_version external_ids fail_mode flood_vlans
> flow_tables ipfix mcast_snooping_enable mirrors name  netflow
> other_config ports                                                      
>          
> 
>  --- --
> -- -   -
> --- --- - - --- -
> --- 
> -
> 
> 006f021a-b14d-49fb-a11d-48df2fa2bca1 []          []        
> "001b21a6ddc4" netdev        ""     {}           []       
> []          {}          []    false                 []      "br0" []   
>   {}           [3bd79dba-777c-40d0-b573-bf9e027326f4,
> 60b0661f-2177-4283-a6cb-a80336
> 
> 
> Controller table
> 
> _uuid connection_mode controller_burst_limit controller_rate_limit
> enable_async_messages external_ids inactivity_probe is_connected
> local_gateway local_ip local_netmask max_backoff other_config role
> status target
> 
> - --- -- -
> -   
> -  - ---  
> -- --
> 
> 
> Flow_Sample_Collector_Set table
> 
> _uuid bridge external_ids id ipfix
> 
> - --  -- -
> 
> 
> Flow_Table table
> 
> _uuid external_ids flow_limit groups name overflow_policy prefixes
> 
> -  -- --  --- 
> 
> 
> IPFIX table
> 
> _uuid cache_active_timeout cache_max_flows external_ids obs_domain_id
> obs_point_id other_config sampling targets
> 
> -  ---  -
>    ---
> 
> 
> Interface table
> 
> _uuid                                admin_state bfd bfd_status
> cfm_fault cfm_fault_status cfm_flap_count cfm_health cfm_mpid
> cfm_remote_mpids cfm_remote_opstate duplex error external_ids ifindex 
> ingress_policing_burst ingress_policing_rate lacp_current link_resets
> link_speed  link_stat 
> 
>  --- --- --
> -  -- -- 
>  -- -- -  
> -- -  ---
> --- --
> 
> 

Re: [ovs-discuss] OVS DPDK performance for TCP traffic versus UDP

[Lam, Tiago]

Hi Onkar,

Thanks for shedding some light.

I don't think your difference in performance will have to do your
OvS-DPDK setup. If you're taking the measurements directly from the
iperf server side you'd be going through the "Internet". Assuming you
don't have a dedicated connection there, things like your connection's
bandwidth, the RTT from end to end start to matter considerably,
specially for TCP.

To get to the bottom of it I'd advise you to take the iperf server and
connect it directly to the first machine (Machine 1). You would be
excluding any "Internet" interference and be able to get the performance
of a pvp scenario first.

Assuming you're using kernel forwarding inside the VMs, if you want to
squeeze in the extra performance it is probably wise to use DPDK testpmd
to forward the traffic inside of the VMs as well, as explained here:
http://docs.openvswitch.org/en/latest/howto/dpdk/#phy-vm-phy-vhost-loopback

Regards,
Tiago.

On 04/10/2018 21:06, Onkar Pednekar wrote:
> Hi Tiago,
> 
> Thanks for your reply.
> 
> Below are the answers to your questions in-line.
> 
> 
> On Thu, Oct 4, 2018 at 4:07 AM Lam, Tiago  <mailto:tiago@intel.com>> wrote:
> 
> Hi Onkar,
> 
> Thanks for your email. Your setup isn't very clear to me, so a few
> queries in-line.
> 
> On 04/10/2018 06:06, Onkar Pednekar wrote:
> > Hi,
> >
> > I have been experimenting with OVS DPDK on 1G interfaces. The
> system has
> > 8 cores (hyperthreading enabled) mix of dpdk and non-dpdk capable
> ports,
> > but the data traffic runs only on dpdk ports.
> >
> > DPDK ports are backed by vhost user netdev and I have configured the
> > system so that hugepages are enabled, CPU cores isolated with PMD
> > threads allocated to them and also pinning the VCPUs.>
> > When I run UDP traffic, I see ~ 1G throughput on dpdk interfaces
> with <
> > 1% packet loss. However, with tcp traffic, I see around 300Mbps
> > thoughput. I see that setting generic receive offload to off
> helps, but
> > still the TCP thpt is very less compared to the nic capabilities.
> I know
> > that there will be some performance degradation for TCP as against UDP
> > but this is way below expected.
> >
> 
> When transmitting traffic between the DPDK ports, what are the flows you
> have setup? Does it follow a p2p or pvp setup? In other words, does the
> traffic flow between the VM and the physical ports, or only between
> physical ports?
> 
>  
>  The traffic is between the VM and the physical ports.
> 
> 
> > I don't see any packets dropped for tcp on the internal VM (virtual)
> > interfaces.
> >
> > I would like to know if there is an settings (offloads) for the
> > interfaces or any other config I might be missing.
> 
> What is the MTU set on the DPDK ports? Both physical and vhost-user?
> 
> $ ovs-vsctl get Interface [dpdk0|vhostuserclient0] mtu
> 
> 
> MTU set on physical ports = 2000 
> MTU set on vhostuser ports = 1500
> 
> 
> This will help to clarify some doubts around your setup first.
> 
> Tiago.
> 
> >
> > Thanks,
> > Onkar
> >
> >
> > ___
> > discuss mailing list
> > disc...@openvswitch.org <mailto:disc...@openvswitch.org>
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >
> 
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] OVS DPDK performance for TCP traffic versus UDP

[Lam, Tiago]

Hi Onkar,

Thanks for your email. Your setup isn't very clear to me, so a few
queries in-line.

On 04/10/2018 06:06, Onkar Pednekar wrote:
> Hi,
> 
> I have been experimenting with OVS DPDK on 1G interfaces. The system has
> 8 cores (hyperthreading enabled) mix of dpdk and non-dpdk capable ports,
> but the data traffic runs only on dpdk ports.
> 
> DPDK ports are backed by vhost user netdev and I have configured the
> system so that hugepages are enabled, CPU cores isolated with PMD
> threads allocated to them and also pinning the VCPUs.>
> When I run UDP traffic, I see ~ 1G throughput on dpdk interfaces with <
> 1% packet loss. However, with tcp traffic, I see around 300Mbps
> thoughput. I see that setting generic receive offload to off helps, but
> still the TCP thpt is very less compared to the nic capabilities. I know
> that there will be some performance degradation for TCP as against UDP
> but this is way below expected.
> 

When transmitting traffic between the DPDK ports, what are the flows you
have setup? Does it follow a p2p or pvp setup? In other words, does the
traffic flow between the VM and the physical ports, or only between
physical ports?

> I don't see any packets dropped for tcp on the internal VM (virtual)
> interfaces.
> 
> I would like to know if there is an settings (offloads) for the
> interfaces or any other config I might be missing.

What is the MTU set on the DPDK ports? Both physical and vhost-user?

$ ovs-vsctl get Interface [dpdk0|vhostuserclient0] mtu

This will help to clarify some doubts around your setup first.

Tiago.

> 
> Thanks,
> Onkar
> 
> 
> ___
> discuss mailing list
> disc...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> 
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Re: [ovs-discuss] DPDK vxlan not work

[Lam, Tiago]

Hi,

On 28/08/2018 02:38, menkeyi wrote:
> [root@compute01 ~]# uname -r
> 3.10.0-862.el7.x86_64
> 
> 1、Ovs vxlan mode is available
> 2、Ovs + DPDK vxlan mode is not available
> 

Can you provide a bit more info here? What in specific do you see not
working, or not available? And when you say "Ovs vxlan" above (without
DPDK), is that the userspace or the kernel datapath you're referring to?

> DPDK ovs info:
> 
> [root@compute01 ~]# ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.9.0
> DPDK 17.11.0
> 

Just a note that the advised version for using with 2.9.0 is (at least)
DPDK v17.11.2, for security concerns. More info here:
http://mails.dpdk.org/archives/announce/2018-April/000192.html

> [root@compute01 ~]# dpdk-devbind --status|head -n10
> Network devices using DPDK-compatible driver
> 
> :81:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> drv=vfio-pci unused=
> :84:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb'
> drv=vfio-pci unused=
> 
> [root@compute01 ~]# ovs-vsctl show
> b79a7e81-2d68-4ecd-8cd8-bc2b7d1c52ef
> Manager "ptcp:6640:127.0.0.1"
> is_connected: true
> Bridge br-tun
> Controller "tcp:127.0.0.1:6633"
> is_connected: true
> fail_mode: secure
> Port patch-int
> Interface patch-int
> type: patch
> options: {peer=patch-tun}
> Port "vxlan-33000169"
> Interface "vxlan-33000169"
> type: vxlan
> options: {df_default="true", in_key=flow,
> local_ip="51.0.1.101", out_key=flow, remote_ip="51.0.1.105"}
> Port br-tun
> Interface br-tun
> type: internal
> Port "vxlan-330001ca"
> Interface "vxlan-330001ca"
> type: vxlan
> options: {df_default="true", in_key=flow,
> local_ip="51.0.1.101", out_key=flow, remote_ip="51.0.1.202"}
> Bridge br-provider
> Controller "tcp:127.0.0.1:6633"
> is_connected: true
> fail_mode: secure
> Port br-provider
> Interface br-provider
> type: internal
> Port phy-br-provider
> Interface phy-br-provider
> type: patch
> options: {peer=int-br-provider}
> Port "Team1"
> Interface "team1-enp132s0f0"
> type: dpdk
> options: {dpdk-devargs=":84:00.0"}
> Interface "team1-enp129s0f0"
> type: dpdk
> options: {dpdk-devargs=":81:00.0"}
> Bridge br-int
> Controller "tcp:127.0.0.1:6633"
> is_connected: true
> fail_mode: secure
> Port "vhu15e890ef-36"
> tag: 1
> Interface "vhu15e890ef-36"
> type: dpdkvhostuserclient
> options:
> {vhost-server-path="/var/run/openvswitch/vhu15e890ef-36"}
> Port patch-tun
> Interface patch-tun
> type: patch
> options: {peer=patch-int}
> Port int-br-provider
> Interface int-br-provider
> type: patch
> options: {peer=phy-br-provider}
> Port br-int
> Interface br-int
> type: internal
> Port "vhu82caa59c-d3"
> tag: 1
> Interface "vhu82caa59c-d3"
> type: dpdkvhostuserclient
> options:
> {vhost-server-path="/var/run/openvswitch/vhu82caa59c-d3"}
> Port "tapadb8cafa-a0"
> tag: 4095
> Interface "tapadb8cafa-a0"
> type: internal
> ovs_version: "2.9.0"
> 
> 
> 
> 
> Create a virtual machine and find that the error is found in the
> ovs-vswitchd.log log. I don't know if it is related to this：
> on vxlan_sys_4789 device failed: No such device
> 
>

This shouldn't be related. I'm seeing this on my system as well even
though vxlan is working. It is a bit misleading though, so I'll check
what can lead to this and try to clean it up.

> 
> 2018-08-25T02:34:54.453Z|01178|dpdk|INFO|VHOST_CONFIG: vhost-user
> client: socket created, fd: 83
> 2018-08-25T02:34:54.453Z|01179|netdev_dpdk|INFO|vHost User device
> 'vhu82caa59c-d3' created in 'client' mode, using client socket
> '/var/run/openvswitch/vhu82caa59c-d3'
> 2018-08-25T02:34:54.453Z|01180|dpdk|WARN|VHOST_CONFIG: failed to connect
> to /var/run/openvswitch/vhu82caa59c-d3: No such file or directory
> 2018-08-25T02:34:54.453Z|01181|dpdk|INFO|VHOST_CONFIG:
> /var/run/openvswitch/vhu82caa59c-d3: reconnecting...
> 2018-08-25T02:34:54.634Z|01182|dpif_netdev|INFO|Core 8 on numa node 1
> assigned port 'team1-enp129s0f0' rx queue 0 (measured processing cycles
> 587888).
> 2018-08-25T02:34:54.634Z|01183|dpif_netdev|INFO|Core 8 on numa node 1
> assigned port 'team1-enp132s0f0' rx queue 0 (measured processing cycles
> 402684).
> 

[ovs-discuss] Facing issues while using OVS-DPDK based

[Lam, Tiago]

Hi Anirudh,

Are you still facing this issue?

I was able to replicate your issues by using your vm1-dpdk.xml config.
You seem to have an extra  node that isn't on the
configuration example pointed out by Ian before [1]. If you take the
following out of your config it should work fine:
  





  

Also, make sure your br0 is appropriately configured and up. Your `$
ovs-ofctl show br0` shows it is down:
   1(dpdk-p0): addr:24:6e:96:13:63:c0
   config: 0
   state:  0
   current:10GB-FD
   speed: 1 Mbps now, 0 Mbps max
   2(vhostuser0): addr:00:00:00:00:00:00
   config: 0
   state:  LINK_DOWN
   speed: 0 Mbps now, 0 Mbps max
   LOCAL(br0): addr:24:6e:96:13:63:c0
   config: PORT_DOWN
   state:  LINK_DOWN
   current:10MB-FD COPPER
   speed: 10 Mbps now, 0 Mbps max

The first Q at [2] can help you with that, i.e. bringing br0 up.

Let us know if this helps,

Tiago

[1]
http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/#adding-vhost-user-ports-to-the-guest-libvirt
[2]
http://docs.openvswitch.org/en/latest/faq/issues/#common-configuration-issues
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss