[ovs-discuss] pmd thread netdev bridge performance
Hi All, Where might i find a performance comparison of dpdk netdev bridging with af_packet or tun pmd's and plain old internal port drivers. thanks. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovs w dpdk live lock?
a transient 3s approx. interruption in > traffic when the cable of another port (not involved in the test traffic) > is unplugged? Then I will try to recreate it here. > > > > Also I’m re-adding the ovs-discuss list to the cc. “Don’t drop the list” > is a regular comment on ovs-*. > > > > Regards, > > Billy > > > > *From:* davidjoshuaev...@gmail.com [mailto:davidjoshuaev...@gmail.com] > *Sent:* Tuesday, August 28, 2018 9:07 PM > *To:* O Mahony, Billy > *Subject:* Re: [ovs-discuss] ovs w dpdk live lock? > > > > also... > > perhaps related to stats collection > > > > #0 0x7fe86789c0e2 in rte_delay_us_block () from /lib64/librte_eal.so.6 > > #1 0x7fe8624940c1 in ixgbe_setup_mac_link_multispeed_fiber () from > /lib64/librte_pmd_ixgbe.so.2 > > #2 0x7fe8624b5684 in ixgbe_dev_link_update () from > /lib64/librte_pmd_ixgbe.so.2 > > #3 0x7fe8673f52ad in rte_eth_link_get_nowait () from > /lib64/librte_ethdev.so.8 > > #4 0x560fa8333905 in check_link_status () > > #5 0x560fa8333a53 in netdev_dpdk_get_carrier () > > #6 0x560fa8333abb in netdev_dpdk_get_stats () > > #7 0x560fa8282adb in netdev_get_stats () > > #8 0x560fa81f1ca8 in iface_refresh_stats.part.26 () > > #9 0x560fa81fad68 in bridge_run () > > #10 0x560fa81efc9d in main () > > > > main thread seems to end up here a lot... > > > > > > On Mon, Aug 27, 2018 at 11:14 AM David Evans > wrote: > > Hi Billy, > > I did some more tests. > > It appears to happen when i unplug the optics from the SFP. > > The dpdk is 17.11.3 and ovs 2.9.2 source,, but built as RPM's and > installed from the RPM. > > I don't know why the RPM build reports the 17.11.0. - this might be a bug > in the RPM script. > > > > As you can see from below, i don't even have rules on the switch. > > If i add rules to the switch just to force traffic from one port to > another, there will be traffic interruption.. multiple second latency > against the ports. > > > > Start up script looks like this after attach all viable devices to igb_uio > > > > ovsdb-tool create $OVS_DBDIR/conf.db $OVS_SCHEMA > > echo "Starting the OVS database server" > > ovsdb-server --remote=punix:"$DB_SOCK" \ > > --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ > > --pidfile --detach > > > > ovs-vsctl --no-wait init > > > > echo "" > > echo "start ovs " > > echo "" > > echo "Telling the OVS controller to start OVS with DPDK using 512MB > hugepage memory and run the ovswitchd daemon on logical core 1" > > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true \ > > other_config:dpdk-lcore-mask=0x1E other_config:dpdk-socket-mem="512" > > > > echo "Starting the OVS daemon..." > > ovs-vswitchd unix:$DB_SOCK --pidfile > --log-file=/var/log/openvswitch/vswitchd.log --detach > > sleep 2 > > echo "Before creating the DPDK ports, tell OVS to use Core 2 for the DPDK > PMD" > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x1E > > ovs-vsctl add-port br0 dpdkx0 -- set Interface dpdkx0 type=dpdk > options:dpdk-devargs=:0a:00.0 options:n_rxq=2 > > ovs-vsctl add-port br0 dpdkx1 -- set Interface dpdkx1 type=dpdk > options:dpdk-devargs=:0a:00.1 options:n_rxq=2 > > ovs-vsctl add-port br0 dpdkx2 -- set Interface dpdkx2 type=dpdk > options:dpdk-devargs=:0b:00.0 > > ovs-vsctl add-port br0 dpdkx3 -- set Interface dpdkx3 type=dpdk > options:dpdk-devargs=:0b:00.1 > > > > echo "**Clearing current flows" > > ovs-ofctl del-flows br0 > > > > You don't need any traffic to get these kind of messages, just un-plug > the fibre. > > 2018-08-27T15:41:27.629Z|00483|timeval|WARN|Unreasonably long 2961ms poll > interval (998ms user, 0ms system) > > 2018-08-27T15:41:27.629Z|00484|timeval|WARN|context switches: 1 voluntary, > 105 involuntary > > 2018-08-27T15:41:35.469Z|00485|timeval|WARN|Unreasonably long 2840ms poll > interval (955ms user, 0ms system) > > 2018-08-27T15:41:35.469Z|00486|timeval|WARN|context switches: 1 voluntary, > 99 involuntary > > 2018-08-27T15:41:43.469Z|00487|timeval|WARN|Unreasonably long 3000ms poll > interval (1000ms user, 0ms system) > > 2018-08-27T15:41:43.469Z|00488|timeval|WARN|context switches: 1 voluntary, > 106 involuntary > > 2018-08-27T15:41:51.389Z|00489|timeval|WARN|Unreasonably long 2920ms poll > interval (996m
Re: [ovs-discuss] ovs w dpdk live lock?
ario here tomorrow and see what > happens. > > > > *From:* davidjoshuaev...@gmail.com [mailto:davidjoshuaev...@gmail.com] > *Sent:* Wednesday, August 29, 2018 2:16 PM > *To:* O Mahony, Billy > *Cc:* ovs-discuss@openvswitch.org > *Subject:* Re: [ovs-discuss] ovs w dpdk live lock? > > > > Thats correct Billy. > > > > Say for instance you have pings from an internal vPort coming through to a > DPDK port on a netdev bridge. (an openflow rule for each direction of > traffic, no mac learning) > > Pings work fine till you unplug the other device. > > Then traffic will come in bursts and have massive latency up to 3 seconds > or thereabouts. > > Now, from the callback check_link_status down the stack... > ixgbe_setup_mac_link_multispeed_fiber is 'supposed' to be non-blocking... > but i see we end up in rte_delay_us_block > > And it looks to me that 'main' thread gets hooked up in that activity. > > > > I hope i am not leading you up the garden path, but that seems to me to be > where it's at. > > > > Dave. > > > > > > > > > > > > > > On Wed, Aug 29, 2018 at 7:37 AM O Mahony, Billy > wrote: > > Hi David, > > > > I just saw this email now. Thanks, that’s interesting. > > > > Can you confirm that the issue is a transient 3s approx. interruption in > traffic when the cable of another port (not involved in the test traffic) > is unplugged? Then I will try to recreate it here. > > > > Also I’m re-adding the ovs-discuss list to the cc. “Don’t drop the list” > is a regular comment on ovs-*. > > > > Regards, > > Billy > > > > *From:* davidjoshuaev...@gmail.com [mailto:davidjoshuaev...@gmail.com] > *Sent:* Tuesday, August 28, 2018 9:07 PM > *To:* O Mahony, Billy > *Subject:* Re: [ovs-discuss] ovs w dpdk live lock? > > > > also... > > perhaps related to stats collection > > > > #0 0x7fe86789c0e2 in rte_delay_us_block () from /lib64/librte_eal.so.6 > > #1 0x7fe8624940c1 in ixgbe_setup_mac_link_multispeed_fiber () from > /lib64/librte_pmd_ixgbe.so.2 > > #2 0x7fe8624b5684 in ixgbe_dev_link_update () from > /lib64/librte_pmd_ixgbe.so.2 > > #3 0x7fe8673f52ad in rte_eth_link_get_nowait () from > /lib64/librte_ethdev.so.8 > > #4 0x560fa8333905 in check_link_status () > > #5 0x560fa8333a53 in netdev_dpdk_get_carrier () > > #6 0x560fa8333abb in netdev_dpdk_get_stats () > > #7 0x560fa8282adb in netdev_get_stats () > > #8 0x560fa81f1ca8 in iface_refresh_stats.part.26 () > > #9 0x560fa81fad68 in bridge_run () > > #10 0x560fa81efc9d in main () > > > > main thread seems to end up here a lot... > > > > > > On Mon, Aug 27, 2018 at 11:14 AM David Evans > wrote: > > Hi Billy, > > I did some more tests. > > It appears to happen when i unplug the optics from the SFP. > > The dpdk is 17.11.3 and ovs 2.9.2 source,, but built as RPM's and > installed from the RPM. > > I don't know why the RPM build reports the 17.11.0. - this might be a bug > in the RPM script. > > > > As you can see from below, i don't even have rules on the switch. > > If i add rules to the switch just to force traffic from one port to > another, there will be traffic interruption.. multiple second latency > against the ports. > > > > Start up script looks like this after attach all viable devices to igb_uio > > > > ovsdb-tool create $OVS_DBDIR/conf.db $OVS_SCHEMA > > echo "Starting the OVS database server" > > ovsdb-server --remote=punix:"$DB_SOCK" \ > > --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ > > --pidfile --detach > > > > ovs-vsctl --no-wait init > > > > echo "" > > echo "start ovs " > > echo "" > > echo "Telling the OVS controller to start OVS with DPDK using 512MB > hugepage memory and run the ovswitchd daemon on logical core 1" > > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true \ > > other_config:dpdk-lcore-mask=0x1E other_config:dpdk-socket-mem="512" > > > > echo "Starting the OVS daemon..." > > ovs-vswitchd unix:$DB_SOCK --pidfile > --log-file=/var/log/openvswitch/vswitchd.log --detach > > sleep 2 > > echo "Before creating the DPDK ports, tell OVS to use Core 2 for the DPDK > PMD" > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x1E > > ovs-vsctl add-port br0 dpdkx0 -- set Interface dpdk
Re: [ovs-discuss] ovs w dpdk live lock?
Thats correct Billy. Say for instance you have pings from an internal vPort coming through to a DPDK port on a netdev bridge. (an openflow rule for each direction of traffic, no mac learning) Pings work fine till you unplug the other device. Then traffic will come in bursts and have massive latency up to 3 seconds or thereabouts. Now, from the callback check_link_status down the stack... ixgbe_setup_mac_link_multispeed_fiber is 'supposed' to be non-blocking... but i see we end up in rte_delay_us_block And it looks to me that 'main' thread gets hooked up in that activity. I hope i am not leading you up the garden path, but that seems to me to be where it's at. Dave. On Wed, Aug 29, 2018 at 7:37 AM O Mahony, Billy wrote: > Hi David, > > > > I just saw this email now. Thanks, that’s interesting. > > > > Can you confirm that the issue is a transient 3s approx. interruption in > traffic when the cable of another port (not involved in the test traffic) > is unplugged? Then I will try to recreate it here. > > > > Also I’m re-adding the ovs-discuss list to the cc. “Don’t drop the list” > is a regular comment on ovs-*. > > > > Regards, > > Billy > > > > *From:* davidjoshuaev...@gmail.com [mailto:davidjoshuaev...@gmail.com] > *Sent:* Tuesday, August 28, 2018 9:07 PM > *To:* O Mahony, Billy > *Subject:* Re: [ovs-discuss] ovs w dpdk live lock? > > > > also... > > perhaps related to stats collection > > > > #0 0x7fe86789c0e2 in rte_delay_us_block () from /lib64/librte_eal.so.6 > > #1 0x7fe8624940c1 in ixgbe_setup_mac_link_multispeed_fiber () from > /lib64/librte_pmd_ixgbe.so.2 > > #2 0x7fe8624b5684 in ixgbe_dev_link_update () from > /lib64/librte_pmd_ixgbe.so.2 > > #3 0x7fe8673f52ad in rte_eth_link_get_nowait () from > /lib64/librte_ethdev.so.8 > > #4 0x560fa8333905 in check_link_status () > > #5 0x560fa8333a53 in netdev_dpdk_get_carrier () > > #6 0x560fa8333abb in netdev_dpdk_get_stats () > > #7 0x560fa8282adb in netdev_get_stats () > > #8 0x560fa81f1ca8 in iface_refresh_stats.part.26 () > > #9 0x560fa81fad68 in bridge_run () > > #10 0x560fa81efc9d in main () > > > > main thread seems to end up here a lot... > > > > > > On Mon, Aug 27, 2018 at 11:14 AM David Evans > wrote: > > Hi Billy, > > I did some more tests. > > It appears to happen when i unplug the optics from the SFP. > > The dpdk is 17.11.3 and ovs 2.9.2 source,, but built as RPM's and > installed from the RPM. > > I don't know why the RPM build reports the 17.11.0. - this might be a bug > in the RPM script. > > > > As you can see from below, i don't even have rules on the switch. > > If i add rules to the switch just to force traffic from one port to > another, there will be traffic interruption.. multiple second latency > against the ports. > > > > Start up script looks like this after attach all viable devices to igb_uio > > > > ovsdb-tool create $OVS_DBDIR/conf.db $OVS_SCHEMA > > echo "Starting the OVS database server" > > ovsdb-server --remote=punix:"$DB_SOCK" \ > > --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ > > --pidfile --detach > > > > ovs-vsctl --no-wait init > > > > echo "" > > echo "start ovs " > > echo "" > > echo "Telling the OVS controller to start OVS with DPDK using 512MB > hugepage memory and run the ovswitchd daemon on logical core 1" > > ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true \ > > other_config:dpdk-lcore-mask=0x1E other_config:dpdk-socket-mem="512" > > > > echo "Starting the OVS daemon..." > > ovs-vswitchd unix:$DB_SOCK --pidfile > --log-file=/var/log/openvswitch/vswitchd.log --detach > > sleep 2 > > echo "Before creating the DPDK ports, tell OVS to use Core 2 for the DPDK > PMD" > > ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x1E > > ovs-vsctl add-port br0 dpdkx0 -- set Interface dpdkx0 type=dpdk > options:dpdk-devargs=:0a:00.0 options:n_rxq=2 > > ovs-vsctl add-port br0 dpdkx1 -- set Interface dpdkx1 type=dpdk > options:dpdk-devargs=:0a:00.1 options:n_rxq=2 > > ovs-vsctl add-port br0 dpdkx2 -- set Interface dpdkx2 type=dpdk > options:dpdk-devargs=:0b:00.0 > > ovs-vsctl add-port br0 dpdkx3 -- set Interface dpdkx3 type=dpdk > options:dpdk-devargs=:0b:00.1 > > > > echo "**Clearing current flows" > > ovs-o
[ovs-discuss] ovs w dpdk live lock?
Dpdk 17.11.3 OVS 2.9.2 Kernel. 3.10.0-862.9.1.el7.x86_64 If i connect intel x553 dpdk 8086:15c4 to ovs bridge, but the ports are down, or sfp not installed, this happens as below in the log traffic through the 'up' port will incur massive latency (up to 3 or more seconds) 'waiting for main to quiesce' If i only add ports that are up, it works fine afaik. 2018-08-24T21:27:12.289Z|1|vlog|INFO|opened log file /var/log/openvswitch/vswitchd.log 2018-08-24T21:27:12.292Z|2|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0 2018-08-24T21:27:12.292Z|3|ovs_numa|INFO|Discovered 1 NUMA nodes and 16 CPU cores 2018-08-24T21:27:12.292Z|4|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2018-08-24T21:27:12.292Z|5|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2018-08-24T21:27:12.295Z|6|dpdk|INFO|Using DPDK 17.11.0 2018-08-24T21:27:12.295Z|7|dpdk|INFO|DPDK Enabled - initializing... 2018-08-24T21:27:12.295Z|8|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-08-24T21:27:12.295Z|9|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2018-08-24T21:27:12.295Z|00010|dpdk|INFO|EAL ARGS: ovs-vswitchd -c 0x1E --socket-mem 512 2018-08-24T21:27:12.298Z|00011|dpdk|INFO|EAL: Detected 16 lcore(s) 2018-08-24T21:27:12.318Z|00012|dpdk|INFO|EAL: Probing VFIO support... 2018-08-24T21:27:13.207Z|00013|dpdk|INFO|EAL: PCI device :02:00.0 on NUMA socket -1 2018-08-24T21:27:13.207Z|00014|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.207Z|00015|dpdk|INFO|EAL: probe driver: 8086:1533 net_e1000_igb 2018-08-24T21:27:13.207Z|00016|dpdk|INFO|EAL: PCI device :03:00.0 on NUMA socket -1 2018-08-24T21:27:13.207Z|00017|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.207Z|00018|dpdk|INFO|EAL: probe driver: 8086:1533 net_e1000_igb 2018-08-24T21:27:13.207Z|00019|dpdk|INFO|EAL: PCI device :04:00.0 on NUMA socket -1 2018-08-24T21:27:13.207Z|00020|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.207Z|00021|dpdk|INFO|EAL: probe driver: 8086:1533 net_e1000_igb 2018-08-24T21:27:13.250Z|00022|dpdk|INFO|EAL: PCI device :05:00.0 on NUMA socket -1 2018-08-24T21:27:13.250Z|00023|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.250Z|00024|dpdk|INFO|EAL: probe driver: 8086:1533 net_e1000_igb 2018-08-24T21:27:13.292Z|00025|dpdk|INFO|EAL: PCI device :06:00.0 on NUMA socket -1 2018-08-24T21:27:13.292Z|00026|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.292Z|00027|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2018-08-24T21:27:13.356Z|00028|dpdk|INFO|EAL: PCI device :06:00.1 on NUMA socket -1 2018-08-24T21:27:13.356Z|00029|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.356Z|00030|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2018-08-24T21:27:13.420Z|00031|dpdk|INFO|EAL: PCI device :06:00.2 on NUMA socket -1 2018-08-24T21:27:13.420Z|00032|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.420Z|00033|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2018-08-24T21:27:13.482Z|00034|dpdk|INFO|EAL: PCI device :06:00.3 on NUMA socket -1 2018-08-24T21:27:13.482Z|00035|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.483Z|00036|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2018-08-24T21:27:13.545Z|00037|dpdk|INFO|EAL: PCI device :0a:00.0 on NUMA socket -1 2018-08-24T21:27:13.545Z|00038|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.545Z|00039|dpdk|INFO|EAL: probe driver: 8086:15c4 net_ixgbe 2018-08-24T21:27:13.987Z|00040|dpdk|INFO|EAL: PCI device :0a:00.1 on NUMA socket -1 2018-08-24T21:27:13.987Z|00041|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:13.987Z|00042|dpdk|INFO|EAL: probe driver: 8086:15c4 net_ixgbe 2018-08-24T21:27:14.419Z|00043|dpdk|INFO|EAL: PCI device :0b:00.0 on NUMA socket -1 2018-08-24T21:27:14.419Z|00044|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:14.419Z|00045|dpdk|INFO|EAL: probe driver: 8086:15c4 net_ixgbe 2018-08-24T21:27:14.846Z|00046|dpdk|INFO|EAL: PCI device :0b:00.1 on NUMA socket -1 2018-08-24T21:27:14.846Z|00047|dpdk|WARN|EAL: Invalid NUMA socket, default to 0 2018-08-24T21:27:14.846Z|00048|dpdk|INFO|EAL: probe driver: 8086:15c4 net_ixgbe 2018-08-24T21:27:15.283Z|00049|dpdk|INFO|DPDK pdump packet capture enabled 2018-08-24T21:27:15.283Z|00050|dpdk|INFO|DPDK Enabled - initialized 2018-08-24T21:27:15.285Z|00051|timeval|WARN|Unreasonably long 2991ms poll interval (2047ms user, 941ms system) 2018-08-24T21:27:15.285Z|00052|timeval|WARN|faults: 887 minor, 0 major 2018-08-24T21:27:15.285Z|00053|timeval|WARN|disk: 0 reads, 8 writes 2018-08-24T21:27:15.285Z|00054|timeval|WARN|context switches: 3 voluntary, 178 involuntary 2018-08-24T21:27:15.285Z|00055|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last
