On 07.03.2016 17:13, Wang, Zhihong wrote: > > >> -----Original Message----- >> From: Ilya Maximets [mailto:i.maxim...@samsung.com] >> Sent: Friday, March 4, 2016 7:14 PM >> To: Wang, Zhihong <zhihong.w...@intel.com>; dev@openvswitch.org >> Cc: Flavio Leitner <f...@redhat.com>; Traynor, Kevin >> <kevin.tray...@intel.com>; >> Dyasly Sergey <s.dya...@samsung.com> >> Subject: Re: vhost-user invalid txqid cause discard of packets >> >> On 04.03.2016 13:50, Wang, Zhihong wrote: >>> >>> >>>> -----Original Message----- >>>> From: Ilya Maximets [mailto:i.maxim...@samsung.com] >>>> Sent: Friday, March 4, 2016 6:00 PM >>>> To: Wang, Zhihong <zhihong.w...@intel.com>; dev@openvswitch.org >>>> Cc: Flavio Leitner <f...@redhat.com>; Traynor, Kevin >> <kevin.tray...@intel.com>; >>>> Dyasly Sergey <s.dya...@samsung.com> >>>> Subject: Re: vhost-user invalid txqid cause discard of packets >>>> >>>> Hi, Zhihong. >>>> I can't reproduce this in my environment. >>>> Could you please provide ovs-vswithd.log with VLOG_DBG enabled >>>> for netdev-dpdk and outputs of following commands: >>>> # ovs-vsctl show >>>> # ovs-appctl dpctl/show >>>> # ovs-appctl dpif-netdev/pmd-rxq-show >>>> in 'good' and 'bad' states? >>>> >>>> Also, are you sure that VM started with exactly 4 queues? >>> >>> >>> Yes it's exact 4 queues. >>> Please see command output below. >>> >>> In "bad" case only vhost txq 0, 1 are sending packets, I believe the other 2 >>> become -1 after the lookup. >> >> I don't think so. Reconfiguration code can't affect vHost queue mapping. >> Only distribution of queues between threads changed here. >> Can you reproduce this issue with another cpu-mask? >> For example : '00ff0' and '003f0'. > > > The results is always the same no matter what 8 cores & 6 cores are used, like > fc0c & f00c. > > The way I launch OVS is like below, could you check if there's anything wrong?
Looks normal. > You should be able to reproduce this issue with it. > The qemu version is 2.5, OVS and DPDK are pulled from the master Mar 1st. > ------------------------------------------------------------- > export DB_SOCK=/var/run/openvswitch/db.sock > #cd ovsdb > ./ovsdb-tool create /etc/openvswitch/conf.db > /usr/share/openvswitch/vswitch.ovsschema > ./ovsdb-server --remote=punix:$DB_SOCK > --remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --detach > > #cd utilities > ./ovs-vsctl --no-wait init > cd $1/vswitchd > ./ovs-vswitchd --dpdk -c 0xf -n 3 --socket-mem 1024 -- unix:$DB_SOCK > --pidfile --log-file=/var/log/openvswitch/ovs-vswitchd.log & > > #cd utilities > ./ovs-vsctl add-br ovsbr0 -- set bridge ovsbr0 datapath_type=netdev > ./ovs-vsctl add-port ovsbr0 vhost-user1 -- set Interface vhost-user1 > type=dpdkvhostuser > ./ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk > ./ovs-vsctl set Interface vhost-user1 options:n_rxq=4 > ./ovs-vsctl set Interface dpdk0 options:n_rxq=4 > ./ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x3fc00 > ./ovs-ofctl del-flows ovsbr0 > ./ovs-ofctl add-flow ovsbr0 in_port=1,action=output:2 > ./ovs-ofctl add-flow ovsbr0 in_port=2,action=output:1 > ------------------------------------------------------------- > > Some logs below, not sure which part you're looking for. I'm looking for something like this: 2016-03-04T09:34:51Z|00459|dpdk(vhost_thread2)|DBG|TX queue mapping for /var/run/openvswitch/vhost-user1 2016-03-04T09:34:51Z|00460|dpdk(vhost_thread2)|DBG| 0 --> 0 2016-03-04T09:34:51Z|00461|dpdk(vhost_thread2)|DBG| 1 --> 1 2016-03-04T09:34:51Z|00462|dpdk(vhost_thread2)|DBG| 2 --> 2 2016-03-04T09:34:51Z|00463|dpdk(vhost_thread2)|DBG| 3 --> 3 > > "bad" 6c: > 2016-03-07T06:37:32.291Z|00001|dpif_netdev(pmd84)|DBG|Core 12 processing port > 'vhost-user1' with queue-id 0 > 2016-03-07T06:37:32.291Z|00002|dpif_netdev(pmd84)|DBG|Core 12 processing port > 'dpdk0' with queue-id 2 > 2016-03-07T06:37:32.292Z|00001|dpif_netdev(pmd80)|DBG|Core 17 processing port > 'dpdk0' with queue-id 1 > 2016-03-07T06:37:32.292Z|00001|dpif_netdev(pmd79)|DBG|Core 16 processing port > 'dpdk0' with queue-id 0 > 2016-03-07T06:37:32.292Z|00001|dpif_netdev(pmd83)|DBG|Core 13 processing port > 'vhost-user1' with queue-id 1 > 2016-03-07T06:37:32.292Z|00002|dpif_netdev(pmd83)|DBG|Core 13 processing port > 'dpdk0' with queue-id 3 > 2016-03-07T06:37:32.293Z|00001|dpif_netdev(pmd81)|DBG|Core 15 processing port > 'vhost-user1' with queue-id 3 > 2016-03-07T06:37:32.294Z|00001|dpif_netdev(pmd82)|DBG|Core 14 processing port > 'vhost-user1' with queue-id 2 > > "good" 8c: > 2016-03-07T06:40:34.200Z|00001|dpif_netdev(pmd86)|DBG|Core 10 processing port > 'vhost-user1' with queue-id 0 > 2016-03-07T06:40:34.202Z|00001|dpif_netdev(pmd79)|DBG|Core 13 processing port > 'vhost-user1' with queue-id 3 > 2016-03-07T06:40:34.202Z|00001|dpif_netdev(pmd80)|DBG|Core 14 processing port > 'dpdk0' with queue-id 0 > 2016-03-07T06:40:34.202Z|00001|dpif_netdev(pmd85)|DBG|Core 11 processing port > 'vhost-user1' with queue-id 1 > 2016-03-07T06:40:34.202Z|00001|dpif_netdev(pmd84)|DBG|Core 12 processing port > 'vhost-user1' with queue-id 2 > 2016-03-07T06:40:34.202Z|00001|dpif_netdev(pmd81)|DBG|Core 15 processing port > 'dpdk0' with queue-id 1 > 2016-03-07T06:40:34.202Z|00001|dpif_netdev(pmd83)|DBG|Core 17 processing port > 'dpdk0' with queue-id 3 > 2016-03-07T06:40:34.202Z|00001|dpif_netdev(pmd82)|DBG|Core 16 processing port > 'dpdk0' with queue-id 2 > > >> >> We will see exact mapping in ovs-vswitchd.log with VLOG_DBG enabled for >> netdev-dpdk. (-vdpdk:all:dbg option to ovs-vswitchd). >> >> One more comment inlined. >> >>> >>> "good": >>> ------------------------------------------------------ >>> [20160301]# ./ovs/utilities/ovs-vsctl show >>> a71febbd-fc2b-4a0a-beb2-d6fe0ae68d58 >>> Bridge "ovsbr0" >>> Port "ovsbr0" >>> Interface "ovsbr0" >>> type: internal >>> Port "vhost-user1" >>> Interface "vhost-user1" >>> type: dpdkvhostuser >>> options: {n_rxq="4"} >>> Port "dpdk0" >>> Interface "dpdk0" >>> type: dpdk >>> options: {n_rxq="4"} >>> [20160301]# ./ovs/utilities/ovs-appctl dpctl/show >>> netdev@ovs-netdev: >>> lookups: hit:2642744165 missed:8 lost:0 >>> flows: 8 >>> port 0: ovs-netdev (internal) >>> port 1: ovsbr0 (tap) >>> port 2: vhost-user1 (dpdkvhostuser: configured_rx_queues=4, >> configured_tx_queues=4, requested_rx_queues=4, requested_tx_queues=73) >>> port 3: dpdk0 (dpdk: configured_rx_queues=4, >> configured_tx_queues=64, requested_rx_queues=4, requested_tx_queues=73) >>> [20160301]# ./ovs/utilities/ovs-appctl dpif-netdev/pmd-rxq-show >>> pmd thread numa_id 0 core_id 16: >>> port: dpdk0 queue-id: 2 >>> pmd thread numa_id 0 core_id 10: >>> port: vhost-user1 queue-id: 0 >>> pmd thread numa_id 0 core_id 12: >>> port: vhost-user1 queue-id: 2 >>> pmd thread numa_id 0 core_id 13: >>> port: vhost-user1 queue-id: 3 >>> pmd thread numa_id 0 core_id 14: >>> port: dpdk0 queue-id: 0 >>> pmd thread numa_id 0 core_id 15: >>> port: dpdk0 queue-id: 1 >>> pmd thread numa_id 0 core_id 11: >>> port: vhost-user1 queue-id: 1 >>> pmd thread numa_id 0 core_id 17: >>> port: dpdk0 queue-id: 3 >>> ------------------------------------------------------ >>> >>> "bad": >>> ------------------------------------------------------ >>> [20160301]# ./ovs/utilities/ovs-vsctl set Open_vSwitch . >> other_config:pmd-cpu-mask=0x3f000 >>> 2016-03-04T03:33:30Z|00041|ovs_numa|WARN|Invalid cpu mask: x >> >> Just in case, have you tried to use the valid mask >> (other_config:pmd-cpu-mask=3f000) ? > > Sure. > >> >>> 2016-03-04T03:33:30Z|00042|dpif_netdev|INFO|Created 6 pmd threads on >> numa node 0 >>> [20160301]# ./ovs/utilities/ovs-vsctl show >>> a71febbd-fc2b-4a0a-beb2-d6fe0ae68d58 >>> Bridge "ovsbr0" >>> Port "ovsbr0" >>> Interface "ovsbr0" >>> type: internal >>> Port "vhost-user1" >>> Interface "vhost-user1" >>> type: dpdkvhostuser >>> options: {n_rxq="4"} >>> Port "dpdk0" >>> Interface "dpdk0" >>> type: dpdk >>> options: {n_rxq="4"} >>> [20160301]# ./ovs/utilities/ovs-appctl dpctl/show >>> netdev@ovs-netdev: >>> lookups: hit:181693955 missed:7 lost:0 >>> flows: 6 >>> port 0: ovs-netdev (internal) >>> port 1: ovsbr0 (tap) >>> port 2: vhost-user1 (dpdkvhostuser: configured_rx_queues=4, >> configured_tx_queues=4, requested_rx_queues=4, requested_tx_queues=73) >>> port 3: dpdk0 (dpdk: configured_rx_queues=4, >> configured_tx_queues=64, requested_rx_queues=4, requested_tx_queues=73) >>> [20160301]# ./ovs/utilities/ovs-appctl dpif-netdev/pmd-rxq-show >>> pmd thread numa_id 0 core_id 13: >>> port: vhost-user1 queue-id: 1 >>> port: dpdk0 queue-id: 3 >>> pmd thread numa_id 0 core_id 14: >>> port: vhost-user1 queue-id: 2 >>> pmd thread numa_id 0 core_id 16: >>> port: dpdk0 queue-id: 0 >>> pmd thread numa_id 0 core_id 17: >>> port: dpdk0 queue-id: 1 >>> pmd thread numa_id 0 core_id 12: >>> port: vhost-user1 queue-id: 0 >>> port: dpdk0 queue-id: 2 >>> pmd thread numa_id 0 core_id 15: >>> port: vhost-user1 queue-id: 3 >>> ------------------------------------------------------ >>> >>> >>>> >>>> Best regards, Ilya Maximets. >>>> >>>> On 03.03.2016 18:24, Wang, Zhihong wrote: >>>>> Hi, >>>>> >>>>> I ran an OVS multiqueue test with very simple traffic topology, basically >>>>> 2 ports each with 4 queues, 8 rxqs in total, like below: >>>>> >>>>> Pktgen <=4q=> PHY <=4q=> OVS <=4q=> testpmd in the guest >>>>> >>>>> First set pmd-cpu-mask to 8 cores, and everything works fine, each rxq >>>>> got a core, and all txqids are valid. >>>>> >>>>> Then I set pmd-cpu-mask to 6 cores, and then 2 txqids become invalid in >>>>> __netdev_dpdk_vhost_send(): >>>>> qid = vhost_dev->tx_q[qid % vhost_dev->real_n_txq].map; >>>>> >>>>> qid returns -1 and this leads to discard of packets. >>>>> >>>>> Consequently in testpmd in VM we see only 2 queues are working and >>>>> throughput drops more than a half. >>>>> >>>>> It works again when I set pmd-cpu-mask to 4 cores. >>>>> >>>>> My OVS and DPDK code are pulled from the repo on March 1st, 2016. >>>>> >>>>> Let me know if you need more info to reproduce this issue. >>>>> >>>>> >>>>> Thanks >>>>> Zhihong >>> >>> >> >> Best regards, Ilya Maximets. > > _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev