On 31.05.2019 1:08, William Tu wrote: > Hi Ilya, > > Thanks for the comments. > > On Thu, May 30, 2019 at 12:47 AM Ilya Maximets <i.maxim...@samsung.com> wrote: >> >> On 29.05.2019 20:57, William Tu wrote: >>> On Wed, May 29, 2019 at 10:51 AM William Tu <u9012...@gmail.com> wrote: >>>> >> >> Thanks for the logs. It looks like a bug in DPDK. >> Does rte_eal_hotplug_remove() works without issues? >> Unfortunately, I have no setup to test myself. > > Right, no issue. reason below: > >> >> You may also use following patch to get more information about detaching >> failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html > > Thanks, -1 is returned > 2019-05-30T19:41:26.467Z|00055|netdev_dpdk|ERR|Device > 'vdev:net_af_xdp0,iface=enp2s0,queue=0' can not be detached: Operation > not permitted. > > The root cause is at > rte_dev_remove > local_dev_remove > ret = dev->bus->unplug(dev); > rte_pmd_af_xdp_remove() > > At rte_pmd_af_xdp_remove() > /* find the ethdev entry */ > eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev)); > if (eth_dev == NULL) > return -1; // failed here
Thanks for detailed investigation. So, it's just a DPDK bug. I think that we need to simply replace 'return -1' with 'return 0' here. The life-cycle of a DPDK port is following: rte_dev_probe() <-- attaches device to dpdk bus. rte_eth_dev_configure() rte_eth_dev_start() rte_eth_dev_stop() rte_eth_dev_close() <-- closes the eth device. rte_dev_remove() <-- detaches device from dpdk bus. af_xdp pmd driver advertises RTE_ETH_DEV_CLOSE_REMOVE, so it frees all the resources on rte_eth_dev_close(). rte_dev_remove() tries to detach device from the vdev bus. It subsequently calls rte_pmd_af_xdp_remove() that tries to free already freed resources and fails. I think that rte_pmd_af_xdp_remove() should just ignore this case reporting successful removal, otherwise device will never be detached from vdev bus reporting failure of driver->remove() callback. Will you prepare the patch for this? If you wish, I could prepare it myself on Monday. Best regards, Ilya Maximets. > > And the reason rte_eth_dev_allocated find no device is that > at _rte_eth_dev_allocated > the rte_eth_devices[i].data is an empty string. > > And the reason is that at rte_eth_dev_close > it calls > if ((dev->data->dev_flags & RTE_ETH_DEV_CLOSE_REMOVE) != 0) { > /* new behaviour: send event + reset state + free all data */ > rte_eth_dev_release_port(dev); > > the rte_eth_dev_release_port clear the dev->data->name > memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data)); > >> >> Another possible issue is that you're using driver base name (net_af_xdp) >> as a name of DPDK device. Maybe this causes some issues. >> Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0". > > Yes, I use the above comment. > > Regards, > William > >> >> Best regards, Ilya Maximets. >> >>>>> >>>>> Another option you may try is: >>>>> >>>>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \ >>>>> options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0" >>>>> >>>> It actually works! When adding the device: >>>> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK >>>> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id: >>>> 0, core id: 11 created. >>>> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd >>>> threads on numa node 0 >>>> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already >>>> stopped >>>> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is >>>> not supported on port 0 >>>> >>>> However , when removing the device, some errors show up. >>>> ovs-vsctl del-port br0 afxdp-p0 >>>> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted >>>> interface afxdp-p0 on port 1 >>>> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id: >>>> 0, core id: 11 destroyed. >>>> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the >>>> device (net_af_xdp) >>>> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device >>>> on primary process >>>> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached >>>> >>>> I also measured the performance, it works fine and shows around 7Mpps. >>>> >>>>> so DPDK will not have to guess the bus type. >>>>> >>>>>> >>>>>> $ ovs-vsctl show >>>>>> Port "afxdp-p0" >>>>>> Interface "afxdp-p0" >>>>>> type: dpdk >>>>>> options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"} >>>>>> >>>>>> The performance of physical device loopback shows around 7Mpps with 64B >>>>>> pkt. >>>>>> >>>>>> $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \ >>>>>> >>>>>> actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0" >>>>>> >>>>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile. >>>>> >>>>> You may use 'dpdk-latest' OVS branch. The proper fix exists there. >>>>> In general, patches that intended to work only with latest DPDK that >>>>> not yet supported on master should be made on top of 'dpdk-latest' >>>>> branch and have [PATCH dpdk-latest] subject prefix. >>>> >>>> Got it, thank you. >>>> William >>>> >>>>> >>>>> Best regards, Ilya Maximets. >>> >>> > > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev