Re: [ovs-discuss] Unable to hotplug interface with DPDK initialized
Hi Ian Thank you for confirming. I tried loading vfio-pci before starting openvswitch and it is working for me. I did not get a change to try out TESTPMD. I will try it out but that might take some time. Thanks for your help. Regards, Shriroop On 6/5/19, 3:12 PM, "Ian Stokes" wrote: On 6/4/2019 6:40 PM, Shriroop Joshi (shrirjos) wrote: > Hi Ian > > I am reaching out to ask if you were able reproduce the issue. Is there anything else I can provide which will help? > > Regards, > Shriroop Hi Shiroop, apologies for the delay, I havent been able to spend as much time as I'd like on the ML lately. I was able to reproduce the issue this evening. It seems specific to i350 devices using VFIO and hotplugging (I was not able to reproduce it with i40e or ixgbe interfaces). I'll have to look further into what is going wrong. Currently the only work around I can think of would be to use igb_uio instead of VFIO if that is an option (hotplug seems to work without issue using igb_uio in this setup although typically VFIO is preferred). Alternatively if VFIO is required, then I would suggest binding the device to VFIO-PCI before starting ovs-dpdk? Do you see the same issue in that case? This could be a bug in DPDK for the net_e100_igb PMD, have you tried to reproduce the issue with a DPDK sample app such as TESTPMD? Regards Ian > > On 5/16/19, 3:29 PM, "Shriroop Joshi (shrirjos)" wrote: > > Hi Ian > > I am using Intel i350 PCIe NIC. I have attached entire ovs-vswitchd.log which shows both error and success. > After attaching the interface, I restarted openvswitch service and it was able to attach the interface. > > Regards > Shriroop > > On 5/16/19, 7:35 AM, "Ian Stokes" wrote: > > On 5/15/2019 10:37 PM, Shriroop Joshi (shrirjos) via discuss wrote: > > Hello ovs community > > > > I am trying to attach an interface (eth4-4) to a bridge (svc-br) with netdev datapath_type. Here are the steps which I follow: > > 1. Load vfio-pci driver > > 2. Bind vfio-pci driver to interface eth4-4 > > (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#dpdk-binding-nics) > > 3. Hotplug this interface to DPDK bridge: > > # ovs-vsctl add-port svc-br eth4-4 -- set interface eth4-4 type=dpdk options:dpdk-devargs=:af:00.3 > > (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#hotplugging) > > > > After completing these steps, I see that it shows 'Error attaching device ':af:00.3' to DPDK' error. > > Here are the error logs: > > 2019-05-15T21:11:20.663Z|00227|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 > > 2019-05-15T21:11:20.663Z|00228|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb > > 2019-05-15T21:11:20.663Z|00229|dpdk|ERR|EAL: Driver cannot attach the device (:af:00.3) > > 2019-05-15T21:11:20.663Z|00230|netdev_dpdk|WARN|Error attaching device ':af:00.3' to DPDK > > 2019-05-15T21:11:20.663Z|00231|netdev|WARN|eth4-4: could not set configuration (Invalid argument) > > > > I tried to reproduce this on my local system but could not. > > What model HW NIC are you attempting to hotplug out of interest? > > Also could you provide the full log for when OVS fails to hotplug as > well as when it succeeds? > > I'd be interested in seeing the OVS configuration commands passed for DPDK. > > Ian > > > I noticed that restarting openvswitch solves this issue, and I can see the interface attached to bridge after restart. > > Here is the snippet from logs during restart: > > 2019-05-15T21:14:54.172Z|00078|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 > > 2019-05-15T21:14:54.172Z|00079|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb > > 2019-05-15T21:14:54.174Z|00080|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) > > > > Am I missing any step? Openvswitch documentation does not specify anything else. > > > > # ovs-vswitchd --version > > ovs-vswitchd (Open vSwitch) 2.9.3 > > DPDK 17.11.0 > > > > Regards, > > Shriroop > > > > > > > > > > ___ > > discuss mailing list
Re: [ovs-discuss] Unable to hotplug interface with DPDK initialized
On 6/4/2019 6:40 PM, Shriroop Joshi (shrirjos) wrote: Hi Ian I am reaching out to ask if you were able reproduce the issue. Is there anything else I can provide which will help? Regards, Shriroop Hi Shiroop, apologies for the delay, I havent been able to spend as much time as I'd like on the ML lately. I was able to reproduce the issue this evening. It seems specific to i350 devices using VFIO and hotplugging (I was not able to reproduce it with i40e or ixgbe interfaces). I'll have to look further into what is going wrong. Currently the only work around I can think of would be to use igb_uio instead of VFIO if that is an option (hotplug seems to work without issue using igb_uio in this setup although typically VFIO is preferred). Alternatively if VFIO is required, then I would suggest binding the device to VFIO-PCI before starting ovs-dpdk? Do you see the same issue in that case? This could be a bug in DPDK for the net_e100_igb PMD, have you tried to reproduce the issue with a DPDK sample app such as TESTPMD? Regards Ian On 5/16/19, 3:29 PM, "Shriroop Joshi (shrirjos)" wrote: Hi Ian I am using Intel i350 PCIe NIC. I have attached entire ovs-vswitchd.log which shows both error and success. After attaching the interface, I restarted openvswitch service and it was able to attach the interface. Regards Shriroop On 5/16/19, 7:35 AM, "Ian Stokes" wrote: On 5/15/2019 10:37 PM, Shriroop Joshi (shrirjos) via discuss wrote: > Hello ovs community > > I am trying to attach an interface (eth4-4) to a bridge (svc-br) with netdev datapath_type. Here are the steps which I follow: > 1. Load vfio-pci driver > 2. Bind vfio-pci driver to interface eth4-4 > (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#dpdk-binding-nics) > 3. Hotplug this interface to DPDK bridge: > # ovs-vsctl add-port svc-br eth4-4 -- set interface eth4-4 type=dpdk options:dpdk-devargs=:af:00.3 > (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#hotplugging) > > After completing these steps, I see that it shows 'Error attaching device ':af:00.3' to DPDK' error. > Here are the error logs: > 2019-05-15T21:11:20.663Z|00227|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 > 2019-05-15T21:11:20.663Z|00228|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb > 2019-05-15T21:11:20.663Z|00229|dpdk|ERR|EAL: Driver cannot attach the device (:af:00.3) > 2019-05-15T21:11:20.663Z|00230|netdev_dpdk|WARN|Error attaching device ':af:00.3' to DPDK > 2019-05-15T21:11:20.663Z|00231|netdev|WARN|eth4-4: could not set configuration (Invalid argument) > I tried to reproduce this on my local system but could not. What model HW NIC are you attempting to hotplug out of interest? Also could you provide the full log for when OVS fails to hotplug as well as when it succeeds? I'd be interested in seeing the OVS configuration commands passed for DPDK. Ian > I noticed that restarting openvswitch solves this issue, and I can see the interface attached to bridge after restart. > Here is the snippet from logs during restart: > 2019-05-15T21:14:54.172Z|00078|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 > 2019-05-15T21:14:54.172Z|00079|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb > 2019-05-15T21:14:54.174Z|00080|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) > > Am I missing any step? Openvswitch documentation does not specify anything else. > > # ovs-vswitchd --version > ovs-vswitchd (Open vSwitch) 2.9.3 > DPDK 17.11.0 > > Regards, > Shriroop > > > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Unable to hotplug interface with DPDK initialized
Hi Ian I am reaching out to ask if you were able reproduce the issue. Is there anything else I can provide which will help? Regards, Shriroop On 5/16/19, 3:29 PM, "Shriroop Joshi (shrirjos)" wrote: Hi Ian I am using Intel i350 PCIe NIC. I have attached entire ovs-vswitchd.log which shows both error and success. After attaching the interface, I restarted openvswitch service and it was able to attach the interface. Regards Shriroop On 5/16/19, 7:35 AM, "Ian Stokes" wrote: On 5/15/2019 10:37 PM, Shriroop Joshi (shrirjos) via discuss wrote: > Hello ovs community > > I am trying to attach an interface (eth4-4) to a bridge (svc-br) with netdev datapath_type. Here are the steps which I follow: > 1. Load vfio-pci driver > 2. Bind vfio-pci driver to interface eth4-4 > (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#dpdk-binding-nics) > 3. Hotplug this interface to DPDK bridge: > # ovs-vsctl add-port svc-br eth4-4 -- set interface eth4-4 type=dpdk options:dpdk-devargs=:af:00.3 > (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#hotplugging) > > After completing these steps, I see that it shows 'Error attaching device ':af:00.3' to DPDK' error. > Here are the error logs: > 2019-05-15T21:11:20.663Z|00227|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 > 2019-05-15T21:11:20.663Z|00228|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb > 2019-05-15T21:11:20.663Z|00229|dpdk|ERR|EAL: Driver cannot attach the device (:af:00.3) > 2019-05-15T21:11:20.663Z|00230|netdev_dpdk|WARN|Error attaching device ':af:00.3' to DPDK > 2019-05-15T21:11:20.663Z|00231|netdev|WARN|eth4-4: could not set configuration (Invalid argument) > I tried to reproduce this on my local system but could not. What model HW NIC are you attempting to hotplug out of interest? Also could you provide the full log for when OVS fails to hotplug as well as when it succeeds? I'd be interested in seeing the OVS configuration commands passed for DPDK. Ian > I noticed that restarting openvswitch solves this issue, and I can see the interface attached to bridge after restart. > Here is the snippet from logs during restart: > 2019-05-15T21:14:54.172Z|00078|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 > 2019-05-15T21:14:54.172Z|00079|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb > 2019-05-15T21:14:54.174Z|00080|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) > > Am I missing any step? Openvswitch documentation does not specify anything else. > > # ovs-vswitchd --version > ovs-vswitchd (Open vSwitch) 2.9.3 > DPDK 17.11.0 > > Regards, > Shriroop > > > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Unable to hotplug interface with DPDK initialized
Hi Ian I am using Intel i350 PCIe NIC. I have attached entire ovs-vswitchd.log which shows both error and success. After attaching the interface, I restarted openvswitch service and it was able to attach the interface. Regards Shriroop On 5/16/19, 7:35 AM, "Ian Stokes" wrote: On 5/15/2019 10:37 PM, Shriroop Joshi (shrirjos) via discuss wrote: > Hello ovs community > > I am trying to attach an interface (eth4-4) to a bridge (svc-br) with netdev datapath_type. Here are the steps which I follow: > 1. Load vfio-pci driver > 2. Bind vfio-pci driver to interface eth4-4 > (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#dpdk-binding-nics) > 3. Hotplug this interface to DPDK bridge: > # ovs-vsctl add-port svc-br eth4-4 -- set interface eth4-4 type=dpdk options:dpdk-devargs=:af:00.3 > (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#hotplugging) > > After completing these steps, I see that it shows 'Error attaching device ':af:00.3' to DPDK' error. > Here are the error logs: > 2019-05-15T21:11:20.663Z|00227|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 > 2019-05-15T21:11:20.663Z|00228|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb > 2019-05-15T21:11:20.663Z|00229|dpdk|ERR|EAL: Driver cannot attach the device (:af:00.3) > 2019-05-15T21:11:20.663Z|00230|netdev_dpdk|WARN|Error attaching device ':af:00.3' to DPDK > 2019-05-15T21:11:20.663Z|00231|netdev|WARN|eth4-4: could not set configuration (Invalid argument) > I tried to reproduce this on my local system but could not. What model HW NIC are you attempting to hotplug out of interest? Also could you provide the full log for when OVS fails to hotplug as well as when it succeeds? I'd be interested in seeing the OVS configuration commands passed for DPDK. Ian > I noticed that restarting openvswitch solves this issue, and I can see the interface attached to bridge after restart. > Here is the snippet from logs during restart: > 2019-05-15T21:14:54.172Z|00078|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 > 2019-05-15T21:14:54.172Z|00079|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb > 2019-05-15T21:14:54.174Z|00080|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) > > Am I missing any step? Openvswitch documentation does not specify anything else. > > # ovs-vswitchd --version > ovs-vswitchd (Open vSwitch) 2.9.3 > DPDK 17.11.0 > > Regards, > Shriroop > > > > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > ovs-vswitchd.log Description: ovs-vswitchd.log ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Unable to hotplug interface with DPDK initialized
On 5/15/2019 10:37 PM, Shriroop Joshi (shrirjos) via discuss wrote: Hello ovs community I am trying to attach an interface (eth4-4) to a bridge (svc-br) with netdev datapath_type. Here are the steps which I follow: 1. Load vfio-pci driver 2. Bind vfio-pci driver to interface eth4-4 (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#dpdk-binding-nics) 3. Hotplug this interface to DPDK bridge: # ovs-vsctl add-port svc-br eth4-4 -- set interface eth4-4 type=dpdk options:dpdk-devargs=:af:00.3 (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#hotplugging) After completing these steps, I see that it shows 'Error attaching device ':af:00.3' to DPDK' error. Here are the error logs: 2019-05-15T21:11:20.663Z|00227|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 2019-05-15T21:11:20.663Z|00228|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2019-05-15T21:11:20.663Z|00229|dpdk|ERR|EAL: Driver cannot attach the device (:af:00.3) 2019-05-15T21:11:20.663Z|00230|netdev_dpdk|WARN|Error attaching device ':af:00.3' to DPDK 2019-05-15T21:11:20.663Z|00231|netdev|WARN|eth4-4: could not set configuration (Invalid argument) I tried to reproduce this on my local system but could not. What model HW NIC are you attempting to hotplug out of interest? Also could you provide the full log for when OVS fails to hotplug as well as when it succeeds? I'd be interested in seeing the OVS configuration commands passed for DPDK. Ian I noticed that restarting openvswitch solves this issue, and I can see the interface attached to bridge after restart. Here is the snippet from logs during restart: 2019-05-15T21:14:54.172Z|00078|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 2019-05-15T21:14:54.172Z|00079|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2019-05-15T21:14:54.174Z|00080|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) Am I missing any step? Openvswitch documentation does not specify anything else. # ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.9.3 DPDK 17.11.0 Regards, Shriroop ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] Unable to hotplug interface with DPDK initialized
Hello ovs community I am trying to attach an interface (eth4-4) to a bridge (svc-br) with netdev datapath_type. Here are the steps which I follow: 1. Load vfio-pci driver 2. Bind vfio-pci driver to interface eth4-4 (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#dpdk-binding-nics) 3. Hotplug this interface to DPDK bridge: # ovs-vsctl add-port svc-br eth4-4 -- set interface eth4-4 type=dpdk options:dpdk-devargs=:af:00.3 (following steps as mentioned here: http://docs.openvswitch.org/en/latest/topics/dpdk/phy/#hotplugging) After completing these steps, I see that it shows 'Error attaching device ':af:00.3' to DPDK' error. Here are the error logs: 2019-05-15T21:11:20.663Z|00227|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 2019-05-15T21:11:20.663Z|00228|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2019-05-15T21:11:20.663Z|00229|dpdk|ERR|EAL: Driver cannot attach the device (:af:00.3) 2019-05-15T21:11:20.663Z|00230|netdev_dpdk|WARN|Error attaching device ':af:00.3' to DPDK 2019-05-15T21:11:20.663Z|00231|netdev|WARN|eth4-4: could not set configuration (Invalid argument) I noticed that restarting openvswitch solves this issue, and I can see the interface attached to bridge after restart. Here is the snippet from logs during restart: 2019-05-15T21:14:54.172Z|00078|dpdk|INFO|EAL: PCI device :af:00.3 on NUMA socket 1 2019-05-15T21:14:54.172Z|00079|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2019-05-15T21:14:54.174Z|00080|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) Am I missing any step? Openvswitch documentation does not specify anything else. # ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.9.3 DPDK 17.11.0 Regards, Shriroop ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
