Re: [ovs-dev] OVS DPDK VF port representors not working

[Ravi Kerur]

Hi Ilya, OvS-DPDK team,

Please correct me if I am wrong. This feature (port representor) seems
broken (non functional) to me as I witnessed 2 basic configuration issues
which should have worked. Moreover, when I look at the patch series it is
from Mellanox and not from OvS/Intel. I am not sure whether this has been
tested correctly. We need this feature and I can volunteer to test it if
needed so kindly let me know what the next steps should be if you agree or
please suggest how to proceed.

Thanks,
Ravi

On Mon, Apr 20, 2020 at 1:04 PM Ravi Kerur  wrote:

>
>
> On Mon, Apr 20, 2020 at 10:37 AM Ravi Kerur  wrote:
>
>>
>>
>> On Mon, Apr 20, 2020 at 3:14 AM Ilya Maximets  wrote:
>>
>>> On 4/19/20 6:08 PM, Ravi Kerur wrote:
>>> >
>>> >
>>> > On Sun, Apr 19, 2020 at 6:14 AM Ilya Maximets >> > wrote:
>>> >
>>> > On 4/18/20 2:33 PM, Ravi Kerur wrote:
>>> > >
>>> > >
>>> > > On Fri, Apr 17, 2020 at 9:57 AM Ilya Maximets <
>>> i.maxim...@ovn.org  >> i.maxim...@ovn.org >> wrote:
>>> > >
>>> > > On 4/16/20 11:00 PM, Ravi Kerur wrote:
>>> > > > Hello OvS-DPDK team,
>>> > > >
>>> > > > We are expanding our usage of OvS-DPDK and want to use VF
>>> port representors
>>> > > > for one use-case. Going through the following document it
>>> is unclear how
>>> > > > this functionality can be achieved
>>> > > >
>>> > > > http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
>>> > > >
>>> > > > I believe following things need to be configured before
>>> configuring OvS, I
>>> > > > am using Intel NICs for evaluation
>>> > > >
>>> > > > (1) Create VFs on Intel NICs(82599). This can only happen
>>> when Intel NIC is
>>> > > > bound to its respective kernel driver (ixgbe) via
>>> > > >
>>> > > > echo 2 > /sys/devices//sriov_numvfs
>>> > > >
>>> > > >
>>> > > > It creates 2 VF interfaces bound to ixgbevf driver
>>> > > >
>>> > > >
>>> > > > (2) Using driverctl, bind VF and PF interfaces to
>>> vfio-pci. Moment PF
>>> > > > device is bound to vfio-pci, VF interfaces vanish
>>> > > >
>>> > > >
>>> > > > Question based on the OvS document shown above
>>> > > >
>>> > > >
>>> > > > (1) Binding PF interface?
>>> > > >
>>> > > >
>>> > > > ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf
>>> type=dpdk
>>> > > > options:dpdk-devargs=:08:00.0
>>> > > >
>>> > > >
>>> > > > What driver interface <:08:00.0> is bound to? igb_uio,
>>> vfio-pci or
>>> > > > kernel driver?
>>> > >
>>> > > I believe that it was tested with igb_uio driver, but I
>>> never tried it myself.
>>> > >
>>> > > In order to have sr-iov configuration support with vfio-pci
>>> driver you need
>>> > > a brand new v5.7-rc1 kernel and DPDK changes that are not
>>> merged yet:
>>> > > https://patchwork.dpdk.org/project/dpdk/list/?series=9356
>>> > > Not sure if it will work smoothly with OVS out of the box,
>>> though.
>>> > >
>>> > >
>>> > > Thanks Ilya for the link. Will look into it.
>>> > >
>>> > > Unfortunately igb_uio doesn't work either. Basics of creating
>>> VF's out of PF and assigning them to DPDK driver itself is not working.
>>> > > My understanding is
>>> > > (1) In order to create VF's NIC should be bound to its kernel
>>> driver. Cannot create VFs when NIC is bound to igb_uio or vfio-pci
>>> > > (2) No mechanism exists to bind both PF and VF to igb_uio to use
>>> this feature
>>> > >
>>> > > Output below shows what I am observing on :18:00.0
>>> > >
>>> > > (1) PCI device has PF and 1 VF, both assigned to kernel driver
>>> > >
>>> > > driverctl list-devices network
>>> > >
>>> > > :04:00.0 igb
>>> > >
>>> > > *:18:00.0 ixgbe*
>>> > >
>>> > > :18:00.1 ixgbe
>>> > >
>>> > > *:18:10.0 ixgbevf*
>>> > >
>>> > > :b3:00.0 vfio-pci [*]
>>> > >
>>> > > :b3:00.1 ixgbe
>>> > >
>>> > >
>>> > > (2) Bind :18:00.0 to igb_uio, VF device is gone
>>> > > # driverctl set-override :18:00.0 igb_uio
>>> > >
>>> > > #driverctl list-devices network
>>> > >
>>> > > :04:00.0 igb
>>> > >
>>> > > *:18:00.0 igb_uio [*]*
>>> > >
>>> > > :18:00.1 ixgbe
>>> > >
>>> > > :b3:00.0 vfio-pci [*]
>>> > >
>>> > > :b3:00.1 ixgbe
>>> > >
>>> > >
>>> > > (3) Try to create VF when bound to igb_uio
>>> > >
>>> > >
>>> > > echo 1 > /sys/bus/pci/devices/\:18\:00.0/sriov_numvfs
>>> > >
>>> > > bash: echo: write error: No such file or directory
>>> >
>>> > OVS and DPDK docs both says that knob shoul

Re: [ovs-dev] OVS DPDK VF port representors not working

[Ravi Kerur]

On Mon, Apr 20, 2020 at 10:37 AM Ravi Kerur  wrote:

>
>
> On Mon, Apr 20, 2020 at 3:14 AM Ilya Maximets  wrote:
>
>> On 4/19/20 6:08 PM, Ravi Kerur wrote:
>> >
>> >
>> > On Sun, Apr 19, 2020 at 6:14 AM Ilya Maximets > > wrote:
>> >
>> > On 4/18/20 2:33 PM, Ravi Kerur wrote:
>> > >
>> > >
>> > > On Fri, Apr 17, 2020 at 9:57 AM Ilya Maximets >   i.maxim...@ovn.org>>> wrote:
>> > >
>> > > On 4/16/20 11:00 PM, Ravi Kerur wrote:
>> > > > Hello OvS-DPDK team,
>> > > >
>> > > > We are expanding our usage of OvS-DPDK and want to use VF
>> port representors
>> > > > for one use-case. Going through the following document it
>> is unclear how
>> > > > this functionality can be achieved
>> > > >
>> > > > http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
>> > > >
>> > > > I believe following things need to be configured before
>> configuring OvS, I
>> > > > am using Intel NICs for evaluation
>> > > >
>> > > > (1) Create VFs on Intel NICs(82599). This can only happen
>> when Intel NIC is
>> > > > bound to its respective kernel driver (ixgbe) via
>> > > >
>> > > > echo 2 > /sys/devices//sriov_numvfs
>> > > >
>> > > >
>> > > > It creates 2 VF interfaces bound to ixgbevf driver
>> > > >
>> > > >
>> > > > (2) Using driverctl, bind VF and PF interfaces to vfio-pci.
>> Moment PF
>> > > > device is bound to vfio-pci, VF interfaces vanish
>> > > >
>> > > >
>> > > > Question based on the OvS document shown above
>> > > >
>> > > >
>> > > > (1) Binding PF interface?
>> > > >
>> > > >
>> > > > ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf
>> type=dpdk
>> > > > options:dpdk-devargs=:08:00.0
>> > > >
>> > > >
>> > > > What driver interface <:08:00.0> is bound to? igb_uio,
>> vfio-pci or
>> > > > kernel driver?
>> > >
>> > > I believe that it was tested with igb_uio driver, but I never
>> tried it myself.
>> > >
>> > > In order to have sr-iov configuration support with vfio-pci
>> driver you need
>> > > a brand new v5.7-rc1 kernel and DPDK changes that are not
>> merged yet:
>> > > https://patchwork.dpdk.org/project/dpdk/list/?series=9356
>> > > Not sure if it will work smoothly with OVS out of the box,
>> though.
>> > >
>> > >
>> > > Thanks Ilya for the link. Will look into it.
>> > >
>> > > Unfortunately igb_uio doesn't work either. Basics of creating
>> VF's out of PF and assigning them to DPDK driver itself is not working.
>> > > My understanding is
>> > > (1) In order to create VF's NIC should be bound to its kernel
>> driver. Cannot create VFs when NIC is bound to igb_uio or vfio-pci
>> > > (2) No mechanism exists to bind both PF and VF to igb_uio to use
>> this feature
>> > >
>> > > Output below shows what I am observing on :18:00.0
>> > >
>> > > (1) PCI device has PF and 1 VF, both assigned to kernel driver
>> > >
>> > > driverctl list-devices network
>> > >
>> > > :04:00.0 igb
>> > >
>> > > *:18:00.0 ixgbe*
>> > >
>> > > :18:00.1 ixgbe
>> > >
>> > > *:18:10.0 ixgbevf*
>> > >
>> > > :b3:00.0 vfio-pci [*]
>> > >
>> > > :b3:00.1 ixgbe
>> > >
>> > >
>> > > (2) Bind :18:00.0 to igb_uio, VF device is gone
>> > > # driverctl set-override :18:00.0 igb_uio
>> > >
>> > > #driverctl list-devices network
>> > >
>> > > :04:00.0 igb
>> > >
>> > > *:18:00.0 igb_uio [*]*
>> > >
>> > > :18:00.1 ixgbe
>> > >
>> > > :b3:00.0 vfio-pci [*]
>> > >
>> > > :b3:00.1 ixgbe
>> > >
>> > >
>> > > (3) Try to create VF when bound to igb_uio
>> > >
>> > >
>> > > echo 1 > /sys/bus/pci/devices/\:18\:00.0/sriov_numvfs
>> > >
>> > > bash: echo: write error: No such file or directory
>> >
>> > OVS and DPDK docs both says that knob should be "max_vfs", not
>> "sriov_numvfs".
>> > I didn't test, but looking at the code of igb_uio driver it
>> actually creates
>> > this sysfs entry.
>> >
>> >
>> > Thanks for your kind help. My mistake. On Ubuntu max_vfs is created in
>> >
>> > /sys/devices/pci:16/:16:02.0/:18:00.0/max_vfs instead
>> of usual
>> >
>> > /sys/bus/pci/devices/\:18\:00.0/
>> >
>> >  I am able to create VF's when PF is attached to igb_uio.
>> > When adding dpdk-pf port to bridge I get following errors (OvS 2.13.90,
>> DPDK 19.11). Please note I have started with empty conf.db in OvS.
>> >
>> > driverctl list-devices network
>> >
>> > :04:00.0 igb
>> >
>> > :18:00.0 igb_uio [*]
>> >
>> > :18:00.1 ixg

Re: [ovs-dev] OVS DPDK VF port representors not working

[Ravi Kerur]

On Mon, Apr 20, 2020 at 3:14 AM Ilya Maximets  wrote:

> On 4/19/20 6:08 PM, Ravi Kerur wrote:
> >
> >
> > On Sun, Apr 19, 2020 at 6:14 AM Ilya Maximets  > wrote:
> >
> > On 4/18/20 2:33 PM, Ravi Kerur wrote:
> > >
> > >
> > > On Fri, Apr 17, 2020 at 9:57 AM Ilya Maximets   >> wrote:
> > >
> > > On 4/16/20 11:00 PM, Ravi Kerur wrote:
> > > > Hello OvS-DPDK team,
> > > >
> > > > We are expanding our usage of OvS-DPDK and want to use VF
> port representors
> > > > for one use-case. Going through the following document it is
> unclear how
> > > > this functionality can be achieved
> > > >
> > > > http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
> > > >
> > > > I believe following things need to be configured before
> configuring OvS, I
> > > > am using Intel NICs for evaluation
> > > >
> > > > (1) Create VFs on Intel NICs(82599). This can only happen
> when Intel NIC is
> > > > bound to its respective kernel driver (ixgbe) via
> > > >
> > > > echo 2 > /sys/devices//sriov_numvfs
> > > >
> > > >
> > > > It creates 2 VF interfaces bound to ixgbevf driver
> > > >
> > > >
> > > > (2) Using driverctl, bind VF and PF interfaces to vfio-pci.
> Moment PF
> > > > device is bound to vfio-pci, VF interfaces vanish
> > > >
> > > >
> > > > Question based on the OvS document shown above
> > > >
> > > >
> > > > (1) Binding PF interface?
> > > >
> > > >
> > > > ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf
> type=dpdk
> > > > options:dpdk-devargs=:08:00.0
> > > >
> > > >
> > > > What driver interface <:08:00.0> is bound to? igb_uio,
> vfio-pci or
> > > > kernel driver?
> > >
> > > I believe that it was tested with igb_uio driver, but I never
> tried it myself.
> > >
> > > In order to have sr-iov configuration support with vfio-pci
> driver you need
> > > a brand new v5.7-rc1 kernel and DPDK changes that are not
> merged yet:
> > > https://patchwork.dpdk.org/project/dpdk/list/?series=9356
> > > Not sure if it will work smoothly with OVS out of the box,
> though.
> > >
> > >
> > > Thanks Ilya for the link. Will look into it.
> > >
> > > Unfortunately igb_uio doesn't work either. Basics of creating VF's
> out of PF and assigning them to DPDK driver itself is not working.
> > > My understanding is
> > > (1) In order to create VF's NIC should be bound to its kernel
> driver. Cannot create VFs when NIC is bound to igb_uio or vfio-pci
> > > (2) No mechanism exists to bind both PF and VF to igb_uio to use
> this feature
> > >
> > > Output below shows what I am observing on :18:00.0
> > >
> > > (1) PCI device has PF and 1 VF, both assigned to kernel driver
> > >
> > > driverctl list-devices network
> > >
> > > :04:00.0 igb
> > >
> > > *:18:00.0 ixgbe*
> > >
> > > :18:00.1 ixgbe
> > >
> > > *:18:10.0 ixgbevf*
> > >
> > > :b3:00.0 vfio-pci [*]
> > >
> > > :b3:00.1 ixgbe
> > >
> > >
> > > (2) Bind :18:00.0 to igb_uio, VF device is gone
> > > # driverctl set-override :18:00.0 igb_uio
> > >
> > > #driverctl list-devices network
> > >
> > > :04:00.0 igb
> > >
> > > *:18:00.0 igb_uio [*]*
> > >
> > > :18:00.1 ixgbe
> > >
> > > :b3:00.0 vfio-pci [*]
> > >
> > > :b3:00.1 ixgbe
> > >
> > >
> > > (3) Try to create VF when bound to igb_uio
> > >
> > >
> > > echo 1 > /sys/bus/pci/devices/\:18\:00.0/sriov_numvfs
> > >
> > > bash: echo: write error: No such file or directory
> >
> > OVS and DPDK docs both says that knob should be "max_vfs", not
> "sriov_numvfs".
> > I didn't test, but looking at the code of igb_uio driver it actually
> creates
> > this sysfs entry.
> >
> >
> > Thanks for your kind help. My mistake. On Ubuntu max_vfs is created in
> >
> > /sys/devices/pci:16/:16:02.0/:18:00.0/max_vfs instead
> of usual
> >
> > /sys/bus/pci/devices/\:18\:00.0/
> >
> >  I am able to create VF's when PF is attached to igb_uio.
> > When adding dpdk-pf port to bridge I get following errors (OvS 2.13.90,
> DPDK 19.11). Please note I have started with empty conf.db in OvS.
> >
> > driverctl list-devices network
> >
> > :04:00.0 igb
> >
> > :18:00.0 igb_uio [*]
> >
> > :18:00.1 ixgbe
> >
> > :18:10.0 vfio-pci [*]
> >
> > :18:10.2 vfio-pci [*]
> >
> > :b3:00.0 vfio-pci [*]
> >
> > :b3:00.1 ixgbe
> >
> >
> > ovs-vsctl add-br br-phy -- set bridge br-phy datapath_type=netd

Re: [ovs-dev] OVS DPDK VF port representors not working

[Ilya Maximets]

On 4/19/20 6:08 PM, Ravi Kerur wrote:
> 
> 
> On Sun, Apr 19, 2020 at 6:14 AM Ilya Maximets  > wrote:
> 
> On 4/18/20 2:33 PM, Ravi Kerur wrote:
> >
> >
> > On Fri, Apr 17, 2020 at 9:57 AM Ilya Maximets    >> wrote:
> >
> >     On 4/16/20 11:00 PM, Ravi Kerur wrote:
> >     > Hello OvS-DPDK team,
> >     >
> >     > We are expanding our usage of OvS-DPDK and want to use VF port 
> representors
> >     > for one use-case. Going through the following document it is 
> unclear how
> >     > this functionality can be achieved
> >     >
> >     > http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
> >     >
> >     > I believe following things need to be configured before 
> configuring OvS, I
> >     > am using Intel NICs for evaluation
> >     >
> >     > (1) Create VFs on Intel NICs(82599). This can only happen when 
> Intel NIC is
> >     > bound to its respective kernel driver (ixgbe) via
> >     >
> >     > echo 2 > /sys/devices//sriov_numvfs
> >     >
> >     >
> >     > It creates 2 VF interfaces bound to ixgbevf driver
> >     >
> >     >
> >     > (2) Using driverctl, bind VF and PF interfaces to vfio-pci. 
> Moment PF
> >     > device is bound to vfio-pci, VF interfaces vanish
> >     >
> >     >
> >     > Question based on the OvS document shown above
> >     >
> >     >
> >     > (1) Binding PF interface?
> >     >
> >     >
> >     > ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk
> >     > options:dpdk-devargs=:08:00.0
> >     >
> >     >
> >     > What driver interface <:08:00.0> is bound to? igb_uio, 
> vfio-pci or
> >     > kernel driver?
> >
> >     I believe that it was tested with igb_uio driver, but I never tried 
> it myself.
> >
> >     In order to have sr-iov configuration support with vfio-pci driver 
> you need
> >     a brand new v5.7-rc1 kernel and DPDK changes that are not merged 
> yet:
> >     https://patchwork.dpdk.org/project/dpdk/list/?series=9356
> >     Not sure if it will work smoothly with OVS out of the box, though.
> >
> >
> > Thanks Ilya for the link. Will look into it. 
> >
> > Unfortunately igb_uio doesn't work either. Basics of creating VF's out 
> of PF and assigning them to DPDK driver itself is not working.
> > My understanding is
> > (1) In order to create VF's NIC should be bound to its kernel driver. 
> Cannot create VFs when NIC is bound to igb_uio or vfio-pci
> > (2) No mechanism exists to bind both PF and VF to igb_uio to use this 
> feature
> >
> > Output below shows what I am observing on :18:00.0
> >
> > (1) PCI device has PF and 1 VF, both assigned to kernel driver
> >
> > driverctl list-devices network
> >
> > :04:00.0 igb
> >
> > *:18:00.0 ixgbe*
> >
> > :18:00.1 ixgbe
> >
> > *:18:10.0 ixgbevf*
> >
> > :b3:00.0 vfio-pci [*]
> >
> > :b3:00.1 ixgbe
> >
> >
> > (2) Bind :18:00.0 to igb_uio, VF device is gone
> > # driverctl set-override :18:00.0 igb_uio 
> >
> > #driverctl list-devices network
> >
> > :04:00.0 igb
> >
> > *:18:00.0 igb_uio [*]*
> >
> > :18:00.1 ixgbe
> >
> > :b3:00.0 vfio-pci [*]
> >
> > :b3:00.1 ixgbe
> >
> >
> > (3) Try to create VF when bound to igb_uio
> >
> >
> > echo 1 > /sys/bus/pci/devices/\:18\:00.0/sriov_numvfs
> >
> > bash: echo: write error: No such file or directory
> 
> OVS and DPDK docs both says that knob should be "max_vfs", not 
> "sriov_numvfs".
> I didn't test, but looking at the code of igb_uio driver it actually 
> creates
> this sysfs entry.
> 
> 
> Thanks for your kind help. My mistake. On Ubuntu max_vfs is created in 
> 
> /sys/devices/pci:16/:16:02.0/:18:00.0/max_vfs instead of usual
> 
> /sys/bus/pci/devices/\:18\:00.0/
> 
>  I am able to create VF's when PF is attached to igb_uio. 
> When adding dpdk-pf port to bridge I get following errors (OvS 2.13.90, DPDK 
> 19.11). Please note I have started with empty conf.db in OvS.
> 
> driverctl list-devices network
> 
> :04:00.0 igb
> 
> :18:00.0 igb_uio [*]
> 
> :18:00.1 ixgbe
> 
> :18:10.0 vfio-pci [*]
> 
> :18:10.2 vfio-pci [*]
> 
> :b3:00.0 vfio-pci [*]
> 
> :b3:00.1 ixgbe
> 
> 
> ovs-vsctl add-br br-phy -- set bridge br-phy datapath_type=netdev
> 
> ovs-vsctl add-port br-phy dpdk-pf -- set Interface dpdk-pf type=dpdk 
> options:dpdk-devargs=:18:00.0
> 
> ovs-vsctl: Error detected while setting up 'dpdk-pf': could not add network 
> device dpdk-pf to ofproto (Invalid argument).  See ovs-vswitchd log for 
> details.
> 
> 

Re: [ovs-dev] OVS DPDK VF port representors not working

[Ravi Kerur]

On Sun, Apr 19, 2020 at 6:14 AM Ilya Maximets  wrote:

> On 4/18/20 2:33 PM, Ravi Kerur wrote:
> >
> >
> > On Fri, Apr 17, 2020 at 9:57 AM Ilya Maximets  > wrote:
> >
> > On 4/16/20 11:00 PM, Ravi Kerur wrote:
> > > Hello OvS-DPDK team,
> > >
> > > We are expanding our usage of OvS-DPDK and want to use VF port
> representors
> > > for one use-case. Going through the following document it is
> unclear how
> > > this functionality can be achieved
> > >
> > > http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
> > >
> > > I believe following things need to be configured before
> configuring OvS, I
> > > am using Intel NICs for evaluation
> > >
> > > (1) Create VFs on Intel NICs(82599). This can only happen when
> Intel NIC is
> > > bound to its respective kernel driver (ixgbe) via
> > >
> > > echo 2 > /sys/devices//sriov_numvfs
> > >
> > >
> > > It creates 2 VF interfaces bound to ixgbevf driver
> > >
> > >
> > > (2) Using driverctl, bind VF and PF interfaces to vfio-pci. Moment
> PF
> > > device is bound to vfio-pci, VF interfaces vanish
> > >
> > >
> > > Question based on the OvS document shown above
> > >
> > >
> > > (1) Binding PF interface?
> > >
> > >
> > > ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk
> > > options:dpdk-devargs=:08:00.0
> > >
> > >
> > > What driver interface <:08:00.0> is bound to? igb_uio,
> vfio-pci or
> > > kernel driver?
> >
> > I believe that it was tested with igb_uio driver, but I never tried
> it myself.
> >
> > In order to have sr-iov configuration support with vfio-pci driver
> you need
> > a brand new v5.7-rc1 kernel and DPDK changes that are not merged yet:
> > https://patchwork.dpdk.org/project/dpdk/list/?series=9356
> > Not sure if it will work smoothly with OVS out of the box, though.
> >
> >
> > Thanks Ilya for the link. Will look into it.
> >
> > Unfortunately igb_uio doesn't work either. Basics of creating VF's out
> of PF and assigning them to DPDK driver itself is not working.
> > My understanding is
> > (1) In order to create VF's NIC should be bound to its kernel driver.
> Cannot create VFs when NIC is bound to igb_uio or vfio-pci
> > (2) No mechanism exists to bind both PF and VF to igb_uio to use this
> feature
> >
> > Output below shows what I am observing on :18:00.0
> >
> > (1) PCI device has PF and 1 VF, both assigned to kernel driver
> >
> > driverctl list-devices network
> >
> > :04:00.0 igb
> >
> > *:18:00.0 ixgbe*
> >
> > :18:00.1 ixgbe
> >
> > *:18:10.0 ixgbevf*
> >
> > :b3:00.0 vfio-pci [*]
> >
> > :b3:00.1 ixgbe
> >
> >
> > (2) Bind :18:00.0 to igb_uio, VF device is gone
> > # driverctl set-override :18:00.0 igb_uio
> >
> > #driverctl list-devices network
> >
> > :04:00.0 igb
> >
> > *:18:00.0 igb_uio [*]*
> >
> > :18:00.1 ixgbe
> >
> > :b3:00.0 vfio-pci [*]
> >
> > :b3:00.1 ixgbe
> >
> >
> > (3) Try to create VF when bound to igb_uio
> >
> >
> > echo 1 > /sys/bus/pci/devices/\:18\:00.0/sriov_numvfs
> >
> > bash: echo: write error: No such file or directory
>
> OVS and DPDK docs both says that knob should be "max_vfs", not
> "sriov_numvfs".
> I didn't test, but looking at the code of igb_uio driver it actually
> creates
> this sysfs entry.
>
>
Thanks for your kind help. My mistake. On Ubuntu max_vfs is created in

/sys/devices/pci:16/:16:02.0/:18:00.0/max_vfs instead of usual

/sys/bus/pci/devices/\:18\:00.0/

 I am able to create VF's when PF is attached to igb_uio.
When adding dpdk-pf port to bridge I get following errors (OvS 2.13.90,
DPDK 19.11). Please note I have started with empty conf.db in OvS.

driverctl list-devices network

:04:00.0 igb

:18:00.0 igb_uio [*]

:18:00.1 ixgbe

:18:10.0 vfio-pci [*]

:18:10.2 vfio-pci [*]

:b3:00.0 vfio-pci [*]

:b3:00.1 ixgbe

ovs-vsctl add-br br-phy -- set bridge br-phy datapath_type=netdev

ovs-vsctl add-port br-phy dpdk-pf -- set Interface dpdk-pf type=dpdk
options:dpdk-devargs=:18:00.0

ovs-vsctl: Error detected while setting up 'dpdk-pf': could not add network
device dpdk-pf to ofproto (Invalid argument).  See ovs-vswitchd log for
details.

ovs-vsctl: The default log directory is "/usr/local/var/log/openvswitch".


Snippets below



2020-04-19T15:50:54.279Z|00170|dpif_netdev|INFO|PMD thread on numa_id: 0,
core id:  9 created.

2020-04-19T15:50:54.448Z|00171|dpif_netdev|INFO|PMD thread on numa_id: 0,
core id:  6 created.

2020-04-19T15:50:54.617Z|00172|dpif_netdev|INFO|PMD thread on numa_id: 0,
core id:  8 created.

2020-04-19T15:50:54.786Z|00173|dpif_netdev|INFO|PMD thread on numa_id: 0,
core id:  7 created.

2020-04-19T15:50:54.786Z|00174|dpif_netdev|INFO|There are 4 pmd threads on
numa node 0

2020-04-19T15:50:54.786Z|00175|dpdk|INFO|Device with port_id=0 already
stopped

Re: [ovs-dev] OVS DPDK VF port representors not working

[Ilya Maximets]

On 4/18/20 2:33 PM, Ravi Kerur wrote:
> 
> 
> On Fri, Apr 17, 2020 at 9:57 AM Ilya Maximets  > wrote:
> 
> On 4/16/20 11:00 PM, Ravi Kerur wrote:
> > Hello OvS-DPDK team,
> >
> > We are expanding our usage of OvS-DPDK and want to use VF port 
> representors
> > for one use-case. Going through the following document it is unclear how
> > this functionality can be achieved
> >
> > http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
> >
> > I believe following things need to be configured before configuring 
> OvS, I
> > am using Intel NICs for evaluation
> >
> > (1) Create VFs on Intel NICs(82599). This can only happen when Intel 
> NIC is
> > bound to its respective kernel driver (ixgbe) via
> >
> > echo 2 > /sys/devices//sriov_numvfs
> >
> >
> > It creates 2 VF interfaces bound to ixgbevf driver
> >
> >
> > (2) Using driverctl, bind VF and PF interfaces to vfio-pci. Moment PF
> > device is bound to vfio-pci, VF interfaces vanish
> >
> >
> > Question based on the OvS document shown above
> >
> >
> > (1) Binding PF interface?
> >
> >
> > ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk
> > options:dpdk-devargs=:08:00.0
> >
> >
> > What driver interface <:08:00.0> is bound to? igb_uio, vfio-pci or
> > kernel driver?
> 
> I believe that it was tested with igb_uio driver, but I never tried it 
> myself.
> 
> In order to have sr-iov configuration support with vfio-pci driver you 
> need
> a brand new v5.7-rc1 kernel and DPDK changes that are not merged yet:
> https://patchwork.dpdk.org/project/dpdk/list/?series=9356
> Not sure if it will work smoothly with OVS out of the box, though.
> 
> 
> Thanks Ilya for the link. Will look into it. 
> 
> Unfortunately igb_uio doesn't work either. Basics of creating VF's out of PF 
> and assigning them to DPDK driver itself is not working.
> My understanding is
> (1) In order to create VF's NIC should be bound to its kernel driver. Cannot 
> create VFs when NIC is bound to igb_uio or vfio-pci
> (2) No mechanism exists to bind both PF and VF to igb_uio to use this feature
> 
> Output below shows what I am observing on :18:00.0
> 
> (1) PCI device has PF and 1 VF, both assigned to kernel driver
> 
> driverctl list-devices network
> 
> :04:00.0 igb
> 
> *:18:00.0 ixgbe*
> 
> :18:00.1 ixgbe
> 
> *:18:10.0 ixgbevf*
> 
> :b3:00.0 vfio-pci [*]
> 
> :b3:00.1 ixgbe
> 
> 
> (2) Bind :18:00.0 to igb_uio, VF device is gone
> # driverctl set-override :18:00.0 igb_uio 
> 
> #driverctl list-devices network
> 
> :04:00.0 igb
> 
> *:18:00.0 igb_uio [*]*
> 
> :18:00.1 ixgbe
> 
> :b3:00.0 vfio-pci [*]
> 
> :b3:00.1 ixgbe
> 
> 
> (3) Try to create VF when bound to igb_uio
> 
> 
> echo 1 > /sys/bus/pci/devices/\:18\:00.0/sriov_numvfs
> 
> bash: echo: write error: No such file or directory

OVS and DPDK docs both says that knob should be "max_vfs", not "sriov_numvfs".
I didn't test, but looking at the code of igb_uio driver it actually creates
this sysfs entry. 

> 
> 
> (4) Unset :18:00.0 to kernel driver, can create VF
> 
> 
> # driverctl unset-override :18:00.0
> 
> #echo 1 > /sys/bus/pci/devices/\:18\:00.0/sriov_numvfs
> 
> #driverctl list-devices network
> 
> :04:00.0 igb
> 
> *:18:00.0 ixgbe*
> 
> :18:00.1 ixgbe
> 
> *:18:10.0 ixgbevf*
> 
> :b3:00.0 vfio-pci [*]
> 
> :b3:00.1 ixgbe
> 
> 
> Without binding both PF and VF to userspace drivers to use DPDK PMD it is not 
> clear to me how this functionality will work?
> 
> 
> >
> >
> > (2) VF interface to representor connection, does representor=[3] 
> represent
> > VF device 3?
> >
> >
> > ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk
> > options:dpdk-devargs=:08:00.0,representor=[3]
> >
> > ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5 type=dpdk
> > options:dpdk-devargs=:08:00.0,representor=[5]
> >
> >
> > Looking into linked DPDK document within OvS is confusing as well.
> >
> >
> > Kindly let me know what I need to do for the setup?
> >
> >
> > Thanks,
> >
> > Ravi
> > ___
> > dev mailing list
> > d...@openvswitch.org 
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
> 

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] OVS DPDK VF port representors not working

[Ravi Kerur]

On Fri, Apr 17, 2020 at 9:57 AM Ilya Maximets  wrote:

> On 4/16/20 11:00 PM, Ravi Kerur wrote:
> > Hello OvS-DPDK team,
> >
> > We are expanding our usage of OvS-DPDK and want to use VF port
> representors
> > for one use-case. Going through the following document it is unclear how
> > this functionality can be achieved
> >
> > http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
> >
> > I believe following things need to be configured before configuring OvS,
> I
> > am using Intel NICs for evaluation
> >
> > (1) Create VFs on Intel NICs(82599). This can only happen when Intel NIC
> is
> > bound to its respective kernel driver (ixgbe) via
> >
> > echo 2 > /sys/devices//sriov_numvfs
> >
> >
> > It creates 2 VF interfaces bound to ixgbevf driver
> >
> >
> > (2) Using driverctl, bind VF and PF interfaces to vfio-pci. Moment PF
> > device is bound to vfio-pci, VF interfaces vanish
> >
> >
> > Question based on the OvS document shown above
> >
> >
> > (1) Binding PF interface?
> >
> >
> > ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk
> > options:dpdk-devargs=:08:00.0
> >
> >
> > What driver interface <:08:00.0> is bound to? igb_uio, vfio-pci or
> > kernel driver?
>
> I believe that it was tested with igb_uio driver, but I never tried it
> myself.
>
> In order to have sr-iov configuration support with vfio-pci driver you need
> a brand new v5.7-rc1 kernel and DPDK changes that are not merged yet:
> https://patchwork.dpdk.org/project/dpdk/list/?series=9356
> Not sure if it will work smoothly with OVS out of the box, though.
>

Thanks Ilya for the link. Will look into it.

Unfortunately igb_uio doesn't work either. Basics of creating VF's out of
PF and assigning them to DPDK driver itself is not working.
My understanding is
(1) In order to create VF's NIC should be bound to its kernel driver.
Cannot create VFs when NIC is bound to igb_uio or vfio-pci
(2) No mechanism exists to bind both PF and VF to igb_uio to use this
feature

Output below shows what I am observing on :18:00.0

(1) PCI device has PF and 1 VF, both assigned to kernel driver

driverctl list-devices network

:04:00.0 igb

*:18:00.0 ixgbe*

:18:00.1 ixgbe

*:18:10.0 ixgbevf*

:b3:00.0 vfio-pci [*]

:b3:00.1 ixgbe

(2) Bind :18:00.0 to igb_uio, VF device is gone
# driverctl set-override :18:00.0 igb_uio

# driverctl list-devices network

:04:00.0 igb

*:18:00.0 igb_uio [*]*

:18:00.1 ixgbe

:b3:00.0 vfio-pci [*]

:b3:00.1 ixgbe


(3) Try to create VF when bound to igb_uio


echo 1 > /sys/bus/pci/devices/\:18\:00.0/sriov_numvfs

bash: echo: write error: No such file or directory


(4) Unset :18:00.0 to kernel driver, can create VF


# driverctl unset-override :18:00.0

# echo 1 > /sys/bus/pci/devices/\:18\:00.0/sriov_numvfs

# driverctl list-devices network

:04:00.0 igb

*:18:00.0 ixgbe*

:18:00.1 ixgbe

*:18:10.0 ixgbevf*

:b3:00.0 vfio-pci [*]

:b3:00.1 ixgbe

Without binding both PF and VF to userspace drivers to use DPDK PMD it is
not clear to me how this functionality will work?

>
> >
> >
> > (2) VF interface to representor connection, does representor=[3]
> represent
> > VF device 3?
> >
> >
> > ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk
> > options:dpdk-devargs=:08:00.0,representor=[3]
> >
> > ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5 type=dpdk
> > options:dpdk-devargs=:08:00.0,representor=[5]
> >
> >
> > Looking into linked DPDK document within OvS is confusing as well.
> >
> >
> > Kindly let me know what I need to do for the setup?
> >
> >
> > Thanks,
> >
> > Ravi
> > ___
> > dev mailing list
> > d...@openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> >
>
>
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

Re: [ovs-dev] OVS DPDK VF port representors not working

[Ilya Maximets]

On 4/16/20 11:00 PM, Ravi Kerur wrote:
> Hello OvS-DPDK team,
> 
> We are expanding our usage of OvS-DPDK and want to use VF port representors
> for one use-case. Going through the following document it is unclear how
> this functionality can be achieved
> 
> http://docs.openvswitch.org/en/latest/topics/dpdk/phy/
> 
> I believe following things need to be configured before configuring OvS, I
> am using Intel NICs for evaluation
> 
> (1) Create VFs on Intel NICs(82599). This can only happen when Intel NIC is
> bound to its respective kernel driver (ixgbe) via
> 
> echo 2 > /sys/devices//sriov_numvfs
> 
> 
> It creates 2 VF interfaces bound to ixgbevf driver
> 
> 
> (2) Using driverctl, bind VF and PF interfaces to vfio-pci. Moment PF
> device is bound to vfio-pci, VF interfaces vanish
> 
> 
> Question based on the OvS document shown above
> 
> 
> (1) Binding PF interface?
> 
> 
> ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk
> options:dpdk-devargs=:08:00.0
> 
> 
> What driver interface <:08:00.0> is bound to? igb_uio, vfio-pci or
> kernel driver?

I believe that it was tested with igb_uio driver, but I never tried it myself.

In order to have sr-iov configuration support with vfio-pci driver you need
a brand new v5.7-rc1 kernel and DPDK changes that are not merged yet:
https://patchwork.dpdk.org/project/dpdk/list/?series=9356
Not sure if it will work smoothly with OVS out of the box, though.

> 
> 
> (2) VF interface to representor connection, does representor=[3] represent
> VF device 3?
> 
> 
> ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk
> options:dpdk-devargs=:08:00.0,representor=[3]
> 
> ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5 type=dpdk
> options:dpdk-devargs=:08:00.0,representor=[5]
> 
> 
> Looking into linked DPDK document within OvS is confusing as well.
> 
> 
> Kindly let me know what I need to do for the setup?
> 
> 
> Thanks,
> 
> Ravi
> ___
> dev mailing list
> d...@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> 

___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev

[ovs-dev] OVS DPDK VF port representors not working

[Ravi Kerur]

Hello OvS-DPDK team,

We are expanding our usage of OvS-DPDK and want to use VF port representors
for one use-case. Going through the following document it is unclear how
this functionality can be achieved

http://docs.openvswitch.org/en/latest/topics/dpdk/phy/

I believe following things need to be configured before configuring OvS, I
am using Intel NICs for evaluation

(1) Create VFs on Intel NICs(82599). This can only happen when Intel NIC is
bound to its respective kernel driver (ixgbe) via

echo 2 > /sys/devices//sriov_numvfs


It creates 2 VF interfaces bound to ixgbevf driver


(2) Using driverctl, bind VF and PF interfaces to vfio-pci. Moment PF
device is bound to vfio-pci, VF interfaces vanish


Question based on the OvS document shown above


(1) Binding PF interface?


ovs-vsctl add-port br0 dpdk-pf -- set Interface dpdk-pf type=dpdk
options:dpdk-devargs=:08:00.0


What driver interface <:08:00.0> is bound to? igb_uio, vfio-pci or
kernel driver?



(2) VF interface to representor connection, does representor=[3] represent
VF device 3?


ovs-vsctl add-port br0 dpdk-rep3 -- set Interface dpdk-rep3 type=dpdk
options:dpdk-devargs=:08:00.0,representor=[3]

ovs-vsctl add-port br0 dpdk-rep5 -- set Interface dpdk-rep5 type=dpdk
options:dpdk-devargs=:08:00.0,representor=[5]


Looking into linked DPDK document within OvS is confusing as well.


Kindly let me know what I need to do for the setup?


Thanks,

Ravi
___
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev