>-----Original Message-----
>From: Wang, Haiyue <haiyue.w...@intel.com>
>Sent: Tuesday, February 23, 2021 7:46 PM
>To: Stephen Hemminger <step...@networkplumber.org>; Xueming(Steven) Li
><xuemi...@nvidia.com>
>Cc: dev@dpdk.org; Slava Ovsiienko <viachesl...@nvidia.com>; Asaf Penso
><as...@nvidia.com>
>Subject: RE: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction representor
>
>> -----Original Message-----
>> From: dev <dev-boun...@dpdk.org> On Behalf Of Stephen Hemminger
>> Sent: Tuesday, February 23, 2021 09:55
>> To: Xueming Li <xuemi...@nvidia.com>
>> Cc: dev@dpdk.org; Viacheslav Ovsiienko <viachesl...@nvidia.com>; Asaf
>> Penso <as...@nvidia.com>
>> Subject: Re: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction
>> representor
>>
>> On Sun, 14 Feb 2021 03:21:30 +0000
>> Xueming Li <xuemi...@nvidia.com> wrote:
>>
>> > SubFunction [1] is a portion of the PCI device, a SF netdev has its
>> > own dedicated queues(txq, rxq). A SF netdev supports E-Switch
>> > representation offload similar to existing PF and VF representors. A
>> > SF shares PCI level resources with other SFs and/or with its parent PCI
>> > function.
>> >
>> > From SmartNIC perspective, when PCI device is shared for multi-host,
>> > representors for host controller and host PF is required.
>> >
>> > This patch set introduces new representor types in addtion to
>> > existing VF representor. Syntax:
>> >
>> > [[c#]pf#]vf#: VF port representor/s from controller/pf
>> > [[c#]pf#]sf#: SF port representor/s from controller/pf
>> > #: VF representor - for backwards compatibility
>> >
>> > "#" is number instance, list or range, valid examples:
>> > 1, [1,3,5], [0-3], [0,2-4,6]
>> >
>> > For backward compatibility, this patch also introduces new netdev
>> > capability to indicate the capability of supportting SF representor.
>> >
>> > Version history:
>> > RFC:
>> > initial version [2]
>> > V2:
>> > - separate patch for represnetor infrastructure, controller, pf and
>> > sf.
>> > - replace representor ID macro with functions:
>> > rte_eth_representor_id_encode()
>> > rte_eth_representor_id_parse()
>> > - new patch to allow devargs with same PCI BDF but different
>> > representors.
>> > - other minor code updates according to comments, thanks Andrew!
>> > - update document
>> > V3:
>> > - improve probing of allowed devargs with same name.
>> > - parse single word of kvargs as key.
>> > - update kvargs test cases.
>> > V4:
>> > - split first representor refactor patch into
>> > 1: add representor type
>> > 2: refector representor list parsing
>> > - push the patch supporting multi-devargs for same device.
>> > V5:
>> > - add comments for parsing functions
>> > - update switch_representation.rst - Thanks Ajit
>> > V6:
>> > - split representor types into different patches, move to
>> > rte_ethdev.h
>> > - improvements of rte_eth_devargs_process_list() according to
>> > Andrew's suggestion
>> > - fixed PF probe failure for Intel i40e
>> > - replace ethdev SF capability with rte_eth_representor_info_get()
>> > - add new ethdev ops api to get representor info from PMD
>> > - replace representor ID encode/decode with conversion from
>> > representor info
>> > - change ethdev representor iterator to use new ID encoding
>> >
>> >
>> > Xueming Li (9):
>> > ethdev: introduce representor type
>> > ethdev: support representor port list
>> > ethdev: support new VF representor syntax
>> > ethdev: support sub function representor
>> > ethdev: support PF index in representor
>> > ethdev: support multi-host in representor
>> > ethdev: new API to get representor info
>> > ethdev: representor iterator compare complete info
>> > kvargs: update parser to support lists
>> >
>> > app/test/test_kvargs.c | 46 ++++-
>> > config/rte_config.h | 1 +
>> > doc/guides/prog_guide/poll_mode_drv.rst | 13 +-
>> > .../prog_guide/switch_representation.rst | 35 +++-
>> > drivers/net/bnxt/bnxt_ethdev.c | 7 +
>> > drivers/net/enic/enic_ethdev.c | 6 +
>> > drivers/net/i40e/i40e_ethdev.c | 7 +
>> > drivers/net/ixgbe/ixgbe_ethdev.c | 7 +
>> > drivers/net/mlx5/linux/mlx5_os.c | 11 ++
>> > lib/librte_ethdev/ethdev_driver.h | 49 ++++-
>> > lib/librte_ethdev/ethdev_private.c | 173 ++++++++++++------
>> > lib/librte_ethdev/ethdev_private.h | 3 -
>> > lib/librte_ethdev/rte_class_eth.c | 40 ++--
>> > lib/librte_ethdev/rte_ethdev.c | 102 ++++++++++-
>> > lib/librte_ethdev/rte_ethdev.h | 53 ++++++
>> > lib/librte_ethdev/version.map | 4 +
>> > lib/librte_kvargs/rte_kvargs.c | 101 +++++++---
>> > 17 files changed, 535 insertions(+), 123 deletions(-)
>> >
>>
>> A couple of higher level design questions:
>> 1. How does Linux and other OS handle this in their API?
Hi Stephen, thanks for looking this!
Representor was a type of ethdev, OS independent.
SF representor is a new type besides existing VF representor type.
>> 2. This solution seems quite tied into a specific implementation on your
>> hardware.
>> Is there a PCI standard for this?
SF has been carved in kernel, supported by devlink. The devlink document in the
link Haiyue posted should be helpful to understand the overall picture:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=24a790da0ac4d9bcce2a9becc8799241716672f6
>
>> 3. Maybe a more general solution where these were represented as buses would
>> allow for other connection methods in the future?
>
>It should be "auxiliary bus", I think. ;-)
>https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/base/auxiliary.c
Good link, Mellanox has exactly implemented SF based auxiliary bus, I'm going
to post another RFC to support auxiliary bus.
But this patchset still needed to support representor of SF.
>
>mlx5 subfunction support
>https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=24a790da0ac4d9bcce2a9becc8799241716672f6