Hi Andrew, Thanks very much all the good suggestions, v7 posted.
Best Regards, Xueming > -----Original Message----- > From: Andrew Rybchenko <andrew.rybche...@oktetlabs.ru> > Sent: Monday, July 5, 2021 12:13 AM > To: Xueming(Steven) Li <xuemi...@nvidia.com> > Cc: dev@dpdk.org; Wang Haiyue <haiyue.w...@intel.com>; NBU-Contact-Thomas > Monjalon <tho...@monjalon.net>; Kinsella Ray > <m...@ashroe.eu>; Neil Horman <nhor...@tuxdriver.com> > Subject: Re: [dpdk-dev] [PATCH v6 2/2] bus/auxiliary: introduce auxiliary bus > > On 6/25/21 2:47 PM, Xueming Li wrote: > > Auxiliary bus [1] provides a way to split function into child-devices > > representing sub-domains of functionality. Each auxiliary device > > represents a part of its parent functionality. > > > > Auxiliary device is identified by unique device name, sysfs path: > > /sys/bus/auxiliary/devices/<name> > > > > Devargs legacy syntax ofauxiliary device: > > Missing space after 'of' > > > -a auxiliary:<name>[,args...] > > Devargs generic syntax of auxiliary device: > > -a bus=auxiliary,name=<name>,,/class=<classs>,,/driver=<driver>,, > > Are two commas above intentionall? What for? > > > > > [1] kernel auxiliary bus document: > > https://www.kernel.org/doc/html/latest/driver-api/auxiliary_bus.html > > > > Signed-off-by: Xueming Li <xuemi...@nvidia.com> > > With my below notes fixed: > > Acked-by: Andrew Rybchenko <andrew.rybche...@oktetlabs.ru> > > > Cc: Wang Haiyue <haiyue.w...@intel.com> > > Cc: Thomas Monjalon <tho...@monjalon.net> > > Cc: Kinsella Ray <m...@ashroe.eu> > > --- > > MAINTAINERS | 5 + > > doc/guides/rel_notes/release_21_08.rst | 6 + > > drivers/bus/auxiliary/auxiliary_common.c | 411 > > ++++++++++++++++++++++ drivers/bus/auxiliary/auxiliary_params.c | 59 ++++ > > drivers/bus/auxiliary/linux/auxiliary.c | 141 ++++++++ > > drivers/bus/auxiliary/meson.build | 16 + > > drivers/bus/auxiliary/private.h | 74 ++++ > > drivers/bus/auxiliary/rte_bus_auxiliary.h | 201 +++++++++++ > > drivers/bus/auxiliary/version.map | 7 + > > drivers/bus/meson.build | 1 + > > 10 files changed, 921 insertions(+) > > create mode 100644 drivers/bus/auxiliary/auxiliary_common.c > > create mode 100644 drivers/bus/auxiliary/auxiliary_params.c > > create mode 100644 drivers/bus/auxiliary/linux/auxiliary.c > > create mode 100644 drivers/bus/auxiliary/meson.build create mode > > 100644 drivers/bus/auxiliary/private.h create mode 100644 > > drivers/bus/auxiliary/rte_bus_auxiliary.h > > create mode 100644 drivers/bus/auxiliary/version.map > > > > diff --git a/MAINTAINERS b/MAINTAINERS index 5877a16971..eaf691ca6a > > 100644 > > --- a/MAINTAINERS > > +++ b/MAINTAINERS > > @@ -525,6 +525,11 @@ F: doc/guides/mempool/octeontx2.rst Bus Drivers > > ----------- > > > > +Auxiliary bus driver > > Shouldn't it be EXPERIMENTAL? > > > +M: Parav Pandit <pa...@nvidia.com> > > +M: Xueming Li <xuemi...@nvidia.com> > > +F: drivers/bus/auxiliary/ > > + > > Intel FPGA bus > > M: Rosen Xu <rosen...@intel.com> > > F: drivers/bus/ifpga/ > > diff --git a/doc/guides/rel_notes/release_21_08.rst > > b/doc/guides/rel_notes/release_21_08.rst > > index a6ecfdf3ce..e7ef4c8a05 100644 > > --- a/doc/guides/rel_notes/release_21_08.rst > > +++ b/doc/guides/rel_notes/release_21_08.rst > > @@ -55,6 +55,12 @@ New Features > > Also, make sure to start the actual text at the margin. > > ======================================================= > > > > +* **Added auxiliary bus support.** > > + > > + Auxiliary bus provides a way to split function into child-devices > > + representing sub-domains of functionality. Each auxiliary device > > + represents a part of its parent functionality. > > + > > > > Removed Items > > ------------- > > diff --git a/drivers/bus/auxiliary/auxiliary_common.c > > b/drivers/bus/auxiliary/auxiliary_common.c > > new file mode 100644 > > index 0000000000..8a75306da5 > > --- /dev/null > > +++ b/drivers/bus/auxiliary/auxiliary_common.c > > @@ -0,0 +1,411 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates */ > > + > > +#include <string.h> > > +#include <inttypes.h> > > +#include <stdint.h> > > +#include <stdbool.h> > > +#include <stdlib.h> > > +#include <stdio.h> > > +#include <sys/queue.h> > > +#include <rte_errno.h> > > +#include <rte_interrupts.h> > > +#include <rte_log.h> > > +#include <rte_bus.h> > > +#include <rte_per_lcore.h> > > +#include <rte_memory.h> > > +#include <rte_eal.h> > > +#include <rte_eal_paging.h> > > +#include <rte_string_fns.h> > > +#include <rte_common.h> > > +#include <rte_devargs.h> > > + > > +#include "private.h" > > +#include "rte_bus_auxiliary.h" > > + > > +static struct rte_devargs * > > +auxiliary_devargs_lookup(const char *name) { > > + struct rte_devargs *devargs; > > + > > + RTE_EAL_DEVARGS_FOREACH(RTE_BUS_AUXILIARY_NAME, devargs) { > > + if (strcmp(devargs->name, name) == 0) > > + return devargs; > > + } > > + return NULL; > > +} > > + > > +/* > > + * Test whether the auxiliary device exist > > Missing full stop above. > > > + * > > + * Stub for OS not supporting auxiliary bus. > > + */ > > +__rte_weak bool > > +auxiliary_dev_exists(const char *name) { > > + RTE_SET_USED(name); > > + return false; > > +} > > + > > +/* > > + * Scan the devices in the auxiliary bus. > > + * > > + * Stub for OS not supporting auxiliary bus. > > + */ > > +__rte_weak int > > +auxiliary_scan(void) > > +{ > > + return 0; > > +} > > + > > +/* > > + * Update a device's devargs being scanned. > > + * > > + * @param aux_dev > > + * AUXILIARY device. > > + */ > > +void > > +auxiliary_on_scan(struct rte_auxiliary_device *aux_dev) { > > + aux_dev->device.devargs = auxiliary_devargs_lookup(aux_dev->name); > > +} > > + > > +/* > > + * Match the auxiliary driver and device using driver function. > > + */ > > +bool > > +auxiliary_match(const struct rte_auxiliary_driver *aux_drv, > > + const struct rte_auxiliary_device *aux_dev) { > > + if (aux_drv->match == NULL) > > + return false; > > + return aux_drv->match(aux_dev->name); } > > + > > +/* > > + * Call the probe() function of the driver. > > + */ > > +static int > > +rte_auxiliary_probe_one_driver(struct rte_auxiliary_driver *drv, > > + struct rte_auxiliary_device *dev) > > +{ > > + enum rte_iova_mode iova_mode; > > + int ret; > > + > > + if ((drv == NULL) || (dev == NULL)) > > Unnecessary internal parenthesis. > > > + return -EINVAL; > > + > > + /* Check if driver supports it. */ > > + if (!auxiliary_match(drv, dev)) > > + /* Match of device and driver failed */ > > + return 1; > > + > > + /* No initialization when marked as blocked, return without error. */ > > + if (dev->device.devargs != NULL && > > + dev->device.devargs->policy == RTE_DEV_BLOCKED) { > > + AUXILIARY_LOG(INFO, "Device is blocked, not initializing"); > > + return -1; > > + } > > + > > + if (dev->device.numa_node < 0) { > > + AUXILIARY_LOG(INFO, "Device is not NUMA-aware, defaulting > > socket to 0"); > > socket -> NUMA node > > > + dev->device.numa_node = 0; > > + } > > + > > + iova_mode = rte_eal_iova_mode(); > > + if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0 && > > + iova_mode != RTE_IOVA_VA) { > > + AUXILIARY_LOG(ERR, "Driver %s expecting VA IOVA mode but > > current mode is PA, not initializing", > > + drv->driver.name); > > + return -EINVAL; > > + } > > + > > + dev->driver = drv; > > + > > + AUXILIARY_LOG(INFO, "Probe auxiliary driver: %s device: %s (socket %i)", > > socket -> NUMA node > > > + drv->driver.name, dev->name, dev->device.numa_node); > > + ret = drv->probe(drv, dev); > > + if (ret != 0) > > + dev->driver = NULL; > > + else > > + dev->device.driver = &drv->driver; > > + > > + return ret; > > +} > > + > > +/* > > + * Call the remove() function of the driver. > > + */ > > +static int > > +rte_auxiliary_driver_remove_dev(struct rte_auxiliary_device *dev) > > +{ > > + struct rte_auxiliary_driver *drv; > > + int ret = 0; > > + > > + if (dev == NULL) > > + return -EINVAL; > > + > > + drv = dev->driver; > > + > > + AUXILIARY_LOG(DEBUG, "Driver %s remove auxiliary device %s on NUMA > > socket %i", > > socket -> node > > > + drv->driver.name, dev->name, dev->device.numa_node); > > + > > + if (drv->remove != NULL) { > > + ret = drv->remove(dev); > > + if (ret < 0) > > + return ret; > > + } > > + > > + /* clear driver structure */ > > + dev->driver = NULL; > > + dev->device.driver = NULL; > > + > > + return 0; > > +} > > + > > +/* > > + * Call the probe() function of all registered driver for the given device. > > + * Return < 0 if initialization failed. > > + * Return 1 if no driver is found for this device. > > + */ > > +static int > > +auxiliary_probe_all_drivers(struct rte_auxiliary_device *dev) > > +{ > > + struct rte_auxiliary_driver *drv; > > + int rc; > > + > > + if (dev == NULL) > > + return -EINVAL; > > + > > + FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) { > > + if (!drv->match(dev->name)) > > + continue; > > + > > + rc = rte_auxiliary_probe_one_driver(drv, dev); > > + if (rc < 0) > > + /* negative value is an error */ > > + return rc; > > + if (rc > 0) > > + /* positive value means driver doesn't support it */ > > + continue; > > + return 0; > > + } > > + return 1; > > +} > > + > > +/* > > + * Scan the content of the auxiliary bus, and call the probe function for > > + * all registered drivers to try to probe discovered devices. > > + */ > > +static int > > +auxiliary_probe(void) > > +{ > > + struct rte_auxiliary_device *dev = NULL; > > + size_t probed = 0, failed = 0; > > + int ret = 0; > > + > > + FOREACH_DEVICE_ON_AUXILIARY_BUS(dev) { > > + probed++; > > + > > + ret = auxiliary_probe_all_drivers(dev); > > + if (ret < 0) { > > + if (ret != -EEXIST) { > > + AUXILIARY_LOG(ERR, "Requested device %s cannot > > be used", > > + dev->name); > > + rte_errno = errno; > > + failed++; > > + } > > + ret = 0; > > + } > > + } > > + > > + return (probed && probed == failed) ? -1 : 0; > > +} > > + > > +static int > > +auxiliary_parse(const char *name, void *addr) > > +{ > > + struct rte_auxiliary_driver *drv = NULL; > > + const char **out = addr; > > + > > + /* Allow empty device name "auxiliary:" to bypass entire bus scan. */ > > + if (strlen(name) == 0) > > + return 0; > > + > > + FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) { > > + if (drv->match(name)) > > + break; > > + } > > + if (drv != NULL && addr != NULL) > > + *out = name; > > + return drv != NULL ? 0 : -1; > > +} > > + > > +/* Register a driver */ > > +void > > +rte_auxiliary_register(struct rte_auxiliary_driver *driver) > > +{ > > + TAILQ_INSERT_TAIL(&auxiliary_bus.driver_list, driver, next); > > + driver->bus = &auxiliary_bus; > > +} > > + > > +/* Unregister a driver */ > > +void > > +rte_auxiliary_unregister(struct rte_auxiliary_driver *driver) > > +{ > > + TAILQ_REMOVE(&auxiliary_bus.driver_list, driver, next); > > + driver->bus = NULL; > > +} > > + > > +/* Add a device to auxiliary bus */ > > +void > > +auxiliary_add_device(struct rte_auxiliary_device *aux_dev) > > +{ > > + TAILQ_INSERT_TAIL(&auxiliary_bus.device_list, aux_dev, next); > > +} > > + > > +/* Insert a device into a predefined position in auxiliary bus */ > > +void > > +auxiliary_insert_device(struct rte_auxiliary_device *exist_aux_dev, > > + struct rte_auxiliary_device *new_aux_dev) > > +{ > > + TAILQ_INSERT_BEFORE(exist_aux_dev, new_aux_dev, next); > > +} > > + > > +/* Remove a device from auxiliary bus */ > > +static void > > +rte_auxiliary_remove_device(struct rte_auxiliary_device *auxiliary_dev) > > +{ > > + TAILQ_REMOVE(&auxiliary_bus.device_list, auxiliary_dev, next); > > +} > > + > > +static struct rte_device * > > +auxiliary_find_device(const struct rte_device *start, rte_dev_cmp_t cmp, > > + const void *data) > > +{ > > + const struct rte_auxiliary_device *pstart; > > + struct rte_auxiliary_device *adev; > > + > > + if (start != NULL) { > > + pstart = RTE_DEV_TO_AUXILIARY_CONST(start); > > + adev = TAILQ_NEXT(pstart, next); > > + } else { > > + adev = TAILQ_FIRST(&auxiliary_bus.device_list); > > + } > > + while (adev != NULL) { > > + if (cmp(&adev->device, data) == 0) > > + return &adev->device; > > + adev = TAILQ_NEXT(adev, next); > > + } > > + return NULL; > > +} > > + > > +static int > > +auxiliary_plug(struct rte_device *dev) > > +{ > > + if (!auxiliary_dev_exists(dev->name)) > > + return -ENOENT; > > + return auxiliary_probe_all_drivers(RTE_DEV_TO_AUXILIARY(dev)); > > +} > > + > > +static int > > +auxiliary_unplug(struct rte_device *dev) > > +{ > > + struct rte_auxiliary_device *adev; > > + int ret; > > + > > + adev = RTE_DEV_TO_AUXILIARY(dev); > > + ret = rte_auxiliary_driver_remove_dev(adev); > > + if (ret == 0) { > > + rte_auxiliary_remove_device(adev); > > + rte_devargs_remove(dev->devargs); > > + free(adev); > > + } > > + return ret; > > +} > > + > > +static int > > +auxiliary_dma_map(struct rte_device *dev, void *addr, uint64_t iova, > > size_t len) > > +{ > > + struct rte_auxiliary_device *aux_dev = RTE_DEV_TO_AUXILIARY(dev); > > + > > + if (dev == NULL || aux_dev->driver == NULL) { > > + rte_errno = EINVAL; > > + return -1; > > + } > > + if (aux_dev->driver->dma_map == NULL) { > > + rte_errno = ENOTSUP; > > + return -1; > > + } > > + return aux_dev->driver->dma_map(aux_dev, addr, iova, len); > > +} > > + > > +static int > > +auxiliary_dma_unmap(struct rte_device *dev, void *addr, uint64_t iova, > > + size_t len) > > +{ > > + struct rte_auxiliary_device *aux_dev = RTE_DEV_TO_AUXILIARY(dev); > > + > > + if (dev == NULL || aux_dev->driver == NULL) { > > + rte_errno = EINVAL; > > + return -1; > > + } > > + if (aux_dev->driver->dma_unmap == NULL) { > > + rte_errno = ENOTSUP; > > + return -1; > > + } > > + return aux_dev->driver->dma_unmap(aux_dev, addr, iova, len); > > +} > > + > > +bool > > +auxiliary_is_ignored_device(const char *name) > > +{ > > + struct rte_devargs *devargs = auxiliary_devargs_lookup(name); > > + > > + switch (auxiliary_bus.bus.conf.scan_mode) { > > + case RTE_BUS_SCAN_ALLOWLIST: > > + if (devargs && devargs->policy == RTE_DEV_ALLOWED) > > + return false; > > + break; > > + case RTE_BUS_SCAN_UNDEFINED: > > + case RTE_BUS_SCAN_BLOCKLIST: > > + if (devargs == NULL || devargs->policy != RTE_DEV_BLOCKED) > > + return false; > > + break; > > + } > > + return true; > > +} > > + > > +static enum rte_iova_mode > > +auxiliary_get_iommu_class(void) > > +{ > > + const struct rte_auxiliary_driver *drv; > > + > > + FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) { > > + if ((drv->drv_flags & RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA) > 0) > > + return RTE_IOVA_VA; > > + } > > + > > + return RTE_IOVA_DC; > > +} > > + > > +struct rte_auxiliary_bus auxiliary_bus = { > > + .bus = { > > + .scan = auxiliary_scan, > > + .probe = auxiliary_probe, > > + .find_device = auxiliary_find_device, > > + .plug = auxiliary_plug, > > + .unplug = auxiliary_unplug, > > + .parse = auxiliary_parse, > > + .dma_map = auxiliary_dma_map, > > + .dma_unmap = auxiliary_dma_unmap, > > + .get_iommu_class = auxiliary_get_iommu_class, > > + .dev_iterate = auxiliary_dev_iterate, > > + }, > > + .device_list = TAILQ_HEAD_INITIALIZER(auxiliary_bus.device_list), > > + .driver_list = TAILQ_HEAD_INITIALIZER(auxiliary_bus.driver_list), > > +}; > > + > > +RTE_REGISTER_BUS(auxiliary, auxiliary_bus.bus); > > +RTE_LOG_REGISTER_DEFAULT(auxiliary_bus_logtype, NOTICE); > > diff --git a/drivers/bus/auxiliary/auxiliary_params.c > > b/drivers/bus/auxiliary/auxiliary_params.c > > new file mode 100644 > > index 0000000000..cd3fa56cb4 > > --- /dev/null > > +++ b/drivers/bus/auxiliary/auxiliary_params.c > > @@ -0,0 +1,59 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates > > + */ > > + > > +#include <string.h> > > + > > +#include <rte_bus.h> > > +#include <rte_dev.h> > > +#include <rte_errno.h> > > +#include <rte_kvargs.h> > > + > > +#include "private.h" > > +#include "rte_bus_auxiliary.h" > > + > > +enum auxiliary_params { > > + RTE_AUXILIARY_PARAM_NAME, > > +}; > > + > > +static const char * const auxiliary_params_keys[] = { > > + [RTE_AUXILIARY_PARAM_NAME] = "name", > > +}; > > + > > +static int > > +auxiliary_dev_match(const struct rte_device *dev, > > + const void *_kvlist) > > +{ > > + const struct rte_kvargs *kvlist = _kvlist; > > + int ret; > > + > > + ret = rte_kvargs_process(kvlist, > > + auxiliary_params_keys[RTE_AUXILIARY_PARAM_NAME], > > + rte_kvargs_strcmp, (void *)(uintptr_t)dev->name); > > + > > + return ret != 0 ? -1 : 0; > > +} > > + > > +void * > > +auxiliary_dev_iterate(const void *start, > > + const char *str, > > + const struct rte_dev_iterator *it __rte_unused) > > +{ > > + rte_bus_find_device_t find_device; > > + struct rte_kvargs *kvargs = NULL; > > + struct rte_device *dev; > > + > > + if (str != NULL) { > > + kvargs = rte_kvargs_parse(str, auxiliary_params_keys); > > + if (kvargs == NULL) { > > + AUXILIARY_LOG(ERR, "cannot parse argument list %s", > > + str); > > + rte_errno = EINVAL; > > + return NULL; > > + } > > + } > > + find_device = auxiliary_bus.bus.find_device; > > + dev = find_device(start, auxiliary_dev_match, kvargs); > > + rte_kvargs_free(kvargs); > > + return dev; > > +} > > diff --git a/drivers/bus/auxiliary/linux/auxiliary.c > > b/drivers/bus/auxiliary/linux/auxiliary.c > > new file mode 100644 > > index 0000000000..8464487971 > > --- /dev/null > > +++ b/drivers/bus/auxiliary/linux/auxiliary.c > > @@ -0,0 +1,141 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates > > + */ > > + > > +#include <string.h> > > +#include <dirent.h> > > + > > +#include <rte_log.h> > > +#include <rte_bus.h> > > +#include <rte_malloc.h> > > +#include <rte_devargs.h> > > +#include <rte_memcpy.h> > > +#include <eal_filesystem.h> > > + > > +#include "../rte_bus_auxiliary.h" > > +#include "../private.h" > > + > > +#define AUXILIARY_SYSFS_PATH "/sys/bus/auxiliary/devices" > > + > > +/* Scan one auxiliary sysfs entry, and fill the devices list from it. */ > > +static int > > +auxiliary_scan_one(const char *dirname, const char *name) > > +{ > > + struct rte_auxiliary_device *dev; > > + struct rte_auxiliary_device *dev2; > > + char filename[PATH_MAX]; > > + unsigned long tmp; > > + int ret; > > + > > + dev = malloc(sizeof(*dev)); > > + if (dev == NULL) > > + return -1; > > + > > + memset(dev, 0, sizeof(*dev)); > > + if (rte_strscpy(dev->name, name, sizeof(dev->name)) < 0) { > > + free(dev); > > + return -1; > > + } > > + dev->device.name = dev->name; > > + dev->device.bus = &auxiliary_bus.bus; > > + > > + /* Get NUMA node, default to 0 if not present */ > > + snprintf(filename, sizeof(filename), "%s/%s/numa_node", > > + dirname, name); > > + if (access(filename, F_OK) != -1) { > > + if (eal_parse_sysfs_value(filename, &tmp) == 0) > > + dev->device.numa_node = tmp; > > + else > > + dev->device.numa_node = -1; > > + } else { > > + dev->device.numa_node = 0; > > + } > > + > > + auxiliary_on_scan(dev); > > + > > + /* Device is valid, add in list (sorted) */ > > + TAILQ_FOREACH(dev2, &auxiliary_bus.device_list, next) { > > + ret = strcmp(dev->name, dev2->name); > > + if (ret > 0) > > + continue; > > + if (ret < 0) { > > + auxiliary_insert_device(dev2, dev); > > + } else { /* already registered */ > > + if (rte_dev_is_probed(&dev2->device) && > > + dev2->device.devargs != dev->device.devargs) { > > + /* To probe device with new devargs. */ > > + rte_devargs_remove(dev2->device.devargs); > > + auxiliary_on_scan(dev2); > > + } > > + free(dev); > > + } > > + return 0; > > + } > > + auxiliary_add_device(dev); > > + return 0; > > +} > > + > > +/* > > + * Test whether the auxiliary device exist > > Missing full stop above. > > > + */ > > +bool > > +auxiliary_dev_exists(const char *name) > > +{ > > + DIR *dir; > > + char dirname[PATH_MAX]; > > + > > + snprintf(dirname, sizeof(dirname), "%s/%s", > > + AUXILIARY_SYSFS_PATH, name); > > + dir = opendir(dirname); > > + if (dir == NULL) > > + return false; > > + closedir(dir); > > + return true; > > +} > > + > > +/* > > + * Scan the devices in the auxiliary bus > > Missing full stop above. > > > + */ > > +int > > +auxiliary_scan(void) > > +{ > > + struct dirent *e; > > + DIR *dir; > > + char dirname[PATH_MAX]; > > + struct rte_auxiliary_driver *drv; > > + > > + dir = opendir(AUXILIARY_SYSFS_PATH); > > + if (dir == NULL) { > > + AUXILIARY_LOG(INFO, "%s not found, is auxiliary module loaded?", > > + AUXILIARY_SYSFS_PATH); > > + return 0; > > + } > > + > > + while ((e = readdir(dir)) != NULL) { > > + if (e->d_name[0] == '.') > > + continue; > > + > > + if (auxiliary_is_ignored_device(e->d_name)) > > + continue; > > + > > + snprintf(dirname, sizeof(dirname), "%s/%s", > > + AUXILIARY_SYSFS_PATH, e->d_name); > > + > > + /* Ignore if no driver can handle. */ > > + FOREACH_DRIVER_ON_AUXILIARY_BUS(drv) { > > + if (drv->match(e->d_name)) > > + break; > > + } > > + if (drv == NULL) > > + continue; > > + > > + if (auxiliary_scan_one(dirname, e->d_name) < 0) > > + goto error; > > + } > > + closedir(dir); > > + return 0; > > + > > +error: > > + closedir(dir); > > + return -1; > > +} > > diff --git a/drivers/bus/auxiliary/meson.build > > b/drivers/bus/auxiliary/meson.build > > new file mode 100644 > > index 0000000000..357550eff7 > > --- /dev/null > > +++ b/drivers/bus/auxiliary/meson.build > > @@ -0,0 +1,16 @@ > > +# SPDX-License-Identifier: BSD-3-Clause > > +# Copyright (c) 2021 NVIDIA Corporation & Affiliates > > + > > +headers = files( > > + 'rte_bus_auxiliary.h', > > +) > > +sources = files( > > + 'auxiliary_common.c', > > + 'auxiliary_params.c', > > +) > > +if is_linux > > + sources += files( > > + 'linux/auxiliary.c', > > + ) > > +endif > > +deps += ['kvargs'] > > diff --git a/drivers/bus/auxiliary/private.h > > b/drivers/bus/auxiliary/private.h > > new file mode 100644 > > index 0000000000..cb3e849993 > > --- /dev/null > > +++ b/drivers/bus/auxiliary/private.h > > @@ -0,0 +1,74 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates > > + */ > > + > > +#ifndef AUXILIARY_PRIVATE_H > > May be add BUS_ prefix at leaat? > > > +#define AUXILIARY_PRIVATE_H > > + > > +#include <stdbool.h> > > +#include <stdio.h> > > + > > +#include "rte_bus_auxiliary.h" > > + > > +extern struct rte_auxiliary_bus auxiliary_bus; > > +extern int auxiliary_bus_logtype; > > + > > +#define AUXILIARY_LOG(level, ...) \ > > + rte_log(RTE_LOG_ ## level, auxiliary_bus_logtype, \ > > + RTE_FMT("auxiliary bus: " RTE_FMT_HEAD(__VA_ARGS__,) "\n", \ > > + RTE_FMT_TAIL(__VA_ARGS__,))) > > + > > +/* Auxiliary bus iterators */ > > +#define FOREACH_DEVICE_ON_AUXILIARY_BUS(p) \ > > + TAILQ_FOREACH(p, &(auxiliary_bus.device_list), next) > > + > > +#define FOREACH_DRIVER_ON_AUXILIARY_BUS(p) \ > > + TAILQ_FOREACH(p, &(auxiliary_bus.driver_list), next) > > + > > +bool auxiliary_dev_exists(const char *name); > > + > > +/* > > + * Scan the content of the auxiliary bus, and the devices in the devices > > + * list. > > + */ > > +int auxiliary_scan(void); > > + > > +/* > > + * Update a device being scanned. > > + */ > > +void auxiliary_on_scan(struct rte_auxiliary_device *aux_dev); > > + > > +/* > > + * Validate whether a device with given auxiliary device should be ignored > > + * or not. > > + */ > > +bool auxiliary_is_ignored_device(const char *name); > > + > > +/* > > + * Add an auxiliary device to the auxiliary bus (append to auxiliary device > > + * list). This function also updates the bus references of the auxiliary > > + * device and the generic device object embedded within. > > + */ > > +void auxiliary_add_device(struct rte_auxiliary_device *aux_dev); > > + > > +/* > > + * Insert an auxiliary device in the auxiliary bus at a particular location > > + * in the device list. It also updates the auxiliary bus reference of the > > + * new devices to be inserted. > > + */ > > +void auxiliary_insert_device(struct rte_auxiliary_device *exist_aux_dev, > > + struct rte_auxiliary_device *new_aux_dev); > > + > > +/* > > + * Match the auxiliary driver and device by driver function > > Missing full stop. > > > + */ > > +bool auxiliary_match(const struct rte_auxiliary_driver *aux_drv, > > + const struct rte_auxiliary_device *aux_dev); > > + > > +/* > > + * Iterate over devices, matching any device against the provided string > > Missing full stop. > > > + */ > > +void *auxiliary_dev_iterate(const void *start, const char *str, > > + const struct rte_dev_iterator *it); > > + > > +#endif /* AUXILIARY_PRIVATE_H */ > > diff --git a/drivers/bus/auxiliary/rte_bus_auxiliary.h > > b/drivers/bus/auxiliary/rte_bus_auxiliary.h > > new file mode 100644 > > index 0000000000..16b147e387 > > --- /dev/null > > +++ b/drivers/bus/auxiliary/rte_bus_auxiliary.h > > @@ -0,0 +1,201 @@ > > +/* SPDX-License-Identifier: BSD-3-Clause > > + * Copyright (c) 2021 NVIDIA Corporation & Affiliates > > + */ > > + > > +#ifndef RTE_BUS_AUXILIARY_H > > +#define RTE_BUS_AUXILIARY_H > > + > > +/** > > + * @file > > + * > > + * Auxiliary Bus Interface. > > + */ > > + > > +#ifdef __cplusplus > > +extern "C" { > > +#endif > > + > > +#include <stdio.h> > > +#include <stdlib.h> > > +#include <limits.h> > > +#include <errno.h> > > +#include <sys/queue.h> > > +#include <stdint.h> > > +#include <inttypes.h> > > + > > +#include <rte_debug.h> > > +#include <rte_interrupts.h> > > +#include <rte_dev.h> > > +#include <rte_bus.h> > > +#include <rte_kvargs.h> > > + > > +#define RTE_BUS_AUXILIARY_NAME "auxiliary" > > + > > +/* Forward declarations */ > > +struct rte_auxiliary_driver; > > +struct rte_auxiliary_bus; > > +struct rte_auxiliary_device; > > + > > +/** > > + * Match function for the driver to decide if device can be handled. > > + * > > + * @param name > > + * Pointer to the auxiliary device name. > > + * @return > > + * Whether the driver can handle the auxiliary device. > > + */ > > +typedef bool(rte_auxiliary_match_t)(const char *name); > > + > > +/** > > + * Initialization function for the driver called during auxiliary probing. > > + * > > + * @param drv > > + * Pointer to the auxiliary driver. > > + * @param dev > > + * Pointer to the auxiliary device. > > + * @return > > + * - 0 On success. > > + * - Negative value and rte_errno is set otherwise. > > + */ > > +typedef int(rte_auxiliary_probe_t)(struct rte_auxiliary_driver *drv, > > + struct rte_auxiliary_device *dev); > > + > > +/** > > + * Uninitialization function for the driver called during hotplugging. > > + * > > + * @param dev > > + * Pointer to the auxiliary device. > > + * @return > > + * - 0 On success. > > + * - Negative value and rte_errno is set otherwise. > > + */ > > +typedef int (rte_auxiliary_remove_t)(struct rte_auxiliary_device *dev); > > + > > +/** > > + * Driver-specific DMA mapping. After a successful call the device > > + * will be able to read/write from/to this segment. > > + * > > + * @param dev > > + * Pointer to the auxiliary device. > > + * @param addr > > + * Starting virtual address of memory to be mapped. > > + * @param iova > > + * Starting IOVA address of memory to be mapped. > > + * @param len > > + * Length of memory segment being mapped. > > + * @return > > + * - 0 On success. > > + * - Negative value and rte_errno is set otherwise. > > + */ > > +typedef int (rte_auxiliary_dma_map_t)(struct rte_auxiliary_device *dev, > > + void *addr, uint64_t iova, size_t len); > > + > > +/** > > + * Driver-specific DMA un-mapping. After a successful call the device > > + * will not be able to read/write from/to this segment. > > + * > > + * @param dev > > + * Pointer to the auxiliary device. > > + * @param addr > > + * Starting virtual address of memory to be unmapped. > > + * @param iova > > + * Starting IOVA address of memory to be unmapped. > > + * @param len > > + * Length of memory segment being unmapped. > > + * @return > > + * - 0 On success. > > + * - Negative value and rte_errno is set otherwise. > > + */ > > +typedef int (rte_auxiliary_dma_unmap_t)(struct rte_auxiliary_device *dev, > > + void *addr, uint64_t iova, size_t len); > > + > > +/** > > + * A structure describing an auxiliary device. > > + */ > > +struct rte_auxiliary_device { > > + TAILQ_ENTRY(rte_auxiliary_device) next; /**< Next probed device. */ > > + struct rte_device device; /**< Inherit core device */ > > + char name[RTE_DEV_NAME_MAX_LEN + 1]; /**< ASCII device name */ > > + struct rte_intr_handle intr_handle; /**< Interrupt handle */ > > + struct rte_auxiliary_driver *driver; /**< Device driver */ > > +}; > > + > > +/** List of auxiliary devices */ > > +TAILQ_HEAD(rte_auxiliary_device_list, rte_auxiliary_device); > > +/** List of auxiliary drivers */ > > +TAILQ_HEAD(rte_auxiliary_driver_list, rte_auxiliary_driver); > > Shouldn't we hide rte_auxiliary_device inside the library take > API/ABI stability into account? Or will be it DPDK internal anyway? If > so, it should be done INTERNAL from the very > beginning. > > > + > > +/** > > + * Structure describing the auxiliary bus > > + */ > > +struct rte_auxiliary_bus { > > + struct rte_bus bus; /**< Inherit the generic class */ > > + struct rte_auxiliary_device_list device_list; /**< List of devices */ > > + struct rte_auxiliary_driver_list driver_list; /**< List of drivers */ > > +}; > > It looks internal. The following forward declaration should be > sufficient to build. > > struct rte_auxiliary_bus; > > > > + > > +/** > > + * A structure describing an auxiliary driver. > > + */ > > +struct rte_auxiliary_driver { > > + TAILQ_ENTRY(rte_auxiliary_driver) next; /**< Next in list. */ > > + struct rte_driver driver; /**< Inherit core driver. */ > > + struct rte_auxiliary_bus *bus; /**< Auxiliary bus reference. */ > > + rte_auxiliary_match_t *match; /**< Device match function. */ > > + rte_auxiliary_probe_t *probe; /**< Device probe function. */ > > + rte_auxiliary_remove_t *remove; /**< Device remove function. */ > > + rte_auxiliary_dma_map_t *dma_map; /**< Device DMA map function. */ > > + rte_auxiliary_dma_unmap_t *dma_unmap; /**< Device DMA unmap function. */ > > + uint32_t drv_flags; /**< Flags RTE_AUXILIARY_DRV_*. */ > > +}; > > + > > +/** > > + * @internal > > + * Helper macro for drivers that need to convert to struct > > rte_auxiliary_device. > > + */ > > +#define RTE_DEV_TO_AUXILIARY(ptr) \ > > + container_of(ptr, struct rte_auxiliary_device, device) > > + > > +#define RTE_DEV_TO_AUXILIARY_CONST(ptr) \ > > + container_of(ptr, const struct rte_auxiliary_device, device) > > + > > +#define RTE_ETH_DEV_TO_AUXILIARY(eth_dev) \ > > + RTE_DEV_TO_AUXILIARY((eth_dev)->device) > > + > > +/** Device driver needs IOVA as VA and cannot work with IOVA as PA */ > > +#define RTE_AUXILIARY_DRV_NEED_IOVA_AS_VA 0x002 > > + > > +/** > > Don't we need EXPERIMENTAL notice here? > > > + * Register an auxiliary driver. > > + * > > + * @param driver > > + * A pointer to a rte_auxiliary_driver structure describing the driver > > + * to be registered. > > + */ > > +__rte_experimental > > +void rte_auxiliary_register(struct rte_auxiliary_driver *driver); > > + > > +/** Helper for auxiliary device registration from driver instance */ > > +#define RTE_PMD_REGISTER_AUXILIARY(nm, auxiliary_drv) \ > > + RTE_INIT(auxiliaryinitfn_ ##nm) \ > > + { \ > > + (auxiliary_drv).driver.name = RTE_STR(nm); \ > > + rte_auxiliary_register(&(auxiliary_drv)); \ > > + } \ > > + RTE_PMD_EXPORT_NAME(nm, __COUNTER__) > > + > > +/** > > Don't we need EXPERIMENTAL notice here? > > > + * Unregister an auxiliary driver. > > + * > > + * @param driver > > + * A pointer to a rte_auxiliary_driver structure describing the driver > > + * to be unregistered. > > + */ > > +__rte_experimental > > +void rte_auxiliary_unregister(struct rte_auxiliary_driver *driver); > > + > > +#ifdef __cplusplus > > +} > > +#endif > > + > > +#endif /* RTE_BUS_AUXILIARY_H */ > > diff --git a/drivers/bus/auxiliary/version.map > > b/drivers/bus/auxiliary/version.map > > new file mode 100644 > > index 0000000000..a52260657c > > --- /dev/null > > +++ b/drivers/bus/auxiliary/version.map > > @@ -0,0 +1,7 @@ > > +EXPERIMENTAL { > > + global: > > + > > + # added in 21.08 > > + rte_auxiliary_register; > > + rte_auxiliary_unregister; > > +}; > > diff --git a/drivers/bus/meson.build b/drivers/bus/meson.build > > index 410058de3a..45eab5233d 100644 > > --- a/drivers/bus/meson.build > > +++ b/drivers/bus/meson.build > > @@ -2,6 +2,7 @@ > > # Copyright(c) 2017 Intel Corporation > > > > drivers = [ > > + 'auxiliary', > > 'dpaa', > > 'fslmc', > > 'ifpga', > >
