Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
On Mon, Mar 8, 2021 at 8:53 PM Chen, Mike Ximing wrote: > > > > -Original Message- > > From: Greg KH > > On Mon, Mar 08, 2021 at 08:00:00PM +, Chen, Mike Ximing wrote: > > > > > > Hi Greg, > > > > > > While waiting for the feedback from the networking maintainers, I am > > > wondering if you have any other comments/suggestions that I should > > > address > > > in parallel. > > > > It's in my "to-review" queue, which is huge at the moment. But the > > networking developers review will determine how this should go forward, > > so I'll just wait for them to get to it. > > > > I see the status of the submission (to netdev) is now marked as "Not > Applicable" > at netdev's patchwork site > https://patchwork.kernel.org/project/netdevbpf/list/?series==197673=*==both= > > Looks like that the patch set is considered as not being networking related. > (?) That's just to clean up their queue of things that need to be pulled into the net tree. This driver is being requested to just be acked-by netdev before Greg considers it for drivers/misc/.
RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> -Original Message- > From: Greg KH > On Mon, Mar 08, 2021 at 08:00:00PM +, Chen, Mike Ximing wrote: > > > > Hi Greg, > > > > While waiting for the feedback from the networking maintainers, I am > > wondering if you have any other comments/suggestions that I should address > > in parallel. > > It's in my "to-review" queue, which is huge at the moment. But the > networking developers review will determine how this should go forward, > so I'll just wait for them to get to it. > I see the status of the submission (to netdev) is now marked as "Not Applicable" at netdev's patchwork site https://patchwork.kernel.org/project/netdevbpf/list/?series==197673=*==both= Looks like that the patch set is considered as not being networking related. (?) Thanks Mike
RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> -Original Message- > From: Dan Williams > Sent: Monday, March 8, 2021 5:06 PM > To: Greg KH > Cc: Chen, Mike Ximing ; Linux Kernel Mailing List > ; Arnd Bergmann ; Pierre-Louis > Bossart ; Brandeburg, Jesse > ; k...@kernel.org; da...@davemloft.net; > Karlsson, Magnus ; Laatz, Kevin > ; maxi...@mellanox.com > Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > [ add Kevin and Maxim ] > > On Mon, Mar 8, 2021 at 12:13 PM Greg KH wrote: > > > > On Mon, Mar 08, 2021 at 08:00:00PM +, Chen, Mike Ximing wrote: > [..] > > > Hi Greg, > > > > > > While waiting for the feedback from the networking maintainers, I am > > > wondering if you have any other comments/suggestions that I should > address > > > in parallel. > > > > It's in my "to-review" queue, which is huge at the moment. But the > > networking developers review will determine how this should go forward, > > so I'll just wait for them to get to it. > > > Mike, > > Perhaps it would help to solicit feedback from other networking > developers that have delivered kernel features to be consumed by DPDK. > A coarse look potentially flagsKevin and Maxim working on AF_XDP > updates that DPDK consumes, Perhaps they would be willing to take a > look? Thanks, Dan. Kevin/Maxim, Please feel free to let me know if you have any question about the submission. Thanks Mike
Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
[ add Kevin and Maxim ] On Mon, Mar 8, 2021 at 12:13 PM Greg KH wrote: > > On Mon, Mar 08, 2021 at 08:00:00PM +, Chen, Mike Ximing wrote: [..] > > Hi Greg, > > > > While waiting for the feedback from the networking maintainers, I am > > wondering if you have any other comments/suggestions that I should address > > in parallel. > > It's in my "to-review" queue, which is huge at the moment. But the > networking developers review will determine how this should go forward, > so I'll just wait for them to get to it. Mike, Perhaps it would help to solicit feedback from other networking developers that have delivered kernel features to be consumed by DPDK. A coarse look potentially flagsKevin and Maxim working on AF_XDP updates that DPDK consumes, Perhaps they would be willing to take a look?
Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
On Mon, Mar 08, 2021 at 08:00:00PM +, Chen, Mike Ximing wrote: > > > > -Original Message- > > From: Dan Williams > > Sent: Tuesday, February 9, 2021 11:30 AM > > To: Greg KH > > Cc: Chen, Mike Ximing ; Linux Kernel Mailing > > List > > ; Arnd Bergmann ; Pierre-Louis > > Bossart ; Gage Eads > > > > Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > > > On Tue, Feb 9, 2021 at 5:36 AM Greg KH wrote: > > > > > > On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote: > > > > Add basic driver functionality (load, unload, probe, and remove > > > > callbacks) > > > > for the DLB driver. > > > > > > > > Add documentation which describes in detail the hardware, the user > > > > interface, device interrupts, and the driver's power-management > > > > strategy. > > > > For more details about the driver see the documentation in the patch. > > > > > > > > Add a DLB entry to the MAINTAINERS file. > > > > > > > > Signed-off-by: Gage Eads > > > > Signed-off-by: Mike Ximing Chen > > > > Reviewed-by: Magnus Karlsson > > > > Reviewed-by: Dan Williams > > > > --- > > > > Documentation/misc-devices/dlb.rst | 259 +++ > > > > Documentation/misc-devices/index.rst | 1 + > > > > MAINTAINERS | 8 + > > > > drivers/misc/Kconfig | 1 + > > > > drivers/misc/Makefile| 1 + > > > > drivers/misc/dlb/Kconfig | 18 ++ > > > > drivers/misc/dlb/Makefile| 9 + > > > > drivers/misc/dlb/dlb_hw_types.h | 32 > > > > drivers/misc/dlb/dlb_main.c | 156 > > > > drivers/misc/dlb/dlb_main.h | 37 > > > > 10 files changed, 522 insertions(+) > > > > create mode 100644 Documentation/misc-devices/dlb.rst > > > > create mode 100644 drivers/misc/dlb/Kconfig > > > > create mode 100644 drivers/misc/dlb/Makefile > > > > create mode 100644 drivers/misc/dlb/dlb_hw_types.h > > > > create mode 100644 drivers/misc/dlb/dlb_main.c > > > > create mode 100644 drivers/misc/dlb/dlb_main.h > > > > > > > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc- > > devices/dlb.rst > > > > new file mode 100644 > > > > index ..aa79be07ee49 > > > > --- /dev/null > > > > +++ b/Documentation/misc-devices/dlb.rst > > > > @@ -0,0 +1,259 @@ > > > > +.. SPDX-License-Identifier: GPL-2.0-only > > > > + > > > > +=== > > > > +Intel(R) Dynamic Load Balancer Overview > > > > +=== > > > > + > > > > +:Authors: Gage Eads and Mike Ximing Chen > > > > + > > > > +Contents > > > > + > > > > + > > > > +- Introduction > > > > +- Scheduling > > > > +- Queue Entry > > > > +- Port > > > > +- Queue > > > > +- Credits > > > > +- Scheduling Domain > > > > +- Interrupts > > > > +- Power Management > > > > +- User Interface > > > > +- Reset > > > > + > > > > +Introduction > > > > + > > > > + > > > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that > > > > +provides load-balanced, prioritized scheduling of core-to-core > > communication. > > > > + > > > > +Intel DLB is an accelerator for the event-driven programming model of > > > > +DPDK's Event Device Library[2]. The library is used in packet > > > > processing > > > > +pipelines that arrange for multi-core scalability, dynamic > > > > load-balancing, > > and > > > > +variety of packet distribution and synchronization schemes. > > > > > > As this is a networking related thing, I would like you to get the > > > proper reviews/acks from the networking maintainers before I can take > > > this. > > > > > > Or, if they think it has nothing to do with networking, that's fine too, > > > but please do not try to route around them. > > > > To be clear, I did not sense any attempt to route around networking > > review as it appeared generically centered around hardware accelerated > > IPC. At the same time I don't know what I don't know about how this > > might interact with networking initiatives so the review trip seems > > reasonable to me. > > Hi Greg, > > While waiting for the feedback from the networking maintainers, I am > wondering if you have any other comments/suggestions that I should address > in parallel. It's in my "to-review" queue, which is huge at the moment. But the networking developers review will determine how this should go forward, so I'll just wait for them to get to it. thanks, greg k-h
RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> -Original Message- > From: Dan Williams > Sent: Tuesday, February 9, 2021 11:30 AM > To: Greg KH > Cc: Chen, Mike Ximing ; Linux Kernel Mailing List > ; Arnd Bergmann ; Pierre-Louis > Bossart ; Gage Eads > > Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > On Tue, Feb 9, 2021 at 5:36 AM Greg KH wrote: > > > > On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote: > > > Add basic driver functionality (load, unload, probe, and remove callbacks) > > > for the DLB driver. > > > > > > Add documentation which describes in detail the hardware, the user > > > interface, device interrupts, and the driver's power-management strategy. > > > For more details about the driver see the documentation in the patch. > > > > > > Add a DLB entry to the MAINTAINERS file. > > > > > > Signed-off-by: Gage Eads > > > Signed-off-by: Mike Ximing Chen > > > Reviewed-by: Magnus Karlsson > > > Reviewed-by: Dan Williams > > > --- > > > Documentation/misc-devices/dlb.rst | 259 +++ > > > Documentation/misc-devices/index.rst | 1 + > > > MAINTAINERS | 8 + > > > drivers/misc/Kconfig | 1 + > > > drivers/misc/Makefile| 1 + > > > drivers/misc/dlb/Kconfig | 18 ++ > > > drivers/misc/dlb/Makefile| 9 + > > > drivers/misc/dlb/dlb_hw_types.h | 32 > > > drivers/misc/dlb/dlb_main.c | 156 > > > drivers/misc/dlb/dlb_main.h | 37 > > > 10 files changed, 522 insertions(+) > > > create mode 100644 Documentation/misc-devices/dlb.rst > > > create mode 100644 drivers/misc/dlb/Kconfig > > > create mode 100644 drivers/misc/dlb/Makefile > > > create mode 100644 drivers/misc/dlb/dlb_hw_types.h > > > create mode 100644 drivers/misc/dlb/dlb_main.c > > > create mode 100644 drivers/misc/dlb/dlb_main.h > > > > > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc- > devices/dlb.rst > > > new file mode 100644 > > > index ..aa79be07ee49 > > > --- /dev/null > > > +++ b/Documentation/misc-devices/dlb.rst > > > @@ -0,0 +1,259 @@ > > > +.. SPDX-License-Identifier: GPL-2.0-only > > > + > > > +=== > > > +Intel(R) Dynamic Load Balancer Overview > > > +=== > > > + > > > +:Authors: Gage Eads and Mike Ximing Chen > > > + > > > +Contents > > > + > > > + > > > +- Introduction > > > +- Scheduling > > > +- Queue Entry > > > +- Port > > > +- Queue > > > +- Credits > > > +- Scheduling Domain > > > +- Interrupts > > > +- Power Management > > > +- User Interface > > > +- Reset > > > + > > > +Introduction > > > + > > > + > > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that > > > +provides load-balanced, prioritized scheduling of core-to-core > communication. > > > + > > > +Intel DLB is an accelerator for the event-driven programming model of > > > +DPDK's Event Device Library[2]. The library is used in packet processing > > > +pipelines that arrange for multi-core scalability, dynamic > > > load-balancing, > and > > > +variety of packet distribution and synchronization schemes. > > > > As this is a networking related thing, I would like you to get the > > proper reviews/acks from the networking maintainers before I can take > > this. > > > > Or, if they think it has nothing to do with networking, that's fine too, > > but please do not try to route around them. > > To be clear, I did not sense any attempt to route around networking > review as it appeared generically centered around hardware accelerated > IPC. At the same time I don't know what I don't know about how this > might interact with networking initiatives so the review trip seems > reasonable to me. Hi Greg, While waiting for the feedback from the networking maintainers, I am wondering if you have any other comments/suggestions that I should address in parallel. Thanks Mike
RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> -Original Message- > From: gre...@linuxfoundation.org > Sent: Thursday, February 18, 2021 2:53 AM > To: Chen, Mike Ximing > Cc: net...@vger.kernel.org; Linux Kernel Mailing List ker...@vger.kernel.org>; da...@davemloft.net; k...@kernel.org; a...@arndb.de; > Williams, Dan J ; > pierre-louis.boss...@linux.intel.com > Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > On Thu, Feb 18, 2021 at 07:34:31AM +, Chen, Mike Ximing wrote: > > > > > > > -Original Message- > > > From: Mike Ximing Chen > > > Sent: Wednesday, February 10, 2021 12:54 PM > > > To: net...@vger.kernel.org > > > Cc: da...@davemloft.net; k...@kernel.org; a...@arndb.de; > > > gre...@linuxfoundation.org; Williams, Dan J ; > pierre- > > > louis.boss...@linux.intel.com; Gage Eads > > > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > > > > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc- > > > devices/dlb.rst > > > new file mode 100644 > > > index ..aa79be07ee49 > > > --- /dev/null > > > +++ b/Documentation/misc-devices/dlb.rst > > > @@ -0,0 +1,259 @@ > > > +.. SPDX-License-Identifier: GPL-2.0-only > > > + > > > +=== > > > +Intel(R) Dynamic Load Balancer Overview > > > +=== > > > + > > > +:Authors: Gage Eads and Mike Ximing Chen > > > + > > > +Contents > > > + > > > + > > > +- Introduction > > > +- Scheduling > > > +- Queue Entry > > > +- Port > > > +- Queue > > > +- Credits > > > +- Scheduling Domain > > > +- Interrupts > > > +- Power Management > > > +- User Interface > > > +- Reset > > > + > > > +Introduction > > > + > > > + > > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that > > > +provides load-balanced, prioritized scheduling of core-to-core > > > communication. > > > + > > > +Intel DLB is an accelerator for the event-driven programming model of > > > +DPDK's Event Device Library[2]. The library is used in packet processing > > > +pipelines that arrange for multi-core scalability, dynamic > > > load-balancing, and > > > +variety of packet distribution and synchronization schemes. > > > + > > > +Intel DLB device consists of queues and arbiters that connect producer > > > +cores and consumer cores. The device implements load-balanced queueing > > > features > > > +including: > > > +- Lock-free multi-producer/multi-consumer operation. > > > +- Multiple priority levels for varying traffic types. > > > +- 'Direct' traffic (i.e. multi-producer/single-consumer) > > > +- Simple unordered load-balanced distribution. > > > +- Atomic lock free load balancing across multiple consumers. > > > +- Queue element reordering feature allowing ordered load-balanced > > > distribution. > > > + > > > > Hi Jakub/Dave, > > This is a device driver for a HW core-to-core communication accelerator. It > > is > submitted > > to "linux-kernel" for a module under device/misc. Greg suggested (see > > below) that > we > > also sent it to you for any potential feedback in case there is any > > interaction with > > networking initiatives. The device is used to handle the load balancing > > among CPU > cores > > after the packets are received and forwarded to CPU. We don't think it > > interferes > > with networking operations, but would appreciate very much your > review/comment on this. > > It's the middle of the merge window, getting maintainers to review new > stuff until after 5.12-rc1 is out is going to be a very difficult thing > to do. > Hi Jakub/Dave, Just wonder if you had a chance to take a look at our patch. With the close of 5.12 merge window, we would like to get the process moving again. > In the meantime, why don't you all help out and review submitted patches > to the mailing lists for the subsystems you all are trying to get this > patch into. I know maintainers would appreciate the help, right? > > thanks, > > greg k-h Did a few reviews last weekend, and will continue to help. Thanks Mike
RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> -Original Message- > From: gre...@linuxfoundation.org > Sent: Thursday, February 18, 2021 2:53 AM > To: Chen, Mike Ximing > Cc: net...@vger.kernel.org; Linux Kernel Mailing List ker...@vger.kernel.org>; da...@davemloft.net; k...@kernel.org; a...@arndb.de; > Williams, Dan J ; > pierre-louis.boss...@linux.intel.com > Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > On Thu, Feb 18, 2021 at 07:34:31AM +, Chen, Mike Ximing wrote: > > > > > > > -Original Message- > > > From: Mike Ximing Chen > > > Sent: Wednesday, February 10, 2021 12:54 PM > > > To: net...@vger.kernel.org > > > Cc: da...@davemloft.net; k...@kernel.org; a...@arndb.de; > > > gre...@linuxfoundation.org; Williams, Dan J ; > pierre- > > > louis.boss...@linux.intel.com; Gage Eads > > > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > > > > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc- > > > devices/dlb.rst > > > new file mode 100644 > > > index ..aa79be07ee49 > > > --- /dev/null > > > +++ b/Documentation/misc-devices/dlb.rst > > > @@ -0,0 +1,259 @@ > > > +.. SPDX-License-Identifier: GPL-2.0-only > > > + > > > +=== > > > +Intel(R) Dynamic Load Balancer Overview > > > +=== > > > + > > > +:Authors: Gage Eads and Mike Ximing Chen > > > + > > > +Contents > > > + > > > + > > > +- Introduction > > > +- Scheduling > > > +- Queue Entry > > > +- Port > > > +- Queue > > > +- Credits > > > +- Scheduling Domain > > > +- Interrupts > > > +- Power Management > > > +- User Interface > > > +- Reset > > > + > > > +Introduction > > > + > > > + > > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that > > > +provides load-balanced, prioritized scheduling of core-to-core > > > communication. > > > + > > > +Intel DLB is an accelerator for the event-driven programming model of > > > +DPDK's Event Device Library[2]. The library is used in packet processing > > > +pipelines that arrange for multi-core scalability, dynamic > > > load-balancing, and > > > +variety of packet distribution and synchronization schemes. > > > + > > > +Intel DLB device consists of queues and arbiters that connect producer > > > +cores and consumer cores. The device implements load-balanced queueing > > > features > > > +including: > > > +- Lock-free multi-producer/multi-consumer operation. > > > +- Multiple priority levels for varying traffic types. > > > +- 'Direct' traffic (i.e. multi-producer/single-consumer) > > > +- Simple unordered load-balanced distribution. > > > +- Atomic lock free load balancing across multiple consumers. > > > +- Queue element reordering feature allowing ordered load-balanced > > > distribution. > > > + > > > > Hi Jakub/Dave, > > This is a device driver for a HW core-to-core communication accelerator. It > > is > submitted > > to "linux-kernel" for a module under device/misc. Greg suggested (see > > below) that > we > > also sent it to you for any potential feedback in case there is any > > interaction with > > networking initiatives. The device is used to handle the load balancing > > among CPU > cores > > after the packets are received and forwarded to CPU. We don't think it > > interferes > > with networking operations, but would appreciate very much your > review/comment on this. > > It's the middle of the merge window, getting maintainers to review new > stuff until after 5.12-rc1 is out is going to be a very difficult thing > to do. > > In the meantime, why don't you all help out and review submitted patches > to the mailing lists for the subsystems you all are trying to get this > patch into. I know maintainers would appreciate the help, right? > > thanks, > > greg k-h Sure. I am a little new to the community and process, but will try to help. Thanks Mike
Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
On Thu, Feb 18, 2021 at 07:34:31AM +, Chen, Mike Ximing wrote: > > > > -Original Message- > > From: Mike Ximing Chen > > Sent: Wednesday, February 10, 2021 12:54 PM > > To: net...@vger.kernel.org > > Cc: da...@davemloft.net; k...@kernel.org; a...@arndb.de; > > gre...@linuxfoundation.org; Williams, Dan J ; > > pierre- > > louis.boss...@linux.intel.com; Gage Eads > > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc- > > devices/dlb.rst > > new file mode 100644 > > index ..aa79be07ee49 > > --- /dev/null > > +++ b/Documentation/misc-devices/dlb.rst > > @@ -0,0 +1,259 @@ > > +.. SPDX-License-Identifier: GPL-2.0-only > > + > > +=== > > +Intel(R) Dynamic Load Balancer Overview > > +=== > > + > > +:Authors: Gage Eads and Mike Ximing Chen > > + > > +Contents > > + > > + > > +- Introduction > > +- Scheduling > > +- Queue Entry > > +- Port > > +- Queue > > +- Credits > > +- Scheduling Domain > > +- Interrupts > > +- Power Management > > +- User Interface > > +- Reset > > + > > +Introduction > > + > > + > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that > > +provides load-balanced, prioritized scheduling of core-to-core > > communication. > > + > > +Intel DLB is an accelerator for the event-driven programming model of > > +DPDK's Event Device Library[2]. The library is used in packet processing > > +pipelines that arrange for multi-core scalability, dynamic load-balancing, > > and > > +variety of packet distribution and synchronization schemes. > > + > > +Intel DLB device consists of queues and arbiters that connect producer > > +cores and consumer cores. The device implements load-balanced queueing > > features > > +including: > > +- Lock-free multi-producer/multi-consumer operation. > > +- Multiple priority levels for varying traffic types. > > +- 'Direct' traffic (i.e. multi-producer/single-consumer) > > +- Simple unordered load-balanced distribution. > > +- Atomic lock free load balancing across multiple consumers. > > +- Queue element reordering feature allowing ordered load-balanced > > distribution. > > + > > Hi Jakub/Dave, > This is a device driver for a HW core-to-core communication accelerator. It > is submitted > to "linux-kernel" for a module under device/misc. Greg suggested (see below) > that we > also sent it to you for any potential feedback in case there is any > interaction with > networking initiatives. The device is used to handle the load balancing among > CPU cores > after the packets are received and forwarded to CPU. We don't think it > interferes > with networking operations, but would appreciate very much your > review/comment on this. It's the middle of the merge window, getting maintainers to review new stuff until after 5.12-rc1 is out is going to be a very difficult thing to do. In the meantime, why don't you all help out and review submitted patches to the mailing lists for the subsystems you all are trying to get this patch into. I know maintainers would appreciate the help, right? thanks, greg k-h
RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> -Original Message- > From: Mike Ximing Chen > Sent: Wednesday, February 10, 2021 12:54 PM > To: net...@vger.kernel.org > Cc: da...@davemloft.net; k...@kernel.org; a...@arndb.de; > gre...@linuxfoundation.org; Williams, Dan J ; > pierre- > louis.boss...@linux.intel.com; Gage Eads > Subject: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc- > devices/dlb.rst > new file mode 100644 > index ..aa79be07ee49 > --- /dev/null > +++ b/Documentation/misc-devices/dlb.rst > @@ -0,0 +1,259 @@ > +.. SPDX-License-Identifier: GPL-2.0-only > + > +=== > +Intel(R) Dynamic Load Balancer Overview > +=== > + > +:Authors: Gage Eads and Mike Ximing Chen > + > +Contents > + > + > +- Introduction > +- Scheduling > +- Queue Entry > +- Port > +- Queue > +- Credits > +- Scheduling Domain > +- Interrupts > +- Power Management > +- User Interface > +- Reset > + > +Introduction > + > + > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that > +provides load-balanced, prioritized scheduling of core-to-core communication. > + > +Intel DLB is an accelerator for the event-driven programming model of > +DPDK's Event Device Library[2]. The library is used in packet processing > +pipelines that arrange for multi-core scalability, dynamic load-balancing, > and > +variety of packet distribution and synchronization schemes. > + > +Intel DLB device consists of queues and arbiters that connect producer > +cores and consumer cores. The device implements load-balanced queueing > features > +including: > +- Lock-free multi-producer/multi-consumer operation. > +- Multiple priority levels for varying traffic types. > +- 'Direct' traffic (i.e. multi-producer/single-consumer) > +- Simple unordered load-balanced distribution. > +- Atomic lock free load balancing across multiple consumers. > +- Queue element reordering feature allowing ordered load-balanced > distribution. > + Hi Jakub/Dave, This is a device driver for a HW core-to-core communication accelerator. It is submitted to "linux-kernel" for a module under device/misc. Greg suggested (see below) that we also sent it to you for any potential feedback in case there is any interaction with networking initiatives. The device is used to handle the load balancing among CPU cores after the packets are received and forwarded to CPU. We don't think it interferes with networking operations, but would appreciate very much your review/comment on this. Thanks for your help. Mike > As this is a networking related thing, I would like you to get the >proper reviews/acks from the networking maintainers before I can take >this. > >Or, if they think it has nothing to do with networking, that's fine too, >but please do not try to route around them. > >thanks, > >greg k-h
RE: [PATCH v10 01/20] dlb: add skeleton for DLB driver
> -Original Message- > From: Dan Williams > Sent: Tuesday, February 9, 2021 11:30 AM > To: Greg KH > Cc: Chen, Mike Ximing ; Linux Kernel Mailing List > ; Arnd Bergmann ; Pierre-Louis > Bossart ; Gage Eads > > Subject: Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver > > On Tue, Feb 9, 2021 at 5:36 AM Greg KH wrote: > > > > On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote: > > > Add basic driver functionality (load, unload, probe, and remove callbacks) > > > for the DLB driver. > > > > > > Add documentation which describes in detail the hardware, the user > > > interface, device interrupts, and the driver's power-management strategy. > > > For more details about the driver see the documentation in the patch. > > > > > > Add a DLB entry to the MAINTAINERS file. > > > > > > Signed-off-by: Gage Eads > > > Signed-off-by: Mike Ximing Chen > > > Reviewed-by: Magnus Karlsson > > > Reviewed-by: Dan Williams > > > --- > > > Documentation/misc-devices/dlb.rst | 259 +++ > > > Documentation/misc-devices/index.rst | 1 + > > > MAINTAINERS | 8 + > > > drivers/misc/Kconfig | 1 + > > > drivers/misc/Makefile| 1 + > > > drivers/misc/dlb/Kconfig | 18 ++ > > > drivers/misc/dlb/Makefile| 9 + > > > drivers/misc/dlb/dlb_hw_types.h | 32 > > > drivers/misc/dlb/dlb_main.c | 156 > > > drivers/misc/dlb/dlb_main.h | 37 > > > 10 files changed, 522 insertions(+) > > > create mode 100644 Documentation/misc-devices/dlb.rst > > > create mode 100644 drivers/misc/dlb/Kconfig > > > create mode 100644 drivers/misc/dlb/Makefile > > > create mode 100644 drivers/misc/dlb/dlb_hw_types.h > > > create mode 100644 drivers/misc/dlb/dlb_main.c > > > create mode 100644 drivers/misc/dlb/dlb_main.h > > > > > > diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc- > devices/dlb.rst > > > new file mode 100644 > > > index ..aa79be07ee49 > > > --- /dev/null > > > +++ b/Documentation/misc-devices/dlb.rst > > > @@ -0,0 +1,259 @@ > > > +.. SPDX-License-Identifier: GPL-2.0-only > > > + > > > +=== > > > +Intel(R) Dynamic Load Balancer Overview > > > +=== > > > + > > > +:Authors: Gage Eads and Mike Ximing Chen > > > + > > > +Contents > > > + > > > + > > > +- Introduction > > > +- Scheduling > > > +- Queue Entry > > > +- Port > > > +- Queue > > > +- Credits > > > +- Scheduling Domain > > > +- Interrupts > > > +- Power Management > > > +- User Interface > > > +- Reset > > > + > > > +Introduction > > > + > > > + > > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that > > > +provides load-balanced, prioritized scheduling of core-to-core > > > communication. > > > + > > > +Intel DLB is an accelerator for the event-driven programming model of > > > +DPDK's Event Device Library[2]. The library is used in packet processing > > > +pipelines that arrange for multi-core scalability, dynamic > > > load-balancing, and > > > +variety of packet distribution and synchronization schemes. > > > > As this is a networking related thing, I would like you to get the > > proper reviews/acks from the networking maintainers before I can take > > this. > > > > Or, if they think it has nothing to do with networking, that's fine too, > > but please do not try to route around them. > > > >thanks, > > > >greg k- > > > To be clear, I did not sense any attempt to route around networking > review as it appeared generically centered around hardware accelerated > IPC. At the same time I don't know what I don't know about how this > might interact with networking initiatives so the review trip seems > reasonable to me. OK. I have forwarded the patch set to netdev. Thanks! Mike
Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
On Tue, Feb 9, 2021 at 5:36 AM Greg KH wrote: > > On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote: > > Add basic driver functionality (load, unload, probe, and remove callbacks) > > for the DLB driver. > > > > Add documentation which describes in detail the hardware, the user > > interface, device interrupts, and the driver's power-management strategy. > > For more details about the driver see the documentation in the patch. > > > > Add a DLB entry to the MAINTAINERS file. > > > > Signed-off-by: Gage Eads > > Signed-off-by: Mike Ximing Chen > > Reviewed-by: Magnus Karlsson > > Reviewed-by: Dan Williams > > --- > > Documentation/misc-devices/dlb.rst | 259 +++ > > Documentation/misc-devices/index.rst | 1 + > > MAINTAINERS | 8 + > > drivers/misc/Kconfig | 1 + > > drivers/misc/Makefile| 1 + > > drivers/misc/dlb/Kconfig | 18 ++ > > drivers/misc/dlb/Makefile| 9 + > > drivers/misc/dlb/dlb_hw_types.h | 32 > > drivers/misc/dlb/dlb_main.c | 156 > > drivers/misc/dlb/dlb_main.h | 37 > > 10 files changed, 522 insertions(+) > > create mode 100644 Documentation/misc-devices/dlb.rst > > create mode 100644 drivers/misc/dlb/Kconfig > > create mode 100644 drivers/misc/dlb/Makefile > > create mode 100644 drivers/misc/dlb/dlb_hw_types.h > > create mode 100644 drivers/misc/dlb/dlb_main.c > > create mode 100644 drivers/misc/dlb/dlb_main.h > > > > diff --git a/Documentation/misc-devices/dlb.rst > > b/Documentation/misc-devices/dlb.rst > > new file mode 100644 > > index ..aa79be07ee49 > > --- /dev/null > > +++ b/Documentation/misc-devices/dlb.rst > > @@ -0,0 +1,259 @@ > > +.. SPDX-License-Identifier: GPL-2.0-only > > + > > +=== > > +Intel(R) Dynamic Load Balancer Overview > > +=== > > + > > +:Authors: Gage Eads and Mike Ximing Chen > > + > > +Contents > > + > > + > > +- Introduction > > +- Scheduling > > +- Queue Entry > > +- Port > > +- Queue > > +- Credits > > +- Scheduling Domain > > +- Interrupts > > +- Power Management > > +- User Interface > > +- Reset > > + > > +Introduction > > + > > + > > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that > > +provides load-balanced, prioritized scheduling of core-to-core > > communication. > > + > > +Intel DLB is an accelerator for the event-driven programming model of > > +DPDK's Event Device Library[2]. The library is used in packet processing > > +pipelines that arrange for multi-core scalability, dynamic load-balancing, > > and > > +variety of packet distribution and synchronization schemes. > > As this is a networking related thing, I would like you to get the > proper reviews/acks from the networking maintainers before I can take > this. > > Or, if they think it has nothing to do with networking, that's fine too, > but please do not try to route around them. To be clear, I did not sense any attempt to route around networking review as it appeared generically centered around hardware accelerated IPC. At the same time I don't know what I don't know about how this might interact with networking initiatives so the review trip seems reasonable to me.
Re: [PATCH v10 01/20] dlb: add skeleton for DLB driver
On Wed, Jan 27, 2021 at 04:56:22PM -0600, Mike Ximing Chen wrote: > Add basic driver functionality (load, unload, probe, and remove callbacks) > for the DLB driver. > > Add documentation which describes in detail the hardware, the user > interface, device interrupts, and the driver's power-management strategy. > For more details about the driver see the documentation in the patch. > > Add a DLB entry to the MAINTAINERS file. > > Signed-off-by: Gage Eads > Signed-off-by: Mike Ximing Chen > Reviewed-by: Magnus Karlsson > Reviewed-by: Dan Williams > --- > Documentation/misc-devices/dlb.rst | 259 +++ > Documentation/misc-devices/index.rst | 1 + > MAINTAINERS | 8 + > drivers/misc/Kconfig | 1 + > drivers/misc/Makefile| 1 + > drivers/misc/dlb/Kconfig | 18 ++ > drivers/misc/dlb/Makefile| 9 + > drivers/misc/dlb/dlb_hw_types.h | 32 > drivers/misc/dlb/dlb_main.c | 156 > drivers/misc/dlb/dlb_main.h | 37 > 10 files changed, 522 insertions(+) > create mode 100644 Documentation/misc-devices/dlb.rst > create mode 100644 drivers/misc/dlb/Kconfig > create mode 100644 drivers/misc/dlb/Makefile > create mode 100644 drivers/misc/dlb/dlb_hw_types.h > create mode 100644 drivers/misc/dlb/dlb_main.c > create mode 100644 drivers/misc/dlb/dlb_main.h > > diff --git a/Documentation/misc-devices/dlb.rst > b/Documentation/misc-devices/dlb.rst > new file mode 100644 > index ..aa79be07ee49 > --- /dev/null > +++ b/Documentation/misc-devices/dlb.rst > @@ -0,0 +1,259 @@ > +.. SPDX-License-Identifier: GPL-2.0-only > + > +=== > +Intel(R) Dynamic Load Balancer Overview > +=== > + > +:Authors: Gage Eads and Mike Ximing Chen > + > +Contents > + > + > +- Introduction > +- Scheduling > +- Queue Entry > +- Port > +- Queue > +- Credits > +- Scheduling Domain > +- Interrupts > +- Power Management > +- User Interface > +- Reset > + > +Introduction > + > + > +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that > +provides load-balanced, prioritized scheduling of core-to-core communication. > + > +Intel DLB is an accelerator for the event-driven programming model of > +DPDK's Event Device Library[2]. The library is used in packet processing > +pipelines that arrange for multi-core scalability, dynamic load-balancing, > and > +variety of packet distribution and synchronization schemes. As this is a networking related thing, I would like you to get the proper reviews/acks from the networking maintainers before I can take this. Or, if they think it has nothing to do with networking, that's fine too, but please do not try to route around them. thanks, greg k-h
[PATCH v10 01/20] dlb: add skeleton for DLB driver
Add basic driver functionality (load, unload, probe, and remove callbacks) for the DLB driver. Add documentation which describes in detail the hardware, the user interface, device interrupts, and the driver's power-management strategy. For more details about the driver see the documentation in the patch. Add a DLB entry to the MAINTAINERS file. Signed-off-by: Gage Eads Signed-off-by: Mike Ximing Chen Reviewed-by: Magnus Karlsson Reviewed-by: Dan Williams --- Documentation/misc-devices/dlb.rst | 259 +++ Documentation/misc-devices/index.rst | 1 + MAINTAINERS | 8 + drivers/misc/Kconfig | 1 + drivers/misc/Makefile| 1 + drivers/misc/dlb/Kconfig | 18 ++ drivers/misc/dlb/Makefile| 9 + drivers/misc/dlb/dlb_hw_types.h | 32 drivers/misc/dlb/dlb_main.c | 156 drivers/misc/dlb/dlb_main.h | 37 10 files changed, 522 insertions(+) create mode 100644 Documentation/misc-devices/dlb.rst create mode 100644 drivers/misc/dlb/Kconfig create mode 100644 drivers/misc/dlb/Makefile create mode 100644 drivers/misc/dlb/dlb_hw_types.h create mode 100644 drivers/misc/dlb/dlb_main.c create mode 100644 drivers/misc/dlb/dlb_main.h diff --git a/Documentation/misc-devices/dlb.rst b/Documentation/misc-devices/dlb.rst new file mode 100644 index ..aa79be07ee49 --- /dev/null +++ b/Documentation/misc-devices/dlb.rst @@ -0,0 +1,259 @@ +.. SPDX-License-Identifier: GPL-2.0-only + +=== +Intel(R) Dynamic Load Balancer Overview +=== + +:Authors: Gage Eads and Mike Ximing Chen + +Contents + + +- Introduction +- Scheduling +- Queue Entry +- Port +- Queue +- Credits +- Scheduling Domain +- Interrupts +- Power Management +- User Interface +- Reset + +Introduction + + +The Intel(r) Dynamic Load Balancer (Intel(r) DLB) is a PCIe device that +provides load-balanced, prioritized scheduling of core-to-core communication. + +Intel DLB is an accelerator for the event-driven programming model of +DPDK's Event Device Library[2]. The library is used in packet processing +pipelines that arrange for multi-core scalability, dynamic load-balancing, and +variety of packet distribution and synchronization schemes. + +Intel DLB device consists of queues and arbiters that connect producer +cores and consumer cores. The device implements load-balanced queueing features +including: +- Lock-free multi-producer/multi-consumer operation. +- Multiple priority levels for varying traffic types. +- 'Direct' traffic (i.e. multi-producer/single-consumer) +- Simple unordered load-balanced distribution. +- Atomic lock free load balancing across multiple consumers. +- Queue element reordering feature allowing ordered load-balanced distribution. + +Note: this document uses 'DLB' when discussing the device hardware and 'dlb' when +discussing the driver implementation. + +Following diagram illustrates the functional blocks of an Intel DLB device. + + ++ + || ++--+ || +---+ + /| IQ |---||--/| | + / +--+ || / | CP | + / ||/ +---+ +++ / || +|| /+--+ | /| +---+ +| PP|--| IQ |---||---| | +++ \+--+ | / | | CP | +\ |/ | +---+ + ... \ ... || +++\ /|| +---+ +|| \+--+ / || | | +| PP|--| IQ |/--||---| CP | +++ +--+ || +---+ + || + ++ ... +PP: Producer Port| +CP: Consumer Port| +IQ: Internal Queue DLB Scheduler + + +Scheduling Types + + +Intel DLB supports four types of scheduling of 'events' (using DPDK +terminology), where an event can represent any type of data (e.g. a network +packet). The first, ``directed``, is multi-producer/single-consumer style +core-to-core communication. The remaining three are +multi-producer/multi-consumer, and support load-balancing across the consumers. + +- ``Directed``: events are scheduled to a single consumer. + +- ``Unordered``: events are load-balanced across consumers without any ordering + guarantees. + +- ``Ordered``: events are load-balanced across consumers, and the consumer can + re-enqueue its events so the device re-orders them into the +
