[dpdk-dev] [PATCH 7/7] examples/eventdev_pipeline: adding example
On Tue, Nov 22, 2016 at 02:04:27PM +, Richardson, Bruce wrote: > > > > -Original Message- > > From: Jerin Jacob [mailto:jerin.jacob at caviumnetworks.com] > > Sent: Tuesday, November 22, 2016 6:02 AM > > To: Van Haaren, Harry > > Cc: dev at dpdk.org; Eads, Gage ; Richardson, Bruce > > > > Subject: Re: [dpdk-dev] [PATCH 7/7] examples/eventdev_pipeline: adding > > example > > > > On Wed, Nov 16, 2016 at 06:00:07PM +, Harry van Haaren wrote: > > > This patch adds a sample app to the examples/ directory, which can be > > > used as a reference application and for general testing. The > > > application requires two ethdev ports and expects traffic to be > > > flowing. The application must be run with the --vdev flags as follows > > > to indicate to EAL that a virtual eventdev device called "evdev_sw0" is > > available to be used: > > > > > > ./build/eventdev_pipeline --vdev evdev_sw0 > > > > > > The general flow of the traffic is as follows: > > > > > > Rx core -> Atomic Queue => 4 worker cores => TX core > > > > > > A scheduler core is required to do the packet scheduling, making this > > > configuration require 7 cores (Rx, Tx, Scheduler, and 4 workers). > > > Finally a master core brings the core count to 8 for this > > > configuration. The > > > > Thanks for the example application.I will try to share my views on ethdev > > integration and usability perspective. Hope we can converge. > > Hi Jerin, > > thanks for the feedback. We'll take it on board for a subsequent version > we produce. Additional comments and queries on your feedback inline below. Thanks Bruce. > > /Bruce > > > > > Some of the high level details first before getting into exact details. > > > > 1) From the HW and ethdev integration perspective, The integrated NIC > > controllers does not need producer core(s) to push the event/packets to > > event queue. So, I was thinking to use 6WIND rte_flow spec to create the > > "ethdev port to event queue wiring" connection by extending the output > > ACTION definition, which specifies event queue its need to enqueued to for > > the given ethdev port (something your are doing in application). > > > > I guess, the producer part of this example can be created as common code, > > somewhere in rte_flow/ethdev to reuse. We would need this scheme also > > where when we deal with external nics + HW event manager use case > > > Yes. This is something to consider. > > For the pure-software model, we also might want to look at the opposite > approach, where we register an ethdev with the scheduler for it to "pull" > new packets from. This would allow it to bypass the port logic for the new > packets. Not sure,I understand this completely. How different its integrating with rte_flow specification ? > > An alternative for this is to extend the schedule API to allow a burst of > packets to be passed in to be scheduled immediately as "NEW" packets. The end > results should be the same, saving cycles by bypassing unneeded processing > for the new packets. > > > The complete event driven model can be verified and exercised without > > integrating with eventdev subsystem. So I think, may be we need to focus > > on functional applications without ethdev to verify the eventdev features > > like(automatic multicore scaling, dynamic load balancing, pipelining, > > packet ingress order maintenance and synchronization services) and then > > integrate with ethdev > > Yes, comprehensive unit tests will be needed too. But an example app that > pulls packets from an external NIC I also think is needed to get a feel > for the performance of the scheduler with real traffic. I agree, we need to have example to show case with real traffic. Please check on ethdev integration aspects. Cavium has both server (that's going to use SW event pmd) and NPU based platform(that's going to use HW event pmd). So we would like to have common approach that makes integration of both models with out changing the application. I was thinking more with "service core" and "rte_flow" based integration methodology to make that happen. > > > > > > + const unsigned cores_needed = num_workers + > > > + /*main*/1 + > > > + /*sched*/1 + > > > + /*TX*/1 + > > > + /*RX*/1; > > > + > > > > 2) One of the prime aims of the event driven model is to remove the fixed
[dpdk-dev] [PATCH 7/7] examples/eventdev_pipeline: adding example
> -Original Message- > From: Jerin Jacob [mailto:jerin.jacob at caviumnetworks.com] > Sent: Tuesday, November 22, 2016 6:02 AM > To: Van Haaren, Harry > Cc: dev at dpdk.org; Eads, Gage ; Richardson, Bruce > > Subject: Re: [dpdk-dev] [PATCH 7/7] examples/eventdev_pipeline: adding > example > > On Wed, Nov 16, 2016 at 06:00:07PM +, Harry van Haaren wrote: > > This patch adds a sample app to the examples/ directory, which can be > > used as a reference application and for general testing. The > > application requires two ethdev ports and expects traffic to be > > flowing. The application must be run with the --vdev flags as follows > > to indicate to EAL that a virtual eventdev device called "evdev_sw0" is > available to be used: > > > > ./build/eventdev_pipeline --vdev evdev_sw0 > > > > The general flow of the traffic is as follows: > > > > Rx core -> Atomic Queue => 4 worker cores => TX core > > > > A scheduler core is required to do the packet scheduling, making this > > configuration require 7 cores (Rx, Tx, Scheduler, and 4 workers). > > Finally a master core brings the core count to 8 for this > > configuration. The > > Thanks for the example application.I will try to share my views on ethdev > integration and usability perspective. Hope we can converge. Hi Jerin, thanks for the feedback. We'll take it on board for a subsequent version we produce. Additional comments and queries on your feedback inline below. /Bruce > > Some of the high level details first before getting into exact details. > > 1) From the HW and ethdev integration perspective, The integrated NIC > controllers does not need producer core(s) to push the event/packets to > event queue. So, I was thinking to use 6WIND rte_flow spec to create the > "ethdev port to event queue wiring" connection by extending the output > ACTION definition, which specifies event queue its need to enqueued to for > the given ethdev port (something your are doing in application). > > I guess, the producer part of this example can be created as common code, > somewhere in rte_flow/ethdev to reuse. We would need this scheme also > where when we deal with external nics + HW event manager use case > Yes. This is something to consider. For the pure-software model, we also might want to look at the opposite approach, where we register an ethdev with the scheduler for it to "pull" new packets from. This would allow it to bypass the port logic for the new packets. An alternative for this is to extend the schedule API to allow a burst of packets to be passed in to be scheduled immediately as "NEW" packets. The end results should be the same, saving cycles by bypassing unneeded processing for the new packets. > The complete event driven model can be verified and exercised without > integrating with eventdev subsystem. So I think, may be we need to focus > on functional applications without ethdev to verify the eventdev features > like(automatic multicore scaling, dynamic load balancing, pipelining, > packet ingress order maintenance and synchronization services) and then > integrate with ethdev Yes, comprehensive unit tests will be needed too. But an example app that pulls packets from an external NIC I also think is needed to get a feel for the performance of the scheduler with real traffic. > > > + const unsigned cores_needed = num_workers + > > + /*main*/1 + > > + /*sched*/1 + > > + /*TX*/1 + > > + /*RX*/1; > > + > > 2) One of the prime aims of the event driven model is to remove the fixed > function core mappings and enable automatic multicore scaling, dynamic > load balancing etc.I will try to use an example in review section to show > the method for removing "consumer core" in this case. Yes, I agree, but unfortunately, for some tasks, distributing those tasks across multiple cores can hurt performance overall do to resource contention. > > > application can be configured for various numbers of flows and worker > > cores. Run the application with -h for details. > > > > Signed-off-by: Gage Eads > > Signed-off-by: Bruce Richardson > > Signed-off-by: Harry van Haaren > > --- > > examples/eventdev_pipeline/Makefile | 49 +++ > > examples/eventdev_pipeline/main.c | 718 > > > 2 files changed, 767 insertions(+) > > create mode 100644 examples/eventdev_pipeline/Makefile > > create mode 100644 examples/eventdev_pipeline/main.c > > > > +static int sched_type = RTE_SCHED_TYPE_ATOMIC; > > RTE_SCHED_T
[dpdk-dev] [PATCH 7/7] examples/eventdev_pipeline: adding example
On Wed, Nov 16, 2016 at 06:00:07PM +, Harry van Haaren wrote:
> This patch adds a sample app to the examples/ directory, which can be used
> as a reference application and for general testing. The application requires
> two ethdev ports and expects traffic to be flowing. The application must be
> run with the --vdev flags as follows to indicate to EAL that a virtual
> eventdev device called "evdev_sw0" is available to be used:
>
> ./build/eventdev_pipeline --vdev evdev_sw0
>
> The general flow of the traffic is as follows:
>
> Rx core -> Atomic Queue => 4 worker cores => TX core
>
> A scheduler core is required to do the packet scheduling, making this
> configuration require 7 cores (Rx, Tx, Scheduler, and 4 workers). Finally
> a master core brings the core count to 8 for this configuration. The
Thanks for the example application.I will try to share my views on
ethdev integration and usability perspective. Hope we can converge.
Some of the high level details first before getting into exact details.
1) From the HW and ethdev integration perspective, The integrated NIC
controllers
does not need producer core(s) to push the event/packets to event queue. So, I
was
thinking to use 6WIND rte_flow spec to create the "ethdev port to event
queue wiring" connection by extending the output ACTION definition, which
specifies event queue its need to enqueued to for the given ethdev port
(something your are doing in application).
I guess, the producer part of this example can be created as common
code, somewhere in rte_flow/ethdev to reuse. We would need this scheme also
where when we deal with external nics + HW event manager use case
The complete event driven model can be verified and exercised without
integrating with eventdev subsystem. So I think, may be we need to
focus on functional applications without ethdev to verify the eventdev
features like(automatic multicore scaling, dynamic load balancing, pipelining,
packet ingress order maintenance and synchronization services) and then
integrate with ethdev
> + const unsigned cores_needed = num_workers +
> + /*main*/1 +
> + /*sched*/1 +
> + /*TX*/1 +
> + /*RX*/1;
> +
2) One of the prime aims of the event driven model is to remove the fixed
function core mappings and enable automatic multicore scaling, dynamic load
balancing etc.I will try to use an example in review section to show the
method for removing "consumer core" in this case.
> application can be configured for various numbers of flows and worker
> cores. Run the application with -h for details.
>
> Signed-off-by: Gage Eads
> Signed-off-by: Bruce Richardson
> Signed-off-by: Harry van Haaren
> ---
> examples/eventdev_pipeline/Makefile | 49 +++
> examples/eventdev_pipeline/main.c | 718
>
> 2 files changed, 767 insertions(+)
> create mode 100644 examples/eventdev_pipeline/Makefile
> create mode 100644 examples/eventdev_pipeline/main.c
>
> +static int sched_type = RTE_SCHED_TYPE_ATOMIC;
RTE_SCHED_TYPE_ORDERED makes sense as a default. Most common case will
have ORDERD at first stage so that it can scale.
> +
> +
> +static int
> +worker(void *arg)
> +{
> + struct rte_event rcv_events[BATCH_SIZE];
> +
> + struct worker_data *data = (struct worker_data *)arg;
> + uint8_t event_dev_id = data->event_dev_id;
> + uint8_t event_port_id = data->event_port_id;
> + int32_t qid = data->qid;
> + size_t sent = 0, received = 0;
> +
> + while (!done) {
> + uint16_t i;
> +
> + uint16_t n = rte_event_dequeue_burst(event_dev_id,
> + event_port_id,
> + rcv_events,
> + RTE_DIM(rcv_events),
> + false);
> + if (n == 0){
> + rte_pause();
> + /* Flush any buffered events */
> + rte_event_dequeue(event_dev_id,
> + event_port_id,
> + NULL,
> + false);
The above can be done in implementation. May not be the candidate for common
code.
> + continue;
> + }
> + received += n;
> +
> + for (i = 0; i < n; i++) {
> + struct ether_hdr *eth;
> + struct ether_addr addr;
> + struct rte_event *ev = &rcv_events[i];
> +
> + ev->queue_id = qid;
> + ev->flow_id = 0;
Another way to deal wit out additional consumer core(it creates issue in
scaling and load balancing) is
in worker:
while(1) {
ev = dequeue(port);
// stage 1 app processing
if (ev.event_type == RTE_EVENT_TYPE_ETHDEV) {
[dpdk-dev] [PATCH 7/7] examples/eventdev_pipeline: adding example
This patch adds a sample app to the examples/ directory, which can be used as a reference application and for general testing. The application requires two ethdev ports and expects traffic to be flowing. The application must be run with the --vdev flags as follows to indicate to EAL that a virtual eventdev device called "evdev_sw0" is available to be used: ./build/eventdev_pipeline --vdev evdev_sw0 The general flow of the traffic is as follows: Rx core -> Atomic Queue => 4 worker cores => TX core A scheduler core is required to do the packet scheduling, making this configuration require 7 cores (Rx, Tx, Scheduler, and 4 workers). Finally a master core brings the core count to 8 for this configuration. The application can be configured for various numbers of flows and worker cores. Run the application with -h for details. Signed-off-by: Gage Eads Signed-off-by: Bruce Richardson Signed-off-by: Harry van Haaren --- examples/eventdev_pipeline/Makefile | 49 +++ examples/eventdev_pipeline/main.c | 718 2 files changed, 767 insertions(+) create mode 100644 examples/eventdev_pipeline/Makefile create mode 100644 examples/eventdev_pipeline/main.c diff --git a/examples/eventdev_pipeline/Makefile b/examples/eventdev_pipeline/Makefile new file mode 100644 index 000..bab8916 --- /dev/null +++ b/examples/eventdev_pipeline/Makefile @@ -0,0 +1,49 @@ +# BSD LICENSE +# +# Copyright(c) 2016 Intel Corporation. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in +# the documentation and/or other materials provided with the +# distribution. +# * Neither the name of Intel Corporation nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +ifeq ($(RTE_SDK),) +$(error "Please define RTE_SDK environment variable") +endif + +# Default target, can be overriden by command line or environment +RTE_TARGET ?= x86_64-native-linuxapp-gcc + +include $(RTE_SDK)/mk/rte.vars.mk + +# binary name +APP = eventdev_pipeline + +# all source are stored in SRCS-y +SRCS-y := main.c + +CFLAGS += -O3 +CFLAGS += $(WERROR_FLAGS) + +include $(RTE_SDK)/mk/rte.extapp.mk diff --git a/examples/eventdev_pipeline/main.c b/examples/eventdev_pipeline/main.c new file mode 100644 index 000..6a8052c --- /dev/null +++ b/examples/eventdev_pipeline/main.c @@ -0,0 +1,718 @@ +/*- + * BSD LICENSE + * + * Copyright(c) 2016 Intel Corporation. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * * Neither the name of Intel Corporation nor the names of its + * contributors may be used to endorse or promote products derived + * from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +
