[ovs-dev] [PATCH v4 6/7] dpif-netdev: Allow pin rxq and non-isolate PMD.
Pinning an rxq to a PMD with pmd-rxq-affinity may be done for various reasons such as reserving a full PMD for an rxq, or to ensure that multiple rxqs from a port are handled on different PMDs. Previously pmd-rxq-affinity always isolated the PMD so no other rxqs could be assigned to it by OVS. There may be cases where there is unused cycles on those pmds and the user would like other rxqs to also be able to be assigned to it by OVS. Add an option to pin the rxq and non-isolate the PMD. The default behaviour is unchanged, which is pin and isolate the PMD. In order to pin and non-isolate: ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false Note this is available only with group assignment type, as pinning conflicts with the operation of the other rxq assignment algorithms. Signed-off-by: Kevin Traynor --- Documentation/topics/dpdk/pmd.rst | 9 ++- NEWS | 3 + lib/dpif-netdev.c | 34 -- tests/pmd.at | 105 ++ vswitchd/vswitch.xml | 19 ++ 5 files changed, 162 insertions(+), 8 deletions(-) diff --git a/Documentation/topics/dpdk/pmd.rst b/Documentation/topics/dpdk/pmd.rst index 29ba53954..30040d703 100644 --- a/Documentation/topics/dpdk/pmd.rst +++ b/Documentation/topics/dpdk/pmd.rst @@ -102,6 +102,11 @@ like so: - Queue #3 pinned to core 8 -PMD threads on cores where Rx queues are *pinned* will become *isolated*. This -means that this thread will only poll the *pinned* Rx queues. +PMD threads on cores where Rx queues are *pinned* will become *isolated* by +default. This means that this thread will only poll the *pinned* Rx queues. + +If using ``pmd-rxq-assign=group`` PMD threads with *pinned* Rxqs can be +*non-isolated* by setting:: + + $ ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false .. warning:: diff --git a/NEWS b/NEWS index b1e186d49..2ee156048 100644 --- a/NEWS +++ b/NEWS @@ -20,4 +20,7 @@ Post-v2.15.0 * Added new 'group' option to pmd-rxq-assign. This will assign rxq to pmds purely based on rxq and pmd load. + * Add new 'pmd-rxq-isolate' option that can be set to 'false' in order + that pmd cores which are pinned with rxqs using 'pmd-rxq-affinity' + are available for assigning other non-pinned rxqs. - ovs-ctl: * New option '--no-record-hostname' to disable hostname configuration diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index c372aa48c..4365b1c2b 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -378,4 +378,5 @@ struct dp_netdev { /* Rxq to pmd assignment type. */ enum sched_assignment_type pmd_rxq_assign_type; +bool pmd_iso; /* Protects the access of the 'struct dp_netdev_pmd_thread' @@ -4390,4 +4391,22 @@ dpif_netdev_set_config(struct dpif *dpif, const struct smap *other_config) } +bool pmd_iso = smap_get_bool(other_config, "pmd-rxq-isolate", true); + +if (pmd_rxq_assign_type != SCHED_GROUP && pmd_iso == false) { +/* Invalid combination. */ +VLOG_WARN("pmd-rxq-isolate can only be set false " + "when using pmd-rxq-assign=group"); +pmd_iso = true; +} +if (dp->pmd_iso != pmd_iso) { +dp->pmd_iso = pmd_iso; +if (pmd_iso) { +VLOG_INFO("pmd-rxq-affinity isolates PMD core"); +} else { +VLOG_INFO("pmd-rxq-affinity does not isolate PMD core"); +} +dp_netdev_request_reconfigure(dp); +} + struct pmd_auto_lb *pmd_alb = &dp->pmd_alb; bool cur_rebalance_requested = pmd_alb->auto_lb_requested; @@ -5115,5 +5134,5 @@ sched_numa_list_assignments(struct sched_numa_list *numa_list, sched_pmd = sched_pmd_find_by_pmd(numa_list, rxq->pmd); if (sched_pmd) { -if (rxq->core_id != OVS_CORE_UNSPEC) { +if (rxq->core_id != OVS_CORE_UNSPEC && dp->pmd_iso) { sched_pmd->isolated = true; } @@ -5382,4 +5401,5 @@ sched_numa_list_schedule(struct sched_numa_list *numa_list, struct dp_netdev_pmd_thread *pmd; struct sched_numa *numa; +bool iso = dp->pmd_iso; uint64_t proc_cycles; char rxq_cyc_log[MAX_RXQ_CYC_STRLEN]; @@ -5404,9 +5424,11 @@ sched_numa_list_schedule(struct sched_numa_list *numa_list, continue; } -/* Mark PMD as isolated if not done already. */ -if (sched_pmd->isolated == false) { -sched_pmd->isolated = true; -numa = sched_pmd->numa; -numa->n_isolated++; +if (iso) { +/* Mark PMD as isolated if not done already. */ +if (sched_pmd->isolated == false) { +sched_pmd->isolated = true; +numa = sched_pmd->numa; +numa->n_
Re: [ovs-dev] [PATCH v4 6/7] dpif-netdev: Allow pin rxq and non-isolate PMD.
Hi Kevin, > Subject: [PATCH v4 6/7] dpif-netdev: Allow pin rxq and non-isolate PMD. > > Pinning an rxq to a PMD with pmd-rxq-affinity may be done for various > reasons such as reserving a full PMD for an rxq, or to ensure that multiple > rxqs from a port are handled on different PMDs. > > Previously pmd-rxq-affinity always isolated the PMD so no other rxqs could > be assigned to it by OVS. There may be cases where there is unused cycles > on those pmds and the user would like other rxqs to also be able to be > assigned to it by OVS. > > Add an option to pin the rxq and non-isolate the PMD. The default behaviour > is unchanged, which is pin and isolate the PMD. > > In order to pin and non-isolate: > ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false > > Note this is available only with group assignment type, as pinning conflicts > with the operation of the other rxq assignment algorithms. > > Signed-off-by: Kevin Traynor > --- > Documentation/topics/dpdk/pmd.rst | 9 ++- > NEWS | 3 + > lib/dpif-netdev.c | 34 -- > tests/pmd.at | 105 ++ > vswitchd/vswitch.xml | 19 ++ > 5 files changed, 162 insertions(+), 8 deletions(-) > LGTM. Acked-by: Sunil Pai G ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v4 6/7] dpif-netdev: Allow pin rxq and non-isolate PMD.
On Thu, Jul 8, 2021 at 3:54 PM Kevin Traynor wrote: > > Pinning an rxq to a PMD with pmd-rxq-affinity may be done for > various reasons such as reserving a full PMD for an rxq, or to > ensure that multiple rxqs from a port are handled on different PMDs. > > Previously pmd-rxq-affinity always isolated the PMD so no other rxqs > could be assigned to it by OVS. There may be cases where there is > unused cycles on those pmds and the user would like other rxqs to > also be able to be assigned to it by OVS. > > Add an option to pin the rxq and non-isolate the PMD. The default > behaviour is unchanged, which is pin and isolate the PMD. > > In order to pin and non-isolate: > ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false > > Note this is available only with group assignment type, as pinning > conflicts with the operation of the other rxq assignment algorithms. > > Signed-off-by: Kevin Traynor Acked-by: David Marchand -- David Marchand ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v4 6/7] dpif-netdev: Allow pin rxq and non-isolate PMD.
> -Original Message- > From: dev On Behalf Of Kevin Traynor > Sent: Thursday, 8 July, 2021 15:54 > To: d...@openvswitch.org > Cc: david.march...@redhat.com > Subject: [ovs-dev] [PATCH v4 6/7] dpif-netdev: Allow pin rxq and non-isolate > PMD. > > Pinning an rxq to a PMD with pmd-rxq-affinity may be done for various reasons > such as reserving a full PMD for an rxq, or to ensure that multiple rxqs from > a > port are handled on different PMDs. > > Previously pmd-rxq-affinity always isolated the PMD so no other rxqs could be > assigned to it by OVS. There may be cases where there is unused cycles on > those pmds and the user would like other rxqs to also be able to be assigned > to > it by OVS. > > Add an option to pin the rxq and non-isolate the PMD. The default behaviour is > unchanged, which is pin and isolate the PMD. > > In order to pin and non-isolate: > ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false > > Note this is available only with group assignment type, as pinning conflicts > with > the operation of the other rxq assignment algorithms. > > Signed-off-by: Kevin Traynor > --- > Documentation/topics/dpdk/pmd.rst | 9 ++- > NEWS | 3 + > lib/dpif-netdev.c | 34 -- > tests/pmd.at | 105 ++ > vswitchd/vswitch.xml | 19 ++ > 5 files changed, 162 insertions(+), 8 deletions(-) > > diff --git a/Documentation/topics/dpdk/pmd.rst > b/Documentation/topics/dpdk/pmd.rst > index 29ba53954..30040d703 100644 > --- a/Documentation/topics/dpdk/pmd.rst > +++ b/Documentation/topics/dpdk/pmd.rst > @@ -102,6 +102,11 @@ like so: > - Queue #3 pinned to core 8 > > -PMD threads on cores where Rx queues are *pinned* will become *isolated*. > This -means that this thread will only poll the *pinned* Rx queues. > +PMD threads on cores where Rx queues are *pinned* will become > +*isolated* by default. This means that this thread will only poll the > *pinned* > Rx queues. > + > +If using ``pmd-rxq-assign=group`` PMD threads with *pinned* Rxqs can be > +*non-isolated* by setting:: > + > + $ ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false Is there any specific reason why the new pmd-rxq-isolate option should be limited to the "group" scheduling algorithm? In my view it would make more sense and simplify documentation and code if these aspects of scheduling were kept orthogonal. Regards, Jan ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v4 6/7] dpif-netdev: Allow pin rxq and non-isolate PMD.
Hi Jan, On 13/07/2021 08:41, Jan Scheurich wrote: > >> -Original Message- >> From: dev On Behalf Of Kevin Traynor >> Sent: Thursday, 8 July, 2021 15:54 >> To: d...@openvswitch.org >> Cc: david.march...@redhat.com >> Subject: [ovs-dev] [PATCH v4 6/7] dpif-netdev: Allow pin rxq and non-isolate >> PMD. >> >> Pinning an rxq to a PMD with pmd-rxq-affinity may be done for various reasons >> such as reserving a full PMD for an rxq, or to ensure that multiple rxqs >> from a >> port are handled on different PMDs. >> >> Previously pmd-rxq-affinity always isolated the PMD so no other rxqs could be >> assigned to it by OVS. There may be cases where there is unused cycles on >> those pmds and the user would like other rxqs to also be able to be assigned >> to >> it by OVS. >> >> Add an option to pin the rxq and non-isolate the PMD. The default behaviour >> is >> unchanged, which is pin and isolate the PMD. >> >> In order to pin and non-isolate: >> ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false >> >> Note this is available only with group assignment type, as pinning conflicts >> with >> the operation of the other rxq assignment algorithms. >> >> Signed-off-by: Kevin Traynor >> --- >> Documentation/topics/dpdk/pmd.rst | 9 ++- >> NEWS | 3 + >> lib/dpif-netdev.c | 34 -- >> tests/pmd.at | 105 ++ >> vswitchd/vswitch.xml | 19 ++ >> 5 files changed, 162 insertions(+), 8 deletions(-) >> >> diff --git a/Documentation/topics/dpdk/pmd.rst >> b/Documentation/topics/dpdk/pmd.rst >> index 29ba53954..30040d703 100644 >> --- a/Documentation/topics/dpdk/pmd.rst >> +++ b/Documentation/topics/dpdk/pmd.rst >> @@ -102,6 +102,11 @@ like so: >> - Queue #3 pinned to core 8 >> >> -PMD threads on cores where Rx queues are *pinned* will become *isolated*. >> This -means that this thread will only poll the *pinned* Rx queues. >> +PMD threads on cores where Rx queues are *pinned* will become >> +*isolated* by default. This means that this thread will only poll the >> *pinned* >> Rx queues. >> + >> +If using ``pmd-rxq-assign=group`` PMD threads with *pinned* Rxqs can be >> +*non-isolated* by setting:: >> + >> + $ ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false > > Is there any specific reason why the new pmd-rxq-isolate option should be > limited to the "group" scheduling algorithm? In my view it would make more > sense and simplify documentation and code if these aspects of scheduling were > kept orthogonal. > David had a similar comment on an earlier version. I will add the fuller reply below. In summary, pinning and the other algorithms (particularly if there was multiple pinnings) conflict in how they operate because they are based on RR pmd's to add equal number of Rxqs/PMD. --- Yes, the main issue is that the other algorithms are based on a pmd order on the assumption that they start from empty. For 'roundrobin' it is to equally distribute num of rxqs on pmd RR - if we pin several rxqs on one pmds, it is not clear what to do etc. Even worse for 'cycles' it is based on placing rxqs in order of busy cycles on the pmds RR. If we pin an rxq initially we are conflicting against how the algorithm operates. Because 'group' algorithm is not tied to a defined pmd order, it is better suited to be able to place a pinned rxq initially and be able to consider that pmd on it's merits along with the others later. --- Kevin. > Regards, Jan > ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Re: [ovs-dev] [PATCH v4 6/7] dpif-netdev: Allow pin rxq and non-isolate PMD.
> >> +If using ``pmd-rxq-assign=group`` PMD threads with *pinned* Rxqs can > >> +be > >> +*non-isolated* by setting:: > >> + > >> + $ ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-isolate=false > > > > Is there any specific reason why the new pmd-rxq-isolate option should be > limited to the "group" scheduling algorithm? In my view it would make more > sense and simplify documentation and code if these aspects of scheduling were > kept orthogonal. > > > > David had a similar comment on an earlier version. I will add the fuller reply > below. In summary, pinning and the other algorithms (particularly if there was > multiple pinnings) conflict in how they operate because they are based on RR > pmd's to add equal number of Rxqs/PMD. > > --- > Yes, the main issue is that the other algorithms are based on a pmd order on > the assumption that they start from empty. For 'roundrobin' it is to equally > distribute num of rxqs on pmd RR - if we pin several rxqs on one pmds, it is > not > clear what to do etc. Even worse for 'cycles' it is based on placing rxqs in > order > of busy cycles on the pmds RR. If we pin an rxq initially we are conflicting > against how the algorithm operates. > > Because 'group' algorithm is not tied to a defined pmd order, it is better > suited > to be able to place a pinned rxq initially and be able to consider that pmd on > it's merits along with the others later. > --- Makes sense. The legacy round-robin and cycles algorithms are not really prepared for pinned rxqs. In our patch set we had replaced the original round-robin and cycles algorithms with a single new algorithm that simply distributed the rxqs to the "least-loaded" non-isolated PMD (based on #rxqs or cycles assigned so far). With pinning=isolation, this exactly reproduced the round-robin behavior and created at least as good balance as the original cycles algorithm. In that new algorithm any pinned rxqs were already considered and didn't lead to distorted results. BR, Jan ___ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev