Re: [PATCH v4 11/13] task_isolation: net: don't flush backlog on CPUs running isolated tasks

2021-01-22 Thread Paolo Abeni 
On Fri, 2021-01-22 at 11:13 -0300, Marcelo Tosatti wrote:
> On Thu, Oct 01, 2020 at 04:47:31PM +0200, Frederic Weisbecker wrote:
> > On Wed, Jul 22, 2020 at 02:58:24PM +, Alex Belits wrote:
> > > From: Yuri Norov 
> > > 
> > > so we don't need to flush it.
> > 
> > What guarantees that we have no backlog on it?
> 
> From Paolo's work to use lockless reading of 
> per-CPU skb lists
> 
> https://www.spinics.net/lists/netdev/msg682693.html
> 
> It also exposed skb queue length to userspace
> 
> https://www.spinics.net/lists/netdev/msg684939.html
> 
> But if i remember correctly waiting for a RCU grace
> period was also necessary to ensure no backlog !?! 
> 
> Paolo would you please remind us what was the sequence of steps?
> (and then also, for the userspace isolation interface, where 
> the application informs the kernel that its entering isolated
> mode, is just confirming the queues have zero length is
> sufficient?).

After commit 2de79ee27fdb52626ac4ac48ec6d8d52ba6f9047, for CONFIG_RPS
enabled build, with no RFS in place to ensure backlog will be empty on
CPU X, the user must:
- configure the RPS map on each device before the device goes up to
explicitly exclude CPU X.

If CPU X is isolated after some network device already went up, to
ensure that the backlog will be empty on CPU X the user must:
- configure RPS on all the network device to exclude CPU X (as in the
previous scenario)
- wait a RCU grace period
- wait untill the backlog len on CPU X reported by procfs is 0

Cheers,

Paolo

Re: [PATCH v4 11/13] task_isolation: net: don't flush backlog on CPUs running isolated tasks

2021-01-22 Thread Marcelo Tosatti 
On Thu, Oct 01, 2020 at 04:47:31PM +0200, Frederic Weisbecker wrote:
> On Wed, Jul 22, 2020 at 02:58:24PM +, Alex Belits wrote:
> > From: Yuri Norov 
> > 

> > so we don't need to flush it.
> 
> What guarantees that we have no backlog on it?

>From Paolo's work to use lockless reading of 
per-CPU skb lists

https://www.spinics.net/lists/netdev/msg682693.html

It also exposed skb queue length to userspace

https://www.spinics.net/lists/netdev/msg684939.html

But if i remember correctly waiting for a RCU grace
period was also necessary to ensure no backlog !?! 

Paolo would you please remind us what was the sequence of steps?
(and then also, for the userspace isolation interface, where 
the application informs the kernel that its entering isolated
mode, is just confirming the queues have zero length is
sufficient?).

TIA!

> 
> > Currently flush_all_backlogs()
> > enqueues corresponding work on all CPUs including ones that run
> > isolated tasks. It leads to breaking task isolation for nothing.
> > 
> > In this patch, backlog flushing is enqueued only on non-isolated CPUs.
> > 
> > Signed-off-by: Yuri Norov 
> > [abel...@marvell.com: use safe task_isolation_on_cpu() implementation]
> > Signed-off-by: Alex Belits 
> > ---
> >  net/core/dev.c | 7 ++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> > 
> > diff --git a/net/core/dev.c b/net/core/dev.c
> > index 90b59fc50dc9..83a282f7453d 100644
> > --- a/net/core/dev.c
> > +++ b/net/core/dev.c
> > @@ -74,6 +74,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  #include 
> >  #include 
> >  #include 
> > @@ -5624,9 +5625,13 @@ static void flush_all_backlogs(void)
> >  
> > get_online_cpus();
> >  
> > -   for_each_online_cpu(cpu)
> > +   smp_rmb();
> 
> What is it ordering?
> 
> > +   for_each_online_cpu(cpu) {
> > +   if (task_isolation_on_cpu(cpu))
> > +   continue;
> > queue_work_on(cpu, system_highpri_wq,
> >   per_cpu_ptr(&flush_works, cpu));
> > +   }
> >  
> > for_each_online_cpu(cpu)
> > flush_work(per_cpu_ptr(&flush_works, cpu));
> 
> Thanks.

Re: [EXT] Re: [PATCH v4 11/13] task_isolation: net: don't flush backlog on CPUs running isolated tasks

2020-10-04 Thread Alex Belits 

On Thu, 2020-10-01 at 16:47 +0200, Frederic Weisbecker wrote:
> External Email
> 
> ---
> ---
> On Wed, Jul 22, 2020 at 02:58:24PM +, Alex Belits wrote:
> > From: Yuri Norov 
> > 
> > If CPU runs isolated task, there's no any backlog on it, and
> > so we don't need to flush it.
> 
> What guarantees that we have no backlog on it?

I believe, the logic was that it is not supposed to have backlog
because it could not be produced while the CPU was in userspace,
because one has to enter kernel to receive (by interrupt) or send (by
syscall) anything.

Now, looking at this patch. I don't think, it can be guaranteed that
there was no backlog before it entered userspace. Then backlog
processing will be delayed until exit from isolation. It won't be
queued, and flush_work() will not wait when no worker is assigned, so
there won't be a deadlock, however this delay may not be such a great
idea.

So it may be better to flush backlog before entering isolation, and in
flush_all_backlogs() instead of skipping all CPUs in isolated mode,
check if their per-CPU softnet_data->input_pkt_queue and softnet_data-
>process_queue are empty, and if they are not, call backlog anyway.
Then, if for whatever reason backlog will appear after flushing (we
can't guarantee that nothing preempted us then), it will cause one
isolation breaking event, and if nothing will be queued before re-
entering isolation, there will be no backlog until exiting isolation.

> 
> > Currently flush_all_backlogs()
> > enqueues corresponding work on all CPUs including ones that run
> > isolated tasks. It leads to breaking task isolation for nothing.
> > 
> > In this patch, backlog flushing is enqueued only on non-isolated
> > CPUs.
> > 
> > Signed-off-by: Yuri Norov 
> > [abel...@marvell.com: use safe task_isolation_on_cpu()
> > implementation]
> > Signed-off-by: Alex Belits 
> > ---
> >  net/core/dev.c | 7 ++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> > 
> > diff --git a/net/core/dev.c b/net/core/dev.c
> > index 90b59fc50dc9..83a282f7453d 100644
> > --- a/net/core/dev.c
> > +++ b/net/core/dev.c
> > @@ -74,6 +74,7 @@
> >  #include 
> >  #include 
> >  #include 
> > +#include 
> >  #include 
> >  #include 
> >  #include 
> > @@ -5624,9 +5625,13 @@ static void flush_all_backlogs(void)
> >  
> > get_online_cpus();
> >  
> > -   for_each_online_cpu(cpu)
> > +   smp_rmb();
> 
> What is it ordering?

Same as with other calls to task_isolation_on_cpu(cpu), it orders
access to ll_isol_flags.

> > +   for_each_online_cpu(cpu) {
> > +   if (task_isolation_on_cpu(cpu))
> > +   continue;
> > queue_work_on(cpu, system_highpri_wq,
> >   per_cpu_ptr(&flush_works, cpu));
> > +   }
> >  
> > for_each_online_cpu(cpu)
> > flush_work(per_cpu_ptr(&flush_works, cpu));
> 
> Thanks.

Re: [PATCH v4 11/13] task_isolation: net: don't flush backlog on CPUs running isolated tasks

2020-10-01 Thread Frederic Weisbecker 
On Wed, Jul 22, 2020 at 02:58:24PM +, Alex Belits wrote:
> From: Yuri Norov 
> 
> If CPU runs isolated task, there's no any backlog on it, and
> so we don't need to flush it.

What guarantees that we have no backlog on it?

> Currently flush_all_backlogs()
> enqueues corresponding work on all CPUs including ones that run
> isolated tasks. It leads to breaking task isolation for nothing.
> 
> In this patch, backlog flushing is enqueued only on non-isolated CPUs.
> 
> Signed-off-by: Yuri Norov 
> [abel...@marvell.com: use safe task_isolation_on_cpu() implementation]
> Signed-off-by: Alex Belits 
> ---
>  net/core/dev.c | 7 ++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/net/core/dev.c b/net/core/dev.c
> index 90b59fc50dc9..83a282f7453d 100644
> --- a/net/core/dev.c
> +++ b/net/core/dev.c
> @@ -74,6 +74,7 @@
>  #include 
>  #include 
>  #include 
> +#include 
>  #include 
>  #include 
>  #include 
> @@ -5624,9 +5625,13 @@ static void flush_all_backlogs(void)
>  
>   get_online_cpus();
>  
> - for_each_online_cpu(cpu)
> + smp_rmb();

What is it ordering?

> + for_each_online_cpu(cpu) {
> + if (task_isolation_on_cpu(cpu))
> + continue;
>   queue_work_on(cpu, system_highpri_wq,
> per_cpu_ptr(&flush_works, cpu));
> + }
>  
>   for_each_online_cpu(cpu)
>   flush_work(per_cpu_ptr(&flush_works, cpu));

Thanks.

[PATCH v4 11/13] task_isolation: net: don't flush backlog on CPUs running isolated tasks

2020-07-22 Thread Alex Belits 
From: Yuri Norov 

If CPU runs isolated task, there's no any backlog on it, and
so we don't need to flush it. Currently flush_all_backlogs()
enqueues corresponding work on all CPUs including ones that run
isolated tasks. It leads to breaking task isolation for nothing.

In this patch, backlog flushing is enqueued only on non-isolated CPUs.

Signed-off-by: Yuri Norov 
[abel...@marvell.com: use safe task_isolation_on_cpu() implementation]
Signed-off-by: Alex Belits 
---
 net/core/dev.c | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index 90b59fc50dc9..83a282f7453d 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -74,6 +74,7 @@
 #include 
 #include 
 #include 
+#include 
 #include 
 #include 
 #include 
@@ -5624,9 +5625,13 @@ static void flush_all_backlogs(void)
 
get_online_cpus();
 
-   for_each_online_cpu(cpu)
+   smp_rmb();
+   for_each_online_cpu(cpu) {
+   if (task_isolation_on_cpu(cpu))
+   continue;
queue_work_on(cpu, system_highpri_wq,
  per_cpu_ptr(&flush_works, cpu));
+   }
 
for_each_online_cpu(cpu)
flush_work(per_cpu_ptr(&flush_works, cpu));
-- 
2.26.2