On Fri, Dec 14, 2018 at 04:39:34PM -0500, Alan Stern wrote:
> On Fri, 14 Dec 2018, Paul E. McKenney wrote:
>
> > I would say that sys_membarrier() has zero-sized read-side critical
> > sections, either comprising a single instruction (as is the case for
> > synchronize_sched(), actually), preempt-disable regions of code
> > (which are irrelevant to userspace execution), or the spaces between
> > consecutive pairs of instructions (as is the case for the newer
> > IPI-based implementation).
> >
> > The model picks the single-instruction option, and I haven't yet found
> > a problem with this -- which is no surprise given that, as you say,
> > an actual implementation makes this same choice.
>
> I believe that for RCU tests the LKMM gives the same results for
> length-zero critical sections interspersed between all the instructions
> and length-one critical sections surrounding all instructions (except
> synchronize_rcu). But the proof is tricky and I haven't checked it
> carefully.
That assertion is completely consistent with my implementation experience,
give or take the usual caveats about idle and offline execution.
> > > > The other thing that took some time to get used to is the possibility
> > > > of long delays during sys_membarrier() execution, allowing significant
> > > > execution and reordering between different CPUs' IPIs. This was key
> > > > to my understanding of the six-process example, and probably needs to
> > > > be clearly called out, including in an example or two.
> > >
> > > In all the examples I'm aware of, no more than one of the IPIs
> > > generated by each sys_membarrier call really matters. (Of course,
> > > there's no way to know in advance which one it will be, so you have to
> > > send an IPI to every CPU.) The execution delays and reordering
> > > between different CPUs' IPIs don't appear to be significant.
> >
> > Well, there are litmus tests that are allowed in which the allowed
> > execution is more easily explained in terms of delays between different
> > CPUs' IPIs, so it seems worth keeping track of.
> >
> > There might be a litmus test that can tell the difference between
> > simultaneous and non-simultaneous IPIs, but I cannot immediately think of
> > one that matters. Might be a failure of imagination on my part, though.
>
> P0 P1 P2
> Wc=1 [mb01] Rb=1
> memb Wa=1 Rc=0
> Ra=0 Wb=1 [mb02]
>
> The IPIs have to appear in the positions shown, which means they cannot
> be simultaneous. The test is allowed because P2's reads can be
> reordered.
OK, so "simultaneous" IPIs could be emulated in a real implementation by
having sys_membarrier() send each IPI (but not wait for a response), then
execute a full memory barrier and set a shared variable. Each IPI handler
would spin waiting for the shared variable to be set, then execute a full
memory barrier and atomically increment yet another shared variable and
return from interrupt. When that other shared variable's value reached
the number of IPIs sent, the sys_membarrier() would execute its final
(already existing) full memory barrier and return. Horribly expensive
and definitely not recommended, but eminently doable.
The difference between current sys_membarrier() and the "simultaneous"
variant described above is similar to the difference between
non-multicopy-atomic and multicopy-atomic memory ordering. So, after
thinking it through, my guess is that pretty much any litmus test that
can discern between multicopy-atomic and non-multicopy-atomic should
be transformable into something that can distinguish between the current
and the "simultaneous" sys_membarrier() implementation.
Seem reasonable?
Or alternatively, may I please apply your Signed-off-by to your earlier
sys_membarrier() patch so that I can queue it? I will probably also
change smp_memb() to membarrier() or some such. Again, within the
Linux kernel, membarrier() can be emulated with smp_call_function()
invoking a handler that does smp_mb().
Thanx, Paul
