On Fri, Mar 23, 2007 at 01:43:53PM +0000, David Howells wrote:
>
> [Resend - this time with a comma in the addresses, not a dot]
>
> Lennert Buytenhek <[EMAIL PROTECTED]> wrote:
>
> > [ background: On ARM, SMP synchronisation does need barriers but device
> > synchronisation does not. The question is that given this, whether
> > mb() and friends can be NOPs on ARM or not (i.e. whether mb() is
> > supposed to sync against other CPUs or not, or whether only smp_mb()
> > can be used for this.) ]
>
> Hmmmm...
>
> I see your problem. I think the right way to deal with this is to get rid of
> mb(), rmb(), wmb() and read_barrier_depends() and replace them with io_mb(),
> io_rmb(), ...
We will get combinatorial explosion if we aren't -extremely- careful:
1. Orders only normal memory accesses, which is all that is required
of smp_*().
2. Orders both normal and device accesses -- mmiowb().
3. Orders memory accesses and device accesses, but not necessarily
the union of the two -- mb(), rmb(), wmb().
4. Orders only device accesses, which is what seems to be looked
for here.
Thanx, Paul
> I think that there are only two places you should be using explicit memory
> barriers:
>
> (1) To control inter-CPU effects on an SMP system.
>
> (2) To control CPU vs device effects.
>
> > On Thu, Mar 22, 2007 at 04:17:44PM +0000, Catalin Marinas wrote:
> >
> > > Is the requirement for mb() to act correctly in the SMP case as well?
> >
> > That's what the docs seem to suggest. A couple of snippets from
> > memory-barriers.txt:
> >
> > [1] A write memory barrier gives a guarantee that all the STORE operations
> > specified before the barrier will appear to happen before all the STORE
> > operations specified after the barrier with respect to the other
> > components of the system.
> >
> > [2] A read barrier is a data dependency barrier plus a guarantee that all
> > the
> > LOAD operations specified before the barrier will appear to happen
> > before
> > all the LOAD operations specified after the barrier with respect to the
> > other components of the system.
> >
> > [3] TYPE MANDATORY SMP CONDITIONAL
> > =============== ======================= ===========================
> > GENERAL mb() smp_mb()
> > WRITE wmb() smp_wmb()
> > READ rmb() smp_rmb()
> > DATA DEPENDENCY read_barrier_depends() smp_read_barrier_depends()
> >
> > [4] Mandatory barriers should not be used to control SMP effects,
> > since mandatory barriers unnecessarily impose overhead on UP
> > systems.
> >
> > Note the wording of 'other components of the system' in [1] and [2] --
> > the way I read it, this includes devices as well as other CPUs.
>
> Yes, but I suppose which "other components" may depend on the class of barrier
> used.
>
> > [4] says that mandatory barriers (i.e. from [3]: mb(), wmb(), rmb(),
> > read_barrier_depends()) SHOULD not be used to control SMP effects, but
> > it does not say that they MUST not.
>
> As it stands, mb() is a superset of smp_mb(), and rmb() of smp_rmb(), etc.,
> so, yes, currently, mb() implies smp_mb(). However, mb() shouldn't be used if
> smb_mb() is sufficient as that may impact performance on a UP system.
>
> Really, mb() should only be used with respect to I/O.
>
> > > The memory-barriers.txt doc says that smp_* must be used for the SMP
> > > case.
> >
> > The exact wording is:
> >
> > [!] Note that SMP memory barriers _must_ be used to control the
> > ordering of references to shared memory on SMP systems, though
> > the use of locking instead is sufficient.
> >
> > This can IMHO be interpreted in two ways:
> > 1. If you want to control ordering of references to shared memory on
> > SMP systems, you must use SMP memory barriers and not any other kind
> > of memory barrier.
>
> If the shared memory is purely an inter-CPU effect, yes. If the shared memory
> is actually a device with side effects, then I/O safe memory barriers are
> required - mb() and co. Note that there must _also_ be safety wrt to other
> CPUs in the system, as other CPUs may also try to access the device.
>
> > 2. If you want to control ordering of references to shared memory on
> > SMP systems, you must use memory barriers, and the SMP memory barrier
> > is the most appropriate barrier type to use.
>
> You may use locking instead to control inter-CPU effects. Locks imply one-way
> permeable SMP-class memory barriers.
>
> > I'm thinking that [2] is what was intended. [1] doesn't seem consistent
> > with the rest of the document, but if [1] _is_ what is what was intended,
> > we're off the hook and mb() and friends can be NOPs on ARM. (But it'd
> > probably still need a thorough audit... :-/ )
>
> I think the best way to do an audit would be to make mb() and co. deprecated,
> pending obsolete, and to replace them with io_mb() and co. That way people
> would have to eyeball any usages of mb() and co.
>
> > > This means that if code uses mb() to control SMP sharing, it is broken.
> >
> > I'm not so sure.
>
> If it's _purely_ to control inter-CPU SMP sharing, then yes, it's broken. It
> must use either a lock or an smp_*mb() barrier.
>
> Of course, Linus may disagree...
>
> David
>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
