Module Name: src
Committed By: riastradh
Date: Sat Apr 9 23:32:53 UTC 2022
Modified Files:
src/common/lib/libc/arch/aarch64/atomic: membar_ops.S
src/common/lib/libc/arch/alpha/atomic: membar_ops.S
src/common/lib/libc/arch/arm/atomic: membar_ops.S
src/common/lib/libc/arch/hppa/atomic: membar_ops.S
src/common/lib/libc/arch/i386/atomic: atomic.S
src/common/lib/libc/arch/ia64/atomic: atomic.S
src/common/lib/libc/arch/mips/atomic: membar_ops.S
src/common/lib/libc/arch/or1k/atomic: membar_ops.S
src/common/lib/libc/arch/powerpc/atomic: membar_ops.S
src/common/lib/libc/arch/riscv/atomic: membar_ops.S
src/common/lib/libc/arch/sparc/atomic: membar_ops.S
src/common/lib/libc/arch/sparc64/atomic: membar_ops.S
src/common/lib/libc/arch/x86_64/atomic: atomic.S
src/common/lib/libc/atomic: membar_ops_nop.c
src/distrib/sets/lists/comp: mi
src/lib/libc/atomic: Makefile.inc membar_ops.3
src/share/man/man9: atomic_loadstore.9
src/sys/sys: atomic.h
src/tests/lib/libc/membar: t_dekker.c t_spinlock.c
Log Message:
Introduce membar_acquire/release. Deprecate membar_enter/exit.
The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.
The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.
Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that
(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.
I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).
Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:
https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html
Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.
With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.
While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:
membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.
The `no stronger than' relation is (the transitive closure of):
- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)
And, for the deprecated membars:
- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)
(membar_exit is identical to membar_release, but the name is
deprecated.)
Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.
Also add membar(3) and atomic(3) man page links.
To generate a diff of this commit:
cvs rdiff -u -r1.3 -r1.4 src/common/lib/libc/arch/aarch64/atomic/membar_ops.S
cvs rdiff -u -r1.8 -r1.9 src/common/lib/libc/arch/alpha/atomic/membar_ops.S
cvs rdiff -u -r1.9 -r1.10 src/common/lib/libc/arch/arm/atomic/membar_ops.S
cvs rdiff -u -r1.2 -r1.3 src/common/lib/libc/arch/hppa/atomic/membar_ops.S
cvs rdiff -u -r1.34 -r1.35 src/common/lib/libc/arch/i386/atomic/atomic.S
cvs rdiff -u -r1.6 -r1.7 src/common/lib/libc/arch/ia64/atomic/atomic.S
cvs rdiff -u -r1.11 -r1.12 src/common/lib/libc/arch/mips/atomic/membar_ops.S
cvs rdiff -u -r1.1 -r1.2 src/common/lib/libc/arch/or1k/atomic/membar_ops.S
cvs rdiff -u -r1.5 -r1.6 src/common/lib/libc/arch/powerpc/atomic/membar_ops.S
cvs rdiff -u -r1.3 -r1.4 src/common/lib/libc/arch/riscv/atomic/membar_ops.S
cvs rdiff -u -r1.7 -r1.8 src/common/lib/libc/arch/sparc/atomic/membar_ops.S
cvs rdiff -u -r1.8 -r1.9 src/common/lib/libc/arch/sparc64/atomic/membar_ops.S
cvs rdiff -u -r1.27 -r1.28 src/common/lib/libc/arch/x86_64/atomic/atomic.S
cvs rdiff -u -r1.7 -r1.8 src/common/lib/libc/atomic/membar_ops_nop.c
cvs rdiff -u -r1.2412 -r1.2413 src/distrib/sets/lists/comp/mi
cvs rdiff -u -r1.4 -r1.5 src/lib/libc/atomic/Makefile.inc
cvs rdiff -u -r1.9 -r1.10 src/lib/libc/atomic/membar_ops.3
cvs rdiff -u -r1.6 -r1.7 src/share/man/man9/atomic_loadstore.9
cvs rdiff -u -r1.22 -r1.23 src/sys/sys/atomic.h
cvs rdiff -u -r1.1 -r1.2 src/tests/lib/libc/membar/t_dekker.c \
src/tests/lib/libc/membar/t_spinlock.c
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
Modified files:
Index: src/common/lib/libc/arch/aarch64/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/aarch64/atomic/membar_ops.S:1.3 src/common/lib/libc/arch/aarch64/atomic/membar_ops.S:1.4
--- src/common/lib/libc/arch/aarch64/atomic/membar_ops.S:1.3 Sat Apr 9 12:07:37 2022
+++ src/common/lib/libc/arch/aarch64/atomic/membar_ops.S Sat Apr 9 23:32:51 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.3 2022/04/09 12:07:37 riastradh Exp $ */
+/* $NetBSD: membar_ops.S,v 1.4 2022/04/09 23:32:51 riastradh Exp $ */
/*-
* Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -32,19 +32,21 @@
#include "atomic_op_asm.h"
ENTRY_NP(_membar_producer)
- dmb ishst
+ dmb ishst /* store-before-store */
ret
END(_membar_producer)
ATOMIC_OP_ALIAS(membar_producer,_membar_producer)
ATOMIC_OP_ALIAS(membar_write,_membar_producer)
STRONG_ALIAS(_membar_write,_membar_producer)
-ENTRY_NP(_membar_consumer)
- dmb ishld
+ENTRY_NP(_membar_acquire)
+ dmb ishld /* load-before-load/store */
ret
-END(_membar_consumer)
-ATOMIC_OP_ALIAS(membar_consumer,_membar_consumer)
+END(_membar_acquire)
+ATOMIC_OP_ALIAS(membar_acquire,_membar_acquire)
+ATOMIC_OP_ALIAS(membar_consumer,_membar_acquire)
ATOMIC_OP_ALIAS(membar_read,_membar_consumer)
+STRONG_ALIAS(_membar_consumer,_membar_acquire)
STRONG_ALIAS(_membar_read,_membar_consumer)
ENTRY_NP(_membar_sync)
@@ -52,8 +54,10 @@ ENTRY_NP(_membar_sync)
ret
END(_membar_sync)
ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
+ATOMIC_OP_ALIAS(membar_release,_membar_sync)
ATOMIC_OP_ALIAS(membar_enter,_membar_sync)
ATOMIC_OP_ALIAS(membar_exit,_membar_sync)
STRONG_ALIAS(__sync_synchronize,_membar_sync)
+STRONG_ALIAS(_membar_release,_membar_sync)
STRONG_ALIAS(_membar_enter,_membar_sync)
STRONG_ALIAS(_membar_exit,_membar_sync)
Index: src/common/lib/libc/arch/alpha/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/alpha/atomic/membar_ops.S:1.8 src/common/lib/libc/arch/alpha/atomic/membar_ops.S:1.9
--- src/common/lib/libc/arch/alpha/atomic/membar_ops.S:1.8 Wed Apr 6 22:47:55 2022
+++ src/common/lib/libc/arch/alpha/atomic/membar_ops.S Sat Apr 9 23:32:51 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.8 2022/04/06 22:47:55 riastradh Exp $ */
+/* $NetBSD: membar_ops.S,v 1.9 2022/04/09 23:32:51 riastradh Exp $ */
/*-
* Copyright (c) 2006, 2007 The NetBSD Foundation, Inc.
@@ -42,45 +42,50 @@
LEAF(_membar_producer, 0)
RET
nop
- END(_membar_producer)
+END(_membar_producer)
EXPORT(_membar_producer_end)
LEAF(_membar_sync, 0)
RET
nop
- END(_membar_sync)
+END(_membar_sync)
EXPORT(_membar_sync_end)
LEAF(_membar_producer_mp, 0)
- wmb
+ wmb /* store-before-store */
RET
- END(_membar_producer_mp)
+END(_membar_producer_mp)
EXPORT(_membar_producer_mp_end)
LEAF(_membar_sync_mp, 0)
- mb
+ mb /* load/store-before-load/store */
RET
- END(_membar_sync_mp)
+END(_membar_sync_mp)
EXPORT(_membar_sync_mp_end)
#else /* _KERNEL */
LEAF(_membar_producer, 0)
- mb
+ mb /* load/store-before-load/store */
RET
- END(_membar_producer)
+END(_membar_producer)
EXPORT(_membar_producer_end)
LEAF(_membar_sync, 0)
- mb
+ mb /* load/store-before-load/store */
RET
- END(_membar_sync)
+END(_membar_sync)
EXPORT(_membar_sync_end)
#endif /* _KERNEL */
ATOMIC_OP_ALIAS(membar_producer,_membar_producer)
ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
+
+ATOMIC_OP_ALIAS(membar_acquire,_membar_sync)
+STRONG_ALIAS(_membar_acquire,_membar_sync)
+ATOMIC_OP_ALIAS(membar_release,_membar_sync)
+STRONG_ALIAS(_membar_release,_membar_sync)
ATOMIC_OP_ALIAS(membar_enter,_membar_sync)
STRONG_ALIAS(_membar_enter,_membar_sync)
ATOMIC_OP_ALIAS(membar_exit,_membar_sync)
Index: src/common/lib/libc/arch/arm/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/arm/atomic/membar_ops.S:1.9 src/common/lib/libc/arch/arm/atomic/membar_ops.S:1.10
--- src/common/lib/libc/arch/arm/atomic/membar_ops.S:1.9 Wed Jul 28 07:32:20 2021
+++ src/common/lib/libc/arch/arm/atomic/membar_ops.S Sat Apr 9 23:32:51 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.9 2021/07/28 07:32:20 skrll Exp $ */
+/* $NetBSD: membar_ops.S,v 1.10 2022/04/09 23:32:51 riastradh Exp $ */
/*-
* Copyright (c) 2008 The NetBSD Foundation, Inc.
* All rights reserved.
@@ -33,7 +33,7 @@
#if defined(_ARM_ARCH_6)
ENTRY_NP(_membar_producer)
- DMBST
+ DMBST /* store-before-store */
RET
END(_membar_producer)
ATOMIC_OP_ALIAS(membar_producer,_membar_producer)
@@ -41,15 +41,19 @@ ATOMIC_OP_ALIAS(membar_write,_membar_pro
STRONG_ALIAS(_membar_write,_membar_producer)
ENTRY_NP(_membar_sync)
- DMB
+ DMB /* load/store-before-load/store */
RET
END(_membar_sync)
ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
+ATOMIC_OP_ALIAS(membar_acquire,_membar_sync)
+ATOMIC_OP_ALIAS(membar_release,_membar_sync)
ATOMIC_OP_ALIAS(membar_enter,_membar_sync)
ATOMIC_OP_ALIAS(membar_exit,_membar_sync)
ATOMIC_OP_ALIAS(membar_consumer,_membar_sync)
ATOMIC_OP_ALIAS(membar_read,_membar_sync)
STRONG_ALIAS(__sync_synchronize,_membar_sync)
+STRONG_ALIAS(_membar_acquire,_membar_sync)
+STRONG_ALIAS(_membar_release,_membar_sync)
STRONG_ALIAS(_membar_enter,_membar_sync)
STRONG_ALIAS(_membar_exit,_membar_sync)
STRONG_ALIAS(_membar_consumer,_membar_sync)
Index: src/common/lib/libc/arch/hppa/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/hppa/atomic/membar_ops.S:1.2 src/common/lib/libc/arch/hppa/atomic/membar_ops.S:1.3
--- src/common/lib/libc/arch/hppa/atomic/membar_ops.S:1.2 Wed Apr 6 22:47:56 2022
+++ src/common/lib/libc/arch/hppa/atomic/membar_ops.S Sat Apr 9 23:32:51 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.2 2022/04/06 22:47:56 riastradh Exp $ */
+/* $NetBSD: membar_ops.S,v 1.3 2022/04/09 23:32:51 riastradh Exp $ */
/*-
* Copyright (c) 2007 The NetBSD Foundation, Inc.
@@ -31,11 +31,11 @@
#include "atomic_op_asm.h"
-RCSID("$NetBSD: membar_ops.S,v 1.2 2022/04/06 22:47:56 riastradh Exp $")
+RCSID("$NetBSD: membar_ops.S,v 1.3 2022/04/09 23:32:51 riastradh Exp $")
.text
-LEAF_ENTRY(_membar_consumer)
+LEAF_ENTRY(_membar_sync)
sync
nop
nop
@@ -44,24 +44,18 @@ LEAF_ENTRY(_membar_consumer)
nop
bv %r0(%rp)
nop
-EXIT(_membar_consumer)
+EXIT(_membar_sync)
+ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
-LEAF_ENTRY(_membar_producer)
- sync
- nop
- nop
- nop
- nop
- nop
- bv %r0(%rp)
- nop
-EXIT(_membar_producer)
-
-ATOMIC_OP_ALIAS(membar_producer,_membar_producer)
-ATOMIC_OP_ALIAS(membar_consumer,_membar_consumer)
-ATOMIC_OP_ALIAS(membar_enter,_membar_consumer)
-STRONG_ALIAS(_membar_enter,_membar_consumer)
-ATOMIC_OP_ALIAS(membar_exit,_membar_producer)
-STRONG_ALIAS(_membar_exit,_membar_producer)
-ATOMIC_OP_ALIAS(membar_sync,_membar_producer)
-STRONG_ALIAS(_membar_sync,_membar_producer)
+ATOMIC_OP_ALIAS(membar_producer,_membar_sync)
+STRONG_ALIAS(_membar_producer,_membar_sync)
+ATOMIC_OP_ALIAS(membar_consumer,_membar_sync)
+STRONG_ALIAS(_membar_consumer,_membar_sync)
+ATOMIC_OP_ALIAS(membar_acquire,_membar_sync)
+STRONG_ALIAS(_membar_acquire,_membar_sync)
+ATOMIC_OP_ALIAS(membar_release,_membar_sync)
+STRONG_ALIAS(_membar_release,_membar_sync)
+ATOMIC_OP_ALIAS(membar_enter,_membar_sync)
+STRONG_ALIAS(_membar_enter,_membar_sync)
+ATOMIC_OP_ALIAS(membar_exit,_membar_sync)
+STRONG_ALIAS(_membar_exit,_membar_sync)
Index: src/common/lib/libc/arch/i386/atomic/atomic.S
diff -u src/common/lib/libc/arch/i386/atomic/atomic.S:1.34 src/common/lib/libc/arch/i386/atomic/atomic.S:1.35
--- src/common/lib/libc/arch/i386/atomic/atomic.S:1.34 Sat Apr 9 22:53:36 2022
+++ src/common/lib/libc/arch/i386/atomic/atomic.S Sat Apr 9 23:32:51 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic.S,v 1.34 2022/04/09 22:53:36 riastradh Exp $ */
+/* $NetBSD: atomic.S,v 1.35 2022/04/09 23:32:51 riastradh Exp $ */
/*-
* Copyright (c) 2007 The NetBSD Foundation, Inc.
@@ -178,23 +178,23 @@ ENTRY(_atomic_cas_32_ni)
ret
END(_atomic_cas_32_ni)
-ENTRY(_membar_consumer)
+ENTRY(_membar_acquire)
/*
* Every load from normal memory is a load-acquire on x86, so
* there is never any need for explicit barriers to order
* load-before-anything.
*/
ret
-END(_membar_consumer)
+END(_membar_acquire)
-ENTRY(_membar_producer)
+ENTRY(_membar_release)
/*
* Every store to normal memory is a store-release on x86, so
* there is never any need for explicit barriers to order
* anything-before-store.
*/
ret
-END(_membar_producer)
+END(_membar_release)
ENTRY(_membar_sync)
/*
@@ -340,10 +340,14 @@ ALIAS(atomic_cas_64_ni,_atomic_cas_64)
ALIAS(__sync_val_compare_and_swap_8,_atomic_cas_64)
#endif /* __HAVE_ATOMIC64_OPS || _KERNEL */
-ALIAS(membar_consumer,_membar_consumer)
-ALIAS(membar_producer,_membar_producer)
+ALIAS(membar_acquire,_membar_acquire)
+ALIAS(membar_release,_membar_release)
+ALIAS(membar_sync,_membar_sync)
+
+ALIAS(membar_consumer,_membar_acquire)
+ALIAS(membar_producer,_membar_release)
ALIAS(membar_enter,_membar_sync)
-ALIAS(membar_exit,_membar_producer)
+ALIAS(membar_exit,_membar_release)
ALIAS(membar_sync,_membar_sync)
STRONG_ALIAS(_atomic_add_int,_atomic_add_32)
@@ -398,8 +402,10 @@ STRONG_ALIAS(_atomic_cas_uint_ni,_atomic
STRONG_ALIAS(_atomic_cas_ulong_ni,_atomic_cas_32_ni)
STRONG_ALIAS(_atomic_cas_ptr_ni,_atomic_cas_32_ni)
+STRONG_ALIAS(_membar_consumer,_membar_acquire)
+STRONG_ALIAS(_membar_producer,_membar_release)
STRONG_ALIAS(_membar_enter,_membar_sync)
-STRONG_ALIAS(_membar_exit,_membar_producer)
+STRONG_ALIAS(_membar_exit,_membar_release)
#ifdef _HARDKERNEL
.section .rodata
Index: src/common/lib/libc/arch/ia64/atomic/atomic.S
diff -u src/common/lib/libc/arch/ia64/atomic/atomic.S:1.6 src/common/lib/libc/arch/ia64/atomic/atomic.S:1.7
--- src/common/lib/libc/arch/ia64/atomic/atomic.S:1.6 Wed Apr 6 22:47:56 2022
+++ src/common/lib/libc/arch/ia64/atomic/atomic.S Sat Apr 9 23:32:51 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic.S,v 1.6 2022/04/06 22:47:56 riastradh Exp $ */
+/* $NetBSD: atomic.S,v 1.7 2022/04/09 23:32:51 riastradh Exp $ */
/*-
* Copyright (c) 2008 The NetBSD Foundation, Inc.
@@ -117,6 +117,16 @@ ENTRY(_membar_producer,0)
br.ret.sptk rp
END(_membar_producer)
+ENTRY(_membar_acquire,0)
+ mf
+ br.ret.sptk rp
+END(_membar_acquire)
+
+ENTRY(_membar_release,0)
+ mf
+ br.ret.sptk rp
+END(_membar_release)
+
ENTRY(_membar_enter,0)
mf
br.ret.sptk rp
@@ -213,6 +223,8 @@ ALIAS(atomic_cas_ptr_ni,_atomic_cas_64)
ALIAS(membar_consumer,_membar_consumer)
ALIAS(membar_producer,_membar_producer)
+ALIAS(membar_acquire,_membar_acquire)
+ALIAS(membar_release,_membar_release)
ALIAS(membar_enter,_membar_enter)
ALIAS(membar_exit,_membar_exit)
ALIAS(membar_sync,_membar_sync)
Index: src/common/lib/libc/arch/mips/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/mips/atomic/membar_ops.S:1.11 src/common/lib/libc/arch/mips/atomic/membar_ops.S:1.12
--- src/common/lib/libc/arch/mips/atomic/membar_ops.S:1.11 Sat Feb 12 17:10:02 2022
+++ src/common/lib/libc/arch/mips/atomic/membar_ops.S Sat Apr 9 23:32:51 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.11 2022/02/12 17:10:02 riastradh Exp $ */
+/* $NetBSD: membar_ops.S,v 1.12 2022/04/09 23:32:51 riastradh Exp $ */
/*-
* Copyright (c) 2006, 2007 The NetBSD Foundation, Inc.
@@ -40,22 +40,40 @@ LEAF(_membar_sync)
END(_membar_sync)
#ifdef __OCTEON__
-LEAF(_membar_producer)
+LEAF(_membar_release)
+ /*
+ * syncw is documented as ordering store-before-store in
+ *
+ * Cavium OCTEON III CN78XX Hardware Reference Manual,
+ * CN78XX-HM-0.99E, September 2014.
+ *
+ * It's unclear from the documentation the architecture
+ * guarantees load-before-store ordering without barriers, but
+ * this code assumes it does. If that assumption is wrong, we
+ * can only use syncw for membar_producer -- membar_release has
+ * to use the full sync.
+ */
j ra
syncw
-END(_membar_producer)
+END(_membar_release)
#endif
ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
+ATOMIC_OP_ALIAS(membar_acquire,_membar_sync)
+STRONG_ALIAS(_membar_acquire,_membar_sync)
ATOMIC_OP_ALIAS(membar_enter,_membar_sync)
STRONG_ALIAS(_membar_enter,_membar_sync)
#ifdef __OCTEON__
-ATOMIC_OP_ALIAS(membar_exit,_membar_producer)
-STRONG_ALIAS(_membar_exit,_membar_producer)
-STRONG_ALIAS(membar_producer,_membar_producer)
+ATOMIC_OP_ALIAS(membar_exit,_membar_release)
+STRONG_ALIAS(_membar_exit,_membar_release)
+ATOMIC_OP_ALIAS(membar_release,_membar_release)
+ATOMIC_OP_ALIAS(membar_producer,_membar_release)
+STRONG_ALIAS(_membar_producer,_membar_release)
#else
ATOMIC_OP_ALIAS(membar_exit,_membar_sync)
STRONG_ALIAS(_membar_exit,_membar_sync)
+ATOMIC_OP_ALIAS(membar_release,_membar_sync)
+STRONG_ALIAS(_membar_release,_membar_sync)
ATOMIC_OP_ALIAS(membar_producer,_membar_sync)
STRONG_ALIAS(_membar_producer,_membar_sync)
#endif
Index: src/common/lib/libc/arch/or1k/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/or1k/atomic/membar_ops.S:1.1 src/common/lib/libc/arch/or1k/atomic/membar_ops.S:1.2
--- src/common/lib/libc/arch/or1k/atomic/membar_ops.S:1.1 Wed Sep 3 19:34:25 2014
+++ src/common/lib/libc/arch/or1k/atomic/membar_ops.S Sat Apr 9 23:32:51 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.1 2014/09/03 19:34:25 matt Exp $ */
+/* $NetBSD: membar_ops.S,v 1.2 2022/04/09 23:32:51 riastradh Exp $ */
/*-
* Copyright (c) 2014 The NetBSD Foundation, Inc.
@@ -31,27 +31,26 @@
#include "atomic_op_asm.h"
-ENTRY_NP(_membar_producer)
- l.msync
- l.jr lr
- l.nop
-END(_membar_producer)
-ATOMIC_OP_ALIAS(membar_producer,_membar_producer)
-ATOMIC_OP_ALIAS(membar_write,_membar_producer)
-STRONG_ALIAS(_membar_write,_membar_producer)
-
ENTRY_NP(_membar_sync)
l.msync
l.jr lr
l.nop
END(_membar_sync)
ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
+ATOMIC_OP_ALIAS(membar_acquire,_membar_sync)
+ATOMIC_OP_ALIAS(membar_release,_membar_sync)
ATOMIC_OP_ALIAS(membar_enter,_membar_sync)
ATOMIC_OP_ALIAS(membar_exit,_membar_sync)
ATOMIC_OP_ALIAS(membar_consumer,_membar_sync)
ATOMIC_OP_ALIAS(membar_read,_membar_sync)
+ATOMIC_OP_ALIAS(membar_producer,_membar_sync)
+ATOMIC_OP_ALIAS(membar_write,_membar_sync)
CRT_ALIAS(__sync_synchronize,_membar_sync)
+STRONG_ALIAS(_membar_acquire,_membar_sync)
+STRONG_ALIAS(_membar_release,_membar_sync)
STRONG_ALIAS(_membar_enter,_membar_sync)
STRONG_ALIAS(_membar_exit,_membar_sync)
STRONG_ALIAS(_membar_consumer,_membar_sync)
STRONG_ALIAS(_membar_read,_membar_sync)
+STRONG_ALIAS(_membar_producer,_membar_sync)
+STRONG_ALIAS(_membar_write,_membar_sync)
Index: src/common/lib/libc/arch/powerpc/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/powerpc/atomic/membar_ops.S:1.5 src/common/lib/libc/arch/powerpc/atomic/membar_ops.S:1.6
--- src/common/lib/libc/arch/powerpc/atomic/membar_ops.S:1.5 Wed Apr 6 22:47:56 2022
+++ src/common/lib/libc/arch/powerpc/atomic/membar_ops.S Sat Apr 9 23:32:52 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.5 2022/04/06 22:47:56 riastradh Exp $ */
+/* $NetBSD: membar_ops.S,v 1.6 2022/04/09 23:32:52 riastradh Exp $ */
/*-
* Copyright (c) 2007 The NetBSD Foundation, Inc.
@@ -31,26 +31,111 @@
#include "atomic_op_asm.h"
-__RCSID("$NetBSD: membar_ops.S,v 1.5 2022/04/06 22:47:56 riastradh Exp $")
+__RCSID("$NetBSD: membar_ops.S,v 1.6 2022/04/09 23:32:52 riastradh Exp $")
.text
-/* These assume Total Store Order (TSO) */
-ENTRY(_membar_consumer)
- isync
+ENTRY(_membar_acquire)
+ /*
+ * It is tempting to use isync to order load-before-load/store.
+ * However, isync orders prior loads only if their value flows
+ * into a control-flow dependency prior to the isync:
+ *
+ * `[I]f an isync follows a conditional Branch instruction
+ * that depends on the value returned by a preceding Load
+ * instruction, the load on which the Branch depends is
+ * performed before any loads caused by instructions
+ * following the isync. This applies even if the effects
+ * of the ``dependency'' are independent of the value
+ * loaded (e.g., the value is compared to itself and the
+ * Branch tests the EQ bit in the selected CR field), and
+ * even if the branch target is the sequentially next
+ * instruction.'
+ *
+ * --PowerPC Virtual Environment Architecture, Book II,
+ * Version 2.01, December 2003, 1.7.1 `Storage Access
+ * Ordering', p. 7.
+ *
+ * We are required here, however, to order _all_ prior loads,
+ * even if they do not flow into any control flow dependency.
+ * For example:
+ *
+ * x = *p;
+ * membar_acquire();
+ * if (x) goto foo;
+ *
+ * This can't be implemented by:
+ *
+ * lwz x, p
+ * isync
+ * cmpwi x, 0
+ * bne foo
+ *
+ * isync doesn't work here because there's no conditional
+ * dependency on x between the lwz x, p and the isync.
+ *
+ * isync would only work if it followed the branch:
+ *
+ * lwz x, p
+ * isync
+ * cmpwi x, 0
+ * bne foo
+ * ...
+ * foo: isync
+ * ...
+ *
+ * lwsync orders everything except store-before-load, so it
+ * serves here -- see below in membar_release in lwsync.
+ * Except we can't use it on booke, so use sync for now.
+ */
+ sync
+ blr
+END(_membar_acquire)
+ATOMIC_OP_ALIAS(membar_acquire,_membar_acquire)
+
+ENTRY(_membar_release)
+ /*
+ * `The memory barrier provides an ordering function for
+ * the storage accesses caused by Load, Store, and dcbz
+ * instructions that are executed by the processor
+ * executing the [lwsync] instruction and for which the
+ * specified storage location is in storage that is
+ * Memory Coherence Required and is neither Write Through
+ * Required nor Caching Inhibited. The applicable pairs
+ * are all pairs a_i, b_j of such accesses except those
+ * in which a_i is an access caused by a Store or dcbz
+ * instruction and b_j is an access caused by a Load
+ * instruction.'
+ *
+ * --PowerPC Virtual Environment Architecture, Book II,
+ * Version 2.01, December 2003, 3.3.3 `Memory Barrier
+ * Instructions', p. 25.
+ *
+ * In brief, lwsync is an acquire-release barrier -- it orders
+ * load-before-load/store and load/store-before-store, but not
+ * store-before-load. Except we can't use it on booke, so use
+ * sync for now.
+ */
+ sync
blr
-END(_membar_consumer)
+END(_membar_release)
+ATOMIC_OP_ALIAS(membar_release,_membar_release)
-ENTRY(_membar_producer)
+ENTRY(_membar_sync)
+ /*
+ * sync, or `heavyweight sync', is a full sequential
+ * consistency barrier.
+ */
sync
blr
-END(_membar_producer)
+END(_membar_sync)
+ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
-ATOMIC_OP_ALIAS(membar_producer,_membar_producer)
-ATOMIC_OP_ALIAS(membar_consumer,_membar_consumer)
-ATOMIC_OP_ALIAS(membar_enter,_membar_consumer)
-STRONG_ALIAS(_membar_enter,_membar_consumer)
-ATOMIC_OP_ALIAS(membar_exit,_membar_producer)
-STRONG_ALIAS(_membar_exit,_membar_producer)
-ATOMIC_OP_ALIAS(membar_sync,_membar_producer)
-STRONG_ALIAS(_membar_sync,_membar_producer)
+ATOMIC_OP_ALIAS(membar_producer,_membar_release)
+STRONG_ALIAS(_membar_producer,_membar_release)
+ATOMIC_OP_ALIAS(membar_consumer,_membar_acquire)
+STRONG_ALIAS(_membar_consumer,_membar_acquire)
+ATOMIC_OP_ALIAS(membar_enter,_membar_sync)
+STRONG_ALIAS(_membar_enter,_membar_sync)
+ATOMIC_OP_ALIAS(membar_exit,_membar_release)
+STRONG_ALIAS(_membar_exit,_membar_release)
Index: src/common/lib/libc/arch/riscv/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/riscv/atomic/membar_ops.S:1.3 src/common/lib/libc/arch/riscv/atomic/membar_ops.S:1.4
--- src/common/lib/libc/arch/riscv/atomic/membar_ops.S:1.3 Sat Apr 9 22:53:53 2022
+++ src/common/lib/libc/arch/riscv/atomic/membar_ops.S Sat Apr 9 23:32:52 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.3 2022/04/09 22:53:53 riastradh Exp $ */
+/* $NetBSD: membar_ops.S,v 1.4 2022/04/09 23:32:52 riastradh Exp $ */
/*-
* Copyright (c) 2006, 2007 The NetBSD Foundation, Inc.
@@ -38,6 +38,18 @@ END(_membar_sync)
ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
CRT_ALIAS(__sync_synchronize,_membar_sync)
+ENTRY_NP(_membar_acquire)
+ fence r,rw
+ ret
+END(_membar_acquire)
+ATOMIC_OP_ALIAS(membar_acquire,_membar_acquire)
+
+ENTRY_NP(_membar_release)
+ fence rw,w
+ ret
+END(_membar_release)
+ATOMIC_OP_ALIAS(membar_release,_membar_release)
+
ENTRY_NP(_membar_enter)
fence rw,rw
ret
Index: src/common/lib/libc/arch/sparc/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/sparc/atomic/membar_ops.S:1.7 src/common/lib/libc/arch/sparc/atomic/membar_ops.S:1.8
--- src/common/lib/libc/arch/sparc/atomic/membar_ops.S:1.7 Sat Apr 9 22:53:17 2022
+++ src/common/lib/libc/arch/sparc/atomic/membar_ops.S Sat Apr 9 23:32:52 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.7 2022/04/09 22:53:17 riastradh Exp $ */
+/* $NetBSD: membar_ops.S,v 1.8 2022/04/09 23:32:52 riastradh Exp $ */
/*-
* Copyright (c) 2007 The NetBSD Foundation, Inc.
@@ -45,14 +45,21 @@
* instruction that implies a sequential consistency barrier.
*
* If we ran with Partial Store Order (PSO), we would also need to
- * issue STBAR for membar_exit (load/store-before-store) and
+ * issue STBAR for membar_release (load/store-before-store) and
* membar_producer (store-before-store).
*/
-ENTRY(_membar_consumer)
+ENTRY(_membar_acquire)
retl
nop
-END(_membar_consumer)
+END(_membar_acquire)
+ATOMIC_OP_ALIAS(membar_acquire,_membar_acquire)
+
+ENTRY(_membar_release)
+ retl
+ nop
+END(_membar_release)
+ATOMIC_OP_ALIAS(membar_release,_membar_release)
ENTRY(_membar_sync)
retl
@@ -62,12 +69,13 @@ ENTRY(_membar_sync)
nop
#endif
END(_membar_sync)
+ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
-ATOMIC_OP_ALIAS(membar_producer,_membar_consumer)
-STRONG_ALIAS(_membar_producer,_membar_consumer)
-ATOMIC_OP_ALIAS(membar_consumer,_membar_consumer)
+ATOMIC_OP_ALIAS(membar_producer,_membar_release)
+STRONG_ALIAS(_membar_producer,_membar_release)
+ATOMIC_OP_ALIAS(membar_consumer,_membar_acquire)
+STRONG_ALIAS(_membar_consumer,_membar_acquire)
ATOMIC_OP_ALIAS(membar_enter,_membar_sync)
STRONG_ALIAS(_membar_enter,_membar_sync)
-ATOMIC_OP_ALIAS(membar_exit,_membar_consumer)
-STRONG_ALIAS(_membar_exit,_membar_consumer)
-ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
+ATOMIC_OP_ALIAS(membar_exit,_membar_release)
+STRONG_ALIAS(_membar_exit,_membar_release)
Index: src/common/lib/libc/arch/sparc64/atomic/membar_ops.S
diff -u src/common/lib/libc/arch/sparc64/atomic/membar_ops.S:1.8 src/common/lib/libc/arch/sparc64/atomic/membar_ops.S:1.9
--- src/common/lib/libc/arch/sparc64/atomic/membar_ops.S:1.8 Sat Apr 9 22:53:25 2022
+++ src/common/lib/libc/arch/sparc64/atomic/membar_ops.S Sat Apr 9 23:32:52 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops.S,v 1.8 2022/04/09 22:53:25 riastradh Exp $ */
+/* $NetBSD: membar_ops.S,v 1.9 2022/04/09 23:32:52 riastradh Exp $ */
/*-
* Copyright (c) 2007 The NetBSD Foundation, Inc.
@@ -44,10 +44,17 @@
* and membar_producer (store-before-store).
*/
-ENTRY(_membar_consumer)
+ENTRY(_membar_acquire)
retl
nop
-END(_membar_consumer)
+END(_membar_acquire)
+ATOMIC_OP_ALIAS(membar_acquire,_membar_acquire)
+
+ENTRY(_membar_release)
+ retl
+ nop
+END(_membar_release)
+ATOMIC_OP_ALIAS(membar_release,_membar_release)
ENTRY(_membar_sync)
/*
@@ -70,12 +77,13 @@ ENTRY(_membar_sync)
retl
nop
END(_membar_sync)
+ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
-ATOMIC_OP_ALIAS(membar_producer,_membar_consumer)
-STRONG_ALIAS(_membar_producer,_membar_consumer)
-ATOMIC_OP_ALIAS(membar_consumer,_membar_consumer)
+ATOMIC_OP_ALIAS(membar_producer,_membar_release)
+STRONG_ALIAS(_membar_producer,_membar_release)
+ATOMIC_OP_ALIAS(membar_consumer,_membar_acquire)
+STRONG_ALIAS(_membar_consumer,_membar_acquire)
ATOMIC_OP_ALIAS(membar_enter,_membar_sync)
STRONG_ALIAS(_membar_enter,_membar_sync)
-ATOMIC_OP_ALIAS(membar_exit,_membar_consumer)
-STRONG_ALIAS(_membar_exit,_membar_consumer)
-ATOMIC_OP_ALIAS(membar_sync,_membar_sync)
+ATOMIC_OP_ALIAS(membar_exit,_membar_release)
+STRONG_ALIAS(_membar_exit,_membar_release)
Index: src/common/lib/libc/arch/x86_64/atomic/atomic.S
diff -u src/common/lib/libc/arch/x86_64/atomic/atomic.S:1.27 src/common/lib/libc/arch/x86_64/atomic/atomic.S:1.28
--- src/common/lib/libc/arch/x86_64/atomic/atomic.S:1.27 Sat Apr 9 22:53:45 2022
+++ src/common/lib/libc/arch/x86_64/atomic/atomic.S Sat Apr 9 23:32:52 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic.S,v 1.27 2022/04/09 22:53:45 riastradh Exp $ */
+/* $NetBSD: atomic.S,v 1.28 2022/04/09 23:32:52 riastradh Exp $ */
/*-
* Copyright (c) 2007 The NetBSD Foundation, Inc.
@@ -253,23 +253,23 @@ END(_atomic_cas_64_ni)
/* memory barriers */
-ENTRY(_membar_consumer)
+ENTRY(_membar_acquire)
/*
* Every load from normal memory is a load-acquire on x86, so
* there is never any need for explicit barriers to order
* load-before-anything.
*/
ret
-END(_membar_consumer)
+END(_membar_acquire)
-ENTRY(_membar_producer)
+ENTRY(_membar_release)
/*
* Every store to normal memory is a store-release on x86, so
* there is never any need for explicit barriers to order
* anything-before-store.
*/
ret
-END(_membar_producer)
+END(_membar_release)
ENTRY(_membar_sync)
/*
@@ -363,10 +363,14 @@ ALIAS(atomic_cas_uint_ni,_atomic_cas_32_
ALIAS(atomic_cas_ulong_ni,_atomic_cas_64_ni)
ALIAS(atomic_cas_ptr_ni,_atomic_cas_64_ni)
-ALIAS(membar_consumer,_membar_consumer)
-ALIAS(membar_producer,_membar_producer)
+ALIAS(membar_acquire,_membar_acquire)
+ALIAS(membar_release,_membar_release)
+ALIAS(membar_sync,_membar_sync)
+
+ALIAS(membar_consumer,_membar_acquire)
+ALIAS(membar_producer,_membar_release)
ALIAS(membar_enter,_membar_sync)
-ALIAS(membar_exit,_membar_producer)
+ALIAS(membar_exit,_membar_release)
ALIAS(membar_sync,_membar_sync)
STRONG_ALIAS(_atomic_add_int,_atomic_add_32)
@@ -421,8 +425,10 @@ STRONG_ALIAS(_atomic_cas_uint_ni,_atomic
STRONG_ALIAS(_atomic_cas_ulong_ni,_atomic_cas_64_ni)
STRONG_ALIAS(_atomic_cas_ptr_ni,_atomic_cas_64_ni)
+STRONG_ALIAS(_membar_consumer,_membar_acquire)
+STRONG_ALIAS(_membar_producer,_membar_release)
STRONG_ALIAS(_membar_enter,_membar_sync)
-STRONG_ALIAS(_membar_exit,_membar_producer)
+STRONG_ALIAS(_membar_exit,_membar_release)
#ifdef _HARDKERNEL
.section .rodata
Index: src/common/lib/libc/atomic/membar_ops_nop.c
diff -u src/common/lib/libc/atomic/membar_ops_nop.c:1.7 src/common/lib/libc/atomic/membar_ops_nop.c:1.8
--- src/common/lib/libc/atomic/membar_ops_nop.c:1.7 Sat Feb 22 17:08:30 2014
+++ src/common/lib/libc/atomic/membar_ops_nop.c Sat Apr 9 23:32:52 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: membar_ops_nop.c,v 1.7 2014/02/22 17:08:30 martin Exp $ */
+/* $NetBSD: membar_ops_nop.c,v 1.8 2022/04/09 23:32:52 riastradh Exp $ */
/*-
* Copyright (c) 2007 The NetBSD Foundation, Inc.
@@ -40,6 +40,12 @@ membar_sync(void)
/* nothing */
}
+#undef membar_acquire
+atomic_op_alias(membar_acquire,_membar_sync)
+__strong_alias(_membar_acquire,_membar_sync)
+#undef membar_release
+atomic_op_alias(membar_release,_membar_sync)
+__strong_alias(_membar_release,_membar_sync)
#undef membar_enter
atomic_op_alias(membar_enter,_membar_sync)
__strong_alias(_membar_enter,_membar_sync)
Index: src/distrib/sets/lists/comp/mi
diff -u src/distrib/sets/lists/comp/mi:1.2412 src/distrib/sets/lists/comp/mi:1.2413
--- src/distrib/sets/lists/comp/mi:1.2412 Tue Feb 15 22:46:29 2022
+++ src/distrib/sets/lists/comp/mi Sat Apr 9 23:32:52 2022
@@ -1,4 +1,4 @@
-# $NetBSD: mi,v 1.2412 2022/02/15 22:46:29 riastradh Exp $
+# $NetBSD: mi,v 1.2413 2022/04/09 23:32:52 riastradh Exp $
#
# Note: don't delete entries from here - mark them as "obsolete" instead.
./etc/mtree/set.comp comp-sys-root
@@ -6307,6 +6307,7 @@
./usr/share/man/cat3/atoi.0 comp-c-catman .cat
./usr/share/man/cat3/atol.0 comp-c-catman .cat
./usr/share/man/cat3/atoll.0 comp-c-catman .cat
+./usr/share/man/cat3/atomic.0 comp-c-catman .cat
./usr/share/man/cat3/atomic_add.0 comp-c-catman .cat
./usr/share/man/cat3/atomic_add_32.0 comp-c-catman .cat
./usr/share/man/cat3/atomic_add_32_nv.0 comp-c-catman .cat
@@ -8913,12 +8914,15 @@
./usr/share/man/cat3/md5.0 comp-c-catman .cat
./usr/share/man/cat3/mdc2.0 comp-obsolete obsolete
./usr/share/man/cat3/mdoc.0 comp-obsolete obsolete
+./usr/share/man/cat3/membar.0 comp-c-catman .cat
+./usr/share/man/cat3/membar_acquire.0 comp-c-catman .cat
./usr/share/man/cat3/membar_consumer.0 comp-c-catman .cat
./usr/share/man/cat3/membar_datadep_consumer.0 comp-c-catman .cat
./usr/share/man/cat3/membar_enter.0 comp-c-catman .cat
./usr/share/man/cat3/membar_exit.0 comp-c-catman .cat
./usr/share/man/cat3/membar_ops.0 comp-c-catman .cat
./usr/share/man/cat3/membar_producer.0 comp-c-catman .cat
+./usr/share/man/cat3/membar_release.0 comp-c-catman .cat
./usr/share/man/cat3/membar_sync.0 comp-c-catman .cat
./usr/share/man/cat3/memccpy.0 comp-c-catman .cat
./usr/share/man/cat3/memchr.0 comp-c-catman .cat
@@ -14668,6 +14672,7 @@
./usr/share/man/html3/atoi.html comp-c-htmlman html
./usr/share/man/html3/atol.html comp-c-htmlman html
./usr/share/man/html3/atoll.html comp-c-htmlman html
+./usr/share/man/html3/atomic.html comp-c-htmlman html
./usr/share/man/html3/atomic_add.html comp-c-htmlman html
./usr/share/man/html3/atomic_add_32.html comp-c-htmlman html
./usr/share/man/html3/atomic_add_32_nv.html comp-c-htmlman html
@@ -17195,12 +17200,15 @@
./usr/share/man/html3/md4.html comp-c-htmlman html
./usr/share/man/html3/md5.html comp-c-htmlman html
./usr/share/man/html3/mdoc.html comp-obsolete obsolete
+./usr/share/man/html3/membar.html comp-c-htmlman html
+./usr/share/man/html3/membar_acquire.html comp-c-htmlman html
./usr/share/man/html3/membar_consumer.html comp-c-htmlman html
./usr/share/man/html3/membar_datadep_consumer.html comp-c-htmlman html
./usr/share/man/html3/membar_enter.html comp-c-htmlman html
./usr/share/man/html3/membar_exit.html comp-c-htmlman html
./usr/share/man/html3/membar_ops.html comp-c-htmlman html
./usr/share/man/html3/membar_producer.html comp-c-htmlman html
+./usr/share/man/html3/membar_release.html comp-c-htmlman html
./usr/share/man/html3/membar_sync.html comp-c-htmlman html
./usr/share/man/html3/memccpy.html comp-c-htmlman html
./usr/share/man/html3/memchr.html comp-c-htmlman html
@@ -22866,6 +22874,7 @@
./usr/share/man/man3/atoi.3 comp-c-man .man
./usr/share/man/man3/atol.3 comp-c-man .man
./usr/share/man/man3/atoll.3 comp-c-man .man
+./usr/share/man/man3/atomic.3 comp-c-man .man
./usr/share/man/man3/atomic_add.3 comp-c-man .man
./usr/share/man/man3/atomic_add_32.3 comp-c-man .man
./usr/share/man/man3/atomic_add_32_nv.3 comp-c-man .man
@@ -25484,12 +25493,15 @@
./usr/share/man/man3/md5.3 comp-c-man .man
./usr/share/man/man3/mdc2.3 comp-obsolete obsolete
./usr/share/man/man3/mdoc.3 comp-obsolete obsolete
+./usr/share/man/man3/membar.3 comp-c-man .man
+./usr/share/man/man3/membar_acquire.3 comp-c-man .man
./usr/share/man/man3/membar_consumer.3 comp-c-man .man
./usr/share/man/man3/membar_datadep_consumer.3 comp-c-man .man
./usr/share/man/man3/membar_enter.3 comp-c-man .man
./usr/share/man/man3/membar_exit.3 comp-c-man .man
./usr/share/man/man3/membar_ops.3 comp-c-man .man
./usr/share/man/man3/membar_producer.3 comp-c-man .man
+./usr/share/man/man3/membar_release.3 comp-c-man .man
./usr/share/man/man3/membar_sync.3 comp-c-man .man
./usr/share/man/man3/memccpy.3 comp-c-man .man
./usr/share/man/man3/memchr.3 comp-c-man .man
Index: src/lib/libc/atomic/Makefile.inc
diff -u src/lib/libc/atomic/Makefile.inc:1.4 src/lib/libc/atomic/Makefile.inc:1.5
--- src/lib/libc/atomic/Makefile.inc:1.4 Sat Dec 7 15:13:59 2019
+++ src/lib/libc/atomic/Makefile.inc Sat Apr 9 23:32:52 2022
@@ -1,4 +1,4 @@
-# $NetBSD: Makefile.inc,v 1.4 2019/12/07 15:13:59 riastradh Exp $
+# $NetBSD: Makefile.inc,v 1.5 2022/04/09 23:32:52 riastradh Exp $
# from: @(#)Makefile.inc 8.6 (Berkeley) 5/4/95
# gen sources
@@ -75,6 +75,10 @@ MLINKS+=atomic_add.3 atomic_add_32.3 \
atomic_swap.3 atomic_swap_ulong.3 \
atomic_swap.3 atomic_swap_ptr.3 \
atomic_swap.3 atomic_swap_64.3 \
+ atomic_ops.3 atomic.3 \
+ membar_ops.3 membar.3 \
+ membar_ops.3 membar_acquire.3 \
+ membar_ops.3 membar_release.3 \
membar_ops.3 membar_enter.3 \
membar_ops.3 membar_exit.3 \
membar_ops.3 membar_producer.3 \
Index: src/lib/libc/atomic/membar_ops.3
diff -u src/lib/libc/atomic/membar_ops.3:1.9 src/lib/libc/atomic/membar_ops.3:1.10
--- src/lib/libc/atomic/membar_ops.3:1.9 Tue Feb 15 22:46:53 2022
+++ src/lib/libc/atomic/membar_ops.3 Sat Apr 9 23:32:52 2022
@@ -1,4 +1,4 @@
-.\" $NetBSD: membar_ops.3,v 1.9 2022/02/15 22:46:53 riastradh Exp $
+.\" $NetBSD: membar_ops.3,v 1.10 2022/04/09 23:32:52 riastradh Exp $
.\"
.\" Copyright (c) 2007, 2008 The NetBSD Foundation, Inc.
.\" All rights reserved.
@@ -27,13 +27,13 @@
.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
-.Dd September 2, 2020
+.Dd March 30, 2022
.Dt MEMBAR_OPS 3
.Os
.Sh NAME
.Nm membar_ops ,
-.Nm membar_enter ,
-.Nm membar_exit ,
+.Nm membar_acquire ,
+.Nm membar_release ,
.Nm membar_producer ,
.Nm membar_consumer ,
.Nm membar_datadep_consumer ,
@@ -45,9 +45,9 @@
.In sys/atomic.h
.\"
.Ft void
-.Fn membar_enter "void"
+.Fn membar_acquire "void"
.Ft void
-.Fn membar_exit "void"
+.Fn membar_release "void"
.Ft void
.Fn membar_producer "void"
.Ft void
@@ -65,9 +65,9 @@ relaxed load and store order.
.Pp
In general, memory barriers must come in pairs \(em a barrier on one
CPU, such as
-.Fn membar_exit ,
+.Fn membar_release ,
must pair with a barrier on another CPU, such as
-.Fn membar_enter ,
+.Fn membar_acquire ,
in order to synchronize anything between the two CPUs.
Code using
.Nm
@@ -92,72 +92,57 @@ not just C11 atomic operations on
.Vt _Atomic\^ Ns -qualified
objects.
.Bl -tag -width abcd
-.It Fn membar_enter
-Any store preceding
-.Fn membar_enter
+.It Fn membar_acquire
+Any load preceding
+.Fn membar_acquire
will happen before all memory operations following it.
.Pp
-An atomic read/modify/write operation
-.Pq Xr atomic_ops 3
-followed by a
-.Fn membar_enter
+A load followed by a
+.Fn membar_acquire
implies a
.Em load-acquire
operation in the language of C11.
+.Fn membar_acquire
+should only be used after atomic read/modify/write, such as
+.Xr atomic_cas_uint 3 .
+For regular loads, instead of
+.Li "x = *p; membar_acquire()" ,
+you should use
+.Li "x = atomic_load_acquire(p)" .
.Pp
-.Sy WARNING :
-A load followed by
-.Fn membar_enter
-.Em does not
-imply a
-.Em load-acquire
-operation, even though
-.Fn membar_exit
-followed by a store implies a
-.Em store-release
-operation; the symmetry of these names and asymmetry of the semantics
-is a historical mistake.
-In the
-.Nx
-kernel, you can use
-.Xr atomic_load_acquire 9
-for a
-.Em load-acquire
-operation without any atomic read/modify/write.
-.Pp
-.Fn membar_enter
+.Fn membar_acquire
is typically used in code that implements locking primitives to ensure
that a lock protects its data, and is typically paired with
-.Fn membar_exit ;
+.Fn membar_release ;
see below for an example.
-.It Fn membar_exit
+.It Fn membar_release
All memory operations preceding
-.Fn membar_exit
+.Fn membar_release
will happen before any store that follows it.
.Pp
A
-.Fn membar_exit
+.Fn membar_release
followed by a store implies a
.Em store-release
operation in the language of C11.
-.Fn membar_exit
+.Fn membar_release
should only be used before atomic read/modify/write, such as
.Xr atomic_inc_uint 3 .
For regular stores, instead of
-.Li "membar_exit(); *p = x" ,
+.Li "membar_release(); *p = x" ,
you should use
.Li "atomic_store_release(p, x)" .
.Pp
-.Fn membar_exit
+.Fn membar_release
is typically paired with
-.Fn membar_enter ,
+.Fn membar_acquire ,
and is typically used in code that implements locking or reference
counting primitives.
Releasing a lock or reference count should use
-.Fn membar_exit ,
+.Fn membar_release ,
and acquiring a lock or handling an object after draining references
should use
-.Fn membar_enter ,
+.Fn membar_acquire ,
so that whatever happened before releasing will also have happened
before acquiring.
For example:
@@ -165,7 +150,7 @@ For example:
/* thread A -- release a reference */
obj->state.mumblefrotz = 42;
KASSERT(valid(&obj->state));
-membar_exit();
+membar_release();
atomic_dec_uint(&obj->refcnt);
/*
@@ -174,7 +159,7 @@ atomic_dec_uint(&obj->refcnt);
*/
while (atomic_cas_uint(&obj->refcnt, 0, -1) != 0)
continue;
-membar_enter();
+membar_acquire();
KASSERT(valid(&obj->state));
obj->state.mumblefrotz--;
.Ed
@@ -188,9 +173,9 @@ in thread B witnesses the store in
in thread A setting the reference count to zero,
.Em then
everything in thread A before the
-.Fn membar_exit
+.Fn membar_release
is guaranteed to happen before everything in thread B after the
-.Fn membar_enter ,
+.Fn membar_acquire ,
as if the machine had sequentially executed:
.Bd -literal -offset abcdefgh
obj->state.mumblefrotz = 42; /* from thread A */
@@ -200,11 +185,11 @@ KASSERT(valid(&obj->state)); /* from thr
obj->state.mumblefrotz--;
.Ed
.Pp
-.Fn membar_exit
+.Fn membar_release
followed by a store, serving as a
.Em store-release
operation, may also be paired with a subsequent load followed by
-.Fn membar_sync ,
+.Fn membar_acquire ,
serving as the corresponding
.Em load-acquire
operation.
@@ -214,7 +199,7 @@ and
.Xr atomic_load_acquire 9
instead in that situation, unless the store is an atomic
read/modify/write which requires a separate
-.Fn membar_exit .
+.Fn membar_release .
.It Fn membar_producer
All stores preceding
.Fn membar_producer
@@ -294,7 +279,7 @@ consume(v);
.Pp
.Fn membar_datadep_consumer
is typically paired with
-.Fn membar_exit
+.Fn membar_release
by code that initializes an object before publishing it.
However, you should use
.Xr atomic_store_release 9
@@ -337,24 +322,48 @@ in C11.
is typically paired with
.Fn membar_sync .
.Pp
-A load followed by
-.Fn membar_sync ,
-serving as a
-.Em load-acquire
-operation, may also be paired with a prior
-.Fn membar_exit
-followed by a store, serving as the corresponding
-.Em store-release
-operation.
-However, you should use
-.Xr atomic_load_acquire 9
-instead of
-.No load-then- Ns Fn membar_sync
-if it is a regular load, or
-.Fn membar_enter
-instead of
.Fn membar_sync
-if the load is in an atomic read/modify/write operation.
+is typically not needed except in exotic synchronization schemes like
+Dekker's algorithm that require store-before-load ordering.
+If you are tempted to reach for it, see if there is another way to do
+what you're trying to do first.
+.El
+.Sh DEPRECATED MEMORY BARRIERS
+The following memory barriers are deprecated.
+They were imported from Solaris, which describes them as providing
+ordering relative to
+.Sq lock acquisition ,
+but the documentation in
+.Nx
+disagreed with the implementation and use on the semantics.
+.Bl -tag -width abcd
+.It Fn membar_enter
+Originally documented as store-before-load/store, this was instead
+implemented as load-before-load/store on some platforms, which is what
+essentially all uses relied on.
+Now this is implemented as an alias for
+.Fn membar_sync
+everywhere, meaning a full load/store-before-load/store sequential
+consistency barrier, in order to guarantee what the documentation
+claimed
+.Em and
+what the implementation actually did.
+.Pp
+New code should use
+.Fn membar_acquire
+for load-before-load/store ordering, which is what most uses need, or
+.Fn membar_sync
+for store-before-load/store ordering, which typically only appears in
+exotic synchronization schemes like Dekker's algorithm.
+.It Fn membar_exit
+Alias for
+.Fn membar_release .
+This was originally meant to be paired with
+.Fn membar_enter .
+.Pp
+New code should use
+.Fn membar_release
+instead.
.El
.Sh SEE ALSO
.Xr atomic_ops 3 ,
@@ -366,7 +375,19 @@ The
.Nm membar_ops
functions first appeared in
.Nx 5.0 .
+.Pp
The data-dependent load barrier,
.Fn membar_datadep_consumer ,
first appeared in
.Nx 7.0 .
+.Pp
+The
+.Fn membar_acquire
+and
+.Fn membar_release
+functions first appeared, and the
+.Fn membar_enter
+and
+.Fn membar_exit
+functions were deprecated, in
+.Nx 10.0 .
Index: src/share/man/man9/atomic_loadstore.9
diff -u src/share/man/man9/atomic_loadstore.9:1.6 src/share/man/man9/atomic_loadstore.9:1.7
--- src/share/man/man9/atomic_loadstore.9:1.6 Thu Sep 3 00:23:57 2020
+++ src/share/man/man9/atomic_loadstore.9 Sat Apr 9 23:32:53 2022
@@ -1,4 +1,4 @@
-.\" $NetBSD: atomic_loadstore.9,v 1.6 2020/09/03 00:23:57 riastradh Exp $
+.\" $NetBSD: atomic_loadstore.9,v 1.7 2022/04/09 23:32:53 riastradh Exp $
.\"
.\" Copyright (c) 2019 The NetBSD Foundation
.\" All rights reserved.
@@ -27,7 +27,7 @@
.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
-.Dd November 25, 2019
+.Dd February 11, 2022
.Dt ATOMIC_LOADSTORE 9
.Os
.Sh NAME
@@ -648,7 +648,7 @@ is an atomic read/modify/write operation
.Xr atomic_ops 3 ,
then
.Bd -literal
- membar_exit();
+ membar_release();
atomic_r/m/w(obj, ...);
.Ed
.Pp
@@ -657,32 +657,34 @@ functions like a release operation on
and
.Bd -literal
atomic_r/m/w(obj, ...);
- membar_enter();
+ membar_acquire();
.Ed
.Pp
functions like a acquire operation on
.Va obj .
.Pp
-.Sy WARNING :
-The combination of
-.Fn atomic_load_relaxed
-and
-.Xr membar_enter 3
-.Em does not
-make an acquire operation; only read/modify/write atomics may be
-combined with
-.Xr membar_enter 3
-this way.
-.Pp
On architectures where
.Dv __HAVE_ATOMIC_AS_MEMBAR
is defined, all the
.Xr atomic_ops 3
imply release and acquire operations, so the
-.Xr membar_enter 3
+.Xr membar_acquire 3
and
-.Xr membar_exit 3
+.Xr membar_release 3
are redundant.
+.Pp
+The combination of
+.Fn atomic_load_relaxed
+and
+.Xr membar_acquire 3
+in that order is equivalent to
+.Fn atomic_load_acquire ,
+and the combination of
+.Xr membar_release 3
+and
+.Fn atomic_store_relaxed
+in that order is equivalent to
+.Fn atomic_store_release .
.Sh EXAMPLES
Maintaining lossy counters.
These may lose some counts, because the read/modify/write cycle as a
Index: src/sys/sys/atomic.h
diff -u src/sys/sys/atomic.h:1.22 src/sys/sys/atomic.h:1.23
--- src/sys/sys/atomic.h:1.22 Sat Dec 18 16:31:40 2021
+++ src/sys/sys/atomic.h Sat Apr 9 23:32:53 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: atomic.h,v 1.22 2021/12/18 16:31:40 riastradh Exp $ */
+/* $NetBSD: atomic.h,v 1.23 2022/04/09 23:32:53 riastradh Exp $ */
/*-
* Copyright (c) 2007, 2008 The NetBSD Foundation, Inc.
@@ -181,12 +181,18 @@ uint8_t atomic_cas_8(volatile uint8_t *
/*
* Memory barrier operations
*/
-void membar_enter(void);
-void membar_exit(void);
+void membar_acquire(void);
+void membar_release(void);
void membar_producer(void);
void membar_consumer(void);
void membar_sync(void);
+/*
+ * Deprecated memory barriers
+ */
+void membar_enter(void);
+void membar_exit(void);
+
#ifdef __HAVE_MEMBAR_DATADEP_CONSUMER
void membar_datadep_consumer(void);
#else
Index: src/tests/lib/libc/membar/t_dekker.c
diff -u src/tests/lib/libc/membar/t_dekker.c:1.1 src/tests/lib/libc/membar/t_dekker.c:1.2
--- src/tests/lib/libc/membar/t_dekker.c:1.1 Fri Apr 8 23:35:52 2022
+++ src/tests/lib/libc/membar/t_dekker.c Sat Apr 9 23:32:53 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: t_dekker.c,v 1.1 2022/04/08 23:35:52 riastradh Exp $ */
+/* $NetBSD: t_dekker.c,v 1.2 2022/04/09 23:32:53 riastradh Exp $ */
/*-
* Copyright (c) 2022 The NetBSD Foundation, Inc.
@@ -27,7 +27,7 @@
*/
#include <sys/cdefs.h>
-__RCSID("$NetBSD: t_dekker.c,v 1.1 2022/04/08 23:35:52 riastradh Exp $");
+__RCSID("$NetBSD: t_dekker.c,v 1.2 2022/04/09 23:32:53 riastradh Exp $");
#include <sys/atomic.h>
#include <sys/param.h>
@@ -44,10 +44,6 @@ __RCSID("$NetBSD: t_dekker.c,v 1.1 2022/
#include <stdio.h>
#include <unistd.h>
-/* XXX */
-#define membar_acquire() membar_enter()
-#define membar_release() membar_exit()
-
#ifdef BROKEN_SYNC
#undef membar_sync
#define membar_sync() asm volatile("" ::: "memory")
Index: src/tests/lib/libc/membar/t_spinlock.c
diff -u src/tests/lib/libc/membar/t_spinlock.c:1.1 src/tests/lib/libc/membar/t_spinlock.c:1.2
--- src/tests/lib/libc/membar/t_spinlock.c:1.1 Fri Apr 8 23:35:52 2022
+++ src/tests/lib/libc/membar/t_spinlock.c Sat Apr 9 23:32:53 2022
@@ -1,4 +1,4 @@
-/* $NetBSD: t_spinlock.c,v 1.1 2022/04/08 23:35:52 riastradh Exp $ */
+/* $NetBSD: t_spinlock.c,v 1.2 2022/04/09 23:32:53 riastradh Exp $ */
/*-
* Copyright (c) 2022 The NetBSD Foundation, Inc.
@@ -27,7 +27,7 @@
*/
#include <sys/cdefs.h>
-__RCSID("$NetBSD: t_spinlock.c,v 1.1 2022/04/08 23:35:52 riastradh Exp $");
+__RCSID("$NetBSD: t_spinlock.c,v 1.2 2022/04/09 23:32:53 riastradh Exp $");
#include <sys/atomic.h>
#include <sys/param.h>
@@ -44,10 +44,6 @@ __RCSID("$NetBSD: t_spinlock.c,v 1.1 202
#include <stdio.h>
#include <unistd.h>
-/* XXX */
-#define membar_acquire() membar_enter()
-#define membar_release() membar_exit()
-
#ifdef BROKEN_ACQUIRE
#undef membar_acquire
#define membar_acquire() asm volatile("" ::: "memory")
