New data structure for tracking objects waiting on an RCU grace period.
Supports regular RCU and SRCU, and possibly other RCU flavors in the
future. Uses radix trees for tracking pending objects, falling back to
linked lists on allocation failure.
This gets us a more general replacement for SLAB_TYPESAFE_BY_RCU, and a
cleaner and slightly faster backend for kvfree_call_rcu(), and
in the future a faster backend for call_rcu() as well.
There's still some small todo items, mentioned in the relevant patches.
Paul - I'm considering putting this into 6.11 for bcachefs (not the
patch that switches kvfree_rcu, of course), as I need it rather
pressingly. Thoughts? I can put it in fs/bcachefs/ if you hate it :)
Kent Overstreet (9):
lib/generic-radix-tree.c: genradix_ptr_inlined()
lib/generic-radix-tree.c: add preallocation
darray: lift from bcachefs
vmalloc: is_vmalloc_addr_inlined()
rcu: delete lockdep_assert_irqs_enabled() assert in
start_poll_synchronize_rcu_common()
rcu: rcu_pending
bcachefs: Rip out freelists from btree key cache
bcachefs: key cache can now allocate from pending
rcu: Switch kvfree_rcu() to new rcu_pending
MAINTAINERS | 7 +
fs/bcachefs/Makefile | 1 -
fs/bcachefs/btree_key_cache.c | 406 +++----------
fs/bcachefs/btree_key_cache_types.h | 18 +-
fs/bcachefs/btree_node_scan_types.h | 2 +-
fs/bcachefs/btree_types.h | 5 +-
fs/bcachefs/btree_update.c | 2 +
fs/bcachefs/btree_write_buffer_types.h | 2 +-
fs/bcachefs/disk_accounting_types.h | 2 +-
fs/bcachefs/fsck.c | 2 +-
fs/bcachefs/journal_io.h | 2 +-
fs/bcachefs/journal_sb.c | 2 +-
fs/bcachefs/sb-downgrade.c | 3 +-
fs/bcachefs/sb-errors_types.h | 2 +-
fs/bcachefs/sb-members.h | 2 +-
fs/bcachefs/subvolume.h | 1 -
fs/bcachefs/subvolume_types.h | 2 +-
fs/bcachefs/thread_with_file_types.h | 2 +-
fs/bcachefs/util.h | 29 +-
{fs/bcachefs => include/linux}/darray.h | 59 +-
include/linux/darray_types.h | 22 +
include/linux/generic-radix-tree.h | 106 +++-
include/linux/mm.h | 7 +
include/linux/rcu_pending.h | 27 +
init/main.c | 2 +
kernel/rcu/Makefile | 2 +-
kernel/rcu/pending.c | 623 ++++++++++++++++++++
kernel/rcu/tree.c | 747 ------------------------
kernel/rcu/update.c | 1 -
lib/Makefile | 2 +-
{fs/bcachefs => lib}/darray.c | 12 +-
lib/generic-radix-tree.c | 80 +--
mm/vmalloc.c | 4 +-
33 files changed, 962 insertions(+), 1224 deletions(-)
rename {fs/bcachefs => include/linux}/darray.h (66%)
create mode 100644 include/linux/darray_types.h
create mode 100644 include/linux/rcu_pending.h
create mode 100644 kernel/rcu/pending.c
rename {fs/bcachefs => lib}/darray.c (57%)
--
2.45.2
