New data structure for tracking objects waiting on an RCU grace period. Supports regular RCU and SRCU, and possibly other RCU flavors in the future. Uses radix trees for tracking pending objects, falling back to linked lists on allocation failure.
This gets us a more general replacement for SLAB_TYPESAFE_BY_RCU, and a cleaner and slightly faster backend for kvfree_call_rcu(), and in the future a faster backend for call_rcu() as well. There's still some small todo items, mentioned in the relevant patches. Paul - I'm considering putting this into 6.11 for bcachefs (not the patch that switches kvfree_rcu, of course), as I need it rather pressingly. Thoughts? I can put it in fs/bcachefs/ if you hate it :) Kent Overstreet (9): lib/generic-radix-tree.c: genradix_ptr_inlined() lib/generic-radix-tree.c: add preallocation darray: lift from bcachefs vmalloc: is_vmalloc_addr_inlined() rcu: delete lockdep_assert_irqs_enabled() assert in start_poll_synchronize_rcu_common() rcu: rcu_pending bcachefs: Rip out freelists from btree key cache bcachefs: key cache can now allocate from pending rcu: Switch kvfree_rcu() to new rcu_pending MAINTAINERS | 7 + fs/bcachefs/Makefile | 1 - fs/bcachefs/btree_key_cache.c | 406 +++---------- fs/bcachefs/btree_key_cache_types.h | 18 +- fs/bcachefs/btree_node_scan_types.h | 2 +- fs/bcachefs/btree_types.h | 5 +- fs/bcachefs/btree_update.c | 2 + fs/bcachefs/btree_write_buffer_types.h | 2 +- fs/bcachefs/disk_accounting_types.h | 2 +- fs/bcachefs/fsck.c | 2 +- fs/bcachefs/journal_io.h | 2 +- fs/bcachefs/journal_sb.c | 2 +- fs/bcachefs/sb-downgrade.c | 3 +- fs/bcachefs/sb-errors_types.h | 2 +- fs/bcachefs/sb-members.h | 2 +- fs/bcachefs/subvolume.h | 1 - fs/bcachefs/subvolume_types.h | 2 +- fs/bcachefs/thread_with_file_types.h | 2 +- fs/bcachefs/util.h | 29 +- {fs/bcachefs => include/linux}/darray.h | 59 +- include/linux/darray_types.h | 22 + include/linux/generic-radix-tree.h | 106 +++- include/linux/mm.h | 7 + include/linux/rcu_pending.h | 27 + init/main.c | 2 + kernel/rcu/Makefile | 2 +- kernel/rcu/pending.c | 623 ++++++++++++++++++++ kernel/rcu/tree.c | 747 ------------------------ kernel/rcu/update.c | 1 - lib/Makefile | 2 +- {fs/bcachefs => lib}/darray.c | 12 +- lib/generic-radix-tree.c | 80 +-- mm/vmalloc.c | 4 +- 33 files changed, 962 insertions(+), 1224 deletions(-) rename {fs/bcachefs => include/linux}/darray.h (66%) create mode 100644 include/linux/darray_types.h create mode 100644 include/linux/rcu_pending.h create mode 100644 kernel/rcu/pending.c rename {fs/bcachefs => lib}/darray.c (57%) -- 2.45.2