Similar to the previous fix for lru_hash maps, the lru_percpu_hash map
implementation also suffers from unnecessary eviction when updating
existing elements.
When updating a key that already exists in a full lru_percpu_hash map,
the current code path calls prealloc_lru_pop() before checking for the
existing key (unless map_flags is BPF_EXIST). This can evict an unrelated
element even though the update is just modifying the per-CPU value of an
existing entry.
Fix this by looking up the key first. If found, update the per-CPU value
in-place using pcpu_copy_value(), refresh the LRU reference, and return
early. Only proceed with node allocation if the key does not exist.
Fixes: 8f8449384ec3 ("bpf: Add BPF_MAP_TYPE_LRU_PERCPU_HASH")
Signed-off-by: Leon Hwang <[email protected]>
---
kernel/bpf/hashtab.c | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index fb624aa76573..af54fc3a9ba9 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -1358,6 +1358,28 @@ static long __htab_lru_percpu_map_update_elem(struct
bpf_map *map, void *key,
b = __select_bucket(htab, hash);
head = &b->head;
+ ret = htab_lock_bucket(b, &flags);
+ if (ret)
+ goto err_lock_bucket;
+
+ l_old = lookup_elem_raw(head, hash, key, key_size);
+
+ ret = check_flags(htab, l_old, map_flags);
+ if (ret)
+ goto err;
+
+ if (l_old) {
+ bpf_lru_node_set_ref(&l_old->lru_node);
+ /* per-cpu hash map can update value in-place */
+ pcpu_copy_value(htab, htab_elem_get_ptr(l_old, key_size),
+ value, onallcpus);
+ }
+
+ htab_unlock_bucket(b, flags);
+
+ if (l_old)
+ return 0;
+
/* For LRU, we need to alloc before taking bucket's
* spinlock because LRU's elem alloc may need
* to remove older elem from htab and this removal
--
2.52.0
