On Thu 25-02-16 19:08:41, Waiman Long wrote:
> Linked list is used everywhere in the Linux kernel. However, if many
> threads are trying to add or delete entries into the same linked list,
> it can create a performance bottleneck.
>
> This patch introduces a new per-cpu list subystem with associated
> per-cpu locks for protecting each of the lists individually. This
> allows list entries insertion and deletion operations to happen in
> parallel instead of being serialized with a global list and lock.
>
> List entry insertion is strictly per cpu. List deletion, however, can
> happen in a cpu other than the one that did the insertion. So we still
> need lock to protect the list. Because of that, there may still be
> a small amount of contention when deletion is being done.
>
> A new header file include/linux/percpu-list.h will be added with the
> associated pcpu_list_head and pcpu_list_node structures. The following
> functions are provided to manage the per-cpu list:
>
> 1. int init_pcpu_list_head(struct pcpu_list_head **ppcpu_head)
> 2. void pcpu_list_add(struct pcpu_list_node *node,
> struct pcpu_list_head *head)
> 3. void pcpu_list_del(struct pcpu_list *node)
>
> Iteration of all the list entries within a group of per-cpu
> lists is done by calling either the pcpu_list_iterate() or
> pcpu_list_iterate_safe() functions in a while loop. They correspond
> to the list_for_each_entry() and list_for_each_entry_safe() macros
> respectively. The iteration states are keep in a pcpu_list_state
> structure that is passed to the iteration functions.
>
> Signed-off-by: Waiman Long <[email protected]>
...
> +static __always_inline bool
> +__pcpu_list_next_cpu(struct pcpu_list_head *head, struct pcpu_list_state
> *state)
> +{
> + if (state->lock)
> + spin_unlock(state->lock);
> +
> + if (state->cpu++ >= 0)
> + return false;
> +
> + if (list_empty(&head->list))
> + return false;
> + state->lock = &head->lock;
> + spin_lock(state->lock);
Thinking about avoiding the overhead for UP case - is there any point in
spin_lock in UP? You could just do preempt_disable(), couldn't you? And
then you don't have to pass around any spinlock pointers, don't you?
> + state->curr = list_entry(head->list.next, struct pcpu_list_node, list);
> + return true;
> +
^^^ Spurious empty line
> +}
> +#else /* NR_CPUS == 1 */
> +/*
> + * Multiprocessor
> + */
> +static inline bool pcpu_list_empty(struct pcpu_list_head *pcpu_head)
> +{
> + int cpu;
> +
> + for_each_possible_cpu(cpu)
> + if (!list_empty(&per_cpu_ptr(pcpu_head, cpu)->list))
> + return false;
> + return true;
> +}
> +
> +/*
> + * Helper function to find the first entry of the next per-cpu list
> + * It works somewhat like for_each_possible_cpu(cpu).
> + *
> + * Return: true if the entry is found, false if all the lists exhausted
> + */
> +static __always_inline bool
> +__pcpu_list_next_cpu(struct pcpu_list_head *head, struct pcpu_list_state
> *state)
> +{
> + if (state->lock)
> + spin_unlock(state->lock);
> +next_cpu:
> + /*
> + * for_each_possible_cpu(cpu)
> + */
> + state->cpu = cpumask_next(state->cpu, cpu_possible_mask);
> + if (state->cpu >= nr_cpu_ids)
> + return false; /* All the per-cpu lists iterated */
> +
> + state->head = &per_cpu_ptr(head, state->cpu)->list;
> + if (list_empty(state->head))
> + goto next_cpu;
> +
> + state->lock = &per_cpu_ptr(head, state->cpu)->lock;
> + spin_lock(state->lock);
> + state->curr = list_entry(state->head->next,
> + struct pcpu_list_node, list);
And I've realized this is still racy. The list can be empty by the time you
acquire state->lock so you have to recheck list_empty() after acquiring the
lock.
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
