Re: [PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add
Hi Matthew, On Tue, Feb 13, 2018 at 07:16:08AM -0800, Matthew Wilcox wrote: > On Tue, Feb 13, 2018 at 12:58:51PM +, Will Deacon wrote: > > This patch resolves the livelock by not taking hlist_bl_lock in > > d_alloc_parallel if the sequence counter is odd, since any subsequent > > masked comparison with i_dir_seq will fail anyway. > > > > Cc: Peter Zijlstra > > Cc: Al Viro > > Signed-off-by: Will Deacon > > Reviewed-by: Matthew Wilcox Thanks! > I wonder whether it makes sense to turn i_dir_seq into a seqcount_t, > which would give us the lockdep checking as well. I'm not sure it's quite as simple as that. start_dir_add looks very much like it's intended to run concurrently, so we'd need a write_seqcount implementation that provides the same atomicity guarantees. Will
Re: [PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add
On Tue, Feb 13, 2018 at 12:58:51PM +, Will Deacon wrote: > This patch resolves the livelock by not taking hlist_bl_lock in > d_alloc_parallel if the sequence counter is odd, since any subsequent > masked comparison with i_dir_seq will fail anyway. > > Cc: Peter Zijlstra > Cc: Al Viro > Signed-off-by: Will Deacon Reviewed-by: Matthew Wilcox I wonder whether it makes sense to turn i_dir_seq into a seqcount_t, which would give us the lockdep checking as well.
Re: [PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add
On Tue, Feb 13, 2018 at 12:58:51PM +, Will Deacon wrote:
> If d_alloc_parallel runs concurrently with __d_add, it is possible for
> d_alloc_parallel to continuously retry whilst i_dir_seq has been
> incremented to an odd value by __d_add:
>
> CPU0:
> __d_add
> n = start_dir_add(dir);
> cmpxchg(&dir->i_dir_seq, n, n + 1) == n
>
> CPU1:
> d_alloc_parallel
> retry:
> seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1;
> hlist_bl_lock(b);
> bit_spin_lock(0, (unsigned long *)b); // Always succeeds
>
> CPU0:
> __d_lookup_done(dentry)
> hlist_bl_lock
> bit_spin_lock(0, (unsigned long *)b); // Never succeeds
>
> CPU1:
> if (unlikely(parent->d_inode->i_dir_seq != seq)) {
> hlist_bl_unlock(b);
> goto retry;
> }
>
> Since the simple bit_spin_lock used to implement hlist_bl_lock does not
And cannot, a single bit is just not enough state.
> provide any fairness guarantees, then CPU1 can starve CPU0 of the lock
> and prevent it from reaching end_dir_add(dir), therefore CPU1 cannot
> exit its retry loop because the sequence number always has the bottom
> bit set.
>
> This patch resolves the livelock by not taking hlist_bl_lock in
> d_alloc_parallel if the sequence counter is odd, since any subsequent
> masked comparison with i_dir_seq will fail anyway.
>
Acked-by: Peter Zijlstra (Intel)
> Cc: Al Viro
> Signed-off-by: Will Deacon
> ---
> fs/dcache.c | 8 +++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/fs/dcache.c b/fs/dcache.c
> index 7c38f39958bc..b243deec298c 100644
> --- a/fs/dcache.c
> +++ b/fs/dcache.c
> @@ -2474,7 +2474,7 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
>
> retry:
> rcu_read_lock();
> - seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1;
> + seq = smp_load_acquire(&parent->d_inode->i_dir_seq);
> r_seq = read_seqbegin(&rename_lock);
> dentry = __d_lookup_rcu(parent, name, &d_seq);
> if (unlikely(dentry)) {
> @@ -2495,6 +2495,12 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
> rcu_read_unlock();
> goto retry;
> }
> +
> + if (unlikely(seq & 1)) {
> + rcu_read_unlock();
> + goto retry;
> + }
> +
> hlist_bl_lock(b);
> if (unlikely(parent->d_inode->i_dir_seq != seq)) {
Also, should that not read:
if (unlikely(READ_ONCE(parent->d_inode->i_dir_seq) != seq)) {
I mean, load-tearing can only result in additional failure, but still.
> hlist_bl_unlock(b);
[PATCH] fs: dcache: Avoid livelock between d_alloc_parallel and __d_add
If d_alloc_parallel runs concurrently with __d_add, it is possible for
d_alloc_parallel to continuously retry whilst i_dir_seq has been
incremented to an odd value by __d_add:
CPU0:
__d_add
n = start_dir_add(dir);
cmpxchg(&dir->i_dir_seq, n, n + 1) == n
CPU1:
d_alloc_parallel
retry:
seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1;
hlist_bl_lock(b);
bit_spin_lock(0, (unsigned long *)b); // Always succeeds
CPU0:
__d_lookup_done(dentry)
hlist_bl_lock
bit_spin_lock(0, (unsigned long *)b); // Never succeeds
CPU1:
if (unlikely(parent->d_inode->i_dir_seq != seq)) {
hlist_bl_unlock(b);
goto retry;
}
Since the simple bit_spin_lock used to implement hlist_bl_lock does not
provide any fairness guarantees, then CPU1 can starve CPU0 of the lock
and prevent it from reaching end_dir_add(dir), therefore CPU1 cannot
exit its retry loop because the sequence number always has the bottom
bit set.
This patch resolves the livelock by not taking hlist_bl_lock in
d_alloc_parallel if the sequence counter is odd, since any subsequent
masked comparison with i_dir_seq will fail anyway.
Cc: Peter Zijlstra
Cc: Al Viro
Signed-off-by: Will Deacon
---
fs/dcache.c | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/fs/dcache.c b/fs/dcache.c
index 7c38f39958bc..b243deec298c 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -2474,7 +2474,7 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
retry:
rcu_read_lock();
- seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1;
+ seq = smp_load_acquire(&parent->d_inode->i_dir_seq);
r_seq = read_seqbegin(&rename_lock);
dentry = __d_lookup_rcu(parent, name, &d_seq);
if (unlikely(dentry)) {
@@ -2495,6 +2495,12 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
rcu_read_unlock();
goto retry;
}
+
+ if (unlikely(seq & 1)) {
+ rcu_read_unlock();
+ goto retry;
+ }
+
hlist_bl_lock(b);
if (unlikely(parent->d_inode->i_dir_seq != seq)) {
hlist_bl_unlock(b);
--
2.1.4
