Re: possible deadlock in fs_reclaim_acquire (2)
On Thu 11-02-21 14:49:43, Dan Carpenter wrote:
> On Thu, Feb 11, 2021 at 11:50:38AM +0100, 'Michal Hocko' via syzkaller-bugs
> wrote:
> > On Thu 11-02-21 11:22:25, Jan Kara wrote:
> > > On Thu 11-02-21 12:07:29, Hillf Danton wrote:
> >
> > I haven't received Hillf's email.
> >
> > [...]
> > > > Fix 71b565ceff37 ("ext4: drop ext4_kvmalloc()") by restoring the
> > > > GFP_NOFS introduced in dec214d00e0d ("ext4: xattr inode deduplication").
> > > >
> > > > Note this may be the fix also to possible deadlock
> > > > Reported-by: syzbot+bfdded10ab7dcd750...@syzkaller.appspotmail.com
> > > >
> > > > https://lore.kernel.org/linux-ext4/563a0205bafb7...@google.com/
> > >
> > > Please no. Ext4 is using scoping API to limit allocations to GFP_NOFS
> > > inside transactions. In this case something didn't work which seems like a
> > > lockdep bug at the first sight but I'll talk to mm guys about it.
> > > Definitely to problem doesn't seem to be in ext4.
> >
> > Agreed. kvmalloc(NOFS) is not even supported because vmalloc doesn't
> > support GFP_KERNEL incompatible requests.
>
> Okay. I have created a new Smatch warning when people pass GFP_NOFS
> to kvmalloc() and friends. We'll see if it finds anything tomorrow.
Thanks! Let me just clarify a bit. The kvmalloc doesn't support
GFP_KERNEL incompatible requests and it simply skips vmalloc fallback.
So this is not a correctness issue. It is the API abuse though.
--
Michal Hocko
SUSE Labs
Re: possible deadlock in fs_reclaim_acquire (2)
On Thu 11-02-21 20:04:14, Hillf Danton wrote:
> On Thu 11-02-21 12:07:29, Jan Kara wrote:
> >> Fix 71b565ceff37 ("ext4: drop ext4_kvmalloc()") by restoring the
> >> GFP_NOFS introduced in dec214d00e0d ("ext4: xattr inode deduplication").
> >>
> >> Note this may be the fix also to possible deadlock
> >> Reported-by: syzbot+bfdded10ab7dcd750...@syzkaller.appspotmail.com
> >>
> >> https://lore.kernel.org/linux-ext4/563a0205bafb7...@google.com/
> >
> >Please no. Ext4 is using scoping API to limit allocations to GFP_NOFS
> >inside transactions. In this case something didn't work which seems like a
> >lockdep bug at the first sight but I'll talk to mm guys about it.
> >Definitely to problem doesn't seem to be in ext4.
>
> Feel free to elaborate why we can find ext4 in the report?
> Why is ext4 special in this case?
Please read my reply to the syzbot report [1]. It has all the details.
[1] https://lore.kernel.org/lkml/20210211104947.gl19...@quack2.suse.cz
Honza
--
Jan Kara
SUSE Labs, CR
Re: possible deadlock in fs_reclaim_acquire (2)
On Thu, Feb 11, 2021 at 11:50:38AM +0100, 'Michal Hocko' via syzkaller-bugs
wrote:
> On Thu 11-02-21 11:22:25, Jan Kara wrote:
> > On Thu 11-02-21 12:07:29, Hillf Danton wrote:
>
> I haven't received Hillf's email.
>
> [...]
> > > Fix 71b565ceff37 ("ext4: drop ext4_kvmalloc()") by restoring the
> > > GFP_NOFS introduced in dec214d00e0d ("ext4: xattr inode deduplication").
> > >
> > > Note this may be the fix also to possible deadlock
> > > Reported-by: syzbot+bfdded10ab7dcd750...@syzkaller.appspotmail.com
> > >
> > > https://lore.kernel.org/linux-ext4/563a0205bafb7...@google.com/
> >
> > Please no. Ext4 is using scoping API to limit allocations to GFP_NOFS
> > inside transactions. In this case something didn't work which seems like a
> > lockdep bug at the first sight but I'll talk to mm guys about it.
> > Definitely to problem doesn't seem to be in ext4.
>
> Agreed. kvmalloc(NOFS) is not even supported because vmalloc doesn't
> support GFP_KERNEL incompatible requests.
Okay. I have created a new Smatch warning when people pass GFP_NOFS
to kvmalloc() and friends. We'll see if it finds anything tomorrow.
(We could probably find the same information with grep, but I run
Smatch every day so it prevents future bugs).
regards,
dan carpenter
Re: possible deadlock in fs_reclaim_acquire (2)
On Thu 11-02-21 11:22:25, Jan Kara wrote:
> On Thu 11-02-21 12:07:29, Hillf Danton wrote:
I haven't received Hillf's email.
[...]
> > Fix 71b565ceff37 ("ext4: drop ext4_kvmalloc()") by restoring the
> > GFP_NOFS introduced in dec214d00e0d ("ext4: xattr inode deduplication").
> >
> > Note this may be the fix also to possible deadlock
> > Reported-by: syzbot+bfdded10ab7dcd750...@syzkaller.appspotmail.com
> > https://lore.kernel.org/linux-ext4/563a0205bafb7...@google.com/
>
> Please no. Ext4 is using scoping API to limit allocations to GFP_NOFS
> inside transactions. In this case something didn't work which seems like a
> lockdep bug at the first sight but I'll talk to mm guys about it.
> Definitely to problem doesn't seem to be in ext4.
Agreed. kvmalloc(NOFS) is not even supported because vmalloc doesn't
support GFP_KERNEL incompatible requests.
>
> Honza
>
> >
> > --- a/fs/ext4/xattr.c
> > +++ b/fs/ext4/xattr.c
> > @@ -1459,7 +1459,7 @@ ext4_xattr_inode_cache_find(struct inode
> > if (!ce)
> > return NULL;
> >
> > - ea_data = kvmalloc(value_len, GFP_KERNEL);
> > + ea_data = kvmalloc(value_len, GFP_NOFS);
> > if (!ea_data) {
> > mb_cache_entry_put(ea_inode_cache, ce);
> > return NULL;
> --
> Jan Kara
> SUSE Labs, CR
--
Michal Hocko
SUSE Labs
Re: possible deadlock in fs_reclaim_acquire (2)
On Thu 11-02-21 12:07:29, Hillf Danton wrote:
> On Wed, 10 Feb 2021 17:25:19 -0800
> > syzbot found the following issue on:
> >
> > HEAD commit:825b5991 Merge tag '5.11-rc6-smb3' of git://git.samba.org/..
> > git tree: upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=117927bf50
> > kernel config: https://syzkaller.appspot.com/x/.config?x=bd1f72220b2e57eb
> > dashboard link: https://syzkaller.appspot.com/bug?extid=a7ab8df042baaf42ae3c
> > userspace arch: i386
> >
> > Unfortunately, I don't have any reproducer for this issue yet.
> >
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+a7ab8df042baaf42a...@syzkaller.appspotmail.com
> >
> > ==
> > WARNING: possible circular locking dependency detected
> > 5.11.0-rc6-syzkaller #0 Not tainted
> > --
> > syz-executor.3/24553 is trying to acquire lock:
> > 8be892c0 (fs_reclaim){+.+.}-{0:0}, at:
> > fs_reclaim_acquire+0xf7/0x150 mm/page_alloc.c:4348
> >
> > but task is already holding lock:
> > 888019b6b2c8 (>xattr_sem){}-{3:3}, at: ext4_write_lock_xattr
> > fs/ext4/xattr.h:142 [inline]
> > 888019b6b2c8 (>xattr_sem){}-{3:3}, at:
> > ext4_xattr_set_handle+0x15c/0x13e0 fs/ext4/xattr.c:2308
> >
> > which lock already depends on the new lock.
> >
> >
> > the existing dependency chain (in reverse order) is:
> >
> > -> #3 (>xattr_sem){}-{3:3}:
> >down_read+0x95/0x440 kernel/locking/rwsem.c:1353
> >ext4_setattr+0x570/0x1fd0 fs/ext4/inode.c:5375
> >notify_change+0xb60/0x10a0 fs/attr.c:336
> >chown_common+0x4a9/0x550 fs/open.c:674
> >vfs_fchown fs/open.c:741 [inline]
> >vfs_fchown fs/open.c:733 [inline]
> >ksys_fchown+0x111/0x170 fs/open.c:752
> >__do_sys_fchown fs/open.c:760 [inline]
> >__se_sys_fchown fs/open.c:758 [inline]
> >__x64_sys_fchown+0x6f/0xb0 fs/open.c:758
> >do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
> >entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> > -> #2 (jbd2_handle){}-{0:0}:
> >start_this_handle+0xfb4/0x1380 fs/jbd2/transaction.c:446
> >jbd2__journal_start+0x399/0x930 fs/jbd2/transaction.c:503
> >__ext4_journal_start_sb+0x227/0x4a0 fs/ext4/ext4_jbd2.c:105
> >ext4_sample_last_mounted fs/ext4/file.c:804 [inline]
> >ext4_file_open+0x613/0xb40 fs/ext4/file.c:832
> >do_dentry_open+0x4b9/0x11b0 fs/open.c:817
> >do_open fs/namei.c:3254 [inline]
> >path_openat+0x1b9a/0x2730 fs/namei.c:3371
> >do_filp_open+0x17e/0x3c0 fs/namei.c:3398
> >do_open_execat+0x116/0x690 fs/exec.c:913
> >bprm_execve fs/exec.c:1801 [inline]
> >bprm_execve+0x4be/0x19a0 fs/exec.c:1788
> >do_execveat_common+0x626/0x7c0 fs/exec.c:1915
> >do_execve fs/exec.c:1983 [inline]
> >__do_sys_execve fs/exec.c:2059 [inline]
> >__se_sys_execve fs/exec.c:2054 [inline]
> >__x64_sys_execve+0x8f/0xc0 fs/exec.c:2054
> >do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
> >entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >
> > -> #1 (sb_internal){.+.+}-{0:0}:
> >percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
> >__sb_start_write include/linux/fs.h:1592 [inline]
> >sb_start_intwrite include/linux/fs.h:1709 [inline]
> >ext4_evict_inode+0xe6f/0x1940 fs/ext4/inode.c:241
> >evict+0x2ed/0x6b0 fs/inode.c:577
> >iput_final fs/inode.c:1653 [inline]
> >iput.part.0+0x57e/0x810 fs/inode.c:1679
> >iput fs/inode.c:1669 [inline]
> >inode_lru_isolate+0x301/0x4f0 fs/inode.c:778
> >__list_lru_walk_one+0x178/0x5c0 mm/list_lru.c:222
> >list_lru_walk_one+0x99/0xd0 mm/list_lru.c:266
> >list_lru_shrink_walk include/linux/list_lru.h:195 [inline]
> >prune_icache_sb+0xdc/0x140 fs/inode.c:803
> >super_cache_scan+0x38d/0x590 fs/super.c:107
> >do_shrink_slab+0x3e4/0x9f0 mm/vmscan.c:511
> >shrink_slab+0x16f/0x5d0 mm/vmscan.c:672
> >shrink_node_memcgs mm/vmscan.c:2665 [inline]
> >shrink_node+0x8cc/0x1de0 mm/vmscan.c:2780
> >kswapd_shrink_node mm/vmscan.c:3523 [inline]
> >balance_pgdat+0x745/0x1270 mm/vmscan.c:3681
> >kswapd+0x5b1/0xdb0 mm/vmscan.c:3938
> >kthread+0x3b1/0x4a0 kernel/kthread.c:292
> >ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:296
> >
> > -> #0 (fs_reclaim){+.+.}-{0:0}:
> >check_prev_add kernel/locking/lockdep.c:2868 [inline]
> >check_prevs_add kernel/locking/lockdep.c:2993 [inline]
> >validate_chain kernel/locking/lockdep.c:3608 [inline]
> >__lock_acquire+0x2b26/0x54f0 kernel/locking/lockdep.c:4832
> >lock_acquire kernel/locking/lockdep.c:5442 [inline]
> >lock_acquire+0x1a8/0x720
possible deadlock in fs_reclaim_acquire (2)
Hello,
syzbot found the following issue on:
HEAD commit:825b5991 Merge tag '5.11-rc6-smb3' of git://git.samba.org/..
git tree: upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=117927bf50
kernel config: https://syzkaller.appspot.com/x/.config?x=bd1f72220b2e57eb
dashboard link: https://syzkaller.appspot.com/bug?extid=a7ab8df042baaf42ae3c
userspace arch: i386
Unfortunately, I don't have any reproducer for this issue yet.
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+a7ab8df042baaf42a...@syzkaller.appspotmail.com
==
WARNING: possible circular locking dependency detected
5.11.0-rc6-syzkaller #0 Not tainted
--
syz-executor.3/24553 is trying to acquire lock:
8be892c0 (fs_reclaim){+.+.}-{0:0}, at: fs_reclaim_acquire+0xf7/0x150
mm/page_alloc.c:4348
but task is already holding lock:
888019b6b2c8 (>xattr_sem){}-{3:3}, at: ext4_write_lock_xattr
fs/ext4/xattr.h:142 [inline]
888019b6b2c8 (>xattr_sem){}-{3:3}, at:
ext4_xattr_set_handle+0x15c/0x13e0 fs/ext4/xattr.c:2308
which lock already depends on the new lock.
the existing dependency chain (in reverse order) is:
-> #3 (>xattr_sem){}-{3:3}:
down_read+0x95/0x440 kernel/locking/rwsem.c:1353
ext4_setattr+0x570/0x1fd0 fs/ext4/inode.c:5375
notify_change+0xb60/0x10a0 fs/attr.c:336
chown_common+0x4a9/0x550 fs/open.c:674
vfs_fchown fs/open.c:741 [inline]
vfs_fchown fs/open.c:733 [inline]
ksys_fchown+0x111/0x170 fs/open.c:752
__do_sys_fchown fs/open.c:760 [inline]
__se_sys_fchown fs/open.c:758 [inline]
__x64_sys_fchown+0x6f/0xb0 fs/open.c:758
do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
entry_SYSCALL_64_after_hwframe+0x44/0xa9
-> #2 (jbd2_handle){}-{0:0}:
start_this_handle+0xfb4/0x1380 fs/jbd2/transaction.c:446
jbd2__journal_start+0x399/0x930 fs/jbd2/transaction.c:503
__ext4_journal_start_sb+0x227/0x4a0 fs/ext4/ext4_jbd2.c:105
ext4_sample_last_mounted fs/ext4/file.c:804 [inline]
ext4_file_open+0x613/0xb40 fs/ext4/file.c:832
do_dentry_open+0x4b9/0x11b0 fs/open.c:817
do_open fs/namei.c:3254 [inline]
path_openat+0x1b9a/0x2730 fs/namei.c:3371
do_filp_open+0x17e/0x3c0 fs/namei.c:3398
do_open_execat+0x116/0x690 fs/exec.c:913
bprm_execve fs/exec.c:1801 [inline]
bprm_execve+0x4be/0x19a0 fs/exec.c:1788
do_execveat_common+0x626/0x7c0 fs/exec.c:1915
do_execve fs/exec.c:1983 [inline]
__do_sys_execve fs/exec.c:2059 [inline]
__se_sys_execve fs/exec.c:2054 [inline]
__x64_sys_execve+0x8f/0xc0 fs/exec.c:2054
do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
entry_SYSCALL_64_after_hwframe+0x44/0xa9
-> #1 (sb_internal){.+.+}-{0:0}:
percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
__sb_start_write include/linux/fs.h:1592 [inline]
sb_start_intwrite include/linux/fs.h:1709 [inline]
ext4_evict_inode+0xe6f/0x1940 fs/ext4/inode.c:241
evict+0x2ed/0x6b0 fs/inode.c:577
iput_final fs/inode.c:1653 [inline]
iput.part.0+0x57e/0x810 fs/inode.c:1679
iput fs/inode.c:1669 [inline]
inode_lru_isolate+0x301/0x4f0 fs/inode.c:778
__list_lru_walk_one+0x178/0x5c0 mm/list_lru.c:222
list_lru_walk_one+0x99/0xd0 mm/list_lru.c:266
list_lru_shrink_walk include/linux/list_lru.h:195 [inline]
prune_icache_sb+0xdc/0x140 fs/inode.c:803
super_cache_scan+0x38d/0x590 fs/super.c:107
do_shrink_slab+0x3e4/0x9f0 mm/vmscan.c:511
shrink_slab+0x16f/0x5d0 mm/vmscan.c:672
shrink_node_memcgs mm/vmscan.c:2665 [inline]
shrink_node+0x8cc/0x1de0 mm/vmscan.c:2780
kswapd_shrink_node mm/vmscan.c:3523 [inline]
balance_pgdat+0x745/0x1270 mm/vmscan.c:3681
kswapd+0x5b1/0xdb0 mm/vmscan.c:3938
kthread+0x3b1/0x4a0 kernel/kthread.c:292
ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:296
-> #0 (fs_reclaim){+.+.}-{0:0}:
check_prev_add kernel/locking/lockdep.c:2868 [inline]
check_prevs_add kernel/locking/lockdep.c:2993 [inline]
validate_chain kernel/locking/lockdep.c:3608 [inline]
__lock_acquire+0x2b26/0x54f0 kernel/locking/lockdep.c:4832
lock_acquire kernel/locking/lockdep.c:5442 [inline]
lock_acquire+0x1a8/0x720 kernel/locking/lockdep.c:5407
__fs_reclaim_acquire mm/page_alloc.c:4326 [inline]
fs_reclaim_acquire+0x117/0x150 mm/page_alloc.c:4340
might_alloc include/linux/sched/mm.h:193 [inline]
slab_pre_alloc_hook mm/slab.h:493 [inline]
slab_alloc_node mm/slub.c:2817 [inline]
__kmalloc_node+0x5f/0x430 mm/slub.c:4015
kmalloc_node include/linux/slab.h:575 [inline]
kvmalloc_node+0x61/0xf0 mm/util.c:587
kvmalloc include/linux/mm.h:781 [inline]
