Re: possible circular locking dependency detected (sb_internal/fs_reclaim)
Hello, -- Původní e-mail -- Od: Nikolay Borisov Komu: Zdenek Sojka , linux-btrfs@vger.kernel.org Datum: 11. 9. 2019 10:16:49 Předmět: Re: possible circular locking dependency detected (sb_internal/fs_reclaim) On 11.09.19 г. 10:54 ч., Zdenek Sojka wrote: > Hello, > > this is my fourth attempt to post this message to the mailing list; this > time, without any attached kernel config (because it has over 100KiB). I also > tried contacting the kernel btrfs maintainers directly by email, but they > probably also didn't receive the message... > > I am running kernel with lock debugging enabled since I am quite often > encountering various lockups and hung tasks. Several of the problems have > been fixed recently, but not all; I don't know if the following backtrace is > related to the hangups, or if it is just a false positive. > > $ uname -a > Linux zso 5.2.11-gentoo #2 SMP Fri Aug 30 07:18:03 CEST 2019 x86_64 Intel(R) > Core(TM) i7-6700 CPU @ 3.40GHz GenuineIntel GNU/Linux > > The kernel has a distro patchset applied (which should not affect this, but > you can never say that for sure) and I am compiling at -O3 -fipa-pta > -march=native instead of default -O2 (gcc-8.3.0). > > Please let me know if I can provide any more information. > > Best regards, > Zdenek Sojka > > The dmesg warning I recently triggered: > This already received a patch in [PATCH] btrfs: nofs inode allocations > It's just not reviewed/merged yet. Care to test that patch if you can > reliably reproduce this? Thank you for the reply, I missed that thread. Applied, and I will start testing. Best regards, Zdenek Sojka
Re: possible circular locking dependency detected (sb_internal/fs_reclaim)
On 11.09.19 г. 10:54 ч., Zdenek Sojka wrote: > Hello, > > this is my fourth attempt to post this message to the mailing list; this > time, without any attached kernel config (because it has over 100KiB). I also > tried contacting the kernel btrfs maintainers directly by email, but they > probably also didn't receive the message... > > I am running kernel with lock debugging enabled since I am quite often > encountering various lockups and hung tasks. Several of the problems have > been fixed recently, but not all; I don't know if the following backtrace is > related to the hangups, or if it is just a false positive. > > $ uname -a > Linux zso 5.2.11-gentoo #2 SMP Fri Aug 30 07:18:03 CEST 2019 x86_64 Intel(R) > Core(TM) i7-6700 CPU @ 3.40GHz GenuineIntel GNU/Linux > > The kernel has a distro patchset applied (which should not affect this, but > you can never say that for sure) and I am compiling at -O3 -fipa-pta > -march=native instead of default -O2 (gcc-8.3.0). > > Please let me know if I can provide any more information. > > Best regards, > Zdenek Sojka > > The dmesg warning I recently triggered: This already received a patch in [PATCH] btrfs: nofs inode allocations It's just not reviewed/merged yet. Care to test that patch if you can reliably reproduce this?
possible circular locking dependency detected (sb_internal/fs_reclaim)
Hello,
this is my fourth attempt to post this message to the mailing list; this time,
without any attached kernel config (because it has over 100KiB). I also tried
contacting the kernel btrfs maintainers directly by email, but they probably
also didn't receive the message...
I am running kernel with lock debugging enabled since I am quite often
encountering various lockups and hung tasks. Several of the problems have been
fixed recently, but not all; I don't know if the following backtrace is related
to the hangups, or if it is just a false positive.
$ uname -a
Linux zso 5.2.11-gentoo #2 SMP Fri Aug 30 07:18:03 CEST 2019 x86_64 Intel(R)
Core(TM) i7-6700 CPU @ 3.40GHz GenuineIntel GNU/Linux
The kernel has a distro patchset applied (which should not affect this, but you
can never say that for sure) and I am compiling at -O3 -fipa-pta -march=native
instead of default -O2 (gcc-8.3.0).
Please let me know if I can provide any more information.
Best regards,
Zdenek Sojka
The dmesg warning I recently triggered:
[30560.303721] ==
[30560.303722] WARNING: possible circular locking dependency detected
[30560.303723] 5.2.11-gentoo #2 Not tainted
[30560.303724] --
[30560.303725] kswapd0/711 is trying to acquire lock:
[30560.303726] a663 (sb_internal){.+.+}, at:
start_transaction+0x3a8/0x500
[30560.303731]
but task is already holding lock:
[30560.303732] 0ba86300 (fs_reclaim){+.+.}, at:
__fs_reclaim_acquire+0x0/0x30
[30560.303735]
which lock already depends on the new lock.
[30560.303736]
the existing dependency chain (in reverse order) is:
[30560.303737]
-> #1 (fs_reclaim){+.+.}:
[30560.303740] kmem_cache_alloc+0x1f/0x1c0
[30560.303742] btrfs_alloc_inode+0x1f/0x260
[30560.303744] alloc_inode+0x16/0xa0
[30560.303745] new_inode+0xe/0xb0
[30560.303747] btrfs_new_inode+0x70/0x610
[30560.303749] btrfs_symlink+0xd0/0x420
[30560.303750] vfs_symlink+0x9c/0x100
[30560.303751] do_symlinkat+0x66/0xe0
[30560.303753] do_syscall_64+0x55/0x1c0
[30560.303756] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[30560.303756]
-> #0 (sb_internal){.+.+}:
[30560.303758] __sb_start_write+0xf6/0x150
[30560.303759] start_transaction+0x3a8/0x500
[30560.303760] btrfs_commit_inode_delayed_inode+0x59/0x110
[30560.303761] btrfs_evict_inode+0x19e/0x4c0
[30560.303763] evict+0xbc/0x1f0
[30560.303763] inode_lru_isolate+0x113/0x190
[30560.303765] __list_lru_walk_one.isra.4+0x5c/0x100
[30560.303766] list_lru_walk_one+0x32/0x50
[30560.303766] prune_icache_sb+0x36/0x80
[30560.303768] super_cache_scan+0x14a/0x1d0
[30560.303769] do_shrink_slab+0x131/0x320
[30560.303770] shrink_node+0xf7/0x380
[30560.303772] balance_pgdat+0x2d5/0x640
[30560.303773] kswapd+0x2ba/0x5e0
[30560.303774] kthread+0x147/0x160
[30560.303775] ret_from_fork+0x24/0x30
[30560.303776]
other info that might help us debug this:
[30560.303776] Possible unsafe locking scenario:
[30560.303777] CPU0 CPU1
[30560.303777]
[30560.303778] lock(fs_reclaim);
[30560.303778] lock(sb_internal);
[30560.303779] lock(fs_reclaim);
[30560.303780] lock(sb_internal);
[30560.303780]
*** DEADLOCK ***
[30560.303781] 3 locks held by kswapd0/711:
[30560.303782] #0: 0ba86300 (fs_reclaim){+.+.}, at:
__fs_reclaim_acquire+0x0/0x30
[30560.303784] #1: 4a5100f8 (shrinker_rwsem){}, at:
shrink_node+0x9a/0x380
[30560.303786] #2: f956fa46 (&type->s_umount_key#30){}, at:
super_cache_scan+0x35/0x1d0
[30560.303788]
stack backtrace:
[30560.303789] CPU: 7 PID: 711 Comm: kswapd0 Not tainted 5.2.11-gentoo #2
[30560.303790] Hardware name: Dell Inc. Precision Tower 3620/0MWYPT, BIOS 2.4.2
09/29/2017
[30560.303791] Call Trace:
[30560.303793] dump_stack+0x85/0xc7
[30560.303795] print_circular_bug.cold.40+0x1d9/0x235
[30560.303796] __lock_acquire+0x18b1/0x1f00
[30560.303797] lock_acquire+0xa6/0x170
[30560.303798] ? start_transaction+0x3a8/0x500
[30560.303799] __sb_start_write+0xf6/0x150
[30560.303800] ? start_transaction+0x3a8/0x500
[30560.303801] start_transaction+0x3a8/0x500
[30560.303802] btrfs_commit_inode_delayed_inode+0x59/0x110
[30560.303804] btrfs_evict_inode+0x19e/0x4c0
[30560.303805] ? var_wake_function+0x20/0x20
[30560.303806] evict+0xbc/0x1f0
[30560.303807] inode_lru_isolate+0x113/0x190
[30560.303808] ? discard_new_inode+0xc0/0xc0
[30560.303809] __list_lru_walk_one.isra.4+0x5c/0x100
[30560.303810] ? discard_new_inode+0xc0/0xc0
[30560.303811] list_lru_walk_one+0x32/0x50
[30560.303812] prune_icache_sb+0x36/0x80
[30560.303813] super_cache_scan+0x14a/0x1d0
[30560.303815] do_shrink_slab+0x131/0x320
[30560.303816] shrink_node+0xf7/0x380
[30560.303817] balance_pgdat+0x2d5/0x640
[30560.303819] kswapd+0x2ba/0x5e0
[30560.303820] ? __wake_up_common_lock+0x90/0x90
[30560.303822] kthread+0x147/0x160
[30560.303823] ? balance_pgdat+0x640/0x640
[30560.303824] ? __kthread_create_on_node+0x160/0x160
[30560.303826] ret_from_fork+0x24/0x30
btrfs/011 possible circular locking dependency detected
btrfs/011 generates possible circular locking dependency warning [1] on
the latest 4.19-rc3, tracing back leads to
1c542f38ab8d mm: Introduce kvcalloc()
[1]
---
[ 233.710021] ==
[ 233.710303] WARNING: possible circular locking dependency detected
[ 233.710574] 4.17.0+ #4 Not tainted
[ 233.710757] --
[ 233.711006] kswapd0/749 is trying to acquire lock:
[ 233.711244] e4661f1f (&delayed_node->mutex){+.+.}, at:
__btrfs_release_delayed_node+0x3a/0x210 [btrfs]
[ 233.711628]
but task is already holding lock:
[ 233.711869] 46873fcd (fs_reclaim){+.+.}, at:
__fs_reclaim_acquire+0x5/0x40
[ 233.712906]
which lock already depends on the new lock.
[ 233.713055]
the existing dependency chain (in reverse order) is:
[ 233.713886]
-> #3 (fs_reclaim){+.+.}:
[ 233.714080]kmem_cache_alloc+0x29/0x240
[ 233.714414]btrfs_alloc_inode+0x24/0x260 [btrfs]
[ 233.715212]alloc_inode+0x1b/0x90
[ 233.716085]new_inode_pseudo+0xc/0x60
[ 233.716503]new_inode+0x12/0x30
[ 233.719126]iget5_locked+0x44/0x80
[ 233.719605]btrfs_iget+0x60/0x6e0 [btrfs]
[ 233.720109]__lookup_free_space_inode+0x102/0x160 [btrfs]
[ 233.720543]lookup_free_space_inode+0x63/0xe0 [btrfs]
[ 233.720885]load_free_space_cache+0x68/0x190 [btrfs]
[ 233.721167]cache_block_group+0x1d1/0x460 [btrfs]
[ 233.721490]find_free_extent+0x10bd/0x1460 [btrfs]
[ 233.721824]btrfs_reserve_extent+0xf6/0x190 [btrfs]
[ 233.722150]btrfs_alloc_tree_block+0x13e/0x520 [btrfs]
[ 233.722483]__btrfs_cow_block+0x128/0x5e0 [btrfs]
[ 233.722807]btrfs_cow_block+0xe9/0x260 [btrfs]
[ 233.723074]commit_cowonly_roots+0x55/0x2b0 [btrfs]
[ 233.723413]btrfs_commit_transaction+0x3b4/0xa20 [btrfs]
[ 233.723760]close_ctree+0x2b1/0x310 [btrfs]
[ 233.724059]generic_shutdown_super+0x67/0x110
[ 233.724361]kill_anon_super+0xe/0x20
[ 233.724663]btrfs_kill_super+0x13/0x100 [btrfs]
[ 233.724973]deactivate_locked_super+0x3a/0x70
[ 233.725289]cleanup_mnt+0x3b/0x70
[ 233.725568]task_work_run+0x89/0xa0
[ 233.725850]exit_to_usermode_loop+0x64/0x9c
[ 233.726151]do_syscall_64+0x12d/0x180
[ 233.726438]entry_SYSCALL_64_after_hwframe+0x49/0xbe
[ 233.726756]
-> #2 (&caching_ctl->mutex){+.+.}:
[ 233.727169]cache_block_group+0x1c6/0x460 [btrfs]
[ 233.727455]find_free_extent+0x10bd/0x1460 [btrfs]
[ 233.727757]btrfs_reserve_extent+0xf6/0x190 [btrfs]
[ 233.728370]btrfs_alloc_tree_block+0x13e/0x520 [btrfs]
[ 233.728850]__btrfs_cow_block+0x128/0x5e0 [btrfs]
[ 233.729364]btrfs_cow_block+0xe9/0x260 [btrfs]
[ 233.729774]btrfs_search_slot+0x1ff/0x9d0 [btrfs]
[ 233.732271]btrfs_insert_empty_items+0x5e/0xb0 [btrfs]
[ 233.732736]btrfs_uuid_tree_add+0x1b6/0x2e0 [btrfs]
[ 233.733089]btrfs_uuid_scan_kthread+0x198/0x330 [btrfs]
[ 233.733418]kthread+0xf8/0x130
[ 233.733698]ret_from_fork+0x3a/0x50
[ 233.733971]
-> #1 (&space_info->groups_sem){}:
[ 233.734429]find_free_extent+0xdac/0x1460 [btrfs]
[ 233.734756]btrfs_reserve_extent+0xf6/0x190 [btrfs]
[ 233.735129]btrfs_alloc_tree_block+0x13e/0x520 [btrfs]
[ 233.735465]split_leaf+0x234/0x6f0 [btrfs]
[ 233.735807]btrfs_search_slot+0x95d/0x9d0 [btrfs]
[ 233.736111]btrfs_insert_empty_items+0x5e/0xb0 [btrfs]
[ 233.736440]btrfs_insert_delayed_items+0x98/0x450 [btrfs]
[ 233.736755]btrfs_async_run_delayed_root+0xed/0x200 [btrfs]
[ 233.737121]normal_work_helper+0x1bd/0x5e0 [btrfs]
[ 233.737492]process_one_work+0x1f4/0x5b0
[ 233.737788]worker_thread+0x46/0x3d0
[ 233.738041]kthread+0xf8/0x130
[ 233.738333]ret_from_fork+0x3a/0x50
[ 233.738629]
-> #0 (&delayed_node->mutex){+.+.}:
[ 233.738991]__mutex_lock+0x6e/0x880
[ 233.739281]__btrfs_release_delayed_node+0x3a/0x210 [btrfs]
[ 233.739592]btrfs_evict_inode+0x232/0x5d0 [btrfs]
[ 233.739893]evict+0xc4/0x190
[ 233.740144]dispose_list+0x35/0x50
[ 233.740395]prune_icache_sb+0x52/0x70
[ 233.740695]super_cache_scan+0x13c/0x190
[ 233.740985]shrink_slab.part.72.constprop.93+0x19b/0x530
[ 233.741372]shrink_node+0x98/0x2e0
[ 233.741619]balance_pgdat+0x167/0x320
[ 233.741896]kswapd+0x189/0x560
[ 233.742149]kthread+0xf8/0x130
[ 233.742422]ret_from_fork+0x3a/0x50
[ 233.742670]
other info that might help us debug this:
[ 233.743146] Chain exists of:
&delayed_node->mutex --> &caching_ctl->mutex --> fs_reclaim
[ 233.743777] P
4.14.35: possible circular locking dependency detected
Hi,
after scrub start, scrub cancel, umount, mount of a two disk raid1
(data + metadata):
[12999.229791] ==
[12999.236029] WARNING: possible circular locking dependency detected
[12999.242261] 4.14.35 #36 Not tainted
[12999.245806] --
[12999.252037] btrfs/4682 is trying to acquire lock:
[12999.256794] ("%s-%s""btrfs", name){+.+.}, at: []
flush_workqueue+0x70/0x480
[12999.265486]
but task is already holding lock:
[12999.271390] (&fs_info->scrub_lock){+.+.}, at: []
btrfs_scrub_dev+0x311/0x650 [btrfs]
[12999.280887]
which lock already depends on the new lock.
[12999.289147]
the existing dependency chain (in reverse order) is:
[12999.296721]
-> #3 (&fs_info->scrub_lock){+.+.}:
[12999.302949]__mutex_lock+0x66/0x9a0
[12999.307287]btrfs_scrub_dev+0x105/0x650 [btrfs]
[12999.312615]btrfs_ioctl+0x19a0/0x2030 [btrfs]
[12999.317706]do_vfs_ioctl+0x8c/0x6a0
[12999.321951]SyS_ioctl+0x6f/0x80
[12999.325860]do_syscall_64+0x64/0x170
[12999.330202]entry_SYSCALL_64_after_hwframe+0x42/0xb7
[12999.335930]
-> #2 (&fs_devs->device_list_mutex){+.+.}:
[12999.342798]__mutex_lock+0x66/0x9a0
[12999.347049]reada_start_machine_worker+0xb0/0x3c0 [btrfs]
[12999.353324]btrfs_worker_helper+0x8b/0x630 [btrfs]
[12999.358926]process_one_work+0x242/0x6a0
[12999.363613]worker_thread+0x32/0x3f0
[12999.367894]kthread+0x11f/0x140
[12999.371792]ret_from_fork+0x3a/0x50
[12999.376038]
-> #1 ((&work->normal_work)){+.+.}:
[12999.382254]process_one_work+0x20c/0x6a0
[12999.386881]worker_thread+0x32/0x3f0
[12999.391216]kthread+0x11f/0x140
[12999.395107]ret_from_fork+0x3a/0x50
[12999.399290]
-> #0 ("%s-%s""btrfs", name){+.+.}:
[12999.405455]lock_acquire+0x93/0x220
[12999.409683]flush_workqueue+0x97/0x480
[12999.414129]drain_workqueue+0xa4/0x180
[12999.418620]destroy_workqueue+0xe/0x1e0
[12999.423202]btrfs_destroy_workqueue+0x57/0x280 [btrfs]
[12999.429137]scrub_workers_put+0x29/0x60 [btrfs]
[12999.434426]btrfs_scrub_dev+0x324/0x650 [btrfs]
[12999.439764]btrfs_ioctl+0x19a0/0x2030 [btrfs]
[12999.444870]do_vfs_ioctl+0x8c/0x6a0
[12999.449115]SyS_ioctl+0x6f/0x80
[12999.453017]do_syscall_64+0x64/0x170
[12999.457367]entry_SYSCALL_64_after_hwframe+0x42/0xb7
[12999.463103]
other info that might help us debug this:
[12999.471370] Chain exists of:
"%s-%s""btrfs", name --> &fs_devs->device_list_mutex -->
&fs_info->scrub_lock
[12999.484396] Possible unsafe locking scenario:
[12999.490524]CPU0CPU1
[12999.495161]
[12999.499840] lock(&fs_info->scrub_lock);
[12999.504001]lock(&fs_devs->device_list_mutex);
[12999.511341]lock(&fs_info->scrub_lock);
[12999.518074] lock("%s-%s""btrfs", name);
[12999.522243]
*** DEADLOCK ***
[12999.528347] 2 locks held by btrfs/4682:
[12999.532330] #0: (sb_writers#15){.+.+}, at: []
mnt_want_write_file+0x33/0xb0
[12999.541310] #1: (&fs_info->scrub_lock){+.+.}, at: []
btrfs_scrub_dev+0x311/0x650 [btrfs]
[12999.551389]
stack backtrace:
[12999.555917] CPU: 3 PID: 4682 Comm: btrfs Not tainted 4.14.35 #36
[12999.562139] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.2a 06/27/2012
[12999.569487] Call Trace:
[12999.572049] dump_stack+0x67/0x95
[12999.575530] print_circular_bug.isra.42+0x1ce/0x1db
[12999.580580] __lock_acquire+0x121c/0x1300
[12999.584758] ? lock_acquire+0x93/0x220
[12999.588668] lock_acquire+0x93/0x220
[12999.592316] ? flush_workqueue+0x70/0x480
[12999.596485] flush_workqueue+0x97/0x480
[12999.600487] ? flush_workqueue+0x70/0x480
[12999.604596] ? find_held_lock+0x2d/0x90
[12999.608541] ? drain_workqueue+0xa4/0x180
[12999.612724] drain_workqueue+0xa4/0x180
[12999.616730] destroy_workqueue+0xe/0x1e0
[12999.620784] btrfs_destroy_workqueue+0x57/0x280 [btrfs]
[12999.626252] scrub_workers_put+0x29/0x60 [btrfs]
[12999.631087] btrfs_scrub_dev+0x324/0x650 [btrfs]
[12999.635814] ? __sb_start_write+0x137/0x1a0
[12999.640156] ? mnt_want_write_file+0x33/0xb0
[12999.644624] btrfs_ioctl+0x19a0/0x2030 [btrfs]
[12999.649194] ? find_held_lock+0x2d/0x90
[12999.653146] ? do_vfs_ioctl+0x8c/0x6a0
[12999.657095] ? btrfs_ioctl_get_supported_features+0x20/0x20 [btrfs]
[12999.663572] do_vfs_ioctl+0x8c/0x6a0
[12999.667203] ? __fget+0x100/0x1f0
[12999.670627] SyS_ioctl+0x6f/0x80
[12999.673954] do_syscall_6
Re: 4.15.0-0.rc2.git2.1.fc28.x86_64: possible circular locking dependency detected
On 12 December 2017 at 06:50, Chris Murphy wrote:
>
> On Mon, Dec 11, 2017 at 10:12 PM, Tomasz Kłoczko
> wrote:
> > Hi,
> >
> > Jut done upgrade on my laptop to the latest Fedora rawhide packages.
> > After reboot and login in dmesg I found logged btrfs issue:
> >
> > [ 94.192907] ======
> > [ 94.192911] WARNING: possible circular locking dependency detected
> > [ 94.192917] 4.15.0-0.rc2.git2.1.fc28.x86_64 #1 Not tainted
>
> I'm not seeing it so far with 4.15.0-0.rc3.git0.1.fc28.x86_64+debug.
[ 90.516496] 4.15.0-0.rc3.git1.1.fc28.x86_64 #1 Not tainted
[ 90.516502] --
[ 90.516509] mysqld/1236 is trying to acquire lock:
[ 90.516515] (&mm->mmap_sem){}, at: [<4969b9d5>]
get_user_pages_unlocked+0x5b/0x1b0
[ 90.516543]
but task is already holding lock:
[ 90.516550] (&ei->dio_sem){}, at: [<0cdc761a>]
btrfs_direct_IO+0x395/0x410 [btrfs]
[ 90.516644]
which lock already depends on the new lock.
[ 90.516650]
the existing dependency chain (in reverse order) is:
[ 90.516654]
-> #5 (&ei->dio_sem){}:
[ 90.516672]down_write+0x4d/0xb0
[ 90.516732]btrfs_log_changed_extents+0x7c/0x680 [btrfs]
[ 90.516807]btrfs_log_inode+0x9c0/0x11a0 [btrfs]
[ 90.516864]btrfs_log_inode_parent+0x2d7/0xaa0 [btrfs]
[ 90.516915]btrfs_log_dentry_safe+0x58/0x80 [btrfs]
[ 90.516973]btrfs_sync_file+0x332/0x4e0 [btrfs]
[ 90.516987]do_fsync+0x38/0x60
[ 90.516997]SyS_fsync+0xc/0x10
[ 90.517008]do_syscall_64+0x66/0x210
[ 90.517020]return_from_SYSCALL_64+0x0/0x75
[ 90.517026]
-> #4 (&ei->log_mutex){+.+.}:
[ 90.517049]__mutex_lock+0x7f/0xa40
[ 90.517126]btrfs_log_inode+0x162/0x11a0 [btrfs]
[ 90.517202]btrfs_log_inode_parent+0x2d7/0xaa0 [btrfs]
[ 90.517277]btrfs_log_dentry_safe+0x58/0x80 [btrfs]
[ 90.517356]btrfs_sync_file+0x332/0x4e0 [btrfs]
[ 90.517367]do_fsync+0x38/0x60
[ 90.517378]SyS_fsync+0xc/0x10
[ 90.517387]do_syscall_64+0x66/0x210
[ 90.517400]return_from_SYSCALL_64+0x0/0x75
[ 90.517406]
-> #3 (sb_internal){.+.+}:
[ 90.517499]free_extent_buffer+0x2f/0x90 [btrfs]
[ 90.517505]
-> #2 ((completion)&async->wait){+.+.}:
[ 90.517529]wait_for_completion+0x63/0x1e0
[ 90.517596]btrfs_async_run_delayed_refs+0x14b/0x170 [btrfs]
[ 90.517668]__btrfs_end_transaction+0x1ee/0x2e0 [btrfs]
[ 90.517739]btrfs_dirty_inode+0x6a/0xd0 [btrfs]
[ 90.517756]file_update_time+0xce/0x110
[ 90.517809]btrfs_page_mkwrite+0xff/0x560 [btrfs]
[ 90.517816]do_page_mkwrite+0x2a/0xc0
[ 90.517822]do_wp_page+0x234/0x5d0
[ 90.517828]__handle_mm_fault+0x658/0x1330
[ 90.517834]handle_mm_fault+0x14c/0x310
[ 90.517840]__do_page_fault+0x28c/0x530
[ 90.517845]do_page_fault+0x32/0x270
[ 90.517853]page_fault+0x22/0x30
[ 90.517856]
-> #1 (sb_pagefaults){.+.+}:
[ 90.517872]__sb_start_write+0x138/0x1b0
[ 90.517915]btrfs_page_mkwrite+0x6a/0x560 [btrfs]
[ 90.517921]do_page_mkwrite+0x2a/0xc0
[ 90.517927]do_wp_page+0x234/0x5d0
[ 90.517932]__handle_mm_fault+0x658/0x1330
[ 90.517938]handle_mm_fault+0x14c/0x310
[ 90.517943]__do_page_fault+0x28c/0x530
[ 90.517949]do_page_fault+0x32/0x270
[ 90.517956]page_fault+0x22/0x30
[ 90.517959]
-> #0 (&mm->mmap_sem){}:
[ 90.517973]lock_acquire+0x9f/0x1f0
[ 90.517979]down_read+0x44/0xa0
[ 90.517984]get_user_pages_unlocked+0x5b/0x1b0
[ 90.517989]get_user_pages_fast+0xcf/0x140
[ 90.517998]iov_iter_get_pages+0xc5/0x2f0
[ 90.518006]do_blockdev_direct_IO+0x193c/0x2950
[ 90.518050]btrfs_direct_IO+0x166/0x410 [btrfs]
[ 90.518059]generic_file_direct_write+0x9d/0x160
[ 90.518104]btrfs_file_write_iter+0x300/0x650 [btrfs]
[ 90.518111]aio_write+0x116/0x1a0
[ 90.518118]do_io_submit+0x42d/0x940
[ 90.518125]entry_SYSCALL_64_fastpath+0x1f/0x96
[ 90.518128]
other info that might help us debug this:
[ 90.518133] Chain exists of:
&mm->mmap_sem --> &ei->log_mutex --> &ei->dio_sem
[ 90.518148] Possible unsafe locking scenario:
[ 90.518152]CPU0CPU1
[ 90.518155]
[ 90.518158] lock(&ei->dio_sem);
[ 90.518164]lock(&ei->log_mutex);
[ 90.518170]
Re: 4.15.0-0.rc2.git2.1.fc28.x86_64: possible circular locking dependency detected
On Mon, Dec 11, 2017 at 10:12 PM, Tomasz Kłoczko wrote: > Hi, > > Jut done upgrade on my laptop to the latest Fedora rawhide packages. > After reboot and login in dmesg I found logged btrfs issue: > > [ 94.192907] == > [ 94.192911] WARNING: possible circular locking dependency detected > [ 94.192917] 4.15.0-0.rc2.git2.1.fc28.x86_64 #1 Not tainted I'm not seeing it so far with 4.15.0-0.rc3.git0.1.fc28.x86_64+debug. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
4.15.0-0.rc2.git2.1.fc28.x86_64: possible circular locking dependency detected
Hi,
Jut done upgrade on my laptop to the latest Fedora rawhide packages.
After reboot and login in dmesg I found logged btrfs issue:
[ 94.192907] ==
[ 94.192911] WARNING: possible circular locking dependency detected
[ 94.192917] 4.15.0-0.rc2.git2.1.fc28.x86_64 #1 Not tainted
[ 94.192921] --
[ 94.192926] mysqld/1225 is trying to acquire lock:
[ 94.192930] (&mm->mmap_sem){}, at: [<65e31c80>]
get_user_pages_unlocked+0x5b/0x1b0
[ 94.192955]
but task is already holding lock:
[ 94.192962] (&ei->dio_sem){}, at: [<08becadb>]
btrfs_direct_IO+0x395/0x410 [btrfs]
[ 94.193059]
which lock already depends on the new lock.
[ 94.193069]
the existing dependency chain (in reverse order) is:
[ 94.193076]
-> #4 (&ei->dio_sem){}:
[ 94.193102]down_write+0x4d/0xb0
[ 94.193183]btrfs_log_changed_extents+0x7c/0x680 [btrfs]
[ 94.193261]btrfs_log_inode+0x9c0/0x11a0 [btrfs]
[ 94.193326]btrfs_log_inode_parent+0x2d7/0xaa0 [btrfs]
[ 94.193395]btrfs_log_dentry_safe+0x58/0x80 [btrfs]
[ 94.193466]btrfs_sync_file+0x332/0x4e0 [btrfs]
[ 94.193477]do_fsync+0x38/0x60
[ 94.193484]SyS_fsync+0xc/0x10
[ 94.193493]do_syscall_64+0x66/0x210
[ 94.193501]return_from_SYSCALL_64+0x0/0x75
[ 94.193505]
-> #3 (&ei->log_mutex){+.+.}:
[ 94.193519]__mutex_lock+0x7f/0xa40
[ 94.193563]btrfs_log_inode+0x162/0x11a0 [btrfs]
[ 94.193604]btrfs_log_inode_parent+0x2d7/0xaa0 [btrfs]
[ 94.193646]btrfs_log_dentry_safe+0x58/0x80 [btrfs]
[ 94.193692]btrfs_sync_file+0x332/0x4e0 [btrfs]
[ 94.193698]do_fsync+0x38/0x60
[ 94.193705]SyS_fsync+0xc/0x10
[ 94.193711]do_syscall_64+0x66/0x210
[ 94.193718]return_from_SYSCALL_64+0x0/0x75
[ 94.193721]
-> #2 (sb_internal){.+.+}:
[ 94.193737]debug_check_no_obj_freed+0xde/0x248
[ 94.193740]
-> #1 ((completion)&async->wait){+.+.}:
[ 94.193753]wait_for_completion+0x63/0x1e0
[ 94.193792]btrfs_async_run_delayed_refs+0x14b/0x170 [btrfs]
[ 94.193837]__btrfs_end_transaction+0x1ee/0x2e0 [btrfs]
[ 94.193881]btrfs_dirty_inode+0x6a/0xd0 [btrfs]
[ 94.193891]touch_atime+0xab/0xd0
[ 94.193934]btrfs_file_mmap+0x3f/0x60 [btrfs]
[ 94.193942]mmap_region+0x3ac/0x5e0
[ 94.193949]do_mmap+0x2e1/0x450
[ 94.193959]vm_mmap_pgoff+0xd5/0x120
[ 94.193966]SyS_mmap_pgoff+0x1c2/0x250
[ 94.193973]entry_SYSCALL_64_fastpath+0x1f/0x96
[ 94.193976]
-> #0 (&mm->mmap_sem){}:
[ 94.193990]lock_acquire+0x9f/0x1f0
[ 94.193995]down_read+0x44/0xa0
[ 94.194001]get_user_pages_unlocked+0x5b/0x1b0
[ 94.194006]get_user_pages_fast+0xcf/0x140
[ 94.194014]iov_iter_get_pages+0xc5/0x2f0
[ 94.194021]do_blockdev_direct_IO+0x193c/0x2950
[ 94.194065]btrfs_direct_IO+0x166/0x410 [btrfs]
[ 94.194074]generic_file_direct_write+0x9d/0x160
[ 94.194117]btrfs_file_write_iter+0x300/0x650 [btrfs]
[ 94.194125]aio_write+0x116/0x1a0
[ 94.194132]do_io_submit+0x42d/0x940
[ 94.194139]entry_SYSCALL_64_fastpath+0x1f/0x96
[ 94.194142]
other info that might help us debug this:
[ 94.194147] Chain exists of:
&mm->mmap_sem --> &ei->log_mutex --> &ei->dio_sem
[ 94.194162] Possible unsafe locking scenario:
[ 94.194166]CPU0CPU1
[ 94.194169]
[ 94.194172] lock(&ei->dio_sem);
[ 94.194178]lock(&ei->log_mutex);
[ 94.194184]lock(&ei->dio_sem);
[ 94.194190] lock(&mm->mmap_sem);
[ 94.194196]
*** DEADLOCK ***
[ 94.194202] 2 locks held by mysqld/1225:
[ 94.194205] #0: (sb_writers#10){.+.+}, at: [<164669fb>]
aio_write+0x191/0x1a0
[ 94.194220] #1: (&ei->dio_sem){}, at: [<08becadb>]
btrfs_direct_IO+0x395/0x410 [btrfs]
[ 94.194268]
stack backtrace:
[ 94.194276] CPU: 1 PID: 1225 Comm: mysqld Not tainted
4.15.0-0.rc2.git2.1.fc28.x86_64 #1
[ 94.194281] Hardware name: Sony Corporation VPCSB2M9E/VAIO, BIOS
R2087H4 06/15/2012
[ 94.194285] Call Trace:
[ 94.194296] dump_stack+0x85/0xbf
[ 94.194304] print_circular_bug+0x1f5/0x2e0
[ 94.194315] ? copy_trace+0x80/0x80
[ 94.194320] check_prev_add+0x3da/0x780
[ 94.194332] ? __lock_acquire+0x11e9/0x1270
[ 94.194338] __lock_acquire+0x11e9/0x1270
[ 94.194349] lock_acquire+0x9f/0x1f
[4.14.0-0.rc2.git3.1.fc28.x86_64]: possible circular locking dependency detected
Raw oops dump:
[ 173.695170] ==
[ 173.695175] WARNING: possible circular locking dependency detected
[ 173.695181] 4.14.0-0.rc2.git3.1.fc28.x86_64 #1 Tainted: G OE
[ 173.695186] --
[ 173.695190] mysqld/1282 is trying to acquire lock:
[ 173.695195] (&mm->mmap_sem){}, at: []
get_user_pages_unlocked+0x5e/0x1b0
[ 173.695217]
[ 173.695222] (&ei->dio_sem){}, at: []
btrfs_direct_IO+0x39f/0x400 [btrfs]
[ 173.695293]
[ 173.695299]
[ 173.695304]
[ 173.695322]__lock_acquire+0x1107/0x11d0
[ 173.695330]lock_acquire+0xa3/0x1f0
[ 173.695338]down_write+0x51/0xc0
[ 173.695397]btrfs_log_changed_extents+0x89/0x640 [btrfs]
[ 173.695449]btrfs_log_inode+0x9c1/0x11d0 [btrfs]
[ 173.695495]btrfs_log_inode_parent+0x2df/0xad0 [btrfs]
[ 173.695540]btrfs_log_dentry_safe+0x60/0x80 [btrfs]
[ 173.695588]btrfs_sync_file+0x344/0x4f0 [btrfs]
[ 173.695598]vfs_fsync_range+0x4b/0xb0
[ 173.695605]do_fsync+0x3d/0x70
[ 173.695612]SyS_fsync+0x10/0x20
[ 173.695621]do_syscall_64+0x6c/0x1f0
[ 173.695629]return_from_SYSCALL_64+0x0/0x7a
[ 173.695632]
[ 173.695648]__lock_acquire+0x1107/0x11d0
[ 173.695656]lock_acquire+0xa3/0x1f0
[ 173.695662]__mutex_lock+0x7f/0xa40
[ 173.695669]mutex_lock_nested+0x1b/0x20
[ 173.695713]btrfs_log_inode+0x159/0x11d0 [btrfs]
[ 173.695755]btrfs_log_inode_parent+0x2df/0xad0 [btrfs]
[ 173.695797]btrfs_log_dentry_safe+0x60/0x80 [btrfs]
[ 173.695842]btrfs_sync_file+0x344/0x4f0 [btrfs]
[ 173.695850]vfs_fsync_range+0x4b/0xb0
[ 173.695857]do_fsync+0x3d/0x70
[ 173.695864]SyS_fsync+0x10/0x20
[ 173.695870]do_syscall_64+0x6c/0x1f0
[ 173.695878]return_from_SYSCALL_64+0x0/0x7a
[ 173.695882]
[ 173.695898]__sb_start_write+0x12b/0x1a0
[ 173.695943]start_transaction+0x368/0x4d0 [btrfs]
[ 173.695985]btrfs_join_transaction+0x1d/0x20 [btrfs]
[ 173.696027]delayed_ref_async_start+0x67/0xd0 [btrfs]
[ 173.696074]btrfs_worker_helper+0x93/0x610 [btrfs]
[ 173.696078]
[ 173.696094]__lock_acquire+0x1107/0x11d0
[ 173.696102]lock_acquire+0xa3/0x1f0
[ 173.696108]wait_for_completion+0x62/0x1d0
[ 173.696147]btrfs_async_run_delayed_refs+0x163/0x180 [btrfs]
[ 173.696190]__btrfs_end_transaction+0x1f2/0x2e0 [btrfs]
[ 173.696229]btrfs_end_transaction+0x10/0x20 [btrfs]
[ 173.696271]btrfs_dirty_inode+0x71/0xd0 [btrfs]
[ 173.696310]btrfs_update_time+0x81/0xc0 [btrfs]
[ 173.696319]file_update_time+0xce/0x110
[ 173.696362]btrfs_page_mkwrite+0xf8/0x530 [btrfs]
[ 173.696369]do_page_mkwrite+0x2f/0xd0
[ 173.696375]do_wp_page+0x23e/0x6e0
[ 173.696382]__handle_mm_fault+0xb23/0x1310
[ 173.696389]handle_mm_fault+0x14d/0x310
[ 173.696395]__do_page_fault+0x26e/0x510
[ 173.696401]do_page_fault+0x38/0x280
[ 173.696406]page_fault+0x22/0x30
[ 173.696409]
[ 173.696425]__lock_acquire+0x1107/0x11d0
[ 173.696432]lock_acquire+0xa3/0x1f0
[ 173.696440]__sb_start_write+0x12b/0x1a0
[ 173.696481]btrfs_page_mkwrite+0x68/0x530 [btrfs]
[ 173.696487]do_page_mkwrite+0x2f/0xd0
[ 173.696494]do_wp_page+0x23e/0x6e0
[ 173.696501]__handle_mm_fault+0xb23/0x1310
[ 173.696507]handle_mm_fault+0x14d/0x310
[ 173.696513]__do_page_fault+0x26e/0x510
[ 173.696518]do_page_fault+0x38/0x280
[ 173.696523]page_fault+0x22/0x30
[ 173.696526]
[ 173.696541]check_prev_add+0x351/0x700
[ 173.696548]__lock_acquire+0x1107/0x11d0
[ 173.696556]lock_acquire+0xa3/0x1f0
[ 173.696562]down_read+0x48/0xb0
[ 173.696568]get_user_pages_unlocked+0x5e/0x1b0
[ 173.696575]get_user_pages_fast+0x7a/0xc0
[ 173.696583]iov_iter_get_pages+0xc9/0x300
[ 173.696591]do_blockdev_direct_IO+0x192b/0x2940
[ 173.696599]__blockdev_direct_IO+0x2e/0x30
[ 173.696640]btrfs_direct_IO+0x171/0x400 [btrfs]
[ 173.696647]generic_file_direct_write+0xa3/0x160
[ 173.696690]btrfs_file_write_iter+0x2fb/0x610 [btrfs]
[ 173.696695]aio_write+0x116/0x1a0
[ 173.696700]do_io_submit+0x42d/0x940
[ 173.696706]SyS_io_submit+0x10/0x20
[ 173.696714]entry_SYSCALL_64_fastpath+0x1f/0xbe
[ 173.696717]
[ 173.696722] Chain exists of:
[ 173.696738] Possible unsafe locking scenario:
[ 173.696742]CPU0CPU1
[ 173.696745]
[ 173.696748] lock(&ei->dio_sem);
[ 173.696755]lock(&ei->log_mutex);
[ 173.696762]lock(&ei->dio_sem);
Re: [BUG] btrfs dev del causes 'possible circular locking dependency detected'
On Thu, 09 Aug 2012 18:44:12 +0200, Stefan Behrens wrote:
> On Thu, 9 Aug 2012 18:01:57 +0200, David Sterba wrote:
>> On Thu, Aug 09, 2012 at 05:36:54PM +0200, Stefan Behrens wrote:
>>> Aug 9 16:02:21 qvarne kernel: [ 543.479460] -> #2
>>> (&fs_info->cleaner_mutex){+.+...}:
>>> Aug 9 16:02:21 qvarne kernel: [ 543.541341][]
>>> lock_acquire+0x95/0x150
>>> Aug 9 16:02:21 qvarne kernel: [ 543.612269][]
>>> mutex_lock_nested+0x71/0x360
>>> Aug 9 16:02:21 qvarne kernel: [ 543.688396][]
>>> btrfs_commit_super+0x1f/0x100 [btrfs]
>>> Aug 9 16:02:21 qvarne kernel: [ 543.773884][]
>>> close_ctree+0x318/0x360 [btrfs]
>>> Aug 9 16:02:21 qvarne kernel: [ 543.853132][]
>>> btrfs_put_super+0x14/0x20 [btrfs]
>>> Aug 9 16:02:21 qvarne kernel: [ 543.934459][]
>>> generic_shutdown_super+0x5d/0xf0
>>> Aug 9 16:02:21 qvarne kernel: [ 544.014747][]
>>> kill_anon_super+0x11/0x20
>>> Aug 9 16:02:21 qvarne kernel: [ 544.087756][]
>>> btrfs_kill_super+0x15/0x90 [btrfs]
>>> Aug 9 16:02:21 qvarne kernel: [ 544.170124][]
>>> deactivate_locked_super+0x3d/0x70
>>> Aug 9 16:02:21 qvarne kernel: [ 544.251452][]
>>> deactivate_super+0x49/0x70
>>> Aug 9 16:02:21 qvarne kernel: [ 544.325500][]
>>> mntput_no_expire+0xfe/0x160
>>> Aug 9 16:02:21 qvarne kernel: [ 544.400588][]
>>> sys_umount+0x69/0x3a0
>>> Aug 9 16:02:21 qvarne kernel: [ 544.469435][]
>>> system_call_fastpath+0x16/0x1b
>>> Aug 9 16:02:21 qvarne kernel: [ 544.547644]
>>> Aug 9 16:02:21 qvarne kernel: [ 544.547644] -> #1
>>> (&type->s_umount_key#31){+.}:
>>> Aug 9 16:02:21 qvarne kernel: [ 544.608588][]
>>> lock_acquire+0x95/0x150
>>> Aug 9 16:02:21 qvarne kernel: [ 544.679516][]
>>> down_read+0x47/0x8e
>>> Aug 9 16:02:21 qvarne kernel: [ 544.746283][]
>>> writeback_inodes_sb_nr_if_idle+0x38/0x60
>>
>> this matches the "umount vs cleaner vs writeback_idle" deadlock pattern,
>> Miao sent
>> patches some time ago, I've tested them succesfully, but they're not
>> merged upstream.
>>
>> http://thread.gmane.org/gmane.linux.file-systems/63900
>> vfs: re-implement writeback_inodes_sb(_nr)_if_idle() and rename them
>>
>> http://thread.gmane.org/gmane.comp.file-systems.btrfs/17040
>> [PATCH 2/2] Btrfs: flush all the dirty pages if
>> try_to_writeback_inodes_sb_nr() fails
>
> After applying these two patches from Miao Xie, logcheck does not
> complain anymore.
>
> Could we start a second attempt to add these VFS, ext4 and btrfs changes?
>
Sorry to reply late. I'm busy to debug some problems recently.
I have re-sent this patchset today, and hope it will be received as soon as
possible.
Thanks
Miao
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] btrfs dev del causes 'possible circular locking dependency detected'
On Thu, 9 Aug 2012 18:01:57 +0200, David Sterba wrote:
> On Thu, Aug 09, 2012 at 05:36:54PM +0200, Stefan Behrens wrote:
>> Aug 9 16:02:21 qvarne kernel: [ 543.479460] -> #2
>> (&fs_info->cleaner_mutex){+.+...}:
>> Aug 9 16:02:21 qvarne kernel: [ 543.541341][]
>> lock_acquire+0x95/0x150
>> Aug 9 16:02:21 qvarne kernel: [ 543.612269][]
>> mutex_lock_nested+0x71/0x360
>> Aug 9 16:02:21 qvarne kernel: [ 543.688396][]
>> btrfs_commit_super+0x1f/0x100 [btrfs]
>> Aug 9 16:02:21 qvarne kernel: [ 543.773884][]
>> close_ctree+0x318/0x360 [btrfs]
>> Aug 9 16:02:21 qvarne kernel: [ 543.853132][]
>> btrfs_put_super+0x14/0x20 [btrfs]
>> Aug 9 16:02:21 qvarne kernel: [ 543.934459][]
>> generic_shutdown_super+0x5d/0xf0
>> Aug 9 16:02:21 qvarne kernel: [ 544.014747][]
>> kill_anon_super+0x11/0x20
>> Aug 9 16:02:21 qvarne kernel: [ 544.087756][]
>> btrfs_kill_super+0x15/0x90 [btrfs]
>> Aug 9 16:02:21 qvarne kernel: [ 544.170124][]
>> deactivate_locked_super+0x3d/0x70
>> Aug 9 16:02:21 qvarne kernel: [ 544.251452][]
>> deactivate_super+0x49/0x70
>> Aug 9 16:02:21 qvarne kernel: [ 544.325500][]
>> mntput_no_expire+0xfe/0x160
>> Aug 9 16:02:21 qvarne kernel: [ 544.400588][]
>> sys_umount+0x69/0x3a0
>> Aug 9 16:02:21 qvarne kernel: [ 544.469435][]
>> system_call_fastpath+0x16/0x1b
>> Aug 9 16:02:21 qvarne kernel: [ 544.547644]
>> Aug 9 16:02:21 qvarne kernel: [ 544.547644] -> #1
>> (&type->s_umount_key#31){+.}:
>> Aug 9 16:02:21 qvarne kernel: [ 544.608588][]
>> lock_acquire+0x95/0x150
>> Aug 9 16:02:21 qvarne kernel: [ 544.679516][]
>> down_read+0x47/0x8e
>> Aug 9 16:02:21 qvarne kernel: [ 544.746283][]
>> writeback_inodes_sb_nr_if_idle+0x38/0x60
>
> this matches the "umount vs cleaner vs writeback_idle" deadlock pattern, Miao
> sent
> patches some time ago, I've tested them succesfully, but they're not
> merged upstream.
>
> http://thread.gmane.org/gmane.linux.file-systems/63900
> vfs: re-implement writeback_inodes_sb(_nr)_if_idle() and rename them
>
> http://thread.gmane.org/gmane.comp.file-systems.btrfs/17040
> [PATCH 2/2] Btrfs: flush all the dirty pages if
> try_to_writeback_inodes_sb_nr() fails
After applying these two patches from Miao Xie, logcheck does not
complain anymore.
Could we start a second attempt to add these VFS, ext4 and btrfs changes?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [BUG] btrfs dev del causes 'possible circular locking dependency detected'
On Thu, Aug 09, 2012 at 05:36:54PM +0200, Stefan Behrens wrote:
> Aug 9 16:02:21 qvarne kernel: [ 543.479460] -> #2
> (&fs_info->cleaner_mutex){+.+...}:
> Aug 9 16:02:21 qvarne kernel: [ 543.541341][]
> lock_acquire+0x95/0x150
> Aug 9 16:02:21 qvarne kernel: [ 543.612269][]
> mutex_lock_nested+0x71/0x360
> Aug 9 16:02:21 qvarne kernel: [ 543.688396][]
> btrfs_commit_super+0x1f/0x100 [btrfs]
> Aug 9 16:02:21 qvarne kernel: [ 543.773884][]
> close_ctree+0x318/0x360 [btrfs]
> Aug 9 16:02:21 qvarne kernel: [ 543.853132][]
> btrfs_put_super+0x14/0x20 [btrfs]
> Aug 9 16:02:21 qvarne kernel: [ 543.934459][]
> generic_shutdown_super+0x5d/0xf0
> Aug 9 16:02:21 qvarne kernel: [ 544.014747][]
> kill_anon_super+0x11/0x20
> Aug 9 16:02:21 qvarne kernel: [ 544.087756][]
> btrfs_kill_super+0x15/0x90 [btrfs]
> Aug 9 16:02:21 qvarne kernel: [ 544.170124][]
> deactivate_locked_super+0x3d/0x70
> Aug 9 16:02:21 qvarne kernel: [ 544.251452][]
> deactivate_super+0x49/0x70
> Aug 9 16:02:21 qvarne kernel: [ 544.325500][]
> mntput_no_expire+0xfe/0x160
> Aug 9 16:02:21 qvarne kernel: [ 544.400588][]
> sys_umount+0x69/0x3a0
> Aug 9 16:02:21 qvarne kernel: [ 544.469435][]
> system_call_fastpath+0x16/0x1b
> Aug 9 16:02:21 qvarne kernel: [ 544.547644]
> Aug 9 16:02:21 qvarne kernel: [ 544.547644] -> #1
> (&type->s_umount_key#31){+.}:
> Aug 9 16:02:21 qvarne kernel: [ 544.608588][]
> lock_acquire+0x95/0x150
> Aug 9 16:02:21 qvarne kernel: [ 544.679516][]
> down_read+0x47/0x8e
> Aug 9 16:02:21 qvarne kernel: [ 544.746283][]
> writeback_inodes_sb_nr_if_idle+0x38/0x60
this matches the "umount vs cleaner vs writeback_idle" deadlock pattern, Miao
sent
patches some time ago, I've tested them succesfully, but they're not
merged upstream.
http://thread.gmane.org/gmane.linux.file-systems/63900
vfs: re-implement writeback_inodes_sb(_nr)_if_idle() and rename them
http://thread.gmane.org/gmane.comp.file-systems.btrfs/17040
[PATCH 2/2] Btrfs: flush all the dirty pages if try_to_writeback_inodes_sb_nr()
fails
david
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[BUG] btrfs dev del causes 'possible circular locking dependency detected'
With chris' for-linus of today.
The script:
echo /dev/sdo, /dev/sdp and /dev/sdq are 1TB disks
mkfs.btrfs -d raid1 -m raid1 -b1600M /dev/sdo /dev/sdp
mount /dev/sdo /mnt2
echo 'copy kernel source and built kernel objects (much larger than 1600MB):'
(cd ~/git/btrfs; tar cf - .) | (cd /mnt2; tar xf -) 2>/dev/null
echo fails with ENOSPC after some seconds
umount /mnt2
mount /dev/sdo /mnt2
btrfs dev add /dev/sdq /mnt2
btrfs dev del /dev/sdp /mnt2
echo causes the 'INFO: possible circular locking dependency detected'
echo fails with 'ERROR: error removing the device '/dev/sdp' - No space left on
device'
The kernel log:
Aug 9 16:02:12 qvarne kernel: [ 542.335716] btrfs: relocating block group
1677721600 flags 20
Aug 9 16:02:12 qvarne kernel: [ 542.548469] btrfs: relocating block group
1639972864 flags 17
Aug 9 16:02:21 qvarne kernel: [ 542.619398]
Aug 9 16:02:21 qvarne kernel: [ 542.637165]
==
Aug 9 16:02:21 qvarne kernel: [ 542.711109] [ INFO: possible circular locking
dependency detected ]
Aug 9 16:02:21 qvarne kernel: [ 542.786094] 3.5.0+ #84 Not tainted
Aug 9 16:02:21 qvarne kernel: [ 542.826757]
---
Aug 9 16:02:21 qvarne kernel: [ 542.901739] btrfs/3790 is trying to acquire
lock:
Aug 9 16:02:21 qvarne kernel: [ 542.958005]
(&sb->s_type->i_mutex_key#11){+.+.+.}, at: []
prealloc_file_extent_cluster+0x57/0x150 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 543.092788]
Aug 9 16:02:21 qvarne kernel: [ 543.092788] but task is already holding lock:
Aug 9 16:02:21 qvarne kernel: [ 543.162571]
(&fs_info->cleaner_mutex){+.+...}, at: []
btrfs_relocate_block_group+0x187/0x2c0 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 543.292053]
Aug 9 16:02:21 qvarne kernel: [ 543.292053] which lock already depends on the
new lock.
Aug 9 16:02:21 qvarne kernel: [ 543.292053]
Aug 9 16:02:21 qvarne kernel: [ 543.389916]
Aug 9 16:02:21 qvarne kernel: [ 543.389916] the existing dependency chain (in
reverse order) is:
Aug 9 16:02:21 qvarne kernel: [ 543.479460]
Aug 9 16:02:21 qvarne kernel: [ 543.479460] -> #2
(&fs_info->cleaner_mutex){+.+...}:
Aug 9 16:02:21 qvarne kernel: [ 543.541341][]
lock_acquire+0x95/0x150
Aug 9 16:02:21 qvarne kernel: [ 543.612269][]
mutex_lock_nested+0x71/0x360
Aug 9 16:02:21 qvarne kernel: [ 543.688396][]
btrfs_commit_super+0x1f/0x100 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 543.773884][]
close_ctree+0x318/0x360 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 543.853132][]
btrfs_put_super+0x14/0x20 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 543.934459][]
generic_shutdown_super+0x5d/0xf0
Aug 9 16:02:21 qvarne kernel: [ 544.014747][]
kill_anon_super+0x11/0x20
Aug 9 16:02:21 qvarne kernel: [ 544.087756][]
btrfs_kill_super+0x15/0x90 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 544.170124][]
deactivate_locked_super+0x3d/0x70
Aug 9 16:02:21 qvarne kernel: [ 544.251452][]
deactivate_super+0x49/0x70
Aug 9 16:02:21 qvarne kernel: [ 544.325500][]
mntput_no_expire+0xfe/0x160
Aug 9 16:02:21 qvarne kernel: [ 544.400588][]
sys_umount+0x69/0x3a0
Aug 9 16:02:21 qvarne kernel: [ 544.469435][]
system_call_fastpath+0x16/0x1b
Aug 9 16:02:21 qvarne kernel: [ 544.547644]
Aug 9 16:02:21 qvarne kernel: [ 544.547644] -> #1
(&type->s_umount_key#31){+.}:
Aug 9 16:02:21 qvarne kernel: [ 544.608588][]
lock_acquire+0x95/0x150
Aug 9 16:02:21 qvarne kernel: [ 544.679516][]
down_read+0x47/0x8e
Aug 9 16:02:21 qvarne kernel: [ 544.746283][]
writeback_inodes_sb_nr_if_idle+0x38/0x60
Aug 9 16:02:21 qvarne kernel: [ 544.834891][]
reserve_metadata_bytes+0x4fd/0x800 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 544.925581][]
btrfs_block_rsv_add+0x32/0x60 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 545.011068][]
start_transaction+0x2e4/0x3b0 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 545.096555][]
btrfs_start_transaction+0xe/0x10 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 545.185165][]
btrfs_create+0x41/0x210 [btrfs]
Aug 9 16:02:21 qvarne kernel: [ 545.264412][]
vfs_create+0xac/0x110
Aug 9 16:02:21 qvarne kernel: [ 545.333259][]
do_last+0x769/0x9b0
Aug 9 16:02:21 qvarne kernel: [ 545.400027][]
path_openat+0xd4/0x450
Aug 9 16:02:21 qvarne kernel: [ 545.469915][]
do_filp_open+0x3d/0xa0
Aug 9 16:02:21 qvarne kernel: [ 545.539803][]
do_sys_open+0xf3/0x1d0
Aug 9 16:02:21 qvarne kernel: [ 545.609692][]
sys_open+0x1c/0x20
Aug 9 16:02:21 qvarne kernel: [ 545.675421][]
system_call_fastpath+0x16/0x1b
Aug 9 16:02:21 qvarne kernel: [ 545.753629]
Aug 9 16:02:21 qvarne kernel: [ 545.753629] -> #0
(&sb->s_type->i_mutex_key#11){+.+.+.}:
Aug
Re: "possible circular locking dependency detected"
On Thu, Nov 20, 2008 at 05:00:15PM +0100, Diego Calleja wrote: > I got this lockdep warning while running tiobench on a clean btrfs filesystem > with the latest code available (commit > 2c41b36dd2f9fb5dee150f20c84895496e0642f2) > But it was a purely read-only workload: only root could write to the > filesystem and I was running tiobench as an user, which was spitting > "denied permission" errors. > > Yeah I sent a patch to the list a few weeks ago to fix this, it should get picked up at some point. Josef -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
"possible circular locking dependency detected"
I got this lockdep warning while running tiobench on a clean btrfs filesystem
with the latest code available (commit 2c41b36dd2f9fb5dee150f20c84895496e0642f2)
But it was a purely read-only workload: only root could write to the
filesystem and I was running tiobench as an user, which was spitting
"denied permission" errors.
[ 5177.201587] ===
[ 5177.201594] [ INFO: possible circular locking dependency detected ]
[ 5177.201601] 2.6.28-rc5-dirty #7
[ 5177.201604] ---
[ 5177.201610] btrfs-transacti/9012 is trying to acquire lock:
[ 5177.201615] (&cache->alloc_mutex){--..}, at: []
btrfs_add_free_space+0x27/0x70 [btrfs]
[ 5177.201648]
[ 5177.201650] but task is already holding lock:
[ 5177.201654] (&fs_info->pinned_mutex){--..}, at: []
btrfs_finish_extent_commit+0x33/0x100 [btrfs]
[ 5177.201692]
[ 5177.201693] which lock already depends on the new lock.
[ 5177.201695]
[ 5177.201698]
[ 5177.201699] the existing dependency chain (in reverse order) is:
[ 5177.201704]
[ 5177.201705] -> #1 (&fs_info->pinned_mutex){--..}:
[ 5177.201712][] __lock_acquire+0x13c9/0x1970
[ 5177.201722][] lock_acquire+0x56/0x80
[ 5177.201728][] mutex_lock_nested+0xb6/0x320
[ 5177.201736][] add_new_free_space+0x37/0xe0 [btrfs]
[ 5177.201755][] cache_block_group+0x2b2/0x320 [btrfs]
[ 5177.201774][] find_free_extent+0x2c4/0x640 [btrfs]
[ 5177.201792][] __btrfs_reserve_extent+0xf4/0x310
[btrfs]
[ 5177.201811][] btrfs_alloc_extent+0x4c/0xc0 [btrfs]
[ 5177.201829][] btrfs_alloc_free_block+0x61/0xa0
[btrfs]
[ 5177.201848][] __btrfs_cow_block+0x966/0xe00 [btrfs]
[ 5177.201865][] btrfs_cow_block+0x136/0x330 [btrfs]
[ 5177.201883][] btrfs_search_slot+0x294/0xb70 [btrfs]
[ 5177.201901][] btrfs_lookup_inode+0x38/0xb0 [btrfs]
[ 5177.201920][] btrfs_update_inode+0x4b/0xd0 [btrfs]
[ 5177.201940][] btrfs_dirty_inode+0x4d/0x70 [btrfs]
[ 5177.201961][] __mark_inode_dirty+0x3a/0x1d0
[ 5177.201968][] touch_atime+0xd8/0x140
[ 5177.201975][] vfs_readdir+0xc6/0xd0
[ 5177.201983][] sys_getdents+0x88/0xe0
[ 5177.201989][] system_call_fastpath+0x16/0x1b
[ 5177.201997][] 0x
[ 5177.202024]
[ 5177.202025] -> #0 (&cache->alloc_mutex){--..}:
[ 5177.202033][] __lock_acquire+0x1512/0x1970
[ 5177.202039][] lock_acquire+0x56/0x80
[ 5177.202046][] mutex_lock_nested+0xb6/0x320
[ 5177.202053][] btrfs_add_free_space+0x27/0x70
[btrfs]
[ 5177.202072][]
btrfs_update_pinned_extents+0x172/0x1d0 [btrfs]
[ 5177.202091][]
btrfs_finish_extent_commit+0x72/0x100 [btrfs]
[ 5177.202110][] btrfs_commit_transaction+0x6f5/0x8b0
[btrfs]
[ 5177.202130][] transaction_kthread+0x231/0x270
[btrfs]
[ 5177.202149][] kthread+0x4e/0x90
[ 5177.202157][] child_rip+0xa/0x11
[ 5177.202163][] 0x
[ 5177.202170]
[ 5177.202171] other info that might help us debug this:
[ 5177.202173]
[ 5177.202177] 3 locks held by btrfs-transacti/9012:
[ 5177.202181] #0: (&fs_info->transaction_kthread_mutex){--..}, at:
[] transaction_kthread+0xf3/0x270 [btrfs]
[ 5177.202206] #1: (&fs_info->tree_reloc_mutex){--..}, at:
[] btrfs_commit_transaction+0x346/0x8b0 [btrfs]
[ 5177.202231] #2: (&fs_info->pinned_mutex){--..}, at: []
btrfs_finish_extent_commit+0x33/0x100 [btrfs]
[ 5177.202254]
[ 5177.202255] stack backtrace:
[ 5177.202260] Pid: 9012, comm: btrfs-transacti Not tainted 2.6.28-rc5-dirty #7
[ 5177.202265] Call Trace:
[ 5177.202271] [] print_circular_bug_tail+0xd8/0xe0
[ 5177.202278] [] __lock_acquire+0x1512/0x1970
[ 5177.202285] [] ? find_usage_backwards+0xdc/0x120
[ 5177.202306] [] ? btrfs_add_free_space+0x27/0x70 [btrfs]
[ 5177.202312] [] lock_acquire+0x56/0x80
[ 5177.202329] [] ? btrfs_add_free_space+0x27/0x70 [btrfs]
[ 5177.202337] [] mutex_lock_nested+0xb6/0x320
[ 5177.202354] [] ? btrfs_add_free_space+0x27/0x70 [btrfs]
[ 5177.202372] [] btrfs_add_free_space+0x27/0x70 [btrfs]
[ 5177.202391] [] btrfs_update_pinned_extents+0x172/0x1d0
[btrfs]
[ 5177.202410] [] btrfs_finish_extent_commit+0x72/0x100
[btrfs]
[ 5177.202429] [] btrfs_commit_transaction+0x6f5/0x8b0
[btrfs]
[ 5177.202437] [] ? autoremove_wake_function+0x0/0x40
[ 5177.202438] [] ? mutex_unlock+0xe/0x10
[ 5177.202438] [] transaction_kthread+0x231/0x270 [btrfs]
[ 5177.202438] [] ? trace_hardirqs_on+0xd/0x10
[ 5177.202438] [] ? _spin_unlock_irqrestore+0x47/0x80
[ 5177.202438] [] ? transaction_kthread+0x0/0x270 [btrfs]
[ 5177.202438] [] kthread+0x4e/0x90
[ 5177.202438] [] child_rip+0xa/0x11
[ 5177.202438] [] ? restore_args+0x0/0x30
[ 5177.202438] [] ? kthread+0x0/0x90
[ 5177.202438] [] ? child_rip+0x0/0x11
--
To unsubscribe from this list: send the line "unsub
