Ilya, so, what is the best action plan now? should I continue using the kernel
that you've sent me? I am running production infrastructure and not sure if
this is the right way forward.
Do you have a patch by any chance against the LTS kernel that I can use to
recompile the ceph module?
Thanks
----- Original Message -----
> From: "Ilya Dryomov" <ilya.dryo...@inktank.com>
> To: "Andrei Mikhailovsky" <and...@arhont.com>
> Cc: "ceph-users" <ceph-users@lists.ceph.com>
> Sent: Saturday, 29 November, 2014 8:45:54 AM
> Subject: Re: [ceph-users] Giant + nfs over cephfs hang tasks
> On Sat, Nov 29, 2014 at 2:13 AM, Andrei Mikhailovsky
> <and...@arhont.com> wrote:
> > Ilya, here is what I got shortly after starting the dd test:
> >
> >
> >
> > [ 288.307993]
> > [ 288.308004]
> > =========================================================
> > [ 288.308008] [ INFO: possible irq lock inversion dependency
> > detected ]
> > [ 288.308014] 3.18.0-rc6-ceph-00024-g72ca172 #1 Tainted: G E
> > [ 288.308019]
> > ---------------------------------------------------------
> > [ 288.308023] kswapd1/87 just changed the state of lock:
> > [ 288.308027] (&xfs_dir_ilock_class){++++-+}, at:
> > [<ffffffffa0682d44>]
> > xfs_ilock+0x134/0x160 [xfs]
> > [ 288.308072] but this lock took another, RECLAIM_FS-unsafe lock in
> > the
> > past:
> > [ 288.308076] (&mm->mmap_sem){++++++}
> > [ 288.308076]
> > [ 288.308076] and interrupts could create inverse lock ordering
> > between
> > them.
> > [ 288.308076]
> > [ 288.308084]
> > [ 288.308084] other info that might help us debug this:
> > [ 288.308089] Possible interrupt unsafe locking scenario:
> > [ 288.308089]
> > [ 288.308094] CPU0 CPU1
> > [ 288.308097] ---- ----
> > [ 288.308100] lock(&mm->mmap_sem);
> > [ 288.308104] local_irq_disable();
> > [ 288.308109] lock(&xfs_dir_ilock_class);
> > [ 288.308114] lock(&mm->mmap_sem);
> > [ 288.308120] <Interrupt>
> > [ 288.308122] lock(&xfs_dir_ilock_class);
> > [ 288.308127]
> > [ 288.308127] *** DEADLOCK ***
> > [ 288.308127]
> > [ 288.308133] 3 locks held by kswapd1/87:
> > [ 288.308136] #0: (shrinker_rwsem){++++..}, at:
> > [<ffffffff8117551f>]
> > shrink_slab+0x3f/0x140
> > [ 288.308151] #1: (&type->s_umount_key#27){++++.+}, at:
> > [<ffffffff811d8c14>] grab_super_passive+0x44/0x90
> > [ 288.308165] #2: (&pag->pag_ici_reclaim_lock){+.+...}, at:
> > [<ffffffffa067acd4>] xfs_reclaim_inodes_ag+0xb4/0x400 [xfs]
> > [ 288.308192]
> > [ 288.308192] the shortest dependencies between 2nd lock and 1st
> > lock:
> > [ 288.308206] -> (&mm->mmap_sem){++++++} ops: 27039227 {
> > [ 288.308214] HARDIRQ-ON-W at:
> > [ 288.308218] [<ffffffff810a7209>]
> > __lock_acquire+0x629/0x1c90
> > [ 288.308229] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.308236] [<ffffffff8173ae99>]
> > down_write+0x49/0x80
> > [ 288.308244] [<ffffffff811dcd03>]
> > do_execve_common.isra.25+0x283/0x6e0
> > [ 288.308253] [<ffffffff811dd178>]
> > do_execve+0x18/0x20
> > [ 288.308259] [<ffffffff8106ff4e>]
> > ____call_usermodehelper+0x11e/0x170
> > [ 288.308269] [<ffffffff8173d66c>]
> > ret_from_fork+0x7c/0xb0
> > [ 288.308276] HARDIRQ-ON-R at:
> > [ 288.308280] [<ffffffff810a6f23>]
> > __lock_acquire+0x343/0x1c90
> > [ 288.308287] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.308294] [<ffffffff8118d833>]
> > might_fault+0x93/0xc0
> > [ 288.308304] [<ffffffff813b7a80>]
> > __clear_user+0x20/0x70
> > [ 288.308314] [<ffffffff813b7afe>]
> > clear_user+0x2e/0x40
> > [ 288.308320] [<ffffffff8122a4cd>] padzero+0x2d/0x40
> > [ 288.308329] [<ffffffff8122b0bf>]
> > load_elf_binary+0x9cf/0x1880
> > [ 288.308336] [<ffffffff811db9f0>]
> > search_binary_handler+0xa0/0x1e0
> > [ 288.308343] [<ffffffff811dcfa2>]
> > do_execve_common.isra.25+0x522/0x6e0
> > [ 288.308351] [<ffffffff811dd178>]
> > do_execve+0x18/0x20
> > [ 288.308358] [<ffffffff8106ff4e>]
> > ____call_usermodehelper+0x11e/0x170
> > [ 288.308366] [<ffffffff8173d66c>]
> > ret_from_fork+0x7c/0xb0
> > [ 288.308373] SOFTIRQ-ON-W at:
> > [ 288.308376] [<ffffffff810a6f54>]
> > __lock_acquire+0x374/0x1c90
> > [ 288.308384] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.308391] [<ffffffff8173ae99>]
> > down_write+0x49/0x80
> > [ 288.308398] [<ffffffff811dcd03>]
> > do_execve_common.isra.25+0x283/0x6e0
> > [ 288.308406] [<ffffffff811dd178>]
> > do_execve+0x18/0x20
> > [ 288.308412] [<ffffffff8106ff4e>]
> > ____call_usermodehelper+0x11e/0x170
> > [ 288.308420] [<ffffffff8173d66c>]
> > ret_from_fork+0x7c/0xb0
> > [ 288.308427] SOFTIRQ-ON-R at:
> > [ 288.308431] [<ffffffff810a6f54>]
> > __lock_acquire+0x374/0x1c90
> > [ 288.308438] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.308445] [<ffffffff8118d833>]
> > might_fault+0x93/0xc0
> > [ 288.308452] [<ffffffff813b7a80>]
> > __clear_user+0x20/0x70
> > [ 288.308458] [<ffffffff813b7afe>]
> > clear_user+0x2e/0x40
> > [ 288.308464] [<ffffffff8122a4cd>] padzero+0x2d/0x40
> > [ 288.308470] [<ffffffff8122b0bf>]
> > load_elf_binary+0x9cf/0x1880
> > [ 288.308477] [<ffffffff811db9f0>]
> > search_binary_handler+0xa0/0x1e0
> > [ 288.308485] [<ffffffff811dcfa2>]
> > do_execve_common.isra.25+0x522/0x6e0
> > [ 288.308493] [<ffffffff811dd178>]
> > do_execve+0x18/0x20
> > [ 288.308499] [<ffffffff8106ff4e>]
> > ____call_usermodehelper+0x11e/0x170
> > [ 288.308507] [<ffffffff8173d66c>]
> > ret_from_fork+0x7c/0xb0
> > [ 288.308514] RECLAIM_FS-ON-W at:
> > [ 288.308518] [<ffffffff810a96fa>]
> > mark_held_locks+0x6a/0x90
> > [ 288.308526] [<ffffffff810a9e17>]
> > lockdep_trace_alloc+0x67/0xd0
> > [ 288.308534] [<ffffffff8116b394>]
> > __alloc_pages_nodemask+0x84/0xba0
> > [ 288.308544] [<ffffffff811af917>]
> > alloc_pages_current+0xf7/0x170
> > [ 288.308553] [<ffffffff81190729>]
> > __pmd_alloc+0x39/0x160
> > [ 288.308559] [<ffffffff8119a69e>]
> > move_page_tables+0x62e/0x6b0
> > [ 288.308566] [<ffffffff811db1cb>]
> > shift_arg_pages+0xcb/0x1a0
> > [ 288.308574] [<ffffffff811db47b>]
> > setup_arg_pages+0x1db/0x200
> > [ 288.308581] [<ffffffff8122ab45>]
> > load_elf_binary+0x455/0x1880
> > [ 288.308588] [<ffffffff811db9f0>]
> > search_binary_handler+0xa0/0x1e0
> > [ 288.308596] [<ffffffff811dcfa2>]
> > do_execve_common.isra.25+0x522/0x6e0
> > [ 288.308603] [<ffffffff811dd178>]
> > do_execve+0x18/0x20
> > [ 288.308610] [<ffffffff8106ff4e>]
> > ____call_usermodehelper+0x11e/0x170
> > [ 288.308617] [<ffffffff8173d66c>]
> > ret_from_fork+0x7c/0xb0
> > [ 288.308624] RECLAIM_FS-ON-R at:
> > [ 288.308628] [<ffffffff810a96fa>]
> > mark_held_locks+0x6a/0x90
> > [ 288.308635] [<ffffffff810a9e17>]
> > lockdep_trace_alloc+0x67/0xd0
> > [ 288.308643] [<ffffffff8116b394>]
> > __alloc_pages_nodemask+0x84/0xba0
> > [ 288.308651] [<ffffffff811af917>]
> > alloc_pages_current+0xf7/0x170
> > [ 288.308658] [<ffffffff81166bee>]
> > __get_free_pages+0xe/0x40
> > [ 288.308666] [<ffffffff81166c36>]
> > get_zeroed_page+0x16/0x20
> > [ 288.308673] [<ffffffff8118ff89>]
> > __pud_alloc+0x29/0xb0
> > [ 288.308679] [<ffffffff81191873>]
> > handle_mm_fault+0xec3/0x10b0
> > [ 288.308686] [<ffffffff81047f32>]
> > __do_page_fault+0x1c2/0x580
> > [ 288.308694] [<ffffffff8104844c>]
> > do_page_fault+0xc/0x10
> > [ 288.308700] [<ffffffff8173f4b2>]
> > page_fault+0x22/0x30
> > [ 288.308707] [<ffffffff813b7afe>]
> > clear_user+0x2e/0x40
> > [ 288.308713] [<ffffffff8122a4cd>]
> > padzero+0x2d/0x40
> > [ 288.308719] [<ffffffff8122b0bf>]
> > load_elf_binary+0x9cf/0x1880
> > [ 288.308726] [<ffffffff811db9f0>]
> > search_binary_handler+0xa0/0x1e0
> > [ 288.308734] [<ffffffff811dcfa2>]
> > do_execve_common.isra.25+0x522/0x6e0
> > [ 288.308741] [<ffffffff811dd178>]
> > do_execve+0x18/0x20
> > [ 288.308748] [<ffffffff8106ff4e>]
> > ____call_usermodehelper+0x11e/0x170
> > [ 288.308755] [<ffffffff8173d66c>]
> > ret_from_fork+0x7c/0xb0
> > [ 288.308762] INITIAL USE at:
> > [ 288.308766] [<ffffffff810a6ff1>]
> > __lock_acquire+0x411/0x1c90
> > [ 288.308773] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.308779] [<ffffffff8173ae99>]
> > down_write+0x49/0x80
> > [ 288.308785] [<ffffffff811dcd03>]
> > do_execve_common.isra.25+0x283/0x6e0
> > [ 288.308793] [<ffffffff811dd178>] do_execve+0x18/0x20
> > [ 288.308800] [<ffffffff8106ff4e>]
> > ____call_usermodehelper+0x11e/0x170
> > [ 288.308807] [<ffffffff8173d66c>]
> > ret_from_fork+0x7c/0xb0
> > [ 288.308814] }
> > [ 288.308816] ... key at: [<ffffffff81eaa388>] __key.49413+0x0/0x8
> > [ 288.308824] ... acquired at:
> > [ 288.308827] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
> > [ 288.308833] [<ffffffff8118d833>] might_fault+0x93/0xc0
> > [ 288.308839] [<ffffffff811e92b4>] filldir+0xa4/0x120
> > [ 288.308845] [<ffffffffa0674308>]
> > xfs_dir2_leaf_getdents+0x2d8/0x370
> > [xfs]
> > [ 288.308866] [<ffffffffa0674556>] xfs_readdir+0x1b6/0x1c0 [xfs]
> > [ 288.308884] [<ffffffffa067606b>] xfs_file_readdir+0x2b/0x30 [xfs]
> > [ 288.308902] [<ffffffff811e93ee>] iterate_dir+0xbe/0x150
> > [ 288.308908] [<ffffffff811e959c>] SyS_getdents+0x8c/0x110
> > [ 288.308914] [<ffffffff8173d716>] system_call_fastpath+0x16/0x1b
> > [ 288.308920]
> > [ 288.308922] -> (&xfs_dir_ilock_class){++++-+} ops: 233164 {
> > [ 288.308930] HARDIRQ-ON-W at:
> > [ 288.308933] [<ffffffff810a7209>]
> > __lock_acquire+0x629/0x1c90
> > [ 288.308941] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.308947] [<ffffffff810a33ef>]
> > down_write_nested+0x4f/0x80
> > [ 288.308955] [<ffffffffa0682d44>]
> > xfs_ilock+0x134/0x160 [xfs]
> > [ 288.308976] [<ffffffffa0685fcd>]
> > xfs_inactive_ifree+0x8d/0x230 [xfs]
> > [ 288.308997] [<ffffffffa0686229>]
> > xfs_inactive+0xb9/0x170 [xfs]
> > [ 288.309017] [<ffffffffa068c0d8>]
> > xfs_fs_evict_inode+0xa8/0x110 [xfs]
> > [ 288.309038] [<ffffffff811f1ed8>] evict+0xb8/0x190
> > [ 288.309045] [<ffffffff811f2815>] iput+0x105/0x190
> > [ 288.309050] [<ffffffff811ecd58>] d_delete+0x168/0x1a0
> > [ 288.309057] [<ffffffff811e0ec5>] vfs_rmdir+0xf5/0x170
> > [ 288.309063] [<ffffffff811e66eb>] do_rmdir+0x1cb/0x1e0
> > [ 288.309069] [<ffffffff811e6ee6>] SyS_rmdir+0x16/0x20
> > [ 288.309075] [<ffffffff8173d716>]
> > system_call_fastpath+0x16/0x1b
> > [ 288.309083] HARDIRQ-ON-R at:
> > [ 288.309086] [<ffffffff810a6f23>]
> > __lock_acquire+0x343/0x1c90
> > [ 288.309093] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.309100] [<ffffffff810a3387>]
> > down_read_nested+0x57/0x70
> > [ 288.309107] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160
> > [xfs]
> > [ 288.309126] [<ffffffffa0682df5>]
> > xfs_ilock_data_map_shared+0x35/0x40 [xfs]
> > [ 288.309145] [<ffffffffa0683fe2>]
> > xfs_lookup+0x62/0x160 [xfs]
> > [ 288.309164] [<ffffffffa068095b>]
> > xfs_vn_lookup+0x5b/0xa0 [xfs]
> > [ 288.309183] [<ffffffff811df12d>]
> > lookup_real+0x1d/0x60
> > [ 288.309189] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
> > [ 288.309195] [<ffffffff811e4c44>]
> > path_openat+0xc4/0x660
> > [ 288.309201] [<ffffffff811e6a33>]
> > do_filp_open+0x43/0xa0
> > [ 288.309207] [<ffffffff811d4462>]
> > do_sys_open+0x132/0x220
> > [ 288.309213] [<ffffffff811d456e>] SyS_open+0x1e/0x20
> > [ 288.309219] [<ffffffff8173d716>]
> > system_call_fastpath+0x16/0x1b
> > [ 288.309226] SOFTIRQ-ON-W at:
> > [ 288.309230] [<ffffffff810a6f54>]
> > __lock_acquire+0x374/0x1c90
> > [ 288.309237] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.309243] [<ffffffff810a33ef>]
> > down_write_nested+0x4f/0x80
> > [ 288.309250] [<ffffffffa0682d44>]
> > xfs_ilock+0x134/0x160 [xfs]
> > [ 288.309268] [<ffffffffa0685fcd>]
> > xfs_inactive_ifree+0x8d/0x230 [xfs]
> > [ 288.309287] [<ffffffffa0686229>]
> > xfs_inactive+0xb9/0x170 [xfs]
> > [ 288.309305] [<ffffffffa068c0d8>]
> > xfs_fs_evict_inode+0xa8/0x110 [xfs]
> > [ 288.309326] [<ffffffff811f1ed8>] evict+0xb8/0x190
> > [ 288.309332] [<ffffffff811f2815>] iput+0x105/0x190
> > [ 288.309338] [<ffffffff811ecd58>] d_delete+0x168/0x1a0
> > [ 288.309344] [<ffffffff811e0ec5>] vfs_rmdir+0xf5/0x170
> > [ 288.309350] [<ffffffff811e66eb>] do_rmdir+0x1cb/0x1e0
> > [ 288.309356] [<ffffffff811e6ee6>] SyS_rmdir+0x16/0x20
> > [ 288.309362] [<ffffffff8173d716>]
> > system_call_fastpath+0x16/0x1b
> > [ 288.309369] SOFTIRQ-ON-R at:
> > [ 288.309373] [<ffffffff810a6f54>]
> > __lock_acquire+0x374/0x1c90
> > [ 288.309380] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.309387] [<ffffffff810a3387>]
> > down_read_nested+0x57/0x70
> > [ 288.309393] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160
> > [xfs]
> > [ 288.309412] [<ffffffffa0682df5>]
> > xfs_ilock_data_map_shared+0x35/0x40 [xfs]
> > [ 288.309431] [<ffffffffa0683fe2>]
> > xfs_lookup+0x62/0x160 [xfs]
> > [ 288.309449] [<ffffffffa068095b>]
> > xfs_vn_lookup+0x5b/0xa0 [xfs]
> > [ 288.309467] [<ffffffff811df12d>]
> > lookup_real+0x1d/0x60
> > [ 288.309473] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
> > [ 288.309479] [<ffffffff811e4c44>]
> > path_openat+0xc4/0x660
> > [ 288.309485] [<ffffffff811e6a33>]
> > do_filp_open+0x43/0xa0
> > [ 288.309491] [<ffffffff811d4462>]
> > do_sys_open+0x132/0x220
> > [ 288.309497] [<ffffffff811d456e>] SyS_open+0x1e/0x20
> > [ 288.309502] [<ffffffff8173d716>]
> > system_call_fastpath+0x16/0x1b
> > [ 288.309510] IN-RECLAIM_FS-W at:
> > [ 288.309513] [<ffffffff810a6fb0>]
> > __lock_acquire+0x3d0/0x1c90
> > [ 288.309521] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.309528] [<ffffffff810a33ef>]
> > down_write_nested+0x4f/0x80
> > [ 288.309535] [<ffffffffa0682d44>]
> > xfs_ilock+0x134/0x160 [xfs]
> > [ 288.309552] [<ffffffffa067aa0c>]
> > xfs_reclaim_inode+0x12c/0x340 [xfs]
> > [ 288.309572] [<ffffffffa067aea7>]
> > xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
> > [ 288.309592] [<ffffffffa067bda3>]
> > xfs_reclaim_inodes_nr+0x33/0x40 [xfs]
> > [ 288.309611] [<ffffffffa068b855>]
> > xfs_fs_free_cached_objects+0x15/0x20 [xfs]
> > [ 288.309632] [<ffffffff811d8dd8>]
> > super_cache_scan+0x178/0x180
> > [ 288.309640] [<ffffffff8117393e>]
> > shrink_slab_node+0x15e/0x310
> > [ 288.309647] [<ffffffff811755e0>]
> > shrink_slab+0x100/0x140
> > [ 288.309654] [<ffffffff81178306>]
> > kswapd_shrink_zone+0x116/0x1a0
> > [ 288.309661] [<ffffffff8117925b>]
> > kswapd+0x4bb/0x9a0
> > [ 288.309667] [<ffffffff8107a664>]
> > kthread+0xe4/0x100
> > [ 288.309674] [<ffffffff8173d66c>]
> > ret_from_fork+0x7c/0xb0
> > [ 288.309682] RECLAIM_FS-ON-R at:
> > [ 288.309685] [<ffffffff810a96fa>]
> > mark_held_locks+0x6a/0x90
> > [ 288.309693] [<ffffffff810a9e17>]
> > lockdep_trace_alloc+0x67/0xd0
> > [ 288.309700] [<ffffffff8116b394>]
> > __alloc_pages_nodemask+0x84/0xba0
> > [ 288.309708] [<ffffffff811b1503>]
> > alloc_pages_vma+0xf3/0x1c0
> > [ 288.309715] [<ffffffff811916ce>]
> > handle_mm_fault+0xd1e/0x10b0
> > [ 288.309722] [<ffffffff81047f32>]
> > __do_page_fault+0x1c2/0x580
> > [ 288.309729] [<ffffffff8104844c>]
> > do_page_fault+0xc/0x10
> > [ 288.309735] [<ffffffff8173f4b2>]
> > page_fault+0x22/0x30
> > [ 288.309741] [<ffffffffa0674308>]
> > xfs_dir2_leaf_getdents+0x2d8/0x370 [xfs]
> > [ 288.309761] [<ffffffffa0674556>]
> > xfs_readdir+0x1b6/0x1c0 [xfs]
> > [ 288.309780] [<ffffffffa067606b>]
> > xfs_file_readdir+0x2b/0x30 [xfs]
> > [ 288.309799] [<ffffffff811e93ee>]
> > iterate_dir+0xbe/0x150
> > [ 288.309806] [<ffffffff811e959c>]
> > SyS_getdents+0x8c/0x110
> > [ 288.309813] [<ffffffff8173d716>]
> > system_call_fastpath+0x16/0x1b
> > [ 288.309821] INITIAL USE at:
> > [ 288.309824] [<ffffffff810a6ff1>]
> > __lock_acquire+0x411/0x1c90
> > [ 288.309831] [<ffffffff810a8e9e>]
> > lock_acquire+0x9e/0x140
> > [ 288.309837] [<ffffffff810a3387>]
> > down_read_nested+0x57/0x70
> > [ 288.309844] [<ffffffffa0682d0a>] xfs_ilock+0xfa/0x160
> > [xfs]
> > [ 288.309863] [<ffffffffa0682df5>]
> > xfs_ilock_data_map_shared+0x35/0x40 [xfs]
> > [ 288.309882] [<ffffffffa0683fe2>] xfs_lookup+0x62/0x160
> > [xfs]
> > [ 288.309900] [<ffffffffa068095b>]
> > xfs_vn_lookup+0x5b/0xa0 [xfs]
> > [ 288.309919] [<ffffffff811df12d>] lookup_real+0x1d/0x60
> > [ 288.309924] [<ffffffff811e2d38>] do_last+0x7a8/0xf80
> > [ 288.309930] [<ffffffff811e4c44>]
> > path_openat+0xc4/0x660
> > [ 288.309936] [<ffffffff811e6a33>]
> > do_filp_open+0x43/0xa0
> > [ 288.309941] [<ffffffff811d4462>]
> > do_sys_open+0x132/0x220
> > [ 288.309947] [<ffffffff811d456e>] SyS_open+0x1e/0x20
> > [ 288.309953] [<ffffffff8173d716>]
> > system_call_fastpath+0x16/0x1b
> > [ 288.309960] }
> > [ 288.309962] ... key at: [<ffffffffa06decf4>]
> > xfs_dir_ilock_class+0x0/0xfffffffffffd130c [xfs]
> > [ 288.309983] ... acquired at:
> > [ 288.309985] [<ffffffff810a607b>] check_usage_forwards+0x10b/0x150
> > [ 288.309992] [<ffffffff810a6a8b>] mark_lock+0x18b/0x2e0
> > [ 288.309997] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
> > [ 288.310003] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
> > [ 288.310009] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
> > [ 288.310015] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
> > [ 288.310032] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340
> > [xfs]
> > [ 288.310050] [<ffffffffa067aea7>]
> > xfs_reclaim_inodes_ag+0x287/0x400
> > [xfs]
> > [ 288.310068] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40
> > [xfs]
> > [ 288.310086] [<ffffffffa068b855>]
> > xfs_fs_free_cached_objects+0x15/0x20
> > [xfs]
> > [ 288.310107] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
> > [ 288.310113] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
> > [ 288.310118] [<ffffffff811755e0>] shrink_slab+0x100/0x140
> > [ 288.310124] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
> > [ 288.310130] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
> > [ 288.310135] [<ffffffff8107a664>] kthread+0xe4/0x100
> > [ 288.310141] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
> > [ 288.310147]
> > [ 288.310149]
> > [ 288.310149] stack backtrace:
> > [ 288.310156] CPU: 8 PID: 87 Comm: kswapd1 Tainted: G E
> > 3.18.0-rc6-ceph-00024-g72ca172 #1
> > [ 288.310162] Hardware name: Supermicro
> > X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014
> > [ 288.310169] ffffffff821208e0 ffff8804676ab608 ffffffff81733b38
> > 0000000000000007
> > [ 288.310177] ffff8804676ab670 ffff8804676ab658 ffffffff810a5f68
> > ffffffff821208e0
> > [ 288.310184] ffffffff81a7cbe0 ffff8804676ab674 0000000000000000
> > ffff88046763cc50
> > [ 288.310192] Call Trace:
> > [ 288.310200] [<ffffffff81733b38>] dump_stack+0x4e/0x68
> > [ 288.310206] [<ffffffff810a5f68>]
> > print_irq_inversion_bug.part.41+0x1e8/0x1f0
> > [ 288.310213] [<ffffffff810a607b>] check_usage_forwards+0x10b/0x150
> > [ 288.310220] [<ffffffff810a6a8b>] mark_lock+0x18b/0x2e0
> > [ 288.310226] [<ffffffff810a5f70>] ?
> > print_irq_inversion_bug.part.41+0x1f0/0x1f0
> > [ 288.310234] [<ffffffff811c9185>] ?
> > __mem_cgroup_threshold+0x5/0x1d0
> > [ 288.310241] [<ffffffff810a6fb0>] __lock_acquire+0x3d0/0x1c90
> > [ 288.310247] [<ffffffff810a6ff1>] ? __lock_acquire+0x411/0x1c90
> > [ 288.310266] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
> > [ 288.310272] [<ffffffff810a8e9e>] lock_acquire+0x9e/0x140
> > [ 288.310289] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
> > [ 288.310295] [<ffffffff810a33ef>] down_write_nested+0x4f/0x80
> > [ 288.310312] [<ffffffffa0682d44>] ? xfs_ilock+0x134/0x160 [xfs]
> > [ 288.310329] [<ffffffffa0682d44>] xfs_ilock+0x134/0x160 [xfs]
> > [ 288.310347] [<ffffffffa067aa0c>] ? xfs_reclaim_inode+0x12c/0x340
> > [xfs]
> > [ 288.310364] [<ffffffffa067aa0c>] xfs_reclaim_inode+0x12c/0x340
> > [xfs]
> > [ 288.310382] [<ffffffffa067aea7>]
> > xfs_reclaim_inodes_ag+0x287/0x400 [xfs]
> > [ 288.310400] [<ffffffffa067ad00>] ?
> > xfs_reclaim_inodes_ag+0xe0/0x400
> > [xfs]
> > [ 288.310418] [<ffffffffa067bda3>] xfs_reclaim_inodes_nr+0x33/0x40
> > [xfs]
> > [ 288.310438] [<ffffffffa068b855>]
> > xfs_fs_free_cached_objects+0x15/0x20
> > [xfs]
> > [ 288.310445] [<ffffffff811d8dd8>] super_cache_scan+0x178/0x180
> > [ 288.310451] [<ffffffff8117393e>] shrink_slab_node+0x15e/0x310
> > [ 288.310457] [<ffffffff811755e0>] shrink_slab+0x100/0x140
> > [ 288.310463] [<ffffffff81178306>] kswapd_shrink_zone+0x116/0x1a0
> > [ 288.310469] [<ffffffff8117925b>] kswapd+0x4bb/0x9a0
> > [ 288.310475] [<ffffffff81178da0>] ?
> > mem_cgroup_shrink_node_zone+0x1c0/0x1c0
> > [ 288.310481] [<ffffffff8107a664>] kthread+0xe4/0x100
> > [ 288.310488] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0
> > [ 288.310494] [<ffffffff8173d66c>] ret_from_fork+0x7c/0xb0
> > [ 288.310500] [<ffffffff8107a580>] ? flush_kthread_worker+0xf0/0xf0
> >
> >
> >
> > I've not seen any hang tasks just yet. The server seems to continue
> > working.
> > I will do more testing and get back to you with more info.
> Yeah, it's not related to rbd.
> Thanks,
> Ilya
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
