Hi all, Since early last summer, we have been running a 2.10.4 filesystem pretty much without incident. Then about 2 weeks ago, it started crashing for no immediately obvious reason. There are no indications of hardware related problems in the logs, and the work loads have not changed significantly as far as we can tell.
I can't rule out hardware or system performance problems, but if that is the case, there are no obvious pointers as to what those would be. We had one workload that seemed to trigger the problem (a couple dozen jobs running du on parts of the filesystem), but that had been running for months, and even after we killed that we had a couple crashes. Since the first crash (on 7 January) we have experience these crashes sporadically. Sometimes days between crashes, other times, hours. The symptoms are the filesystem becoming unresponsive, and a load spike on the MDS and one OSS (we have 8x OSS). The OSS affected seems to be somewhat random. In the system logs, we see hung_task timeouts and stack traces, followed shortly by lustre-log dumps. The only real commonality I have seen is that on the MDS, the first hung task is in jbd2_journal_commit_transaction. To recover the filesystems, I have done e2fsck on the MDT, and any affected OSTs. They have come back cleanly every time. I have attached snippets of the log files at the time of the most recent crash. A high level summary of our system: MDS (1x) & OSS (8x) OS: CentOS Linux release 7.6.1810 (Core) kernel: 3.10.0-862.2.3.el7_lustre.x86_64 Lustre: 2.10.4 (ldiskfs) Clients: a mix, but predominantly CentOS 7.x running 2.10.4 Any insights would be greatly appreciated. There are lots of logs, so if they would be helpful, I can certainly make them available. In particular, that first lustre-log is pretty big, so I just grabbed the lines in closest proximity to the crash. Also, if there's a way to get more debugging level information from lustre, I'm happy to try that as well. And I realize this is all at a very high level, so I'll be happy to provide any additional info needed to help me figure this out. Thanks much for taking the time! Best, ---Steve
Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io00_002:152121 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io00_002 D ffff914c50f4eeb0 0 152121 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task jbd2/dm-10-8:152218 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: jbd2/dm-10-8 D ffff914c44ac2f70 0 152218 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01fb2ac>] jbd2_journal_commit_transaction+0x23c/0x19b0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed1f75>] ? sched_clock_cpu+0x85/0xc0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4e2959e>] ? __switch_to+0xce/0x580 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551294c>] ? __schedule+0x41c/0xa20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ea376e>] ? try_to_del_timer_sync+0x5e/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0201ab9>] kjournald2+0xc9/0x260 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc02019f0>] ? commit_timeout+0x10/0x10 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec9667>] ? finish_task_switch+0x57/0x170 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_006:193341 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_006 D ffff914c509bbf40 0 193341 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_007:193344 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_007 D ffff914b61bdeeb0 0 193344 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_009:193346 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_009 D ffff914b61bdaf70 0 193346 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff5239>] ? ___slab_alloc+0x209/0x4f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_out00_00:205734 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_out00_00 D ffff916453cabf40 0 205734 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5514865>] rwsem_down_write_failed+0x225/0x3a0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb515aa77>] call_rwsem_down_write_failed+0x17/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb55120ed>] down_write+0x2d/0x3d Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d30a8c>] osd_write_lock+0x5c/0xe0 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0aa2119>] out_tx_attr_set_exec+0x69/0x3f0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9c591>] out_tx_end+0xe1/0x5c0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0aa06d3>] out_handle+0x1453/0x1bc0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a30f72>] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a94c69>] ? tgt_request_preprocess.isra.26+0x299/0x790 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io00_009:261208 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io00_009 D ffff9148bb720000 0 261208 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff5239>] ? ___slab_alloc+0x209/0x4f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io00_011:261718 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io00_011 D ffff9148bb723f40 0 261718 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_017:410044 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_017 D ffff9142da8a0fd0 0 410044 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: INFO: task ll_ost_io01_020:3987 blocked for more than 120 seconds. Jan 18 04:27:02 lfs6-oss-4 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:02 lfs6-oss-4 kernel: ll_ost_io01_020 D ffff914ec23e8fd0 0 3987 2 0x00000000 Jan 18 04:27:02 lfs6-oss-4 kernel: Call Trace: Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb5512f79>] schedule+0x29/0x70 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f85e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d358ce>] ? osd_declare_xattr_set+0xee/0x370 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ff73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc01f8a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0c71009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0d3bc8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7327e>] ofd_trans_start+0x6e/0xf0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7972b>] ofd_commitrw_write+0x94b/0x1c90 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0e7d779>] ofd_commitrw+0x4c9/0xae0 [ofd] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0ac7864>] obd_commitrw+0x2f3/0x336 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9a38d>] tgt_brw_write+0xffd/0x17d0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed4ade>] ? account_entity_dequeue+0xae/0xd0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ed3838>] ? __enqueue_entity+0x78/0x80 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4eda2fc>] ? enqueue_entity+0x26c/0xb60 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc09ebf30>] ? target_send_reply_msg+0x170/0x170 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a9638a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3ee4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a3b478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ecee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ec4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a42592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffc0a41b00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebae31>] kthread+0xd1/0xe0 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb551f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:02 lfs6-oss-4 kernel: [<ffffffffb4ebad60>] ? insert_kthread_work+0x40/0x40
Jan 18 04:26:32 lfs6-mds-1 kernel: LNet: Service thread pid 63647 was inactive for 200.55s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 18 04:26:32 lfs6-mds-1 kernel: LNet: 63574:0:(linux-debug.c:185:libcfs_call_trace()) can't show stack: kernel doesn't export show_task Jan 18 04:26:32 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807192.63647 Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: Service thread pid 40323 was inactive for 200.38s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: 63574:0:(linux-debug.c:185:libcfs_call_trace()) can't show stack: kernel doesn't export show_task Jan 18 04:26:37 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807197.40323 Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: Service thread pid 113300 was inactive for 200.42s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: Service thread pid 434500 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 18 04:26:37 lfs6-mds-1 kernel: LNet: Skipped 118 previous similar messages Jan 18 04:26:38 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807198.434571 Jan 18 04:26:38 lfs6-mds-1 kernel: LNet: Service thread pid 434632 was inactive for 200.36s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 18 04:26:38 lfs6-mds-1 kernel: LNet: Skipped 99 previous similar messages Jan 18 04:26:39 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807199.434732 Jan 18 04:26:40 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807200.434844 Jan 18 04:26:40 lfs6-mds-1 kernel: LNet: Service thread pid 63654 was inactive for 200.27s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 18 04:26:40 lfs6-mds-1 kernel: LNet: Skipped 35 previous similar messages Jan 18 04:26:41 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807201.103920 Jan 18 04:26:42 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807202.434796 Jan 18 04:26:43 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807203.434917 Jan 18 04:26:44 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807204.434919 Jan 18 04:26:44 lfs6-mds-1 kernel: LNet: Service thread pid 434927 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 18 04:26:44 lfs6-mds-1 kernel: LNet: Skipped 30 previous similar messages Jan 18 04:26:45 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807205.434934 Jan 18 04:26:46 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807206.434944 Jan 18 04:26:47 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807207.434945 Jan 18 04:26:48 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807208.434948 Jan 18 04:26:49 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807209.434960 Jan 18 04:26:50 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807210.63656 Jan 18 04:26:51 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807211.434967 Jan 18 04:26:52 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807212.434998 Jan 18 04:26:52 lfs6-mds-1 kernel: LNet: Service thread pid 434999 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 18 04:26:52 lfs6-mds-1 kernel: LNet: Skipped 36 previous similar messages Jan 18 04:26:53 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807213.434992 Jan 18 04:26:54 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807214.242587 Jan 18 04:26:55 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807215.434986 Jan 18 04:26:56 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807216.435002 Jan 18 04:26:57 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807217.435020 Jan 18 04:26:58 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807218.435021 Jan 18 04:26:59 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807219.435013 Jan 18 04:27:00 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807220.435036 Jan 18 04:27:01 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807221.435026 Jan 18 04:27:02 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807222.435007 Jan 18 04:27:03 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807223.435041 Jan 18 04:27:04 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807224.434715 Jan 18 04:27:05 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807225.435050 Jan 18 04:27:06 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807226.435055 Jan 18 04:27:07 lfs6-mds-1 kernel: LustreError: dumping log to /tmp/lustre-log.1547807227.435062 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task jbd2/dm-3-8:63269 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: jbd2/dm-3-8 D ffff9d7e5877eeb0 0 63269 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc04102ac>] jbd2_journal_commit_transaction+0x23c/0x19b0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896d4ade>] ? account_entity_dequeue+0xae/0xd0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896d865c>] ? dequeue_entity+0x11c/0x5e0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896d1f75>] ? sched_clock_cpu+0x85/0xc0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff8962959e>] ? __switch_to+0xce/0x580 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1294c>] ? __schedule+0x41c/0xa20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896a376e>] ? try_to_del_timer_sync+0x5e/0x90 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0416ab9>] kjournald2+0xc9/0x260 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc04169f0>] ? commit_timeout+0x10/0x10 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt00_000:63633 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: mdt00_000 D ffff9d7e3ae18fd0 0 63633 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt00_001:63634 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: mdt00_001 D ffff9d7e3ae19fa0 0 63634 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee80>] ? wake_up_state+0x20/0x20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt00_002:63635 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: mdt00_002 D ffff9d7e3ae1af70 0 63635 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt01_000:63636 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: mdt01_000 D ffff9d7e3ae1bf40 0 63636 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt01_001:63637 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: mdt01_001 D ffff9d7e3ae1cf10 0 63637 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt01_002:63638 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: mdt01_002 D ffff9d7e3ae1dee0 0 63638 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt02_000:63639 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: mdt02_000 D ffff9d7e3ae1eeb0 0 63639 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt02_001:63640 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: mdt02_001 D ffff9d7df65f0000 0 63640 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: INFO: task mdt02_002:63641 blocked for more than 120 seconds. Jan 18 04:27:08 lfs6-mds-1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 18 04:27:08 lfs6-mds-1 kernel: mdt02_002 D ffff9d7df65f0fd0 0 63641 2 0x00000000 Jan 18 04:27:08 lfs6-mds-1 kernel: Call Trace: Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0aa20a0>] ? ldlm_pool_add+0x80/0xf0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d12f79>] schedule+0x29/0x70 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d085>] wait_transaction_locked+0x85/0xd0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bbe20>] ? wake_up_atomic_t+0x30/0x30 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d368>] add_transaction_credits+0x268/0x2f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040d5e1>] start_this_handle+0x1a1/0x430 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0dcef5a>] ? osd_declare_write+0x1ea/0x450 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff897f73a2>] ? kmem_cache_alloc+0x1c2/0x1f0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc040da93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ce6009>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0db0c8e>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f59d97>] mdt_empty_transno+0xf7/0x840 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5cece>] mdt_mfd_open+0x8de/0xe70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f35042>] ? mdt_pack_acl2body+0x1b2/0x800 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5d9db>] mdt_finish_open+0x57b/0x690 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f5f2b8>] mdt_reint_open+0x17c8/0x3190 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08a93d1>] ? upcall_cache_get_entry+0x211/0x8f0 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc08ae32e>] ? lu_ucred+0x1e/0x30 [obdclass] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f44a25>] ? mdt_ucred+0x15/0x20 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f452f1>] ? mdt_root_squash+0x21/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f54b33>] mdt_reint_rec+0x83/0x210 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f3636b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f36892>] mdt_intent_reint+0x162/0x430 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0f41671>] mdt_intent_policy+0x441/0xc70 [mdt] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a741db>] ? ldlm_resource_get+0xab/0xa60 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a6d2ba>] ldlm_lock_enqueue+0x38a/0x980 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0a96b53>] ldlm_handle_enqueue0+0x9d3/0x16a0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0abee00>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b1c452>] tgt_enqueue+0x62/0x210 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0b2038a>] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac8e4b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0ac5478>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896cee92>] ? default_wake_function+0x12/0x20 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896c4abb>] ? __wake_up_common+0x5b/0x90 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acc592>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffffc0acbb00>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bae31>] kthread+0xd1/0xe0 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff89d1f5dd>] ret_from_fork_nospec_begin+0x7/0x21 Jan 18 04:27:08 lfs6-mds-1 kernel: [<ffffffff896bad60>] ? insert_kthread_work+0x40/0x40 Jan 18 04:27:09 lfs6-mds-1 kernel: LNet: Service thread pid 435054 was inactive for 200.69s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jan 18 04:27:09 lfs6-mds-1 kernel: LNet: Skipped 34 previous similar messages
lustre-log.1547807192.63647-snippet.gz
Description: GNU Zip compressed data
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org