Hi, i have a strange issue on CentOS 6.5
If i install a new vm on node1 it works well. If i install a new vm on node2 it gets stuck. Same if i do a dd if=/dev/zero of=/dev/DATEN/vm-test (on node2) On node1 it works: dd if=/dev/zero of=vm-test Schreiben in „vm-test“: Auf dem Gerät ist kein Speicherplatz mehr verfügbar 83886081+0 Datensätze ein 83886080+0 Datensätze aus 42949672960 Bytes (43 GB) kopiert, 2338,15 s, 18,4 MB/s dmesg shows the following (while dd'ing on node2): INFO: task flush-253:18:9820 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-253:18 D 0000000000000000 0 9820 2 0x00000080 ffff8805f9bc3490 0000000000000046 ffff8805f9bc3458 ffff8805f9bc3454 ffff880620768c00 ffff88062fc23080 0000006fc80f5a1e ffff8800282959c0 00000000000002aa 000000010002be11 ffff8806241d65f8 ffff8805f9bc3fd8 Call Trace: [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd] [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd] [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod] [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd] [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60 [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0 [<ffffffff81276544>] ? blk_queue_bio+0x494/0x610 [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod] [<ffffffff81274910>] generic_make_request+0x240/0x5a0 [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff81274ce0>] submit_bio+0x70/0x120 [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0 [<ffffffff811ca0a8>] __block_write_full_page+0x1c8/0x330 [<ffffffff811c9070>] ? end_buffer_async_write+0x0/0x190 [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20 [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20 [<ffffffff811ca2f0>] block_write_full_page_endio+0xe0/0x120 [<ffffffff81126ff0>] ? find_get_pages_tag+0x40/0x130 [<ffffffff811ca345>] block_write_full_page+0x15/0x20 [<ffffffff811cf108>] blkdev_writepage+0x18/0x20 [<ffffffff8113b387>] __writepage+0x17/0x40 [<ffffffff8113c64d>] write_cache_pages+0x1fd/0x4c0 [<ffffffff8113b370>] ? __writepage+0x0/0x40 [<ffffffff8113c934>] generic_writepages+0x24/0x30 [<ffffffff8113c961>] do_writepages+0x21/0x40 [<ffffffff811beb3d>] writeback_single_inode+0xdd/0x290 [<ffffffff811bef3d>] writeback_sb_inodes+0xbd/0x170 [<ffffffff811bf09b>] writeback_inodes_wb+0xab/0x1b0 [<ffffffff811bf493>] wb_writeback+0x2f3/0x410 [<ffffffff8108a752>] ? del_timer_sync+0x22/0x30 [<ffffffff811bf755>] wb_do_writeback+0x1a5/0x240 [<ffffffff811bf853>] bdi_writeback_task+0x63/0x1b0 [<ffffffff810a12e7>] ? bit_waitqueue+0x17/0xd0 [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100 [<ffffffff8114b7c6>] bdi_start_fn+0x86/0x100 [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100 [<ffffffff810a0fce>] kthread+0x9e/0xc0 [<ffffffff8100c28a>] child_rip+0xa/0x20 [<ffffffff810a0f30>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 INFO: task flush-253:18:9820 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-253:18 D 0000000000000000 0 9820 2 0x00000080 ffff8805f9bc3490 0000000000000046 ffff8805f9bc3458 ffff8805f9bc3454 ffff880620768c00 ffff88062fc23080 0000006fc80f5a1e ffff8800282959c0 00000000000002aa 000000010002be11 ffff8806241d65f8 ffff8805f9bc3fd8 Call Trace: [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd] [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd] [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod] [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd] [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60 [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0 [<ffffffff81276544>] ? blk_queue_bio+0x494/0x610 [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod] [<ffffffff81274910>] generic_make_request+0x240/0x5a0 [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff81274ce0>] submit_bio+0x70/0x120 [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0 [<ffffffff811ca0a8>] __block_write_full_page+0x1c8/0x330 [<ffffffff811c9070>] ? end_buffer_async_write+0x0/0x190 [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20 [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20 [<ffffffff811ca2f0>] block_write_full_page_endio+0xe0/0x120 [<ffffffff81126ff0>] ? find_get_pages_tag+0x40/0x130 [<ffffffff811ca345>] block_write_full_page+0x15/0x20 [<ffffffff811cf108>] blkdev_writepage+0x18/0x20 [<ffffffff8113b387>] __writepage+0x17/0x40 [<ffffffff8113c64d>] write_cache_pages+0x1fd/0x4c0 [<ffffffff8113b370>] ? __writepage+0x0/0x40 [<ffffffff8113c934>] generic_writepages+0x24/0x30 [<ffffffff8113c961>] do_writepages+0x21/0x40 [<ffffffff811beb3d>] writeback_single_inode+0xdd/0x290 [<ffffffff811bef3d>] writeback_sb_inodes+0xbd/0x170 [<ffffffff811bf09b>] writeback_inodes_wb+0xab/0x1b0 [<ffffffff811bf493>] wb_writeback+0x2f3/0x410 [<ffffffff8108a752>] ? del_timer_sync+0x22/0x30 [<ffffffff811bf755>] wb_do_writeback+0x1a5/0x240 [<ffffffff811bf853>] bdi_writeback_task+0x63/0x1b0 [<ffffffff810a12e7>] ? bit_waitqueue+0x17/0xd0 [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100 [<ffffffff8114b7c6>] bdi_start_fn+0x86/0x100 [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100 [<ffffffff810a0fce>] kthread+0x9e/0xc0 [<ffffffff8100c28a>] child_rip+0xa/0x20 [<ffffffff810a0f30>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 INFO: task glock_workqueue:7144 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. glock_workque D 0000000000000001 0 7144 2 0x00000080 ffff880602d13740 0000000000000046 0000000000000000 0000000000000000 0000000000000000 ffff880622eb3080 0000009d020d96ca 0000000000000246 ffff880602d13720 000000010005b5aa ffff880622cbbad8 ffff880602d13fd8 Call Trace: [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd] [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd] [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod] [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd] [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60 [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0 [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod] [<ffffffff81274910>] generic_make_request+0x240/0x5a0 [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff81274ce0>] submit_bio+0x70/0x120 [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0 [<ffffffffa04f0ddf>] buf_lo_before_commit+0x10f/0x1c0 [gfs2] [<ffffffffa04ef6c8>] gfs2_log_flush+0x1d8/0x600 [gfs2] [<ffffffff8100969d>] ? __switch_to+0x7d/0x340 [<ffffffffa04ec4bd>] inode_go_sync+0x7d/0x160 [gfs2] [<ffffffffa04eafd6>] do_xmote+0x156/0x280 [gfs2] [<ffffffffa04eb1f1>] run_queue+0xf1/0x1d0 [gfs2] [<ffffffffa04eb9fa>] glock_work_func+0x7a/0x1d0 [gfs2] [<ffffffffa04eb980>] ? glock_work_func+0x0/0x1d0 [gfs2] [<ffffffff8109a780>] worker_thread+0x170/0x2a0 [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 [<ffffffff8109a610>] ? worker_thread+0x0/0x2a0 [<ffffffff810a0fce>] kthread+0x9e/0xc0 [<ffffffff8100c28a>] child_rip+0xa/0x20 [<ffffffff810a0f30>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 INFO: task gfs2_logd:7162 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. gfs2_logd D 0000000000000003 0 7162 2 0x00000080 ffff880600d13d00 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 000000a727fdf37d 0000000000000000 0000000000000000 0000000100066027 ffff88062257a5f8 ffff880600d13fd8 Call Trace: [<ffffffff8153b665>] rwsem_down_failed_common+0x95/0x1d0 [<ffffffff8153b7c3>] rwsem_down_write_failed+0x23/0x30 [<ffffffff8129d6f3>] call_rwsem_down_write_failed+0x13/0x20 [<ffffffff8153acc2>] ? down_write+0x32/0x40 [<ffffffffa04ef51f>] gfs2_log_flush+0x2f/0x600 [gfs2] [<ffffffffa04ee57f>] ? gfs2_ail1_empty+0x2f/0x1b0 [gfs2] [<ffffffffa04efbc9>] gfs2_logd+0xd9/0x140 [gfs2] [<ffffffffa04efaf0>] ? gfs2_logd+0x0/0x140 [gfs2] [<ffffffff810a0fce>] kthread+0x9e/0xc0 [<ffffffff8100c28a>] child_rip+0xa/0x20 [<ffffffff810a0f30>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 INFO: task gfs2_quotad:7163 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. gfs2_quotad D 0000000000000001 0 7163 2 0x00000080 ffff880600d17c20 0000000000000046 0000000000000000 0000000400000018 aa838d160000001c ffffffffa0508110 000000a03da58698 ffffffffa05080c0 ffff880600000005 000000010005eb43 ffff880624105068 ffff880600d17fd8 Call Trace: [<ffffffffa0508110>] ? gdlm_ast+0x0/0x210 [gfs2] [<ffffffffa05080c0>] ? gdlm_bast+0x0/0x50 [gfs2] [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2] [<ffffffffa04e710e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2] [<ffffffff815397ff>] __wait_on_bit+0x5f/0x90 [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2] [<ffffffff815398a8>] out_of_line_wait_on_bit+0x78/0x90 [<ffffffff810a14e0>] ? wake_bit_function+0x0/0x50 [<ffffffffa04e80e5>] gfs2_glock_wait+0x45/0x90 [gfs2] [<ffffffffa04eb619>] gfs2_glock_nq+0x2c9/0x410 [gfs2] [<ffffffff8108a6cb>] ? try_to_del_timer_sync+0x7b/0xe0 [<ffffffffa0504549>] gfs2_statfs_sync+0x59/0x1c0 [gfs2] [<ffffffff815393ca>] ? schedule_timeout+0x19a/0x2e0 [<ffffffffa0504541>] ? gfs2_statfs_sync+0x51/0x1c0 [gfs2] [<ffffffffa04fbce7>] quotad_check_timeo+0x57/0xb0 [gfs2] [<ffffffffa04fbf74>] gfs2_quotad+0x234/0x2b0 [gfs2] [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa04fbd40>] ? gfs2_quotad+0x0/0x2b0 [gfs2] [<ffffffff810a0fce>] kthread+0x9e/0xc0 [<ffffffff8100c28a>] child_rip+0xa/0x20 [<ffffffff810a0f30>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 INFO: task flush-253:18:9820 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. flush-253:18 D 0000000000000000 0 9820 2 0x00000080 ffff8805f9bc3490 0000000000000046 ffff8805f9bc3458 ffff8805f9bc3454 ffff880620768c00 ffff88062fc23080 0000006fc80f5a1e ffff8800282959c0 00000000000002aa 000000010002be11 ffff8806241d65f8 ffff8805f9bc3fd8 Call Trace: [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd] [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd] [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod] [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd] [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60 [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0 [<ffffffff81276544>] ? blk_queue_bio+0x494/0x610 [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod] [<ffffffff81274910>] generic_make_request+0x240/0x5a0 [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff8100bc0e>] ? apic_timer_interrupt+0xe/0x20 [<ffffffff81274ce0>] submit_bio+0x70/0x120 [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0 [<ffffffff811ca0a8>] __block_write_full_page+0x1c8/0x330 [<ffffffff811c9070>] ? end_buffer_async_write+0x0/0x190 [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20 [<ffffffff811cdf70>] ? blkdev_get_block+0x0/0x20 [<ffffffff811ca2f0>] block_write_full_page_endio+0xe0/0x120 [<ffffffff81126ff0>] ? find_get_pages_tag+0x40/0x130 [<ffffffff811ca345>] block_write_full_page+0x15/0x20 [<ffffffff811cf108>] blkdev_writepage+0x18/0x20 [<ffffffff8113b387>] __writepage+0x17/0x40 [<ffffffff8113c64d>] write_cache_pages+0x1fd/0x4c0 [<ffffffff8113b370>] ? __writepage+0x0/0x40 [<ffffffff8113c934>] generic_writepages+0x24/0x30 [<ffffffff8113c961>] do_writepages+0x21/0x40 [<ffffffff811beb3d>] writeback_single_inode+0xdd/0x290 [<ffffffff811bef3d>] writeback_sb_inodes+0xbd/0x170 [<ffffffff811bf09b>] writeback_inodes_wb+0xab/0x1b0 [<ffffffff811bf493>] wb_writeback+0x2f3/0x410 [<ffffffff8108a752>] ? del_timer_sync+0x22/0x30 [<ffffffff811bf755>] wb_do_writeback+0x1a5/0x240 [<ffffffff811bf853>] bdi_writeback_task+0x63/0x1b0 [<ffffffff810a12e7>] ? bit_waitqueue+0x17/0xd0 [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100 [<ffffffff8114b7c6>] bdi_start_fn+0x86/0x100 [<ffffffff8114b740>] ? bdi_start_fn+0x0/0x100 [<ffffffff810a0fce>] kthread+0x9e/0xc0 [<ffffffff8100c28a>] child_rip+0xa/0x20 [<ffffffff810a0f30>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 INFO: task clusterfs.sh:11652 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. clusterfs.sh D 0000000000000001 0 11652 11588 0x00000080 ffff8804f9363c58 0000000000000082 ffff88061f81a6c0 0000000600000000 0000000000000000 ffff880400000000 0000000000000001 ffff8804f9363db8 ffff8804f9363ce8 ffff8804f9363cd8 ffff880624ead068 ffff8804f9363fd8 Call Trace: [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2] [<ffffffffa04e710e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2] [<ffffffff815397ff>] __wait_on_bit+0x5f/0x90 [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2] [<ffffffff815398a8>] out_of_line_wait_on_bit+0x78/0x90 [<ffffffff810a14e0>] ? wake_bit_function+0x0/0x50 [<ffffffffa04e80e5>] gfs2_glock_wait+0x45/0x90 [gfs2] [<ffffffffa04eb619>] gfs2_glock_nq+0x2c9/0x410 [gfs2] [<ffffffffa04f9f97>] gfs2_getattr+0xd7/0x1d0 [gfs2] [<ffffffffa04f9f8f>] ? gfs2_getattr+0xcf/0x1d0 [gfs2] [<ffffffff81197691>] vfs_getattr+0x51/0x80 [<ffffffff811974d4>] ? cp_new_stat+0xe4/0x100 [<ffffffff81197724>] vfs_fstatat+0x64/0xa0 [<ffffffff811977ce>] vfs_lstat+0x1e/0x20 [<ffffffff811977f4>] sys_newlstat+0x24/0x50 [<ffffffff810e8a57>] ? audit_syscall_entry+0x1d7/0x200 [<ffffffff810e884e>] ? __audit_syscall_exit+0x25e/0x290 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b INFO: task nrpe:11727 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. nrpe D 0000000000000000 0 11727 11696 0x00000080 ffff8804f91d7988 0000000000000082 ffffea0011683018 0000000114570cc0 ffff880626a297c0 00000002000022c0 000000000000004b 00000040ffffffff 0000000000000000 ffff880000049b08 ffff880624a9b068 ffff8804f91d7fd8 Call Trace: [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2] [<ffffffffa04e710e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2] [<ffffffff815397ff>] __wait_on_bit+0x5f/0x90 [<ffffffffa04e7100>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2] [<ffffffff815398a8>] out_of_line_wait_on_bit+0x78/0x90 [<ffffffff810a14e0>] ? wake_bit_function+0x0/0x50 [<ffffffffa04e80e5>] gfs2_glock_wait+0x45/0x90 [gfs2] [<ffffffffa04eb619>] gfs2_glock_nq+0x2c9/0x410 [gfs2] [<ffffffff811ad4a2>] ? dput+0x32/0x160 [<ffffffffa04f922f>] gfs2_permission+0x17f/0x210 [gfs2] [<ffffffffa04f9226>] ? gfs2_permission+0x176/0x210 [gfs2] [<ffffffff811a1da3>] __link_path_walk+0xb3/0x1000 [<ffffffff811a21eb>] __link_path_walk+0x4fb/0x1000 [<ffffffff811a2faa>] path_walk+0x6a/0xe0 [<ffffffff811a31bb>] filename_lookup+0x6b/0xc0 [<ffffffff811a42e7>] user_path_at+0x57/0xa0 [<ffffffff8119fd1b>] ? putname+0x2b/0x40 [<ffffffff811a42f2>] ? user_path_at+0x62/0xa0 [<ffffffff811974d4>] ? cp_new_stat+0xe4/0x100 [<ffffffff81197710>] vfs_fstatat+0x50/0xa0 [<ffffffff8119788b>] vfs_stat+0x1b/0x20 [<ffffffff811978b4>] sys_newstat+0x24/0x50 [<ffffffff810e8a57>] ? audit_syscall_entry+0x1d7/0x200 [<ffffffff810e884e>] ? __audit_syscall_exit+0x25e/0x290 [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b INFO: task glock_workqueue:7144 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. glock_workque D 0000000000000001 0 7144 2 0x00000080 ffff880602d13740 0000000000000046 0000000000000000 0000000000000000 0000000000000000 ffff880622eb3080 0000009d020d96ca 0000000000000246 ffff880602d13720 000000010005b5aa ffff880622cbbad8 ffff880602d13fd8 Call Trace: [<ffffffffa04a6885>] drbd_al_begin_io+0x1a5/0x240 [drbd] [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa04a3e4c>] drbd_make_request_common+0xf5c/0x14a0 [drbd] [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff811cdd1b>] ? bio_alloc_bioset+0x5b/0xf0 [<ffffffffa000256d>] ? __map_bio+0xad/0x140 [dm_mod] [<ffffffffa04a48c1>] drbd_make_request+0x531/0x870 [drbd] [<ffffffff81284376>] ? throtl_find_tg+0x46/0x60 [<ffffffff8128566a>] ? blk_throtl_bio+0x1ea/0x5f0 [<ffffffffa00042c2>] ? dm_make_request+0x122/0x180 [dm_mod] [<ffffffff81274910>] generic_make_request+0x240/0x5a0 [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20 [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 [<ffffffff81274ce0>] submit_bio+0x70/0x120 [<ffffffff811c78ed>] submit_bh+0x11d/0x1f0 [<ffffffffa04f0ddf>] buf_lo_before_commit+0x10f/0x1c0 [gfs2] [<ffffffffa04ef6c8>] gfs2_log_flush+0x1d8/0x600 [gfs2] [<ffffffff8100969d>] ? __switch_to+0x7d/0x340 [<ffffffffa04ec4bd>] inode_go_sync+0x7d/0x160 [gfs2] [<ffffffffa04eafd6>] do_xmote+0x156/0x280 [gfs2] [<ffffffffa04eb1f1>] run_queue+0xf1/0x1d0 [gfs2] [<ffffffffa04eb9fa>] glock_work_func+0x7a/0x1d0 [gfs2] [<ffffffffa04eb980>] ? glock_work_func+0x0/0x1d0 [gfs2] [<ffffffff8109a780>] worker_thread+0x170/0x2a0 [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 [<ffffffff8109a610>] ? worker_thread+0x0/0x2a0 [<ffffffff810a0fce>] kthread+0x9e/0xc0 [<ffffffff8100c28a>] child_rip+0xa/0x20 [<ffffffff810a0f30>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 INFO: task gfs2_logd:7162 blocked for more than 120 seconds. Not tainted 2.6.32-573.7.1.el6.x86_64 #1 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. gfs2_logd D 0000000000000003 0 7162 2 0x00000080 ffff880600d13d00 0000000000000046 0000000000000000 0000000000000000 0000000000000000 0000000000000000 000000a727fdf37d 0000000000000000 0000000000000000 0000000100066027 ffff88062257a5f8 ffff880600d13fd8 Call Trace: [<ffffffff8153b665>] rwsem_down_failed_common+0x95/0x1d0 [<ffffffff8153b7c3>] rwsem_down_write_failed+0x23/0x30 [<ffffffff8129d6f3>] call_rwsem_down_write_failed+0x13/0x20 [<ffffffff8153acc2>] ? down_write+0x32/0x40 [<ffffffffa04ef51f>] gfs2_log_flush+0x2f/0x600 [gfs2] [<ffffffffa04ee57f>] ? gfs2_ail1_empty+0x2f/0x1b0 [gfs2] [<ffffffffa04efbc9>] gfs2_logd+0xd9/0x140 [gfs2] [<ffffffffa04efaf0>] ? gfs2_logd+0x0/0x140 [gfs2] [<ffffffff810a0fce>] kthread+0x9e/0xc0 [<ffffffff8100c28a>] child_rip+0xa/0x20 [<ffffffff810a0f30>] ? kthread+0x0/0xc0 [<ffffffff8100c280>] ? child_rip+0x0/0x20 any hint on fixing that? _______________________________________________ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org