For the (roughly) past 12 months, I've been experiencing the following (repeatable, nightly) kernel traces on btrfs filesystems which are under heavy (nightly backups) btrfs-receive traffic (over the network).
The problems intensify when the amount of data per second increases. If the amount of data per second is high enough, either the whole kernel locks up, or the btrfs filesystem hangs on all operations. The oopses below were generated under moderate traffic levels and did not cause a full kernel lockup and did not make the btrfs filesystem permanently inaccessible (possibly temporarily unresponsive though). My current way to mitigate this problem is by limiting the amount of network traffic to below 3MB/s (which is, ridiculously low, I think). Nonetheless, it should not be possible to lock up a system by increasing bandwidth to the btrfs filesystem with btrfs-receive. Any ideas? I can apply custom patches to either test and get more output or to test possible fixes. It is on a production system, so dataloss is not an option, but I can reboot it several times a day, if necessary. I can increase the datarate to any desirable level to test if the problem reoccurs or not. (The most recent ones are from v4.19.7): Total devices 8 FS bytes used 8.98TiB devid 1 size 5.46TiB used 2.49TiB path /dev/sdf devid 2 size 5.46TiB used 2.49TiB path /dev/sdi devid 3 size 5.46TiB used 2.49TiB path /dev/sdh devid 4 size 5.46TiB used 2.49TiB path /dev/sdg devid 5 size 5.46TiB used 2.49TiB path /dev/sdj devid 6 size 5.46TiB used 2.49TiB path /dev/sdc devid 7 size 5.46TiB used 2.49TiB path /dev/sda devid 8 size 3.64TiB used 689.00GiB path /dev/sdb [64389.127561] INFO: task btrfs:7518 blocked for more than 120 seconds. [64389.132750] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64389.138421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64389.145126] btrfs D 0 7518 7489 0x00000000 [64389.149481] Call Trace: [64389.150755] ? __schedule+0x4db/0x524 [64389.153220] ? schedule+0x60/0x71 [64389.155342] ? __rwsem_down_read_failed_common+0x180/0x1df [64389.159679] ? call_rwsem_down_read_failed+0x14/0x30 [64389.163541] ? down_read+0x1c/0x2f [64389.165755] ? btrfs_ioctl_subvol_getflags+0x67/0xbb [64389.169592] ? btrfs_ioctl+0x2d8/0x2a51 [64389.172237] ? current_time+0x67/0x88 [64389.174705] ? pipe_write+0x356/0x369 [64389.177222] ? vfs_ioctl+0x1e/0x2b [64389.179439] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64389.183814] ? vfs_ioctl+0x1e/0x2b [64389.186048] ? do_vfs_ioctl+0x523/0x54c [64389.188742] ? vfs_write+0xb2/0xdf [64389.190958] ? ksys_ioctl+0x3e/0x5d [64389.193268] ? __x64_sys_ioctl+0x16/0x19 [64389.196000] ? do_syscall_64+0x4e/0x5b [64389.198591] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [64512.019259] INFO: task btrfs:7828 blocked for more than 120 seconds. [64512.024455] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64512.030109] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64512.036781] btrfs D 0 7828 7771 0x00000000 [64512.041123] Call Trace: [64512.042397] ? __schedule+0x4db/0x524 [64512.044865] ? schedule+0x60/0x71 [64512.046988] ? __rwsem_down_write_failed_common+0x2ff/0x382 [64512.051443] ? update_cfs_rq_load_avg+0x124/0x150 [64512.054964] ? check_preempt_wakeup+0xfd/0x146 [64512.058222] ? call_rwsem_down_write_failed+0x13/0x20 [64512.062125] ? down_write+0x29/0x38 [64512.064429] ? _btrfs_ioctl_set_received_subvol+0x90/0x2ab [64512.068725] ? btrfs_ioctl+0x1572/0x2a51 [64512.071508] ? current_time+0x67/0x88 [64512.073992] ? __switch_to_asm+0x40/0x70 [64512.076723] ? __switch_to_asm+0x34/0x70 [64512.079504] ? __switch_to_asm+0x40/0x70 [64512.082235] ? __switch_to_asm+0x34/0x70 [64512.084963] ? __switch_to_asm+0x40/0x70 [64512.087687] ? __switch_to_asm+0x34/0x70 [64512.090458] ? __switch_to_asm+0x40/0x70 [64512.093202] ? __switch_to_asm+0x34/0x70 [64512.095927] ? __switch_to_asm+0x40/0x70 [64512.098650] ? __switch_to_asm+0x34/0x70 [64512.101427] ? __switch_to_asm+0x40/0x70 [64512.104164] ? __switch_to_asm+0x34/0x70 [64512.106891] ? __switch_to_asm+0x40/0x70 [64512.109668] ? vfs_ioctl+0x1e/0x2b [64512.111887] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64512.116274] ? vfs_ioctl+0x1e/0x2b [64512.118479] ? do_vfs_ioctl+0x523/0x54c [64512.121159] ? __switch_to_asm+0x34/0x70 [64512.123905] ? __switch_to_asm+0x40/0x70 [64512.126637] ? __switch_to+0x1c8/0x37f [64512.129237] ? finish_task_switch+0x113/0x1e6 [64512.132413] ? ksys_ioctl+0x3e/0x5d [64512.134703] ? __x64_sys_ioctl+0x16/0x19 [64512.137426] ? do_syscall_64+0x4e/0x5b [64512.140043] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [64512.144084] INFO: task btrfs:7840 blocked for more than 120 seconds. [64512.149282] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64512.154895] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64512.161563] btrfs D 0 7840 7691 0x00000000 [64512.165860] Call Trace: [64512.167118] ? __schedule+0x4db/0x524 [64512.169630] ? schedule+0x60/0x71 [64512.171757] ? __rwsem_down_read_failed_common+0x180/0x1df [64512.176045] ? call_rwsem_down_read_failed+0x14/0x30 [64512.179853] ? down_read+0x1c/0x2f [64512.182072] ? btrfs_ioctl_subvol_getflags+0x67/0xbb [64512.185858] ? btrfs_ioctl+0x2d8/0x2a51 [64512.188503] ? current_time+0x67/0x88 [64512.191026] ? pipe_write+0x356/0x369 [64512.193505] ? vfs_ioctl+0x1e/0x2b [64512.195733] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64512.200162] ? vfs_ioctl+0x1e/0x2b [64512.202376] ? do_vfs_ioctl+0x523/0x54c [64512.205035] ? vfs_write+0xb2/0xdf [64512.207260] ? ksys_ioctl+0x3e/0x5d [64512.209600] ? __x64_sys_ioctl+0x16/0x19 [64512.212353] ? do_syscall_64+0x4e/0x5b [64512.214936] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [64512.219027] INFO: task btrfs:15145 blocked for more than 120 seconds. [64512.224292] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64512.229923] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64512.236564] btrfs D 0 15145 15124 0x00000000 [64512.240886] Call Trace: [64512.242157] ? __schedule+0x4db/0x524 [64512.244638] ? schedule+0x60/0x71 [64512.246769] ? __rwsem_down_read_failed_common+0x180/0x1df [64512.251116] ? call_rwsem_down_read_failed+0x14/0x30 [64512.254907] ? down_read+0x1c/0x2f [64512.257144] ? btrfs_mksubvol+0x110/0x453 [64512.259997] ? btrfs_ioctl_snap_create_transid+0x165/0x199 [64512.264301] ? btrfs_ioctl_snap_create_v2+0xf1/0x154 [64512.268069] ? btrfs_ioctl+0x87f/0x2a51 [64512.270755] ? do_filp_open+0x89/0xaf [64512.273235] ? get_page+0x5/0xa [64512.275178] ? vfs_ioctl+0x1e/0x2b [64512.277388] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64512.281811] ? vfs_ioctl+0x1e/0x2b [64512.284020] ? do_vfs_ioctl+0x523/0x54c [64512.286677] ? cache_from_obj+0x22/0x57 [64512.289373] ? ksys_ioctl+0x3e/0x5d [64512.291673] ? __x64_sys_ioctl+0x16/0x19 [64512.294400] ? do_syscall_64+0x4e/0x5b [64512.296959] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [64634.890975] INFO: task btrfs:7823 blocked for more than 120 seconds. [64634.896137] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64634.901800] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64634.908461] btrfs D 0 7823 7641 0x00000000 [64634.912806] Call Trace: [64634.914070] ? __schedule+0x4db/0x524 [64634.916540] ? schedule+0x60/0x71 [64634.918656] ? __rwsem_down_write_failed_common+0x2ff/0x382 [64634.923081] ? apic_timer_interrupt+0xa/0x20 [64634.926193] ? call_rwsem_down_write_failed+0x13/0x20 [64634.930053] ? down_write+0x29/0x38 [64634.932398] ? btrfs_ioctl_subvol_setflags+0xc5/0x1fb [64634.936259] ? btrfs_ioctl+0x13a/0x2a51 [64634.938943] ? current_time+0x67/0x88 [64634.941467] ? pipe_write+0x356/0x369 [64634.943937] ? vfs_ioctl+0x1e/0x2b [64634.946174] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64634.950597] ? vfs_ioctl+0x1e/0x2b [64634.952804] ? do_vfs_ioctl+0x523/0x54c [64634.965173] ? vfs_write+0xb2/0xdf [64634.967417] ? ksys_ioctl+0x3e/0x5d [64634.969712] ? __x64_sys_ioctl+0x16/0x19 [64634.972482] ? do_syscall_64+0x4e/0x5b [64634.975043] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [64634.979075] INFO: task btrfs:7831 blocked for more than 120 seconds. [64634.984266] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64634.989885] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64634.996560] btrfs D 0 7831 7755 0x00000000 [64635.000865] Call Trace: [64635.002126] ? __schedule+0x4db/0x524 [64635.004608] ? schedule+0x60/0x71 [64635.006757] ? __rwsem_down_read_failed_common+0x180/0x1df [64635.011092] ? call_rwsem_down_read_failed+0x14/0x30 [64635.014876] ? down_read+0x1c/0x2f [64635.017087] ? btrfs_ioctl_subvol_getflags+0x67/0xbb [64635.020893] ? btrfs_ioctl+0x2d8/0x2a51 [64635.023544] ? current_time+0x67/0x88 [64635.026012] ? __switch_to_asm+0x40/0x70 [64635.028770] ? __switch_to_asm+0x34/0x70 [64635.031529] ? __switch_to_asm+0x40/0x70 [64635.034261] ? __switch_to_asm+0x34/0x70 [64635.036993] ? __switch_to_asm+0x40/0x70 [64635.039720] ? __switch_to_asm+0x34/0x70 [64635.042476] ? __switch_to_asm+0x40/0x70 [64635.045210] ? __switch_to_asm+0x34/0x70 [64635.047964] ? __switch_to_asm+0x40/0x70 [64635.050731] ? __switch_to_asm+0x34/0x70 [64635.053459] ? __switch_to_asm+0x40/0x70 [64635.056188] ? __switch_to_asm+0x34/0x70 [64635.058910] ? __switch_to_asm+0x40/0x70 [64635.061670] ? vfs_ioctl+0x1e/0x2b [64635.063878] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64635.068281] ? vfs_ioctl+0x1e/0x2b [64635.070491] ? do_vfs_ioctl+0x523/0x54c [64635.073179] ? __switch_to_asm+0x34/0x70 [64635.075913] ? __switch_to_asm+0x40/0x70 [64635.078636] ? __switch_to+0x1c8/0x37f [64635.081219] ? finish_task_switch+0x113/0x1e6 [64635.084384] ? ksys_ioctl+0x3e/0x5d [64635.086679] ? __x64_sys_ioctl+0x16/0x19 [64635.089418] ? do_syscall_64+0x4e/0x5b [64635.092018] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [64635.096062] INFO: task btrfs:7840 blocked for more than 120 seconds. [64635.101255] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64635.106866] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64635.113532] btrfs D 0 7840 7691 0x00000000 [64635.117827] Call Trace: [64635.119083] ? __schedule+0x4db/0x524 [64635.121583] ? schedule+0x60/0x71 [64635.123715] ? __rwsem_down_read_failed_common+0x180/0x1df [64635.128002] ? call_rwsem_down_read_failed+0x14/0x30 [64635.131810] ? down_read+0x1c/0x2f [64635.134031] ? btrfs_ioctl_subvol_getflags+0x67/0xbb [64635.137815] ? btrfs_ioctl+0x2d8/0x2a51 [64635.140457] ? current_time+0x67/0x88 [64635.142965] ? pipe_write+0x356/0x369 [64635.145433] ? vfs_ioctl+0x1e/0x2b [64635.147643] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64635.152051] ? vfs_ioctl+0x1e/0x2b [64635.154262] ? do_vfs_ioctl+0x523/0x54c [64635.156919] ? vfs_write+0xb2/0xdf [64635.159126] ? ksys_ioctl+0x3e/0x5d [64635.161448] ? __x64_sys_ioctl+0x16/0x19 [64635.164180] ? do_syscall_64+0x4e/0x5b [64635.166736] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [64635.170813] INFO: task btrfs:15145 blocked for more than 120 seconds. [64635.176068] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64635.181703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64635.188354] btrfs D 0 15145 15124 0x00000000 [64635.192676] Call Trace: [64635.193939] ? __schedule+0x4db/0x524 [64635.196410] ? schedule+0x60/0x71 [64635.198527] ? __rwsem_down_read_failed_common+0x180/0x1df [64635.202854] ? call_rwsem_down_read_failed+0x14/0x30 [64635.206651] ? down_read+0x1c/0x2f [64635.208858] ? btrfs_mksubvol+0x110/0x453 [64635.211710] ? btrfs_ioctl_snap_create_transid+0x165/0x199 [64635.216009] ? btrfs_ioctl_snap_create_v2+0xf1/0x154 [64635.219792] ? btrfs_ioctl+0x87f/0x2a51 [64635.222484] ? do_filp_open+0x89/0xaf [64635.224967] ? get_page+0x5/0xa [64635.226911] ? vfs_ioctl+0x1e/0x2b [64635.229121] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64635.233567] ? vfs_ioctl+0x1e/0x2b [64635.235778] ? do_vfs_ioctl+0x523/0x54c [64635.238443] ? cache_from_obj+0x22/0x57 [64635.241132] ? ksys_ioctl+0x3e/0x5d [64635.243457] ? __x64_sys_ioctl+0x16/0x19 [64635.246187] ? do_syscall_64+0x4e/0x5b [64635.248742] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [64635.252813] INFO: task btrfs:16588 blocked for more than 120 seconds. [64635.258065] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64635.263717] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64635.270357] btrfs D 0 16588 16585 0x00000000 [64635.274703] Call Trace: [64635.275964] ? __schedule+0x4db/0x524 [64635.278434] ? mem_cgroup_commit_charge+0xb1/0xf3 [64635.281995] ? schedule+0x60/0x71 [64635.284121] ? __rwsem_down_write_failed_common+0x2ff/0x382 [64635.288496] ? call_rwsem_down_write_failed+0x13/0x20 [64635.292393] ? down_write+0x29/0x38 [64635.294696] ? btrfs_delete_subvolume+0x9b/0x4d9 [64635.298116] ? d_lookup+0x2d/0x3d [64635.300239] ? btrfs_ioctl_snap_destroy+0x245/0x2a3 [64635.303984] ? btrfs_ioctl+0x8aa/0x2a51 [64635.306629] ? __count_memcg_events+0x11/0x4e [64635.309792] ? mem_cgroup_commit_charge+0xb1/0xf3 [64635.313341] ? get_page+0x5/0xa [64635.315293] ? __lru_cache_add+0x1e/0x4d [64635.318015] ? __handle_mm_fault+0x5f0/0x9cf [64635.321123] ? vfs_ioctl+0x1e/0x2b [64635.323348] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64635.327731] ? vfs_ioctl+0x1e/0x2b [64635.329932] ? do_vfs_ioctl+0x523/0x54c [64635.332619] ? handle_mm_fault+0x12f/0x17e [64635.335523] ? ksys_ioctl+0x3e/0x5d [64635.337818] ? __x64_sys_ioctl+0x16/0x19 [64635.340550] ? do_syscall_64+0x4e/0x5b [64635.343154] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 [64757.782746] INFO: task btrfs:7823 blocked for more than 120 seconds. [64757.788007] Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 [64757.793652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [64757.800296] btrfs D 0 7823 7641 0x00000000 [64757.804628] Call Trace: [64757.805897] ? __schedule+0x4db/0x524 [64757.808366] ? schedule+0x60/0x71 [64757.810491] ? __rwsem_down_write_failed_common+0x2ff/0x382 [64757.814892] ? apic_timer_interrupt+0xa/0x20 [64757.817971] ? call_rwsem_down_write_failed+0x13/0x20 [64757.821825] ? down_write+0x29/0x38 [64757.824139] ? btrfs_ioctl_subvol_setflags+0xc5/0x1fb [64757.827999] ? btrfs_ioctl+0x13a/0x2a51 [64757.830641] ? current_time+0x67/0x88 [64757.833142] ? pipe_write+0x356/0x369 [64757.835607] ? vfs_ioctl+0x1e/0x2b [64757.837819] ? btrfs_ioctl_get_supported_features+0x22/0x22 [64757.842202] ? vfs_ioctl+0x1e/0x2b [64757.844437] ? do_vfs_ioctl+0x523/0x54c [64757.847084] ? vfs_write+0xb2/0xdf [64757.849293] ? ksys_ioctl+0x3e/0x5d [64757.851590] ? __x64_sys_ioctl+0x16/0x19 [64757.854346] ? do_syscall_64+0x4e/0x5b [64757.856902] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 The following ones are from the sister system, starting at an older date, the older ones are from v4.19.1 and the newer ones from v4.19.7: Total devices 5 FS bytes used 2.48TiB devid 1 size 5.46TiB used 1.41TiB path /dev/sdb devid 2 size 5.46TiB used 1.41TiB path /dev/sdg devid 3 size 5.46TiB used 1.41TiB path /dev/sdf devid 4 size 5.46TiB used 1.41TiB path /dev/sde devid 5 size 3.64TiB used 0.00B path /dev/sda Nov 11 07:43:15 argos kernel: INFO: task btrfs-transacti:1235 blocked for more than 120 seconds. Nov 11 07:43:15 argos kernel: Not tainted 94.19.1-srb-asrock-00001-gdffa15601442 #132 Nov 11 07:43:15 argos kernel: btrfs-transacti D 0 1235 2 0x80000000 Nov 11 07:43:15 argos kernel: Call Trace: Nov 11 07:43:15 argos kernel: ? __schedule+0x4db/0x524 Nov 11 07:43:15 argos kernel: ? schedule+0x60/0x71 Nov 11 07:45:18 argos kernel: ? btrfs_start_ordered_extent+0xd3/0x105 Nov 11 07:45:18 argos kernel: ? finish_wait+0x60/0x60 Nov 11 07:45:18 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Nov 11 07:45:18 argos kernel: ? __btrfs_wait_cache_io+0x47/0x16d Nov 11 07:45:18 argos kernel: ? btrfs_start_dirty_block_groups+0x111/0x390 Nov 11 07:45:18 argos kernel: ? btrfs_commit_transaction+0xd3/0x73b Nov 11 07:47:21 argos kernel: ? kthread+0xeb/0xf0 Nov 11 07:47:21 argos kernel: ? __kthread_cancel_work+0x68/0x68 Nov 11 07:47:21 argos kernel: ? ret_from_fork+0x35/0x40 Nov 11 07:49:23 argos kernel: ? __btrfs_wait_cache_io+0x47/0x16d Nov 11 07:49:23 argos kernel: ? btrfs_start_dirty_block_groups+0x111/0x390 Nov 11 07:49:23 argos kernel: ? start_transaction+0x2b8/0x329 Nov 11 07:49:23 argos kernel: ? transaction_kthread+0xc6/0x132 Nov 11 07:49:23 argos kernel: ? btrfs_cleanup_transaction+0x3ca/0x3ca Nov 11 07:51:26 argos kernel: btrfs-transacti D 0 1235 2 0x80000000 Nov 11 07:51:26 argos kernel: Call Trace: Nov 11 07:51:26 argos kernel: ? __schedule+0x4db/0x524 Nov 11 07:51:26 argos kernel: ? schedule+0x60/0x71 Nov 11 07:51:26 argos kernel: ? __btrfs_wait_cache_io+0x47/0x16d Nov 11 07:51:26 argos kernel: ? btrfs_start_dirty_block_groups+0x111/0x390 Nov 11 07:51:26 argos kernel: ? btrfs_commit_transaction+0xd3/0x73b Nov 11 07:51:26 argos kernel: ? kthread+0xeb/0xf0 Nov 11 07:51:26 argos kernel: ? __kthread_cancel_work+0x68/0x68 Nov 11 07:51:26 argos kernel: ? ret_from_fork+0x35/0x40 Nov 11 07:53:29 argos kernel: INFO: task btrfs-transacti:1235 blocked for more than 120 seconds. Nov 11 07:53:29 argos kernel: Not tainted 94.19.1-srb-asrock-00001-gdffa15601442 #132 Nov 11 07:53:29 argos kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 11 07:57:35 argos kernel: ? btrfs_start_ordered_extent+0xd3/0x105 Nov 11 07:57:35 argos kernel: ? finish_wait+0x60/0x60 Nov 11 07:57:35 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Nov 13 12:02:01 argos kernel: ? btrfs_start_ordered_extent+0xd3/0x105 Nov 13 12:06:07 argos kernel: INFO: task btrfs:26969 blocked for more than 120 seconds. Nov 13 12:06:07 argos kernel: Not tainted 94.19.1-srb-asrock-00001-gdffa15601442 #135 Nov 13 12:08:10 argos kernel: Call Trace: Nov 13 12:30:06 argos kernel: perf: interrupt took too long (2506 > 2500), lowering kernel.perf_event_max_sample_rate to 79800 Nov 13 14:29:19 argos kernel: perf: interrupt took too long (3144 > 3132), lowering kernel.perf_event_max_sample_rate to 63600 Nov 14 07:41:40 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Nov 14 07:47:49 argos kernel: ? ret_from_fork+0x35/0x40 Nov 14 13:34:18 argos kernel: INFO: task btrfs:6654 blocked for more than 120 seconds. Nov 14 13:34:18 argos kernel: Not tainted 94.19.1-srb-asrock-00001-gdffa15601442 #135 Nov 14 13:34:18 argos kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 14 13:34:19 argos kernel: btrfs D 0 6654 6649 0x00000000 Nov 14 13:34:19 argos kernel: Call Trace: Nov 14 13:34:19 argos kernel: ? do_truncate+0x82/0xb6 Nov 14 13:34:19 argos kernel: ? kmem_cache_alloc+0x9a/0x100 Nov 14 13:34:19 argos kernel: ? vfs_truncate+0xbc/0xda Nov 14 13:34:19 argos kernel: ? do_sys_truncate+0x60/0xa9 Nov 14 13:34:19 argos kernel: ? do_syscall_64+0x4e/0x5b Nov 14 13:34:19 argos kernel: ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 14 13:36:21 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Nov 14 13:36:21 argos kernel: ? unmap_mapping_pages+0x64/0x113 Nov 14 13:36:21 argos kernel: ? btrfs_setattr+0x20b/0x499 Nov 18 09:48:00 argos kernel: btrfs D 0 26152 26147 0x00000000 Nov 18 09:48:00 argos kernel: ? __schedule+0x4db/0x524 Nov 18 09:48:00 argos kernel: ? finish_wait+0x60/0x60 Nov 18 09:48:00 argos kernel: ? notify_change+0x268/0x347 Nov 18 09:48:00 argos kernel: ? do_truncate+0x82/0xb6 Nov 18 09:48:00 argos kernel: ? kmem_cache_alloc+0x9a/0x100 Nov 18 09:48:00 argos kernel: ? vfs_truncate+0xbc/0xda Nov 19 10:02:05 argos kernel: ? do_sys_truncate+0x60/0xa9 Nov 19 10:02:05 argos kernel: ? do_syscall_64+0x4e/0x5b Nov 19 10:02:05 argos kernel: ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 19 10:04:08 argos kernel: ? vfs_truncate+0xbc/0xda Nov 22 12:22:53 argos kernel: INFO: task btrfs:19688 blocked for more than 120 seconds. Nov 22 12:22:53 argos kernel: Not tainted 94.19.1-srb-asrock-00001-gdffa15601442 #135 Nov 22 12:22:53 argos kernel: btrfs D 0 19688 19683 0x00000000 Nov 22 12:22:53 argos kernel: Call Trace: Nov 22 12:22:53 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Nov 22 12:22:53 argos kernel: ? unmap_mapping_pages+0x64/0x113 Nov 23 07:46:09 argos kernel: INFO: task btrfs-transacti:1234 blocked for more than 120 seconds. Nov 23 07:46:09 argos kernel: Not tainted 94.19.1-srb-asrock-00001-gdffa15601442 #135 Nov 23 07:46:09 argos kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 23 07:46:09 argos kernel: btrfs-transacti D 0 1234 2 0x80000000 Nov 23 07:46:09 argos kernel: ? btrfs_start_ordered_extent+0xd3/0x105 Nov 23 07:46:09 argos kernel: ? finish_wait+0x60/0x60 Nov 23 07:46:09 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Nov 23 07:46:09 argos kernel: ? __kthread_cancel_work+0x68/0x68 Nov 23 07:48:12 argos kernel: INFO: task btrfs-transacti:1234 blocked for more than 120 seconds. Nov 23 07:48:12 argos kernel: Not tainted 94.19.1-srb-asrock-00001-gdffa15601442 #135 Nov 23 07:48:12 argos kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 23 07:48:12 argos kernel: btrfs-transacti D 0 1234 2 0x80000000 Nov 23 07:48:12 argos kernel: ? btrfs_start_dirty_block_groups+0x111/0x390 Nov 23 07:48:12 argos kernel: ? btrfs_commit_transaction+0xd3/0x73b Nov 23 07:50:15 argos kernel: ? btrfs_clean_one_deleted_snapshot+0xa8/0xb2 Nov 23 07:50:15 argos kernel: INFO: task btrfs-transacti:1234 blocked for more than 120 seconds. Nov 23 07:50:15 argos kernel: Not tainted 94.19.1-srb-asrock-00001-gdffa15601442 #135 Nov 23 07:52:18 argos kernel: ? finish_wait+0x60/0x60 Nov 25 09:53:44 argos kernel: Not tainted 94.19.1-srb-asrock-00001-gdffa15601442 #135 Nov 25 09:53:44 argos kernel: btrfs D 0 7623 7618 0x00000000 Nov 25 09:53:44 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Nov 25 09:53:44 argos kernel: ? unmap_mapping_pages+0x64/0x113 Nov 25 09:53:44 argos kernel: ? btrfs_setattr+0x20b/0x499 Nov 26 10:26:14 argos kernel: INFO: task btrfs:18234 blocked for more than 120 seconds. Nov 27 10:22:54 argos kernel: ? __schedule+0x4db/0x524 Nov 27 10:22:54 argos kernel: ? schedule+0x60/0x71 Nov 27 10:22:54 argos kernel: ? btrfs_start_ordered_extent+0xd3/0x105 Nov 27 10:22:54 argos kernel: ? notify_change+0x268/0x347 Nov 27 10:22:54 argos kernel: ? do_truncate+0x82/0xb6 Nov 27 10:22:54 argos kernel: ? kmem_cache_alloc+0x9a/0x100 Nov 27 10:22:54 argos kernel: ? vfs_truncate+0xbc/0xda Nov 27 10:22:54 argos kernel: ? do_sys_truncate+0x60/0xa9 Nov 27 10:22:54 argos kernel: ? do_syscall_64+0x4e/0x5b Nov 27 10:22:54 argos kernel: ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 27 10:24:57 argos kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 28 10:04:13 argos kernel: ? __schedule+0x4db/0x524 Nov 28 10:04:13 argos kernel: ? schedule+0x60/0x71 Nov 28 10:04:13 argos kernel: ? btrfs_start_ordered_extent+0xd3/0x105 Nov 28 10:04:13 argos kernel: ? finish_wait+0x60/0x60 Nov 28 10:06:16 argos kernel: ? btrfs_start_ordered_extent+0xd3/0x105 Nov 28 10:06:16 argos kernel: ? finish_wait+0x60/0x60 Nov 28 10:06:16 argos kernel: ? btrfs_setattr+0x20b/0x499 Nov 28 10:08:18 argos kernel: ? vfs_truncate+0xbc/0xda Nov 29 10:24:26 argos kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 29 10:24:26 argos kernel: ? do_sys_truncate+0x60/0xa9 Nov 29 10:24:26 argos kernel: ? do_syscall_64+0x4e/0x5b Nov 29 10:24:26 argos kernel: ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 Nov 29 10:26:29 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Nov 29 10:26:29 argos kernel: ? unmap_mapping_pages+0x64/0x113 Nov 29 10:26:29 argos kernel: ? btrfs_setattr+0x20b/0x499 Nov 29 10:28:32 argos kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 29 10:28:32 argos kernel: btrfs D 0 24965 24960 0x00000000 Nov 29 10:28:32 argos kernel: Call Trace: Nov 29 13:08:31 argos kernel: Code: 1f 80 00 00 00 00 e9 7b ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 90 48 81 ec f8 ff 00 00 8b 05 8f 2e 00 00 48 8d 35 40 0e 00 00 <48> 89 3c 24 48 8b 3d 81 2e 00 00 8d 90 00 00 01 00 31 c0 89 15 6f Nov 30 10:13:56 argos kernel: ? btrfs_setattr+0x20b/0x499 Nov 30 10:13:56 argos kernel: ? vfs_truncate+0xbc/0xda Dec 3 17:08:40 argos kernel: Code: 1f 80 00 00 00 00 e9 7b ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 90 48 81 ec f8 ff 00 00 8b 05 8f 2e 00 00 48 8d 35 40 0e 00 00 <48> 89 3c 24 48 8b 3d 81 2e 00 00 8d 90 00 00 01 00 31 c0 89 15 6f Dec 6 11:41:13 argos kernel: perf: interrupt took too long (2514 > 2500), lowering kernel.perf_event_max_sample_rate to 79500 Dec 6 22:42:18 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Dec 6 22:42:18 argos kernel: ? unmap_mapping_pages+0x64/0x113 Dec 6 22:42:18 argos kernel: ? btrfs_setattr+0x20b/0x499 Dec 6 22:42:18 argos kernel: ? notify_change+0x268/0x347 Dec 6 22:42:18 argos kernel: ? do_truncate+0x82/0xb6 Dec 6 22:42:18 argos kernel: ? kmem_cache_alloc+0x9a/0x100 Dec 6 22:42:18 argos kernel: ? vfs_truncate+0xbc/0xda Dec 6 22:58:41 argos kernel: Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 Dec 6 22:58:41 argos kernel: Call Trace: Dec 6 22:58:41 argos kernel: ? btrfs_setattr+0x20b/0x499 Dec 7 00:26:26 argos kernel: Linux version 94.19.7-srb-asrock-00001-g22c8f45edcaf (root@devel2) (gcc version 8.2.0 (Debian 8.2.0-9)) #137 SMP Thu Dec 6 09:36:50 CET 2018 Dec 7 03:49:45 argos kernel: perf: interrupt took too long (3958 > 3937), lowering kernel.perf_event_max_sample_rate to 50500 Dec 7 11:35:20 argos kernel: ? finish_wait+0x60/0x60 Dec 7 11:39:26 argos kernel: ? notify_change+0x268/0x347 Dec 7 11:39:26 argos kernel: ? do_truncate+0x82/0xb6 Dec 7 11:39:26 argos kernel: ? kmem_cache_alloc+0x9a/0x100 Dec 7 11:39:26 argos kernel: ? vfs_truncate+0xbc/0xda Dec 7 14:37:55 argos kernel: in libxul.so[7f3f410b2000+3dd6000] Dec 7 14:37:55 argos kernel: Code: 15 7a 84 a3 04 48 89 10 c7 04 25 00 00 00 00 e0 09 00 00 e8 47 57 ff ff 90 48 8b 05 c7 75 ec 05 48 8d 0d c0 84 a3 04 48 89 08 <c7> 04 25 00 00 00 00 66 0a 00 00 e8 25 57 ff ff e8 2a f3 ff ff 48 Dec 7 14:49:20 argos kernel: Code: 15 7a 84 a3 04 48 89 10 c7 04 25 00 00 00 00 e0 09 00 00 e8 47 57 ff ff 90 48 8b 05 c7 75 ec 05 48 8d 0d c0 84 a3 04 48 89 08 <c7> 04 25 00 00 00 00 66 0a 00 00 e8 25 57 ff ff e8 2a f3 ff ff 48 Dec 8 12:03:45 argos kernel: ? unmap_mapping_pages+0x64/0x113 Dec 8 12:05:48 argos kernel: ? __schedule+0x4db/0x524 Dec 8 12:05:48 argos kernel: ? schedule+0x60/0x71 Dec 8 12:05:48 argos kernel: ? btrfs_start_ordered_extent+0xd3/0x105 Dec 8 12:05:48 argos kernel: ? finish_wait+0x60/0x60 Dec 8 12:07:51 argos kernel: btrfs D 0 17853 17848 0x00000000 Dec 8 12:07:51 argos kernel: Call Trace: Dec 8 12:07:51 argos kernel: ? vfs_truncate+0xbc/0xda Dec 9 10:37:29 argos kernel: Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 Dec 9 10:37:29 argos kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 9 10:37:29 argos kernel: btrfs D 0 409 404 0x00000000 Dec 9 10:37:29 argos kernel: Call Trace: Dec 9 10:37:29 argos kernel: ? vfs_truncate+0xbc/0xda Dec 9 10:39:32 argos kernel: btrfs D 0 409 404 0x00000000 Dec 9 10:39:32 argos kernel: Call Trace: Dec 9 10:39:32 argos kernel: ? btrfs_wait_ordered_range+0xa4/0x100 Dec 9 10:39:32 argos kernel: ? unmap_mapping_pages+0x64/0x113 Dec 9 10:39:32 argos kernel: ? btrfs_setattr+0x20b/0x499 Dec 9 10:39:32 argos kernel: ? notify_change+0x268/0x347 Dec 9 10:39:32 argos kernel: ? do_truncate+0x82/0xb6 Dec 9 10:41:34 argos kernel: ? __schedule+0x4db/0x524 Dec 9 10:41:34 argos kernel: ? schedule+0x60/0x71 Dec 9 10:41:34 argos kernel: ? btrfs_start_ordered_extent+0xd3/0x105 Dec 9 10:41:34 argos kernel: ? finish_wait+0x60/0x60 Dec 9 10:43:37 argos kernel: INFO: task btrfs:409 blocked for more than 120 seconds. Dec 9 10:43:37 argos kernel: Not tainted 94.19.7-srb-asrock-00001-g22c8f45edcaf #137 Dec 9 10:43:37 argos kernel: ? finish_wait+0x60/0x60 Dec 9 12:08:42 argos kernel: Code: 1f 80 00 00 00 00 e9 7b ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 90 48 81 ec f8 ff 00 00 8b 05 8f 2e 00 00 48 8d 35 40 0e 00 00 <48> 89 3c 24 48 8b 3d 81 2e 00 00 8d 90 00 00 01 00 31 c0 89 15 6f -- Stephen.