On Wed, 13 Apr 2016, Eric Wheeler wrote: > Hello all, > > We just got this backtrace in 4.4.6 on an ARM AM335x (beaglebone > compatible). The trace looks similar to this one: > http://permalink.gmane.org/gmane.comp.file-systems.btrfs/54734
We solved the problem so I thought I would report back in case anyone else runs in to the same issue. This is running on a memory constrained 700mhz ARM (single core) so we had these mount options that worked fine until we went from a single volume to raid1, at which time it hung after the first reboot (or remount?): ssd,ssd_spread,thread_pool=1,noatime,compress-force=lzo,nospace_cache,commit=37 so we switched to these on the raid1 system: noatime,compress-force=lzo,commit=37 and it stopped hanging. Ultimately my guess is either of thread_pool=1 causing deadlock (no progress), or nospace_cache causing space churn. We used thread_pool=1 which worked for a long time on a single volume. We just added a 2nd volume and went to raid1, so perhaps the minimum thread_pool should be the number of volumes? I cannot be certain, so FYI in case others run into this. -- Eric Wheeler but I > don't have nice backtraces on this hardware (maybe hang traces are a > compile option?). > > In my case, the system appeared to hang but sysrq functions still worked > and I was able to send a sysrq-(c)rash over serial. The filesystem was > just formatted in RAID1, and while I cannot access it because this hangs > at boot, there can't be very much data yet. > > This looks like the top of the relevant section, full trace below: > > [ 80.738518] [<bf187910>] (setup_cluster_no_bitmap [btrfs]) from > [<bf18b3b8>] (btrfs_find_space_cluster+0x10c/0x1dc [btrfs]) > > Any help you can offer would be greatly appreciated! > > -Eric > > [ 80.005546] sysrq: SysRq : Trigger a crash > [ 80.018339] pgd = c0004000 > [ 80.021160] [00000000] *pgd=00000000 > [ 80.024897] Internal error: Oops: 817 [#1] ARM > [ 80.029531] Modules linked in: btrfs raid10 raid456 async_raid6_recov > async_memcpy async_pq async_xor async_tx xor > raid6_pq raid0 multipath linear zram lz4_compress zsmalloc dm_thin_pool > dm_persistent_data dm_bio_prison dm_snapshot > dm_bufio dm_zero dm_mod raid1 md_mod > [ 80.054411] CPU: 0 PID: 6 Comm: kworker/u2:0 Not tainted > 4.4.6-vr3-00004-g2dfa78e #15 > [ 80.062579] Hardware name: Generic AM33XX (Flattened Device Tree) > [ 80.069476] Workqueue: btrfs-delalloc btrfs_delalloc_helper [btrfs] > [ 80.076026] task: cc8344c0 ti: cc858000 task.ti: cc858000 > [ 80.081669] PC is at sysrq_handle_crash+0x28/0x30 > [ 80.086576] LR is at sysrq_handle_crash+0x24/0x30 > [ 80.091484] pc : [<c02936f4>] lr : [<c02936f0>] psr: 60000193 > [ 80.091484] sp : cc8599e8 ip : cc8599e8 fp : cc8599fc > [ 80.103460] r10: cc8ecb80 r9 : c06008ce r8 : 00000001 > [ 80.108909] r7 : 00000007 r6 : 00000063 r5 : c05d6308 r4 : 00000001 > [ 80.115717] r3 : 00000000 r2 : c05d62c8 r1 : 00000000 r0 : 00000063 > [ 80.122529] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment > none > [ 80.130063] Control: 10c5387d Table: 8a42c019 DAC: 00000051 > [ 80.136057] Process kworker/u2:0 (pid: 6, stack limit = 0xcc858210) > [ 80.142595] Stack: (0xcc8599e8 to 0xcc85a000) > [ 80.147145] 99e0: c02936cc c05e34f4 cc859a24 cc859a00 > c0293d00 c02936d8 > [ 80.155683] 9a00: c05b2bc4 cca49010 00000000 00000101 00000002 00000005 > cc859a34 cc859a28 > [ 80.164220] 9a20: c0294208 c0293c64 cc859a74 cc859a38 c02ad3fc c02941e4 > 00000000 00000000 > [ 80.172758] 9a40: 00000001 00000001 00015200 cca22380 cc8ecb90 00000000 > 00000000 0000009b > [ 80.181296] 9a60: c06008ce cc8ecb80 cc859aac cc859a78 c006652c c02ad050 > c00409f4 c00405e0 > [ 80.189833] 9a80: 00000000 cc8ecb80 cc8ecb90 00000000 00000000 cc804000 > cc15ec50 ccd02518 > [ 80.198373] 9aa0: cc859ac4 cc859ab0 c0066668 c00664d4 00022000 cc8ecb80 > cc859adc cc859ac8 > [ 80.206912] 9ac0: c00690ec c0066644 0000009b c05de0a0 cc859aec cc859ae0 > c0065e58 c006905c > [ 80.215451] 9ae0: cc859b14 cc859af0 c0065f7c c0065e44 cc859b30 c06342c0 > 60000013 ffffffff > [ 80.223990] 9b00: cc859b64 00201000 cc859b2c cc859b18 c00093b8 c0065f34 > bf187974 bf18799c > [ 80.232529] 9b20: cc859bcc cc859b30 c0412e14 c000938c cc15ec50 cc15fba8 > 00201000 00000000 > [ 80.241066] 9b40: 00005000 00000000 cc859bf8 ca144880 00201000 cc15ec50 > ccd02518 cc859bcc > [ 80.249604] 9b60: cc859b80 cc859b80 bf187974 bf18799c 60000013 ffffffff > 00000051 c020aba4 > [ 80.258142] 9b80: 00000000 00000001 00000000 cc859bac cc859bf8 c0203d08 > 20000013 ccd02518 > [ 80.266680] 9ba0: 00000051 00201000 00000000 00201000 00000000 00201000 > 00000000 cc859bf8 > [ 80.275218] 9bc0: cc859c2c cc859bd0 bf18b3b8 bf18791c 41c00000 00000000 > 00201000 00000000 > [ 80.283756] 9be0: 00201000 00000000 00201000 00000000 ccd02518 ccb31200 > cc15fbd4 cc15fbd4 > [ 80.292293] 9c00: cce624d0 00001000 00000000 00000000 00000001 ccb31200 > 00000001 ccd02518 > [ 80.300832] 9c20: cc859cdc cc859c30 bf12c868 bf18b2b8 41c00000 00000000 > 00001000 00000000 > [ 80.309369] 9c40: 00200000 00000000 00000000 cc13aef0 cc859c8c 00000001 > 00000002 00000001 > [ 80.317906] 9c60: 00000001 00000001 00000001 cce62570 00000001 ca157800 > 00001000 00000000 > [ 80.326442] 9c80: cce62500 00000001 bf11e508 bf11da44 00000001 00000000 > 00001000 00000000 > [ 80.334979] 9ca0: 00000000 00000000 00200000 00000000 00000001 00000011 > 00000001 cc859daf > [ 80.343517] 9cc0: 00000001 00001000 00000000 ccdf7000 cc859d4c cc859ce0 > bf12cff4 bf12c1f8 > [ 80.352054] 9ce0: 00000000 00000000 41c00000 00000000 cc859daf 00000001 > 00000011 00000000 > [ 80.360590] 9d00: 00000001 00000000 00001000 00000000 00000000 00000001 > 00000011 00000000 > [ 80.369127] 9d20: 00000000 00000000 cc859daf 00000001 00000000 cc126350 > 00000000 ccdf7000 > [ 80.377664] 9d40: cc859dec cc859d50 bf145250 bf12cef4 00001000 00000000 > 00000000 00000000 > [ 80.386201] 9d60: 41c00000 00000000 cc859daf 00000001 00000001 ca264f80 > 00000000 00000000 > [ 80.394737] 9d80: cc8e0480 00000000 ccdf7000 cfdc3e60 00000fff 00000000 > 00001000 00000000 > [ 80.403275] 9da0: 41c00000 00000000 ca40a9a4 0040a980 00000000 a8000000 > 00000000 00000000 > [ 80.411812] 9dc0: bf15e2a8 00000000 ca2ff980 00000200 ca40a9a4 ca40a980 > cc126350 cc126268 > [ 80.420349] 9de0: cc859e94 cc859df0 bf146838 bf1450e0 00000fff 00000000 > cc859e4c cc859e50 > [ 80.428887] 9e00: 00000000 00000000 cfdc3e60 00000000 00001000 00000000 > 00001000 00000002 > [ 80.437423] 9e20: 00001000 00000000 00001000 ccdf7000 ca2ff9a0 cc126258 > 00000000 00000000 > [ 80.445959] 9e40: ca2ff980 ccdf7000 00001000 00000000 00000000 00000fff > 00000000 ca2ff9a8 > [ 80.454496] 9e60: ca2ff980 00000000 cc81c000 00000000 00000000 ca2ff9a8 > ca2ff980 00000100 > [ 80.463033] 9e80: cc81c000 ca269340 cc859eb4 cc859e98 bf146cc4 bf146680 > ca2ff9c4 ca2ff9a8 > [ 80.471570] 9ea0: ca269344 00000200 cc859edc cc859eb8 bf171f20 bf146c38 > cc81c000 ca2ff9b4 > [ 80.480107] 9ec0: cc802c00 cc83c300 00000000 00000000 cc859eec cc859ee0 > bf171ff4 bf171dfc > [ 80.488645] 9ee0: cc859f2c cc859ef0 c00509c0 bf171fec cc81c018 cc802c10 > 00000001 cc802c00 > [ 80.497181] 9f00: 00000088 cc802c10 00000001 cc81c018 00000088 cc802c00 > cc81c000 cc802c00 > [ 80.505719] 9f20: cc859f64 cc859f30 c0050d78 c00508cc cc838b80 cc802c10 > c0050c10 00000000 > [ 80.514257] 9f40: cc838b80 cc81c000 c0050c10 00000000 00000000 00000000 > cc859fac cc859f68 > [ 80.522794] 9f60: c0055948 c0050c1c ffffffff 00000000 cc838b80 cc81c000 > 00000000 cc859f7c > [ 80.531331] 9f80: cc859f7c 00000000 cc859f88 cc859f88 cc838b80 c0055874 > 00000000 00000000 > [ 80.539869] 9fa0: 00000000 cc859fb0 c00155d8 c0055880 00000000 00000000 > 00000000 00000000 > [ 80.548406] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 > 00000000 00000000 > [ 80.556944] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 > fffff9ff fbfffbff > [ 80.565472] Backtrace: > [ 80.568035] [<c02936cc>] (sysrq_handle_crash) from [<c0293d00>] > (__handle_sysrq+0xa8/0x164) > [ 80.576747] r4:c05e34f4 r3:c02936cc > [ 80.580488] [<c0293c58>] (__handle_sysrq) from [<c0294208>] > (handle_sysrq+0x30/0x38) > [ 80.588564] r8:00000005 r7:00000002 r6:00000101 r5:00000000 r4:cca49010 > r3:c05b2bc4 > [ 80.596686] [<c02941d8>] (handle_sysrq) from [<c02ad3fc>] > (serial_omap_irq+0x3b8/0x418) > [ 80.605060] [<c02ad044>] (serial_omap_irq) from [<c006652c>] > (handle_irq_event_percpu+0x64/0x170) > [ 80.614315] r10:cc8ecb80 r9:c06008ce r8:0000009b r7:00000000 r6:00000000 > r5:cc8ecb90 > [ 80.622505] r4:cca22380 > [ 80.625156] [<c00664c8>] (handle_irq_event_percpu) from [<c0066668>] > (handle_irq_event+0x30/0x44) > [ 80.634410] r10:ccd02518 r9:cc15ec50 r8:cc804000 r7:00000000 r6:00000000 > r5:cc8ecb90 > [ 80.642602] r4:cc8ecb80 > [ 80.645255] [<c0066638>] (handle_irq_event) from [<c00690ec>] > (handle_level_irq+0x9c/0x120) > [ 80.653965] r4:cc8ecb80 r3:00022000 > [ 80.657707] [<c0069050>] (handle_level_irq) from [<c0065e58>] > (generic_handle_irq+0x20/0x30) > [ 80.666507] r5:c05de0a0 r4:0000009b > [ 80.670250] [<c0065e38>] (generic_handle_irq) from [<c0065f7c>] > (__handle_domain_irq+0x54/0xac) > [ 80.679334] [<c0065f28>] (__handle_domain_irq) from [<c00093b8>] > (omap_intc_handle_irq+0x38/0x98) > [ 80.688588] r8:00201000 r7:cc859b64 r6:ffffffff r5:60000013 r4:c06342c0 > r3:cc859b30 > [ 80.696710] [<c0009380>] (omap_intc_handle_irq) from [<c0412e14>] > (__irq_svc+0x54/0x70) > [ 80.705060] Exception stack(0xcc859b30 to 0xcc859b78) > [ 80.710331] 9b20: cc15ec50 cc15fba8 > 00201000 00000000 > [ 80.718868] 9b40: 00005000 00000000 cc859bf8 ca144880 00201000 cc15ec50 > ccd02518 cc859bcc > [ 80.727403] 9b60: cc859b80 cc859b80 bf187974 bf18799c 60000013 ffffffff > [ 80.734301] r4:bf18799c r3:bf187974 > [ 80.738518] [<bf187910>] (setup_cluster_no_bitmap [btrfs]) from > [<bf18b3b8>] (btrfs_find_space_cluster+0x10c/0x1dc > [btrfs]) > [ 80.750131] r10:cc859bf8 r9:00000000 r8:00201000 r7:00000000 r6:00201000 > r5:00000000 > [ 80.758322] r4:00201000 > [ 80.761346] [<bf18b2ac>] (btrfs_find_space_cluster [btrfs]) from > [<bf12c868>] (find_free_extent+0x67c/0xcfc [btrfs > ]) > [ 80.772323] r10:ccd02518 r9:00000001 r8:ccb31200 r7:00000001 r6:00000000 > r5:00000000 > [ 80.780514] r4:00001000 > [ 80.783528] [<bf12c1ec>] (find_free_extent [btrfs]) from [<bf12cff4>] > (btrfs_reserve_extent+0x10c/0x210 [btrfs]) > [ 80.794142] r10:ccdf7000 r9:00000000 r8:00001000 r7:00000001 r6:cc859daf > r5:00000001 > [ 80.802333] r4:00000011 > [ 80.805350] [<bf12cee8>] (btrfs_reserve_extent [btrfs]) from [<bf145250>] > (cow_file_range+0x17c/0x5f0 [btrfs]) > [ 80.815783] r10:ccdf7000 r9:00000000 r8:cc126350 r7:00000000 r6:00000001 > r5:cc859daf > [ 80.823973] r4:00000000 > [ 80.826994] [<bf1450d4>] (cow_file_range [btrfs]) from [<bf146838>] > (submit_compressed_extents+0x1c4/0x5b8 [btrfs] > ) > [ 80.837881] r10:cc126268 r9:cc126350 r8:ca40a980 r7:ca40a9a4 r6:00000200 > r5:ca2ff980 > [ 80.846071] r4:00000000 > [ 80.849091] [<bf146674>] (submit_compressed_extents [btrfs]) from > [<bf146cc4>] (async_cow_submit+0x98/0x9c [btrfs] > ) > [ 80.859978] r10:ca269340 r9:cc81c000 r8:00000100 r7:ca2ff980 r6:ca2ff9a8 > r5:00000000 > [ 80.868169] r4:00000000 > [ 80.871191] [<bf146c2c>] (async_cow_submit [btrfs]) from [<bf171f20>] > (normal_work_helper+0x130/0x1d8 [btrfs]) > [ 80.881623] r7:00000200 r6:ca269344 r5:ca2ff9a8 r4:ca2ff9c4 > [ 80.887924] [<bf171df0>] (normal_work_helper [btrfs]) from [<bf171ff4>] > (btrfs_delalloc_helper+0x14/0x18 [btrfs]) > [ 80.898630] r10:00000000 r8:00000000 r7:cc83c300 r6:cc802c00 r5:ca2ff9b4 > r4:cc81c000 > [ 80.907023] [<bf171fe0>] (btrfs_delalloc_helper [btrfs]) from [<c00509c0>] > (process_one_work+0x100/0x314) > [ 80.917010] [<c00508c0>] (process_one_work) from [<c0050d78>] > (worker_thread+0x168/0x4c8) > [ 80.925539] r10:cc802c00 r9:cc81c000 r8:cc802c00 r7:00000088 r6:cc81c018 > r5:00000001 > [ 80.933729] r4:cc802c10 > [ 80.936385] [<c0050c10>] (worker_thread) from [<c0055948>] > (kthread+0xd4/0xf0) > [ 80.943916] r10:00000000 r9:00000000 r8:00000000 r7:c0050c10 r6:cc81c000 > r5:cc838b80 > [ 80.952105] r4:00000000 > [ 80.954766] [<c0055874>] (kthread) from [<c00155d8>] > (ret_from_fork+0x14/0x3c) > [ 80.962298] r7:00000000 r6:00000000 r5:c0055874 r4:cc838b80 > [ 80.968227] Code: e5834000 f57ff04e ebf6357c e3a03000 (e5c34000) > [ 80.974592] ---[ end trace 18ae8f27c987efb8 ]--- > [ 80.979408] Kernel panic - not syncing: Fatal exception in interrupt > [ 80.986039] ---[ end Kernel panic - not syncing: Fatal exception in > interrupt > > > > -- > Eric Wheeler > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
