Hi Guys, I hit the OOM issues with as Box running 4.4.x kernel, so I tried to build a 4.14.3 kernel to try that. The testbed is : NAS box with 2G memory, and a single disk raid , I setup a nfs server with sync mode, add the storage on ESXi servers 6.0 and backup all the VMs on it by the ghettoVCB script , after about 14 hours, Inpot/Output error happened, checked the box , found OOM. # cat /etc/exports "/data/Videos" *(insecure,insecure_locks,no_subtree_check,crossmnt,anonuid=99,anongid=99,root_squash,rw,sync) # uname -a Linux lzx-314-desk 4.14.3.x86_64.1 #1 SMP Fri Dec 1 01:31:25 UTC 2017 x86_64 GNU/Linux # btrfs fi show /data/ Label: '43f611ae:data' uuid: 6fefb319-a21d-476e-9642-565e0600a049 Total devices 1 FS bytes used 292.78GiB devid 1 size 1.81TiB used 296.02GiB path /dev/md127
The stack is : Dec 04 23:47:43 lzx-314-desk kernel: nfsd: page allocation stalls for 621031ms, order:0, mode:0x14000c0(GFP_KERNEL), nodemask=(null) Dec 04 23:47:43 lzx-314-desk kernel: nfsd cpuset=/ mems_allowed=0 Dec 04 23:47:43 lzx-314-desk kernel: CPU: 0 PID: 3376 Comm: nfsd Not tainted 4.14.3.x86_64.1 #1 Dec 04 23:47:43 lzx-314-desk kernel: Hardware name: NETGEAR ReadyNAS 314/To be filled by O.E.M., BIOS 4.6.5 11/05/2013 Dec 04 23:47:43 lzx-314-desk kernel: Call Trace: Dec 04 23:47:43 lzx-314-desk kernel: dump_stack+0x4d/0x6a Dec 04 23:47:43 lzx-314-desk kernel: warn_alloc+0xe3/0x180 Dec 04 23:47:43 lzx-314-desk kernel: __alloc_pages_nodemask+0xb1e/0xed0 Dec 04 23:47:43 lzx-314-desk kernel: svc_recv+0x99/0x900 Dec 04 23:47:43 lzx-314-desk kernel: ? svc_process+0x241/0x690 Dec 04 23:47:43 lzx-314-desk kernel: nfsd+0xd2/0x150 Dec 04 23:47:43 lzx-314-desk kernel: kthread+0x11a/0x150 Dec 04 23:47:43 lzx-314-desk kernel: ? nfsd_destroy+0x60/0x60 Dec 04 23:47:43 lzx-314-desk kernel: ? kthread_create_on_node+0x40/0x40 Dec 04 23:47:43 lzx-314-desk kernel: ret_from_fork+0x22/0x30 Dec 04 23:47:43 lzx-314-desk kernel: readynasd invoked oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=-1000 Dec 04 23:47:43 lzx-314-desk kernel: readynasd cpuset=/ mems_allowed=0 Dec 04 23:47:43 lzx-314-desk kernel: CPU: 3 PID: 3307 Comm: readynasd Not tainted 4.14.3.x86_64.1 #1 Dec 04 23:47:43 lzx-314-desk kernel: Hardware name: NETGEAR ReadyNAS 314/To be filled by O.E.M., BIOS 4.6.5 11/05/2013 Dec 04 23:47:43 lzx-314-desk kernel: Call Trace: Dec 04 23:47:43 lzx-314-desk kernel: dump_stack+0x4d/0x6a Dec 04 23:47:43 lzx-314-desk kernel: dump_header+0x9a/0x21b Dec 04 23:47:43 lzx-314-desk kernel: ? pick_next_task_fair+0x1d5/0x4b0 Dec 04 23:47:43 lzx-314-desk kernel: ? security_capable_noaudit+0x40/0x60 Dec 04 23:47:43 lzx-314-desk kernel: oom_kill_process+0x216/0x430 Dec 04 23:47:43 lzx-314-desk kernel: out_of_memory+0xf9/0x2e0 Dec 04 23:47:43 lzx-314-desk kernel: __alloc_pages_nodemask+0xd6c/0xed0 Dec 04 23:47:43 lzx-314-desk kernel: __read_swap_cache_async+0x11d/0x190 Dec 04 23:47:43 lzx-314-desk kernel: read_swap_cache_async+0x17/0x40 Dec 04 23:47:43 lzx-314-desk kernel: swapin_readahead+0x1f1/0x230 Dec 04 23:47:43 lzx-314-desk kernel: ? find_get_entry+0x19/0xf0 Dec 04 23:47:43 lzx-314-desk kernel: ? pagecache_get_page+0x27/0x210 Dec 04 23:47:43 lzx-314-desk kernel: do_swap_page+0x432/0x590 Dec 04 23:47:43 lzx-314-desk kernel: ? do_swap_page+0x432/0x590 Dec 04 23:47:43 lzx-314-desk kernel: ? poll_select_copy_remaining+0x120/0x120 Dec 04 23:47:43 lzx-314-desk kernel: __handle_mm_fault+0x33e/0xa20 Dec 04 23:47:43 lzx-314-desk kernel: handle_mm_fault+0x14a/0x1d0 Dec 04 23:47:43 lzx-314-desk kernel: __do_page_fault+0x212/0x440 Dec 04 23:47:43 lzx-314-desk kernel: page_fault+0x22/0x30 Dec 04 23:47:43 lzx-314-desk kernel: RIP: 0010:copy_user_generic_unrolled+0x89/0xc0 Dec 04 23:47:43 lzx-314-desk kernel: RSP: 0000:ffffc90000cdbd70 EFLAGS: 00010202 Dec 04 23:47:43 lzx-314-desk kernel: RAX: 0000000000000000 RBX: 0000000000000008 RCX: 0000000000000001 Dec 04 23:47:43 lzx-314-desk kernel: RDX: 0000000000000000 RSI: ffffc90000cdbdd8 RDI: 00007ff45e7fba80 Dec 04 23:47:43 lzx-314-desk kernel: RBP: ffffc90000cdbee8 R08: 0000000000000000 R09: 0000000000000104 Dec 04 23:47:43 lzx-314-desk kernel: R10: ffffc90000cdbd78 R11: 0000000000000104 R12: 0000000000000000 Dec 04 23:47:43 lzx-314-desk kernel: R13: ffffc90000cdbdc0 R14: 00007ff45e7fba80 R15: ffffc90000cdbdc0 Dec 04 23:47:43 lzx-314-desk kernel: ? core_sys_select+0x208/0x2a0 Dec 04 23:47:43 lzx-314-desk kernel: ? __handle_mm_fault+0x4fc/0xa20 Dec 04 23:47:43 lzx-314-desk kernel: ? ktime_get_ts64+0x44/0xe0 Dec 04 23:47:43 lzx-314-desk kernel: SyS_select+0xa6/0xe0 Dec 04 23:47:43 lzx-314-desk kernel: entry_SYSCALL_64_fastpath+0x13/0x94 Dec 04 23:47:43 lzx-314-desk kernel: RIP: 0033:0x7ff488f05893 Dec 04 23:47:43 lzx-314-desk kernel: RSP: 002b:00007ff45e7fb9f0 EFLAGS: 00000293 ORIG_RAX: 0000000000000017 Dec 04 23:47:43 lzx-314-desk kernel: RAX: ffffffffffffffda RBX: 0000000000000018 RCX: 00007ff488f05893 Dec 04 23:47:43 lzx-314-desk kernel: RDX: 0000000000000000 RSI: 00007ff45e7fba80 RDI: 0000000000000019 Dec 04 23:47:43 lzx-314-desk kernel: RBP: 00007ff48fc4b0f0 R08: 00007ff45e7fba70 R09: 00007ff4480008c0 Dec 04 23:47:43 lzx-314-desk kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000 Dec 04 23:47:43 lzx-314-desk kernel: R13: 0000000000000000 R14: 0000000001999670 R15: 00007ff4480008c0 Dec 04 23:47:43 lzx-314-desk kernel: Mem-Info: Dec 04 23:47:43 lzx-314-desk kernel: active_anon:0 inactive_anon:0 isolated_anon:0 active_file:322 inactive_file:461978 isolated_file:352 unevictable:0 dirty:136 writeback:0 unstable:0 slab_reclaimable:15780 slab_unreclaimable:5246 mapped:1 shmem:0 pagetables:1165 bounce:0 free:13867 free_pcp:60 free_cma:0 ...... I tried to fomat the md device to ext4, then it's fine to backup all the VMs, And if I use async option for nfs, it works too, so seems like btrfs is more memory consuming sometimes. I attatched the full logs. Any one hit similar issue or have any ideas ? thanks so much. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
