Re: OOM: Better, but still there on 4.9
On Fri 16-12-16 17:47:25, Chris Mason wrote: > On 12/16/2016 05:14 PM, Michal Hocko wrote: > > On Fri 16-12-16 13:15:18, Chris Mason wrote: > > > On 12/16/2016 02:39 AM, Michal Hocko wrote: > > [...] > > > > I believe the right way to go around this is to pursue what I've started > > > > in [1]. I will try to prepare something for testing today for you. Stay > > > > tuned. But I would be really happy if somebody from the btrfs camp could > > > > check the NOFS aspect of this allocation. We have already seen > > > > allocation stalls from this path quite recently > > > > > > Just double checking, are you asking why we're using GFP_NOFS to avoid > > > going > > > into btrfs from the btrfs writepages call, or are you asking why we aren't > > > allowing highmem? > > > > I am more interested in the NOFS part. Why cannot this be a full > > GFP_KERNEL context? What kind of locks we would lock up when recursing > > to the fs via slab shrinkers? > > > > Since this is our writepages call, any jump into direct reclaim would go to > writepage, which would end up calling the same set of code to read metadata > blocks, which would do a GFP_KERNEL allocation and end up back in writepage > again. But we are not doing pageout on the page cache from the direct reclaim for a long time. So basically the only way to recurse back to the fs code is via slab ([di]cache) shrinkers. Are those a problem as well? -- Michal Hocko SUSE Labs
Re: OOM: Better, but still there on 4.9
On Fri 16-12-16 17:47:25, Chris Mason wrote: > On 12/16/2016 05:14 PM, Michal Hocko wrote: > > On Fri 16-12-16 13:15:18, Chris Mason wrote: > > > On 12/16/2016 02:39 AM, Michal Hocko wrote: > > [...] > > > > I believe the right way to go around this is to pursue what I've started > > > > in [1]. I will try to prepare something for testing today for you. Stay > > > > tuned. But I would be really happy if somebody from the btrfs camp could > > > > check the NOFS aspect of this allocation. We have already seen > > > > allocation stalls from this path quite recently > > > > > > Just double checking, are you asking why we're using GFP_NOFS to avoid > > > going > > > into btrfs from the btrfs writepages call, or are you asking why we aren't > > > allowing highmem? > > > > I am more interested in the NOFS part. Why cannot this be a full > > GFP_KERNEL context? What kind of locks we would lock up when recursing > > to the fs via slab shrinkers? > > > > Since this is our writepages call, any jump into direct reclaim would go to > writepage, which would end up calling the same set of code to read metadata > blocks, which would do a GFP_KERNEL allocation and end up back in writepage > again. But we are not doing pageout on the page cache from the direct reclaim for a long time. So basically the only way to recurse back to the fs code is via slab ([di]cache) shrinkers. Are those a problem as well? -- Michal Hocko SUSE Labs
Re: OOM: Better, but still there on 4.9
On 12/16/2016 05:14 PM, Michal Hocko wrote: On Fri 16-12-16 13:15:18, Chris Mason wrote: On 12/16/2016 02:39 AM, Michal Hocko wrote: [...] I believe the right way to go around this is to pursue what I've started in [1]. I will try to prepare something for testing today for you. Stay tuned. But I would be really happy if somebody from the btrfs camp could check the NOFS aspect of this allocation. We have already seen allocation stalls from this path quite recently Just double checking, are you asking why we're using GFP_NOFS to avoid going into btrfs from the btrfs writepages call, or are you asking why we aren't allowing highmem? I am more interested in the NOFS part. Why cannot this be a full GFP_KERNEL context? What kind of locks we would lock up when recursing to the fs via slab shrinkers? Since this is our writepages call, any jump into direct reclaim would go to writepage, which would end up calling the same set of code to read metadata blocks, which would do a GFP_KERNEL allocation and end up back in writepage again. We'd also have issues with blowing through transaction reservations since the writepage recursion would have to nest into the running transaction. -chris
Re: OOM: Better, but still there on 4.9
On 12/16/2016 05:14 PM, Michal Hocko wrote: On Fri 16-12-16 13:15:18, Chris Mason wrote: On 12/16/2016 02:39 AM, Michal Hocko wrote: [...] I believe the right way to go around this is to pursue what I've started in [1]. I will try to prepare something for testing today for you. Stay tuned. But I would be really happy if somebody from the btrfs camp could check the NOFS aspect of this allocation. We have already seen allocation stalls from this path quite recently Just double checking, are you asking why we're using GFP_NOFS to avoid going into btrfs from the btrfs writepages call, or are you asking why we aren't allowing highmem? I am more interested in the NOFS part. Why cannot this be a full GFP_KERNEL context? What kind of locks we would lock up when recursing to the fs via slab shrinkers? Since this is our writepages call, any jump into direct reclaim would go to writepage, which would end up calling the same set of code to read metadata blocks, which would do a GFP_KERNEL allocation and end up back in writepage again. We'd also have issues with blowing through transaction reservations since the writepage recursion would have to nest into the running transaction. -chris
Re: OOM: Better, but still there on 4.9
On Fri 16-12-16 13:15:18, Chris Mason wrote: > On 12/16/2016 02:39 AM, Michal Hocko wrote: [...] > > I believe the right way to go around this is to pursue what I've started > > in [1]. I will try to prepare something for testing today for you. Stay > > tuned. But I would be really happy if somebody from the btrfs camp could > > check the NOFS aspect of this allocation. We have already seen > > allocation stalls from this path quite recently > > Just double checking, are you asking why we're using GFP_NOFS to avoid going > into btrfs from the btrfs writepages call, or are you asking why we aren't > allowing highmem? I am more interested in the NOFS part. Why cannot this be a full GFP_KERNEL context? What kind of locks we would lock up when recursing to the fs via slab shrinkers? -- Michal Hocko SUSE Labs
Re: OOM: Better, but still there on 4.9
On Fri 16-12-16 13:15:18, Chris Mason wrote: > On 12/16/2016 02:39 AM, Michal Hocko wrote: [...] > > I believe the right way to go around this is to pursue what I've started > > in [1]. I will try to prepare something for testing today for you. Stay > > tuned. But I would be really happy if somebody from the btrfs camp could > > check the NOFS aspect of this allocation. We have already seen > > allocation stalls from this path quite recently > > Just double checking, are you asking why we're using GFP_NOFS to avoid going > into btrfs from the btrfs writepages call, or are you asking why we aren't > allowing highmem? I am more interested in the NOFS part. Why cannot this be a full GFP_KERNEL context? What kind of locks we would lock up when recursing to the fs via slab shrinkers? -- Michal Hocko SUSE Labs
Re: OOM: Better, but still there on 4.9
On 12/16/2016 02:39 AM, Michal Hocko wrote: [CC linux-mm and btrfs guys] On Thu 15-12-16 23:57:04, Nils Holland wrote: [...] Of course, none of this are workloads that are new / special in any way - prior to 4.8, I never experienced any issues doing the exact same things. Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, oom_score_adj=0 Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0 Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted 4.9.0-gentoo #2 Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014 Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1) Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 eff0b634 c1163332 0292 Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 e7fa2900 c1b58785 eff0b734 Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 0007 Dec 15 19:02:18 teela kernel: Call Trace: Dec 15 19:02:18 teela kernel: [] dump_stack+0x47/0x69 Dec 15 19:02:18 teela kernel: [] dump_header+0x60/0x178 Dec 15 19:02:18 teela kernel: [] ? ___ratelimit+0x86/0xe0 Dec 15 19:02:18 teela kernel: [] oom_kill_process+0x20f/0x3d0 Dec 15 19:02:18 teela kernel: [] ? has_capability_noaudit+0x15/0x20 Dec 15 19:02:18 teela kernel: [] ? oom_badness.part.13+0xb7/0x130 Dec 15 19:02:18 teela kernel: [] out_of_memory+0xd9/0x260 Dec 15 19:02:18 teela kernel: [] __alloc_pages_nodemask+0xbfb/0xc80 Dec 15 19:02:18 teela kernel: [] pagecache_get_page+0xad/0x270 Dec 15 19:02:18 teela kernel: [] alloc_extent_buffer+0x116/0x3e0 Dec 15 19:02:18 teela kernel: [] btrfs_find_create_tree_block+0xe/0x10 Dec 15 19:02:18 teela kernel: [] btrfs_alloc_tree_block+0x1ef/0x5f0 Dec 15 19:02:18 teela kernel: [] __btrfs_cow_block+0x143/0x5f0 Dec 15 19:02:18 teela kernel: [] btrfs_cow_block+0x13a/0x220 Dec 15 19:02:18 teela kernel: [] btrfs_search_slot+0x1d1/0x870 Dec 15 19:02:18 teela kernel: [] btrfs_lookup_file_extent+0x4d/0x60 Dec 15 19:02:18 teela kernel: [] __btrfs_drop_extents+0x176/0x1070 Dec 15 19:02:18 teela kernel: [] ? kmem_cache_alloc+0xb7/0x190 Dec 15 19:02:18 teela kernel: [] ? start_transaction+0x65/0x4b0 Dec 15 19:02:18 teela kernel: [] ? __kmalloc+0x147/0x1e0 Dec 15 19:02:18 teela kernel: [] cow_file_range_inline+0x215/0x6b0 Dec 15 19:02:18 teela kernel: [] cow_file_range.isra.49+0x55c/0x6d0 Dec 15 19:02:18 teela kernel: [] ? lock_extent_bits+0x75/0x1e0 Dec 15 19:02:18 teela kernel: [] run_delalloc_range+0x441/0x470 Dec 15 19:02:18 teela kernel: [] writepage_delalloc.isra.47+0x144/0x1e0 Dec 15 19:02:18 teela kernel: [] __extent_writepage+0xd8/0x2b0 Dec 15 19:02:18 teela kernel: [] extent_writepages+0x25c/0x380 Dec 15 19:02:18 teela kernel: [] ? btrfs_real_readdir+0x610/0x610 Dec 15 19:02:18 teela kernel: [] btrfs_writepages+0x1f/0x30 Dec 15 19:02:18 teela kernel: [] do_writepages+0x15/0x40 Dec 15 19:02:18 teela kernel: [] __writeback_single_inode+0x35/0x2f0 Dec 15 19:02:18 teela kernel: [] writeback_sb_inodes+0x16e/0x340 Dec 15 19:02:18 teela kernel: [] wb_writeback+0xaa/0x280 Dec 15 19:02:18 teela kernel: [] wb_workfn+0xd8/0x3e0 Dec 15 19:02:18 teela kernel: [] process_one_work+0x114/0x3e0 Dec 15 19:02:18 teela kernel: [] worker_thread+0x2f/0x4b0 Dec 15 19:02:18 teela kernel: [] ? create_worker+0x180/0x180 Dec 15 19:02:18 teela kernel: [] kthread+0x97/0xb0 Dec 15 19:02:18 teela kernel: [] ? __kthread_parkme+0x60/0x60 Dec 15 19:02:18 teela kernel: [] ret_from_fork+0x1b/0x28 Dec 15 19:02:18 teela kernel: Mem-Info: Dec 15 19:02:18 teela kernel: active_anon:58685 inactive_anon:90 isolated_anon:0 active_file:274324 inactive_file:281962 isolated_file:0 OK, so there is still some anonymous memory that could be swapped out and quite a lot of page cache. This might be harder to reclaim because the allocation is a GFP_NOFS request which is limited in its reclaim capabilities. It might be possible that those pagecache pages are pinned in some way by the the filesystem. Reading harder, its possible those pagecache pages are all from the btree inode. They shouldn't be pinned by btrfs, kswapd should be able to wander in and free a good chunk. What btrfs wants to happen is for this allocation to sit and wait for kswapd to make progress. -chris
Re: OOM: Better, but still there on 4.9
On 12/16/2016 02:39 AM, Michal Hocko wrote: [CC linux-mm and btrfs guys] On Thu 15-12-16 23:57:04, Nils Holland wrote: [...] Of course, none of this are workloads that are new / special in any way - prior to 4.8, I never experienced any issues doing the exact same things. Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, oom_score_adj=0 Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0 Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted 4.9.0-gentoo #2 Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014 Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1) Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 eff0b634 c1163332 0292 Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 e7fa2900 c1b58785 eff0b734 Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 0007 Dec 15 19:02:18 teela kernel: Call Trace: Dec 15 19:02:18 teela kernel: [] dump_stack+0x47/0x69 Dec 15 19:02:18 teela kernel: [] dump_header+0x60/0x178 Dec 15 19:02:18 teela kernel: [] ? ___ratelimit+0x86/0xe0 Dec 15 19:02:18 teela kernel: [] oom_kill_process+0x20f/0x3d0 Dec 15 19:02:18 teela kernel: [] ? has_capability_noaudit+0x15/0x20 Dec 15 19:02:18 teela kernel: [] ? oom_badness.part.13+0xb7/0x130 Dec 15 19:02:18 teela kernel: [] out_of_memory+0xd9/0x260 Dec 15 19:02:18 teela kernel: [] __alloc_pages_nodemask+0xbfb/0xc80 Dec 15 19:02:18 teela kernel: [] pagecache_get_page+0xad/0x270 Dec 15 19:02:18 teela kernel: [] alloc_extent_buffer+0x116/0x3e0 Dec 15 19:02:18 teela kernel: [] btrfs_find_create_tree_block+0xe/0x10 Dec 15 19:02:18 teela kernel: [] btrfs_alloc_tree_block+0x1ef/0x5f0 Dec 15 19:02:18 teela kernel: [] __btrfs_cow_block+0x143/0x5f0 Dec 15 19:02:18 teela kernel: [] btrfs_cow_block+0x13a/0x220 Dec 15 19:02:18 teela kernel: [] btrfs_search_slot+0x1d1/0x870 Dec 15 19:02:18 teela kernel: [] btrfs_lookup_file_extent+0x4d/0x60 Dec 15 19:02:18 teela kernel: [] __btrfs_drop_extents+0x176/0x1070 Dec 15 19:02:18 teela kernel: [] ? kmem_cache_alloc+0xb7/0x190 Dec 15 19:02:18 teela kernel: [] ? start_transaction+0x65/0x4b0 Dec 15 19:02:18 teela kernel: [] ? __kmalloc+0x147/0x1e0 Dec 15 19:02:18 teela kernel: [] cow_file_range_inline+0x215/0x6b0 Dec 15 19:02:18 teela kernel: [] cow_file_range.isra.49+0x55c/0x6d0 Dec 15 19:02:18 teela kernel: [] ? lock_extent_bits+0x75/0x1e0 Dec 15 19:02:18 teela kernel: [] run_delalloc_range+0x441/0x470 Dec 15 19:02:18 teela kernel: [] writepage_delalloc.isra.47+0x144/0x1e0 Dec 15 19:02:18 teela kernel: [] __extent_writepage+0xd8/0x2b0 Dec 15 19:02:18 teela kernel: [] extent_writepages+0x25c/0x380 Dec 15 19:02:18 teela kernel: [] ? btrfs_real_readdir+0x610/0x610 Dec 15 19:02:18 teela kernel: [] btrfs_writepages+0x1f/0x30 Dec 15 19:02:18 teela kernel: [] do_writepages+0x15/0x40 Dec 15 19:02:18 teela kernel: [] __writeback_single_inode+0x35/0x2f0 Dec 15 19:02:18 teela kernel: [] writeback_sb_inodes+0x16e/0x340 Dec 15 19:02:18 teela kernel: [] wb_writeback+0xaa/0x280 Dec 15 19:02:18 teela kernel: [] wb_workfn+0xd8/0x3e0 Dec 15 19:02:18 teela kernel: [] process_one_work+0x114/0x3e0 Dec 15 19:02:18 teela kernel: [] worker_thread+0x2f/0x4b0 Dec 15 19:02:18 teela kernel: [] ? create_worker+0x180/0x180 Dec 15 19:02:18 teela kernel: [] kthread+0x97/0xb0 Dec 15 19:02:18 teela kernel: [] ? __kthread_parkme+0x60/0x60 Dec 15 19:02:18 teela kernel: [] ret_from_fork+0x1b/0x28 Dec 15 19:02:18 teela kernel: Mem-Info: Dec 15 19:02:18 teela kernel: active_anon:58685 inactive_anon:90 isolated_anon:0 active_file:274324 inactive_file:281962 isolated_file:0 OK, so there is still some anonymous memory that could be swapped out and quite a lot of page cache. This might be harder to reclaim because the allocation is a GFP_NOFS request which is limited in its reclaim capabilities. It might be possible that those pagecache pages are pinned in some way by the the filesystem. Reading harder, its possible those pagecache pages are all from the btree inode. They shouldn't be pinned by btrfs, kswapd should be able to wander in and free a good chunk. What btrfs wants to happen is for this allocation to sit and wait for kswapd to make progress. -chris
Re: OOM: Better, but still there on 4.9
On 12/16/2016 02:39 AM, Michal Hocko wrote: [CC linux-mm and btrfs guys] On Thu 15-12-16 23:57:04, Nils Holland wrote: [...] Of course, none of this are workloads that are new / special in any way - prior to 4.8, I never experienced any issues doing the exact same things. Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, oom_score_adj=0 Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0 Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted 4.9.0-gentoo #2 Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014 Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1) Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 eff0b634 c1163332 0292 Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 e7fa2900 c1b58785 eff0b734 Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 0007 Dec 15 19:02:18 teela kernel: Call Trace: Dec 15 19:02:18 teela kernel: [] dump_stack+0x47/0x69 Dec 15 19:02:18 teela kernel: [] dump_header+0x60/0x178 Dec 15 19:02:18 teela kernel: [] ? ___ratelimit+0x86/0xe0 Dec 15 19:02:18 teela kernel: [] oom_kill_process+0x20f/0x3d0 Dec 15 19:02:18 teela kernel: [] ? has_capability_noaudit+0x15/0x20 Dec 15 19:02:18 teela kernel: [] ? oom_badness.part.13+0xb7/0x130 Dec 15 19:02:18 teela kernel: [] out_of_memory+0xd9/0x260 Dec 15 19:02:18 teela kernel: [] __alloc_pages_nodemask+0xbfb/0xc80 Dec 15 19:02:18 teela kernel: [] pagecache_get_page+0xad/0x270 Dec 15 19:02:18 teela kernel: [] alloc_extent_buffer+0x116/0x3e0 Dec 15 19:02:18 teela kernel: [] btrfs_find_create_tree_block+0xe/0x10 Dec 15 19:02:18 teela kernel: [] btrfs_alloc_tree_block+0x1ef/0x5f0 Dec 15 19:02:18 teela kernel: [] __btrfs_cow_block+0x143/0x5f0 Dec 15 19:02:18 teela kernel: [] btrfs_cow_block+0x13a/0x220 Dec 15 19:02:18 teela kernel: [] btrfs_search_slot+0x1d1/0x870 Dec 15 19:02:18 teela kernel: [] btrfs_lookup_file_extent+0x4d/0x60 Dec 15 19:02:18 teela kernel: [] __btrfs_drop_extents+0x176/0x1070 Dec 15 19:02:18 teela kernel: [] ? kmem_cache_alloc+0xb7/0x190 Dec 15 19:02:18 teela kernel: [] ? start_transaction+0x65/0x4b0 Dec 15 19:02:18 teela kernel: [] ? __kmalloc+0x147/0x1e0 Dec 15 19:02:18 teela kernel: [] cow_file_range_inline+0x215/0x6b0 Dec 15 19:02:18 teela kernel: [] cow_file_range.isra.49+0x55c/0x6d0 Dec 15 19:02:18 teela kernel: [] ? lock_extent_bits+0x75/0x1e0 Dec 15 19:02:18 teela kernel: [] run_delalloc_range+0x441/0x470 Dec 15 19:02:18 teela kernel: [] writepage_delalloc.isra.47+0x144/0x1e0 Dec 15 19:02:18 teela kernel: [] __extent_writepage+0xd8/0x2b0 Dec 15 19:02:18 teela kernel: [] extent_writepages+0x25c/0x380 Dec 15 19:02:18 teela kernel: [] ? btrfs_real_readdir+0x610/0x610 Dec 15 19:02:18 teela kernel: [] btrfs_writepages+0x1f/0x30 Dec 15 19:02:18 teela kernel: [] do_writepages+0x15/0x40 Dec 15 19:02:18 teela kernel: [] __writeback_single_inode+0x35/0x2f0 Dec 15 19:02:18 teela kernel: [] writeback_sb_inodes+0x16e/0x340 Dec 15 19:02:18 teela kernel: [] wb_writeback+0xaa/0x280 Dec 15 19:02:18 teela kernel: [] wb_workfn+0xd8/0x3e0 Dec 15 19:02:18 teela kernel: [] process_one_work+0x114/0x3e0 Dec 15 19:02:18 teela kernel: [] worker_thread+0x2f/0x4b0 Dec 15 19:02:18 teela kernel: [] ? create_worker+0x180/0x180 Dec 15 19:02:18 teela kernel: [] kthread+0x97/0xb0 Dec 15 19:02:18 teela kernel: [] ? __kthread_parkme+0x60/0x60 Dec 15 19:02:18 teela kernel: [] ret_from_fork+0x1b/0x28 Dec 15 19:02:18 teela kernel: Mem-Info: Dec 15 19:02:18 teela kernel: active_anon:58685 inactive_anon:90 isolated_anon:0 active_file:274324 inactive_file:281962 isolated_file:0 OK, so there is still some anonymous memory that could be swapped out and quite a lot of page cache. This might be harder to reclaim because the allocation is a GFP_NOFS request which is limited in its reclaim capabilities. It might be possible that those pagecache pages are pinned in some way by the the filesystem. unevictable:0 dirty:649 writeback:0 unstable:0 slab_reclaimable:40662 slab_unreclaimable:17754 mapped:7382 shmem:202 pagetables:351 bounce:0 free:206736 free_pcp:332 free_cma:0 Dec 15 19:02:18 teela kernel: Node 0 active_anon:234740kB inactive_anon:360kB active_file:1097296kB inactive_file:1127848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:29528kB dirty:2596kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 184320kB anon_thp: 808kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no Dec 15 19:02:18 teela kernel: DMA free:3952kB min:788kB low:984kB high:1180kB active_anon:0kB inactive_anon:0kB active_file:7316kB
Re: OOM: Better, but still there on 4.9
On 12/16/2016 02:39 AM, Michal Hocko wrote: [CC linux-mm and btrfs guys] On Thu 15-12-16 23:57:04, Nils Holland wrote: [...] Of course, none of this are workloads that are new / special in any way - prior to 4.8, I never experienced any issues doing the exact same things. Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, oom_score_adj=0 Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0 Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted 4.9.0-gentoo #2 Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014 Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1) Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 eff0b634 c1163332 0292 Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 e7fa2900 c1b58785 eff0b734 Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 0007 Dec 15 19:02:18 teela kernel: Call Trace: Dec 15 19:02:18 teela kernel: [] dump_stack+0x47/0x69 Dec 15 19:02:18 teela kernel: [] dump_header+0x60/0x178 Dec 15 19:02:18 teela kernel: [] ? ___ratelimit+0x86/0xe0 Dec 15 19:02:18 teela kernel: [] oom_kill_process+0x20f/0x3d0 Dec 15 19:02:18 teela kernel: [] ? has_capability_noaudit+0x15/0x20 Dec 15 19:02:18 teela kernel: [] ? oom_badness.part.13+0xb7/0x130 Dec 15 19:02:18 teela kernel: [] out_of_memory+0xd9/0x260 Dec 15 19:02:18 teela kernel: [] __alloc_pages_nodemask+0xbfb/0xc80 Dec 15 19:02:18 teela kernel: [] pagecache_get_page+0xad/0x270 Dec 15 19:02:18 teela kernel: [] alloc_extent_buffer+0x116/0x3e0 Dec 15 19:02:18 teela kernel: [] btrfs_find_create_tree_block+0xe/0x10 Dec 15 19:02:18 teela kernel: [] btrfs_alloc_tree_block+0x1ef/0x5f0 Dec 15 19:02:18 teela kernel: [] __btrfs_cow_block+0x143/0x5f0 Dec 15 19:02:18 teela kernel: [] btrfs_cow_block+0x13a/0x220 Dec 15 19:02:18 teela kernel: [] btrfs_search_slot+0x1d1/0x870 Dec 15 19:02:18 teela kernel: [] btrfs_lookup_file_extent+0x4d/0x60 Dec 15 19:02:18 teela kernel: [] __btrfs_drop_extents+0x176/0x1070 Dec 15 19:02:18 teela kernel: [] ? kmem_cache_alloc+0xb7/0x190 Dec 15 19:02:18 teela kernel: [] ? start_transaction+0x65/0x4b0 Dec 15 19:02:18 teela kernel: [] ? __kmalloc+0x147/0x1e0 Dec 15 19:02:18 teela kernel: [] cow_file_range_inline+0x215/0x6b0 Dec 15 19:02:18 teela kernel: [] cow_file_range.isra.49+0x55c/0x6d0 Dec 15 19:02:18 teela kernel: [] ? lock_extent_bits+0x75/0x1e0 Dec 15 19:02:18 teela kernel: [] run_delalloc_range+0x441/0x470 Dec 15 19:02:18 teela kernel: [] writepage_delalloc.isra.47+0x144/0x1e0 Dec 15 19:02:18 teela kernel: [] __extent_writepage+0xd8/0x2b0 Dec 15 19:02:18 teela kernel: [] extent_writepages+0x25c/0x380 Dec 15 19:02:18 teela kernel: [] ? btrfs_real_readdir+0x610/0x610 Dec 15 19:02:18 teela kernel: [] btrfs_writepages+0x1f/0x30 Dec 15 19:02:18 teela kernel: [] do_writepages+0x15/0x40 Dec 15 19:02:18 teela kernel: [] __writeback_single_inode+0x35/0x2f0 Dec 15 19:02:18 teela kernel: [] writeback_sb_inodes+0x16e/0x340 Dec 15 19:02:18 teela kernel: [] wb_writeback+0xaa/0x280 Dec 15 19:02:18 teela kernel: [] wb_workfn+0xd8/0x3e0 Dec 15 19:02:18 teela kernel: [] process_one_work+0x114/0x3e0 Dec 15 19:02:18 teela kernel: [] worker_thread+0x2f/0x4b0 Dec 15 19:02:18 teela kernel: [] ? create_worker+0x180/0x180 Dec 15 19:02:18 teela kernel: [] kthread+0x97/0xb0 Dec 15 19:02:18 teela kernel: [] ? __kthread_parkme+0x60/0x60 Dec 15 19:02:18 teela kernel: [] ret_from_fork+0x1b/0x28 Dec 15 19:02:18 teela kernel: Mem-Info: Dec 15 19:02:18 teela kernel: active_anon:58685 inactive_anon:90 isolated_anon:0 active_file:274324 inactive_file:281962 isolated_file:0 OK, so there is still some anonymous memory that could be swapped out and quite a lot of page cache. This might be harder to reclaim because the allocation is a GFP_NOFS request which is limited in its reclaim capabilities. It might be possible that those pagecache pages are pinned in some way by the the filesystem. unevictable:0 dirty:649 writeback:0 unstable:0 slab_reclaimable:40662 slab_unreclaimable:17754 mapped:7382 shmem:202 pagetables:351 bounce:0 free:206736 free_pcp:332 free_cma:0 Dec 15 19:02:18 teela kernel: Node 0 active_anon:234740kB inactive_anon:360kB active_file:1097296kB inactive_file:1127848kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:29528kB dirty:2596kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 184320kB anon_thp: 808kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no Dec 15 19:02:18 teela kernel: DMA free:3952kB min:788kB low:984kB high:1180kB active_anon:0kB inactive_anon:0kB active_file:7316kB
Re: OOM: Better, but still there on 4.9
[CC linux-mm and btrfs guys] On Thu 15-12-16 23:57:04, Nils Holland wrote: [...] > Of course, none of this are workloads that are new / special in any > way - prior to 4.8, I never experienced any issues doing the exact > same things. > > Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: > gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, > oom_score_adj=0 > Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0 > Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted > 4.9.0-gentoo #2 > Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 > Notebook PC/21F7, BIOS F.22 08/06/2014 > Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1) > Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 eff0b634 > c1163332 0292 > Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 > e7fa2900 c1b58785 eff0b734 > Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 > 0007 > Dec 15 19:02:18 teela kernel: Call Trace: > Dec 15 19:02:18 teela kernel: [] dump_stack+0x47/0x69 > Dec 15 19:02:18 teela kernel: [] dump_header+0x60/0x178 > Dec 15 19:02:18 teela kernel: [] ? ___ratelimit+0x86/0xe0 > Dec 15 19:02:18 teela kernel: [] oom_kill_process+0x20f/0x3d0 > Dec 15 19:02:18 teela kernel: [] ? has_capability_noaudit+0x15/0x20 > Dec 15 19:02:18 teela kernel: [] ? oom_badness.part.13+0xb7/0x130 > Dec 15 19:02:18 teela kernel: [] out_of_memory+0xd9/0x260 > Dec 15 19:02:18 teela kernel: [] __alloc_pages_nodemask+0xbfb/0xc80 > Dec 15 19:02:18 teela kernel: [] pagecache_get_page+0xad/0x270 > Dec 15 19:02:18 teela kernel: [] alloc_extent_buffer+0x116/0x3e0 > Dec 15 19:02:18 teela kernel: [] > btrfs_find_create_tree_block+0xe/0x10 > Dec 15 19:02:18 teela kernel: [] btrfs_alloc_tree_block+0x1ef/0x5f0 > Dec 15 19:02:18 teela kernel: [] __btrfs_cow_block+0x143/0x5f0 > Dec 15 19:02:18 teela kernel: [] btrfs_cow_block+0x13a/0x220 > Dec 15 19:02:18 teela kernel: [] btrfs_search_slot+0x1d1/0x870 > Dec 15 19:02:18 teela kernel: [] btrfs_lookup_file_extent+0x4d/0x60 > Dec 15 19:02:18 teela kernel: [] __btrfs_drop_extents+0x176/0x1070 > Dec 15 19:02:18 teela kernel: [] ? kmem_cache_alloc+0xb7/0x190 > Dec 15 19:02:18 teela kernel: [] ? start_transaction+0x65/0x4b0 > Dec 15 19:02:18 teela kernel: [] ? __kmalloc+0x147/0x1e0 > Dec 15 19:02:18 teela kernel: [] cow_file_range_inline+0x215/0x6b0 > Dec 15 19:02:18 teela kernel: [] cow_file_range.isra.49+0x55c/0x6d0 > Dec 15 19:02:18 teela kernel: [] ? lock_extent_bits+0x75/0x1e0 > Dec 15 19:02:18 teela kernel: [] run_delalloc_range+0x441/0x470 > Dec 15 19:02:18 teela kernel: [] > writepage_delalloc.isra.47+0x144/0x1e0 > Dec 15 19:02:18 teela kernel: [] __extent_writepage+0xd8/0x2b0 > Dec 15 19:02:18 teela kernel: [] extent_writepages+0x25c/0x380 > Dec 15 19:02:18 teela kernel: [] ? btrfs_real_readdir+0x610/0x610 > Dec 15 19:02:18 teela kernel: [] btrfs_writepages+0x1f/0x30 > Dec 15 19:02:18 teela kernel: [] do_writepages+0x15/0x40 > Dec 15 19:02:18 teela kernel: [] > __writeback_single_inode+0x35/0x2f0 > Dec 15 19:02:18 teela kernel: [] writeback_sb_inodes+0x16e/0x340 > Dec 15 19:02:18 teela kernel: [] wb_writeback+0xaa/0x280 > Dec 15 19:02:18 teela kernel: [] wb_workfn+0xd8/0x3e0 > Dec 15 19:02:18 teela kernel: [] process_one_work+0x114/0x3e0 > Dec 15 19:02:18 teela kernel: [] worker_thread+0x2f/0x4b0 > Dec 15 19:02:18 teela kernel: [] ? create_worker+0x180/0x180 > Dec 15 19:02:18 teela kernel: [] kthread+0x97/0xb0 > Dec 15 19:02:18 teela kernel: [] ? __kthread_parkme+0x60/0x60 > Dec 15 19:02:18 teela kernel: [] ret_from_fork+0x1b/0x28 > Dec 15 19:02:18 teela kernel: Mem-Info: > Dec 15 19:02:18 teela kernel: active_anon:58685 inactive_anon:90 > isolated_anon:0 >active_file:274324 inactive_file:281962 > isolated_file:0 OK, so there is still some anonymous memory that could be swapped out and quite a lot of page cache. This might be harder to reclaim because the allocation is a GFP_NOFS request which is limited in its reclaim capabilities. It might be possible that those pagecache pages are pinned in some way by the the filesystem. >unevictable:0 dirty:649 writeback:0 unstable:0 >slab_reclaimable:40662 slab_unreclaimable:17754 >mapped:7382 shmem:202 pagetables:351 bounce:0 >free:206736 free_pcp:332 free_cma:0 > Dec 15 19:02:18 teela kernel: Node 0 active_anon:234740kB inactive_anon:360kB > active_file:1097296kB inactive_file:1127848kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB mapped:29528kB dirty:2596kB > writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 184320kB anon_thp: > 808kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no > Dec 15 19:02:18 teela
Re: OOM: Better, but still there on 4.9
[CC linux-mm and btrfs guys] On Thu 15-12-16 23:57:04, Nils Holland wrote: [...] > Of course, none of this are workloads that are new / special in any > way - prior to 4.8, I never experienced any issues doing the exact > same things. > > Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: > gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, > oom_score_adj=0 > Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0 > Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted > 4.9.0-gentoo #2 > Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 > Notebook PC/21F7, BIOS F.22 08/06/2014 > Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1) > Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 eff0b634 > c1163332 0292 > Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 > e7fa2900 c1b58785 eff0b734 > Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 > 0007 > Dec 15 19:02:18 teela kernel: Call Trace: > Dec 15 19:02:18 teela kernel: [] dump_stack+0x47/0x69 > Dec 15 19:02:18 teela kernel: [] dump_header+0x60/0x178 > Dec 15 19:02:18 teela kernel: [] ? ___ratelimit+0x86/0xe0 > Dec 15 19:02:18 teela kernel: [] oom_kill_process+0x20f/0x3d0 > Dec 15 19:02:18 teela kernel: [] ? has_capability_noaudit+0x15/0x20 > Dec 15 19:02:18 teela kernel: [] ? oom_badness.part.13+0xb7/0x130 > Dec 15 19:02:18 teela kernel: [] out_of_memory+0xd9/0x260 > Dec 15 19:02:18 teela kernel: [] __alloc_pages_nodemask+0xbfb/0xc80 > Dec 15 19:02:18 teela kernel: [] pagecache_get_page+0xad/0x270 > Dec 15 19:02:18 teela kernel: [] alloc_extent_buffer+0x116/0x3e0 > Dec 15 19:02:18 teela kernel: [] > btrfs_find_create_tree_block+0xe/0x10 > Dec 15 19:02:18 teela kernel: [] btrfs_alloc_tree_block+0x1ef/0x5f0 > Dec 15 19:02:18 teela kernel: [] __btrfs_cow_block+0x143/0x5f0 > Dec 15 19:02:18 teela kernel: [] btrfs_cow_block+0x13a/0x220 > Dec 15 19:02:18 teela kernel: [] btrfs_search_slot+0x1d1/0x870 > Dec 15 19:02:18 teela kernel: [] btrfs_lookup_file_extent+0x4d/0x60 > Dec 15 19:02:18 teela kernel: [] __btrfs_drop_extents+0x176/0x1070 > Dec 15 19:02:18 teela kernel: [] ? kmem_cache_alloc+0xb7/0x190 > Dec 15 19:02:18 teela kernel: [] ? start_transaction+0x65/0x4b0 > Dec 15 19:02:18 teela kernel: [] ? __kmalloc+0x147/0x1e0 > Dec 15 19:02:18 teela kernel: [] cow_file_range_inline+0x215/0x6b0 > Dec 15 19:02:18 teela kernel: [] cow_file_range.isra.49+0x55c/0x6d0 > Dec 15 19:02:18 teela kernel: [] ? lock_extent_bits+0x75/0x1e0 > Dec 15 19:02:18 teela kernel: [] run_delalloc_range+0x441/0x470 > Dec 15 19:02:18 teela kernel: [] > writepage_delalloc.isra.47+0x144/0x1e0 > Dec 15 19:02:18 teela kernel: [] __extent_writepage+0xd8/0x2b0 > Dec 15 19:02:18 teela kernel: [] extent_writepages+0x25c/0x380 > Dec 15 19:02:18 teela kernel: [] ? btrfs_real_readdir+0x610/0x610 > Dec 15 19:02:18 teela kernel: [] btrfs_writepages+0x1f/0x30 > Dec 15 19:02:18 teela kernel: [] do_writepages+0x15/0x40 > Dec 15 19:02:18 teela kernel: [] > __writeback_single_inode+0x35/0x2f0 > Dec 15 19:02:18 teela kernel: [] writeback_sb_inodes+0x16e/0x340 > Dec 15 19:02:18 teela kernel: [] wb_writeback+0xaa/0x280 > Dec 15 19:02:18 teela kernel: [] wb_workfn+0xd8/0x3e0 > Dec 15 19:02:18 teela kernel: [] process_one_work+0x114/0x3e0 > Dec 15 19:02:18 teela kernel: [] worker_thread+0x2f/0x4b0 > Dec 15 19:02:18 teela kernel: [] ? create_worker+0x180/0x180 > Dec 15 19:02:18 teela kernel: [] kthread+0x97/0xb0 > Dec 15 19:02:18 teela kernel: [] ? __kthread_parkme+0x60/0x60 > Dec 15 19:02:18 teela kernel: [] ret_from_fork+0x1b/0x28 > Dec 15 19:02:18 teela kernel: Mem-Info: > Dec 15 19:02:18 teela kernel: active_anon:58685 inactive_anon:90 > isolated_anon:0 >active_file:274324 inactive_file:281962 > isolated_file:0 OK, so there is still some anonymous memory that could be swapped out and quite a lot of page cache. This might be harder to reclaim because the allocation is a GFP_NOFS request which is limited in its reclaim capabilities. It might be possible that those pagecache pages are pinned in some way by the the filesystem. >unevictable:0 dirty:649 writeback:0 unstable:0 >slab_reclaimable:40662 slab_unreclaimable:17754 >mapped:7382 shmem:202 pagetables:351 bounce:0 >free:206736 free_pcp:332 free_cma:0 > Dec 15 19:02:18 teela kernel: Node 0 active_anon:234740kB inactive_anon:360kB > active_file:1097296kB inactive_file:1127848kB unevictable:0kB > isolated(anon):0kB isolated(file):0kB mapped:29528kB dirty:2596kB > writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 184320kB anon_thp: > 808kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no > Dec 15 19:02:18 teela
OOM: Better, but still there on 4.9
Hi folks, I've been reading quite a bit about OOM related issues in recent kernels, and as I've been experiencing some of these myself for quite a while, I thought I'd send in my report in case in the hope that it is useful. Of course, if there's ever anything to test, like some patches or something, I'd be glad to help! Now, my situation: I have two different x86 machines, both equipped with 4 GB of RAM, running 32 bit kernels. I've never really observed any OOM issues until kernel 4.8, but with that kernel, it was enough to unpack a bigger source tarball (like the firefox sources) on a freshly booted system and subsequently compile them, and with very high certainty, the OOL killer would kick in during the compile, killing a whole lot of processes, with the machine then becoming unresponsive and finally a kernel panic taking place. With kernel 4.9, these OOM events seem to be somewhat harder to trigger, but in most cases, unpacking some larger tarballs and then launching a build process on a freshly booted system without many other processed (not even X) running seems to do the trick. However, the consequences don't seem to be as severe as they were in 4.8: The machines did, in fact, become unresponsive in the way that logging in locally (after I'm being thrown out when my bash gets killed by the OOM reaper) is no longer possible, sshing into the machine also doesn't work anymore (most likely because sshd has also been killed), but in all cases the machine was still pingable and the magic SysRequest key combo was still working. I've not yet seen a single real kernel panic as I did with 4.8, still, the only way to get the machine back into action was a hard reboot via the magic SysRequest commands or a power cycle. For the reference, I'm attaching an OOM I've observed under 4.9 at the end of this machine. This one happened after I had been using the machine in a normal fashion for a short time, and then had portage, Gentoo's build system, unpack the firefox sources - compiling hadn't even started yet at this point. Oh yes, I'm using btrfs in case that might make a differences - at least I've seen some references to it in other similar reports I've found on the web. Of course, none of this are workloads that are new / special in any way - prior to 4.8, I never experienced any issues doing the exact same things. Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, oom_score_adj=0 Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0 Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted 4.9.0-gentoo #2 Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014 Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1) Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 eff0b634 c1163332 0292 Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 e7fa2900 c1b58785 eff0b734 Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 0007 Dec 15 19:02:18 teela kernel: Call Trace: Dec 15 19:02:18 teela kernel: [] dump_stack+0x47/0x69 Dec 15 19:02:18 teela kernel: [] dump_header+0x60/0x178 Dec 15 19:02:18 teela kernel: [] ? ___ratelimit+0x86/0xe0 Dec 15 19:02:18 teela kernel: [] oom_kill_process+0x20f/0x3d0 Dec 15 19:02:18 teela kernel: [] ? has_capability_noaudit+0x15/0x20 Dec 15 19:02:18 teela kernel: [] ? oom_badness.part.13+0xb7/0x130 Dec 15 19:02:18 teela kernel: [] out_of_memory+0xd9/0x260 Dec 15 19:02:18 teela kernel: [] __alloc_pages_nodemask+0xbfb/0xc80 Dec 15 19:02:18 teela kernel: [] pagecache_get_page+0xad/0x270 Dec 15 19:02:18 teela kernel: [] alloc_extent_buffer+0x116/0x3e0 Dec 15 19:02:18 teela kernel: [] btrfs_find_create_tree_block+0xe/0x10 Dec 15 19:02:18 teela kernel: [] btrfs_alloc_tree_block+0x1ef/0x5f0 Dec 15 19:02:18 teela kernel: [] __btrfs_cow_block+0x143/0x5f0 Dec 15 19:02:18 teela kernel: [] btrfs_cow_block+0x13a/0x220 Dec 15 19:02:18 teela kernel: [] btrfs_search_slot+0x1d1/0x870 Dec 15 19:02:18 teela kernel: [] btrfs_lookup_file_extent+0x4d/0x60 Dec 15 19:02:18 teela kernel: [] __btrfs_drop_extents+0x176/0x1070 Dec 15 19:02:18 teela kernel: [] ? kmem_cache_alloc+0xb7/0x190 Dec 15 19:02:18 teela kernel: [] ? start_transaction+0x65/0x4b0 Dec 15 19:02:18 teela kernel: [] ? __kmalloc+0x147/0x1e0 Dec 15 19:02:18 teela kernel: [] cow_file_range_inline+0x215/0x6b0 Dec 15 19:02:18 teela kernel: [] cow_file_range.isra.49+0x55c/0x6d0 Dec 15 19:02:18 teela kernel: [] ? lock_extent_bits+0x75/0x1e0 Dec 15 19:02:18 teela kernel: [] run_delalloc_range+0x441/0x470 Dec 15 19:02:18 teela kernel: [] writepage_delalloc.isra.47+0x144/0x1e0 Dec 15 19:02:18 teela kernel: [] __extent_writepage+0xd8/0x2b0 Dec 15 19:02:18 teela kernel: [] extent_writepages+0x25c/0x380 Dec 15 19:02:18 teela kernel: [] ?
OOM: Better, but still there on 4.9
Hi folks, I've been reading quite a bit about OOM related issues in recent kernels, and as I've been experiencing some of these myself for quite a while, I thought I'd send in my report in case in the hope that it is useful. Of course, if there's ever anything to test, like some patches or something, I'd be glad to help! Now, my situation: I have two different x86 machines, both equipped with 4 GB of RAM, running 32 bit kernels. I've never really observed any OOM issues until kernel 4.8, but with that kernel, it was enough to unpack a bigger source tarball (like the firefox sources) on a freshly booted system and subsequently compile them, and with very high certainty, the OOL killer would kick in during the compile, killing a whole lot of processes, with the machine then becoming unresponsive and finally a kernel panic taking place. With kernel 4.9, these OOM events seem to be somewhat harder to trigger, but in most cases, unpacking some larger tarballs and then launching a build process on a freshly booted system without many other processed (not even X) running seems to do the trick. However, the consequences don't seem to be as severe as they were in 4.8: The machines did, in fact, become unresponsive in the way that logging in locally (after I'm being thrown out when my bash gets killed by the OOM reaper) is no longer possible, sshing into the machine also doesn't work anymore (most likely because sshd has also been killed), but in all cases the machine was still pingable and the magic SysRequest key combo was still working. I've not yet seen a single real kernel panic as I did with 4.8, still, the only way to get the machine back into action was a hard reboot via the magic SysRequest commands or a power cycle. For the reference, I'm attaching an OOM I've observed under 4.9 at the end of this machine. This one happened after I had been using the machine in a normal fashion for a short time, and then had portage, Gentoo's build system, unpack the firefox sources - compiling hadn't even started yet at this point. Oh yes, I'm using btrfs in case that might make a differences - at least I've seen some references to it in other similar reports I've found on the web. Of course, none of this are workloads that are new / special in any way - prior to 4.8, I never experienced any issues doing the exact same things. Dec 15 19:02:16 teela kernel: kworker/u4:5 invoked oom-killer: gfp_mask=0x2400840(GFP_NOFS|__GFP_NOFAIL), nodemask=0, order=0, oom_score_adj=0 Dec 15 19:02:18 teela kernel: kworker/u4:5 cpuset=/ mems_allowed=0 Dec 15 19:02:18 teela kernel: CPU: 1 PID: 2603 Comm: kworker/u4:5 Not tainted 4.9.0-gentoo #2 Dec 15 19:02:18 teela kernel: Hardware name: Hewlett-Packard Compaq 15 Notebook PC/21F7, BIOS F.22 08/06/2014 Dec 15 19:02:18 teela kernel: Workqueue: writeback wb_workfn (flush-btrfs-1) Dec 15 19:02:18 teela kernel: eff0b604 c142bcce eff0b734 eff0b634 c1163332 0292 Dec 15 19:02:18 teela kernel: eff0b634 c1431876 eff0b638 e7fb0b00 e7fa2900 e7fa2900 c1b58785 eff0b734 Dec 15 19:02:18 teela kernel: eff0b678 c110795f c1043895 eff0b664 c11075c7 0007 Dec 15 19:02:18 teela kernel: Call Trace: Dec 15 19:02:18 teela kernel: [] dump_stack+0x47/0x69 Dec 15 19:02:18 teela kernel: [] dump_header+0x60/0x178 Dec 15 19:02:18 teela kernel: [] ? ___ratelimit+0x86/0xe0 Dec 15 19:02:18 teela kernel: [] oom_kill_process+0x20f/0x3d0 Dec 15 19:02:18 teela kernel: [] ? has_capability_noaudit+0x15/0x20 Dec 15 19:02:18 teela kernel: [] ? oom_badness.part.13+0xb7/0x130 Dec 15 19:02:18 teela kernel: [] out_of_memory+0xd9/0x260 Dec 15 19:02:18 teela kernel: [] __alloc_pages_nodemask+0xbfb/0xc80 Dec 15 19:02:18 teela kernel: [] pagecache_get_page+0xad/0x270 Dec 15 19:02:18 teela kernel: [] alloc_extent_buffer+0x116/0x3e0 Dec 15 19:02:18 teela kernel: [] btrfs_find_create_tree_block+0xe/0x10 Dec 15 19:02:18 teela kernel: [] btrfs_alloc_tree_block+0x1ef/0x5f0 Dec 15 19:02:18 teela kernel: [] __btrfs_cow_block+0x143/0x5f0 Dec 15 19:02:18 teela kernel: [] btrfs_cow_block+0x13a/0x220 Dec 15 19:02:18 teela kernel: [] btrfs_search_slot+0x1d1/0x870 Dec 15 19:02:18 teela kernel: [] btrfs_lookup_file_extent+0x4d/0x60 Dec 15 19:02:18 teela kernel: [] __btrfs_drop_extents+0x176/0x1070 Dec 15 19:02:18 teela kernel: [] ? kmem_cache_alloc+0xb7/0x190 Dec 15 19:02:18 teela kernel: [] ? start_transaction+0x65/0x4b0 Dec 15 19:02:18 teela kernel: [] ? __kmalloc+0x147/0x1e0 Dec 15 19:02:18 teela kernel: [] cow_file_range_inline+0x215/0x6b0 Dec 15 19:02:18 teela kernel: [] cow_file_range.isra.49+0x55c/0x6d0 Dec 15 19:02:18 teela kernel: [] ? lock_extent_bits+0x75/0x1e0 Dec 15 19:02:18 teela kernel: [] run_delalloc_range+0x441/0x470 Dec 15 19:02:18 teela kernel: [] writepage_delalloc.isra.47+0x144/0x1e0 Dec 15 19:02:18 teela kernel: [] __extent_writepage+0xd8/0x2b0 Dec 15 19:02:18 teela kernel: [] extent_writepages+0x25c/0x380 Dec 15 19:02:18 teela kernel: [] ?
