Re: [Bug 199931] New: systemd/rtorrent file data corruption when using echo 3 >/proc/sys/vm/drop_caches

2018-06-06 Thread Michal Hocko
afterwards run into > > rtorrent/journald corruption issues. Without the echo in place, mysql > > usually > > gets oom-killed after a few days of uptime. Do you have any oom reports to share? -- Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line "unsubscrib

Re: [External] Re: [RFC PATCH v2 00/12] get rid of GFP_ZONE_TABLE/BAD

2018-05-28 Thread Michal Hocko
On Fri 25-05-18 09:43:09, Huaisheng HS1 Ye wrote: > From: Michal Hocko [mailto:mho...@kernel.org] > Sent: Thursday, May 24, 2018 8:19 PM> > > > Let me try to reply your questions. > > > Exactly, GFP_ZONE_TABLE is too complicated. I think there are two > > &

Re: [RFC PATCH v2 00/12] get rid of GFP_ZONE_TABLE/BAD

2018-05-28 Thread Michal Hocko
On Fri 25-05-18 05:00:44, Matthew Wilcox wrote: > On Thu, May 24, 2018 at 05:29:43PM +0200, Michal Hocko wrote: > > > ie if we had more, > > > could we solve our pain by making them more generic? > > > > Well, if you have more you will consume more bi

Re: [RFC PATCH v2 00/12] get rid of GFP_ZONE_TABLE/BAD

2018-05-24 Thread Michal Hocko
On Thu 24-05-18 08:18:18, Matthew Wilcox wrote: > On Thu, May 24, 2018 at 02:23:23PM +0200, Michal Hocko wrote: > > > If we had eight ZONEs, we could offer: > > > > No, please no more zones. What we have is quite a maint. burden on its > > own. Ideally we sh

Re: [RFC PATCH v2 00/12] get rid of GFP_ZONE_TABLE/BAD

2018-05-24 Thread Michal Hocko
On Wed 23-05-18 22:19:19, Matthew Wilcox wrote: > On Tue, May 22, 2018 at 08:37:28PM +0200, Michal Hocko wrote: > > So why is this any better than the current code. Sure I am not a great > > fan of GFP_ZONE_TABLE because of how it is incomprehensible but this > > doesn't look

Re: [External] Re: [RFC PATCH v2 00/12] get rid of GFP_ZONE_TABLE/BAD

2018-05-24 Thread Michal Hocko
On Wed 23-05-18 16:07:16, Huaisheng HS1 Ye wrote: > From: Michal Hocko [mailto:mho...@kernel.org] > Sent: Wednesday, May 23, 2018 2:37 AM > > > > On Mon 21-05-18 23:20:21, Huaisheng Ye wrote: > > > From: Huaisheng Ye <ye...@lenovo.com> > > > >

Re: [RFC PATCH v2 00/12] get rid of GFP_ZONE_TABLE/BAD

2018-05-22 Thread Michal Hocko
-- > include/linux/highmem.h | 4 +- > mm/vmpressure.c | 2 +- > mm/zsmalloc.c| 4 +- > 12 files changed, 26 insertions(+), 103 deletions(-) > > -- > 1.8.3.1 > -- Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH] mm: save/restore current->journal_info in handle_mm_fault

2017-12-14 Thread Michal Hocko
ags); > else > ret = __handle_mm_fault(vma, address, flags); > > + current->journal_info = old_journal_info; > + > if (flags & FAULT_FLAG_USER) { > mem_cgroup_oom_disable(); > /* > -- > 2.13.6 >

Re: [PATCH 4/7] mm: introduce memalloc_nofs_{save,restore} API

2017-03-07 Thread Michal Hocko
On Mon 06-03-17 13:22:14, Andrew Morton wrote: > On Mon, 6 Mar 2017 14:14:05 +0100 Michal Hocko <mho...@kernel.org> wrote: [...] > > --- a/include/linux/gfp.h > > +++ b/include/linux/gfp.h > > @@ -210,8 +210,16 @@ struct vm_area_struct; > > * > > * GFP_

[PATCH 2/7] lockdep: allow to disable reclaim lockup detection

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> The current implementation of the reclaim lockup detection can lead to false positives and those even happen and usually lead to tweak the code to silence the lockdep by using GFP_NOFS even though the context can use __GFP_FS just fine. Se

[PATCH 7/7] jbd2: make the whole kjournald2 kthread NOFS safe

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kjournald2 is central to the transaction commit processing. As such any potential allocation from this kernel thread has to be GFP_NOFS. Make sure to mark the whole kernel thread GFP_NOFS by the memalloc_nofs_save. Suggested-by: Jan Kara <j..

[PATCH 6/7] jbd2: mark the transaction context with the scope GFP_NOFS context

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> now that we have memalloc_nofs_{save,restore} api we can mark the whole transaction context as implicitly GFP_NOFS. All allocations will automatically inherit GFP_NOFS this way. This means that we do not have to mark any of those requests with GF

[PATCH 3/7] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more generic name PF_MEMALLOC_NOFS which is i

[PATCH 1/7] lockdep: teach lockdep about memalloc_noio_save

2017-03-06 Thread Michal Hocko
[ 644.174012] SyS_lsetxattr+0x11/0x20 [ 644.174012] entry_SYSCALL_64_fastpath+0x23/0xc6 Let's fix this by making lockdep explicitly do the shaving of respective GFP flags. Fixes: 934f3072c17c ("mm: clear __GFP_FS when PF_MEMALLOC_NOIO is set") Acked-by: Michal Hocko <mho..

[PATCH 0/7 v5] scope GFP_NOFS api

2017-03-06 Thread Michal Hocko
6 files changed, 106 insertions(+), 37 deletions(-) Shortlog: Michal Hocko (6): lockdep: allow to disable reclaim lockup detection xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS mm: introduce memalloc_nofs_{save,restore} API xfs: use memalloc_nofs_{save,restore} instead of memalloc_no

[PATCH 4/7] mm: introduce memalloc_nofs_{save,restore} API

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> GFP_NOFS context is used for the following 5 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the r

[PATCH 5/7] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2017-03-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} API to prevent from reclaim recursion into the fs because vmalloc can invoke unconditional GFP_KERNEL allocations and these functions might be called from the NOFS co

Re: [PATCH 7/8] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2017-03-06 Thread Michal Hocko
On Tue 17-01-17 08:54:50, Michal Hocko wrote: > On Mon 16-01-17 22:01:18, Theodore Ts'o wrote: > > On Fri, Jan 06, 2017 at 03:11:06PM +0100, Michal Hocko wrote: > > > From: Michal Hocko <mho...@suse.com> > > > > > > This reverts commit c45653

Re: [PATCH 4/6] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2017-02-06 Thread Michal Hocko
On Tue 07-02-17 09:51:50, Dave Chinner wrote: > On Mon, Feb 06, 2017 at 07:47:43PM +0100, Michal Hocko wrote: > > On Mon 06-02-17 10:32:37, Darrick J. Wong wrote: [...] > > > I prefer to keep the "...yet we are likely to be under GFP_NOFS..." > > > wording o

Re: [PATCH 4/6] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2017-02-06 Thread Michal Hocko
On Mon 06-02-17 11:51:11, Darrick J. Wong wrote: > On Mon, Feb 06, 2017 at 07:47:43PM +0100, Michal Hocko wrote: > > On Mon 06-02-17 10:32:37, Darrick J. Wong wrote: > > > On Mon, Feb 06, 2017 at 06:44:15PM +0100, Michal Hocko wrote: > > > > On Mon 06-02-17

Re: [PATCH 4/6] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2017-02-06 Thread Michal Hocko
On Mon 06-02-17 10:32:37, Darrick J. Wong wrote: > On Mon, Feb 06, 2017 at 06:44:15PM +0100, Michal Hocko wrote: > > On Mon 06-02-17 07:39:23, Matthew Wilcox wrote: > > > On Mon, Feb 06, 2017 at 03:07:16PM +0100, Michal Hocko wrote: > > > > +++ b/fs/xfs/xfs_bu

Re: [PATCH 4/6] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2017-02-06 Thread Michal Hocko
On Mon 06-02-17 07:39:23, Matthew Wilcox wrote: > On Mon, Feb 06, 2017 at 03:07:16PM +0100, Michal Hocko wrote: > > +++ b/fs/xfs/xfs_buf.c > > @@ -442,17 +442,17 @@ _xfs_buf_map_pages( > > bp->b_addr = NULL; > > } else { > > int r

Re: [PATCH 1/6] lockdep: allow to disable reclaim lockup detection

2017-02-06 Thread Michal Hocko
On Mon 06-02-17 07:24:00, Matthew Wilcox wrote: > On Mon, Feb 06, 2017 at 03:34:50PM +0100, Michal Hocko wrote: > > This part is not needed for the patch, strictly speaking but I wanted to > > make the code more future proof. > > Understood. I took an extra bit myself for m

Re: [PATCH 1/6] lockdep: allow to disable reclaim lockup detection

2017-02-06 Thread Michal Hocko
On Mon 06-02-17 06:26:41, Matthew Wilcox wrote: > On Mon, Feb 06, 2017 at 03:07:13PM +0100, Michal Hocko wrote: > > While we are at it also make sure that the radix tree doesn't > > accidentaly override tags stored in the upper part of the gfp_mask. > > > diff --git a/lib/

[PATCH 1/6] lockdep: allow to disable reclaim lockup detection

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> The current implementation of the reclaim lockup detection can lead to false positives and those even happen and usually lead to tweak the code to silence the lockdep by using GFP_NOFS even though the context can use __GFP_FS just fine. Se

[PATCH 2/6] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more generic name PF_MEMALLOC_NOFS which is i

[PATCH 0/6 v4] scope GFP_NOFS api

2017-02-06 Thread Michal Hocko
| 10 ++ mm/vmscan.c | 6 +++--- 15 files changed, 100 insertions(+), 36 deletions(-) Shortlog: Michal Hocko (6): lockdep: allow to disable reclaim lockup detection xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS mm: introduce memalloc_nofs_{save,restore

[PATCH 5/6] jbd2: mark the transaction context with the scope GFP_NOFS context

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> now that we have memalloc_nofs_{save,restore} api we can mark the whole transaction context as implicitly GFP_NOFS. All allocations will automatically inherit GFP_NOFS this way. This means that we do not have to mark any of those requests with GF

[PATCH 4/6] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} API to prevent from reclaim recursion into the fs because vmalloc can invoke unconditional GFP_KERNEL allocations and these functions might be called from the NOFS co

[PATCH 6/6] jbd2: make the whole kjournald2 kthread NOFS safe

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kjournald2 is central to the transaction commit processing. As such any potential allocation from this kernel thread has to be GFP_NOFS. Make sure to mark the whole kernel thread GFP_NOFS by the memalloc_nofs_save. Suggested-by: Jan Kara <j..

[PATCH 3/6] mm: introduce memalloc_nofs_{save,restore} API

2017-02-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> GFP_NOFS context is used for the following 5 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the r

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-02-03 Thread Michal Hocko
On Mon 30-01-17 09:12:10, Michal Hocko wrote: > On Fri 27-01-17 11:40:42, Theodore Ts'o wrote: > > On Fri, Jan 27, 2017 at 10:37:35AM +0100, Michal Hocko wrote: > > > If this ever turn out to be a problem and with the vmapped stacks we > > > have good chances t

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-30 Thread Michal Hocko
On Fri 27-01-17 11:40:42, Theodore Ts'o wrote: > On Fri, Jan 27, 2017 at 10:37:35AM +0100, Michal Hocko wrote: > > If this ever turn out to be a problem and with the vmapped stacks we > > have good chances to get a proper stack traces on a potential overflow > > we can add

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-27 Thread Michal Hocko
On Fri 27-01-17 01:13:18, Theodore Ts'o wrote: > On Thu, Jan 26, 2017 at 08:44:55AM +0100, Michal Hocko wrote: > > > > I'm convinced the current series is OK, only real life will tell us > > > > whether > > > > we missed something or not ;) > >

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-25 Thread Michal Hocko
On Thu 19-01-17 10:44:05, Michal Hocko wrote: > On Thu 19-01-17 10:22:36, Jan Kara wrote: > > On Thu 19-01-17 09:39:56, Michal Hocko wrote: > > > On Tue 17-01-17 18:29:25, Jan Kara wrote: > > > > On Tue 17-01-17 17:16:19, Michal Hocko wrote: > > > > >

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Thu 19-01-17 10:22:36, Jan Kara wrote: > On Thu 19-01-17 09:39:56, Michal Hocko wrote: > > On Tue 17-01-17 18:29:25, Jan Kara wrote: > > > On Tue 17-01-17 17:16:19, Michal Hocko wrote: > > > > > > But before going to play with that I am really wo

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-19 Thread Michal Hocko
On Tue 17-01-17 18:29:25, Jan Kara wrote: > On Tue 17-01-17 17:16:19, Michal Hocko wrote: > > > > But before going to play with that I am really wondering whether we need > > > > all this with no journal at all. AFAIU what Jack told me it is the > > > > journ

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-18 Thread Michal Hocko
On Tue 17-01-17 14:04:03, Andreas Dilger wrote: > On Jan 17, 2017, at 8:59 AM, Theodore Ts'o <ty...@mit.edu> wrote: > > > > On Tue, Jan 17, 2017 at 04:18:17PM +0100, Michal Hocko wrote: > >> > >> OK, so I've been staring into the code and AFAIU current-&

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-17 Thread Michal Hocko
On Tue 17-01-17 10:59:16, Theodore Ts'o wrote: > On Tue, Jan 17, 2017 at 04:18:17PM +0100, Michal Hocko wrote: > > > > OK, so I've been staring into the code and AFAIU current->journal_info > > can contain my stored information. I could either hijack part of the >

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-17 Thread Michal Hocko
On Tue 17-01-17 09:24:25, Michal Hocko wrote: > On Mon 16-01-17 21:56:07, Theodore Ts'o wrote: > > On Fri, Jan 06, 2017 at 03:11:07PM +0100, Michal Hocko wrote: > > > From: Michal Hocko <mho...@suse.com> > > > > > > This reverts commit 216

Re: [PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-17 Thread Michal Hocko
On Mon 16-01-17 21:56:07, Theodore Ts'o wrote: > On Fri, Jan 06, 2017 at 03:11:07PM +0100, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > This reverts commit 216553c4b7f3e3e2beb4981cddca9b2027523928. Now that > > the transaction contex

Re: [PATCH 7/8] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2017-01-16 Thread Michal Hocko
On Mon 16-01-17 22:01:18, Theodore Ts'o wrote: > On Fri, Jan 06, 2017 at 03:11:06PM +0100, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > This reverts commit c45653c341f5c8a0ce19c8f0ad4678640849cb86 because > > sb_getblk_gfp is n

Re: [PATCH 1/2] etrfs: fix up misleading GFP_NOFS usage in btrfs_releasepage

2017-01-11 Thread Michal Hocko
On Wed 11-01-17 14:55:50, David Sterba wrote: > On Mon, Jan 09, 2017 at 03:39:02PM +0100, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > b335b0034e25 ("Btrfs: Avoid using __GFP_HIGHMEM with slab allocator") > > has reduced the alloc

Re: [PATCH 1/2] etrfs: fix up misleading GFP_NOFS usage in btrfs_releasepage

2017-01-11 Thread Michal Hocko
On Wed 11-01-17 14:55:50, David Sterba wrote: [...] > But otherwise looks ok to me, I'm going to merge the patch. Thanks. I have only now noticed typo in the subject. s@etrfs:@btrfs:@ -- Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs"

[PATCH 1/2] etrfs: fix up misleading GFP_NOFS usage in btrfs_releasepage

2017-01-09 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> b335b0034e25 ("Btrfs: Avoid using __GFP_HIGHMEM with slab allocator") has reduced the allocation mask in btrfs_releasepage to GFP_NOFS just to prevent from giving an unappropriate gfp mask to the slab allocator deeper down the callchain (in al

[PATCH 2/2] btrfs: drop gfp mask tweaking in try_release_extent_state

2017-01-09 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> try_release_extent_state reduces the gfp mask to GFP_NOFS if it is compatible. This is true for GFP_KERNEL as well. There is no real reason to do that though. There is no new lock taken down the the only consumer of the gfp mask

Re: [PATCH 2/8] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2017-01-09 Thread Michal Hocko
On Mon 09-01-17 13:59:05, Vlastimil Babka wrote: > On 01/06/2017 03:11 PM, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite > > some time ago. We would like to make

Re: [PATCH 4/8] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2017-01-09 Thread Michal Hocko
On Mon 09-01-17 15:08:27, Vlastimil Babka wrote: > On 01/06/2017 03:11 PM, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} > > API to prevent from reclaim recursio

Re: [PATCH 3/8] mm: introduce memalloc_nofs_{save,restore} API

2017-01-09 Thread Michal Hocko
On Mon 09-01-17 14:42:10, Michal Hocko wrote: > On Mon 09-01-17 14:04:21, Vlastimil Babka wrote: [...] > Now that you have opened this I have noticed that the code is wrong > here because GFP_HIGHUSER_MOVABLE & ~GFP_RECLAIM_MASK would overwrite > the removed GFP_FS. Blee, it

Re: [PATCH 3/8] mm: introduce memalloc_nofs_{save,restore} API

2017-01-09 Thread Michal Hocko
struct scan_control sc = { .nr_to_reclaim = max(nr_pages, SWAP_CLUSTER_MAX), - .gfp_mask = (gfp_mask = current_gfp_context(gfp_mask)), + .gfp_mask = gfp_mask, .order = order, .priority = NODE_RECLAIM_PRIORITY, .may_wr

[DEBUG PATCH 2/2] silent warnings which we cannot do anything about

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE There are some code paths used by all the filesystems which we cannot change to drop the GFP_NOFS, yet they generate a lot of warnings. Provide {disable,enable}_scope_gfp_check to silence

[DEBUG PATCH 1/2] mm, debug: report when GFP_NO{FS,IO} is used explicitly from memalloc_no{fs,io}_{save,restore} context

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE It is desirable to reduce the direct GFP_NO{FS,IO} usage at minimum and prefer scope usage defined by memalloc_no{fs,io}_{save,restore} API. Let's help this process and add a debuggin

[DEBUG PATCH 0/2] debug explicit GFP_NO{FS,IO} usage from the scope context

2017-01-06 Thread Michal Hocko
These two patches should help to identify explicit GFP_NO{FS,IO} usage from withing a scope context and reduce such a usage as a result. Such a usage can be changed to the full GFP_KERNEL because all the calls from within the NO{FS,IO} scope will drop the __GFP_FS resp. __GFP_IO automatically and

[PATCH 1/8] lockdep: allow to disable reclaim lockup detection

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> The current implementation of the reclaim lockup detection can lead to false positives and those even happen and usually lead to tweak the code to silence the lockdep by using GFP_NOFS even though the context can use __GFP_FS just fine. Se

[PATCH 7/8] Revert "ext4: avoid deadlocks in the writeback path by using sb_getblk_gfp"

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> This reverts commit c45653c341f5c8a0ce19c8f0ad4678640849cb86 because sb_getblk_gfp is not really needed as sb_getblk __getblk_gfp __getblk_slow grow_buffers grow_dev_page gfp_mask = mapping_gfp_constraint(inode->

[PATCH 3/8] mm: introduce memalloc_nofs_{save,restore} API

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> GFP_NOFS context is used for the following 5 reasons currently - to prevent from deadlocks when the lock held by the allocation context would be needed during the memory reclaim - to prevent from stack overflows during the r

[PATCH 0/8 v3] scope GFP_NOFS api

2017-01-06 Thread Michal Hocko
++-- kernel/locking/lockdep.c | 6 +- lib/radix-tree.c | 2 ++ mm/page_alloc.c | 8 +--- mm/vmscan.c | 6 +++--- 19 files changed, 109 insertions(+), 45 deletions(-) Shortlog: Michal Hocko (8): lockdep: allow to disable reclaim lockup detection xfs

[PATCH 6/8] jbd2: make the whole kjournald2 kthread NOFS safe

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kjournald2 is central to the transaction commit processing. As such any potential allocation from this kernel thread has to be GFP_NOFS. Make sure to mark the whole kernel thread GFP_NOFS by the memalloc_nofs_save. Suggested-by: Jan Kara <j...@suse.c

[PATCH 2/8] xfs: abstract PF_FSTRANS to PF_MEMALLOC_NOFS

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> xfs has defined PF_FSTRANS to declare a scope GFP_NOFS semantic quite some time ago. We would like to make this concept more generic and use it for other filesystems as well. Let's start by giving the flag a more generic name PF_MEMALLOC_NOFS which is i

[PATCH 4/8] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> kmem_zalloc_large and _xfs_buf_map_pages use memalloc_noio_{save,restore} API to prevent from reclaim recursion into the fs because vmalloc can invoke unconditional GFP_KERNEL allocations and these functions might be called from the NOFS co

[PATCH 5/8] jbd2: mark the transaction context with the scope GFP_NOFS context

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> now that we have memalloc_nofs_{save,restore} api we can mark the whole transaction context as implicitly GFP_NOFS. All allocations will automatically inherit GFP_NOFS this way. This means that we do not have to mark any of those requests with GF

[PATCH 8/8] Revert "ext4: fix wrong gfp type under transaction"

2017-01-06 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> This reverts commit 216553c4b7f3e3e2beb4981cddca9b2027523928. Now that the transaction context uses memalloc_nofs_save and all allocations within the this context inherit GFP_NOFS automatically, there is no reason to mark specific allocations expl

Re: [RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-30 Thread Michal Hocko
On Fri 30-12-16 10:19:26, Mel Gorman wrote: > On Mon, Dec 26, 2016 at 01:48:40PM +0100, Michal Hocko wrote: > > On Fri 23-12-16 23:26:00, Nils Holland wrote: > > > On Fri, Dec 23, 2016 at 03:47:39PM +0100, Michal Hocko wrote: > > > > > > > > Nils, eve

Re: [RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-30 Thread Michal Hocko
On Fri 30-12-16 11:05:22, Minchan Kim wrote: > On Thu, Dec 29, 2016 at 10:04:32AM +0100, Michal Hocko wrote: > > On Thu 29-12-16 10:20:26, Minchan Kim wrote: > > > On Tue, Dec 27, 2016 at 04:55:33PM +0100, Michal Hocko wrote: [...] > > > > + * given zone_idx > &g

Re: [RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-29 Thread Michal Hocko
On Thu 29-12-16 10:20:26, Minchan Kim wrote: > On Tue, Dec 27, 2016 at 04:55:33PM +0100, Michal Hocko wrote: > > Hi, > > could you try to run with the following patch on top of the previous > > one? I do not think it will make a large change in your workload but > > I t

Re: [RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-29 Thread Michal Hocko
RU_FILE, > zid); > > WARNING: line over 80 characters > #202: FILE: mm/vmscan.c:2100: > + active_zone = lruvec_zone_lru_size(lruvec, (file * LRU_FILE) > + LRU_ACTIVE, zid); I would prefer to have those on the same line though. It will make them easier to follow. --

Re: [RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-28 Thread Michal Hocko
On Tue 27-12-16 20:33:09, Nils Holland wrote: > On Tue, Dec 27, 2016 at 04:55:33PM +0100, Michal Hocko wrote: > > Hi, > > could you try to run with the following patch on top of the previous > > one? I do not think it will make a large change in your workload but > > I t

Re: [PATCH] mm, vmscan: consider eligible zones in get_scan_count

2016-12-28 Thread Michal Hocko
; > url: > https://github.com/0day-ci/linux/commits/Michal-Hocko/mm-vmscan-consider-eligible-zones-in-get_scan_count/20161228-000917 > base: git://git.cmpxchg.org/linux-mmotm.git master > config: i386-tinyconfig (attached as .config) > compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20

Re: [RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-27 Thread Michal Hocko
balancing in get_scan_count. I suspect we need but need to think more about that. Thanks a lot again! --- >From b51f50340fe9e40b68be198b012f8ab9869c1850 Mon Sep 17 00:00:00 2001 From: Michal Hocko <mho...@suse.com> Date: Tue, 27 Dec 2016 16:28:44 +0100 Subject: [PATCH] mm, vmscan: consider

Re: [RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-27 Thread Michal Hocko
On Tue 27-12-16 12:23:13, Nils Holland wrote: > On Tue, Dec 27, 2016 at 09:08:38AM +0100, Michal Hocko wrote: > > On Mon 26-12-16 19:57:03, Nils Holland wrote: > > > On Mon, Dec 26, 2016 at 01:48:40PM +0100, Michal Hocko wrote: > > > > On Fri 23-12-16 23:26:00, Nils H

Re: [RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-27 Thread Michal Hocko
On Mon 26-12-16 19:57:03, Nils Holland wrote: > On Mon, Dec 26, 2016 at 01:48:40PM +0100, Michal Hocko wrote: > > On Fri 23-12-16 23:26:00, Nils Holland wrote: > > > On Fri, Dec 23, 2016 at 03:47:39PM +0100, Michal Hocko wrote: > > > > > > > > Nils, eve

Re: [lkp-developer] [mm, memcg] d18e2b2aca: WARNING:at_mm/memcontrol.c:#mem_cgroup_update_lru_size

2016-12-26 Thread Michal Hocko
On Mon 26-12-16 13:26:51, Michal Hocko wrote: > On Mon 26-12-16 06:25:56, kernel test robot wrote: [...] > > [ 95.226364] init: tty6 main process (990) killed by TERM signal > > [ 95.314020] init: plymouth-upstart-bridge main process (1039) terminated > > with statu

Re: [RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-26 Thread Michal Hocko
On Fri 23-12-16 23:26:00, Nils Holland wrote: > On Fri, Dec 23, 2016 at 03:47:39PM +0100, Michal Hocko wrote: > > > > Nils, even though this is still highly experimental, could you give it a > > try please? > > Yes, no problem! So I kept the very first patch y

Re: [lkp-developer] [mm, memcg] d18e2b2aca: WARNING:at_mm/memcontrol.c:#mem_cgroup_update_lru_size

2016-12-26 Thread Michal Hocko
On Mon 26-12-16 06:25:56, kernel test robot wrote: > > FYI, we noticed the following commit: > > commit: d18e2b2aca0396849f588241e134787a829c707d ("mm, memcg: fix (Re: OOM: > Better, but still there on)") > url: > https://github.com/0day-ci/linux/commits/Michal-H

[RFC PATCH] mm, memcg: fix (Re: OOM: Better, but still there on)

2016-12-23 Thread Michal Hocko
-12-16 13:57:28, Michal Hocko wrote: > On Fri 23-12-16 13:18:51, Nils Holland wrote: > > On Fri, Dec 23, 2016 at 11:51:57AM +0100, Michal Hocko wrote: > > > TL;DR > > > drop the last patch, check whether memory cgroup is enabled and retest > > > with cgroup_disa

Re: OOM: Better, but still there on

2016-12-23 Thread Michal Hocko
On Fri 23-12-16 13:18:51, Nils Holland wrote: > On Fri, Dec 23, 2016 at 11:51:57AM +0100, Michal Hocko wrote: > > TL;DR > > drop the last patch, check whether memory cgroup is enabled and retest > > with cgroup_disable=memory to see whether this is memcg related and if >

Re: OOM: Better, but still there on

2016-12-23 Thread Michal Hocko
lruvec *lruvec, struct mem_cgroup *memcg, unsigned long size; unsigned long scan; - size = lruvec_lru_size(lruvec, lru); + size = lruvec_lru_size_zone_idx(lruvec, lru, sc->reclaim_idx); scan = size >

Re: OOM: Better, but still there on

2016-12-22 Thread Michal Hocko
ne = zone_page_state_snapshot(zone, NR_ZONE_LRU_BASE + (file * LRU_FILE) + LRU_ACTIVE); inactive -= min(inactive, inactive_zone); active -= min(active, active_zone); } + spin_unlock_irq(>lru_lock); gb = (inactive + active) >> (3

Re: OOM: Better, but still there on

2016-12-22 Thread Michal Hocko
On Thu 22-12-16 11:10:29, Nils Holland wrote: > On Wed, Dec 21, 2016 at 08:36:59AM +0100, Michal Hocko wrote: > > TL;DR > > there is another version of the debugging patch. Just revert the > > previous one and apply this one instead. It's still not clear what > > is go

Re: [PATCH 0/9 v2] scope GFP_NOFS api

2016-12-22 Thread Michal Hocko
this API. Can we find a way to use it? -- Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: OOM: Better, but still there on

2016-12-21 Thread Michal Hocko
On Wed 21-12-16 20:00:38, Tetsuo Handa wrote: > Michal Hocko wrote: > > TL;DR > > there is another version of the debugging patch. Just revert the > > previous one and apply this one instead. It's still not clear what > > is going on but I suspect either some misaccoun

Re: OOM: Better, but still there on

2016-12-20 Thread Michal Hocko
mm_vmscan_inactive_list_is_low. Hopefully the additional data will tell us more. On Tue 20-12-16 03:08:29, Nils Holland wrote: > On Mon, Dec 19, 2016 at 02:45:34PM +0100, Michal Hocko wrote: > > > Unfortunatelly shrink_active_list doesn't have any tracepoint so we do > > not know whether we

Re: [PATCH 2/9] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-20 Thread Michal Hocko
On Tue 20-12-16 08:24:13, Dave Chinner wrote: > On Thu, Dec 15, 2016 at 03:07:08PM +0100, Michal Hocko wrote: > > From: Michal Hocko <mho...@suse.com> > > > > Now that the page allocator offers __GFP_NOLOCKDEP let's introduce > > KM_NOLOCKDEP alias for

Re: OOM: Better, but still there on

2016-12-19 Thread Michal Hocko
ode_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); spin_unlock_irq(>lru_lock); mem_cgroup_uncharge_list(_hold); - free_hot_cold_page_list(_hold, true); + nr_freed = free_hot_cold_page_list(_hold, true); + trace_mm_vmscan_lru_shrink_active(pgdat->node_id, n

Re: [PATCH 2/2] mm, oom: do not enfore OOM killer for __GFP_NOFAIL automatically

2016-12-18 Thread Michal Hocko
familiar with this code much so there might be some reasons for that. The fact that it might stall under a heavy memory pressure is sad but who actually cares? > Do we want to add __GFP_NOFAIL to this GFP_NOIO allocation request > in order to allow access to memory reserves as well as GFP_N

Re: OOM: Better, but still there on

2016-12-16 Thread Michal Hocko
0kB free_cma:0kB same here. All that suggests that the page cache cannot be reclaimed for some reason. It is hard to tell why but there is definitely something bad going on. -- Michal Hocko SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: OOM: Better, but still there on 4.9

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 17:47:25, Chris Mason wrote: > On 12/16/2016 05:14 PM, Michal Hocko wrote: > > On Fri 16-12-16 13:15:18, Chris Mason wrote: > > > On 12/16/2016 02:39 AM, Michal Hocko wrote: > > [...] > > > > I believe the right way to go around this is to pursue

Re: OOM: Better, but still there on 4.9

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 13:15:18, Chris Mason wrote: > On 12/16/2016 02:39 AM, Michal Hocko wrote: [...] > > I believe the right way to go around this is to pursue what I've started > > in [1]. I will try to prepare something for testing today for you. Stay > > tuned. But I

Re: [PATCH 2/2] mm, oom: do not enfore OOM killer for __GFP_NOFAIL automatically

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 12:31:51, Johannes Weiner wrote: > On Fri, Dec 16, 2016 at 04:58:08PM +0100, Michal Hocko wrote: > > @@ -1013,7 +1013,7 @@ bool out_of_memory(struct oom_control *oc) > > * make sure exclude 0 mask - all other users should have at least > > * ___GFP_

Re: [PATCH 2/9 v2] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 11:37:50, Brian Foster wrote: > On Fri, Dec 16, 2016 at 04:40:41PM +0100, Michal Hocko wrote: > > Updated patch after Mike noticed a BUG_ON when KM_NOLOCKDEP is used. > > --- > > From 1497e713e11639157aef21cae29052cb3dc7ab44 Mon Sep 17 00:00:00 2001 > &

[PATCH 5/9 v2] xfs: use memalloc_nofs_{save,restore} instead of memalloc_noio*

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 11:38:11, Brian Foster wrote: > On Thu, Dec 15, 2016 at 03:07:11PM +0100, Michal Hocko wrote: [...] > > @@ -459,7 +459,7 @@ _xfs_buf_map_pages( > > break; > > vm_unmap_aliases(); > >

[PATCH 2/2] mm, oom: do not enfore OOM killer for __GFP_NOFAIL automatically

2016-12-16 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> __alloc_pages_may_oom makes sure to skip the OOM killer depending on the allocation request. This includes lowmem requests, costly high order requests and others. For a long time __GFP_NOFAIL acted as an override for all those rules. This is not docu

[PATCH 1/2] mm: consolidate GFP_NOFAIL checks in the allocator slowpath

2016-12-16 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> Tetsuo Handa has pointed out that 0a0337e0d1d1 ("mm, oom: rework oom detection") has subtly changed semantic for costly high order requests with __GFP_NOFAIL and withtout __GFP_REPEAT and those can fail right now. My code inspection d

Re: OOM: Better, but still there on

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 08:39:41, Michal Hocko wrote: [...] > That being said, the OOM killer invocation is clearly pointless and > pre-mature. We normally do not invoke it normally for GFP_NOFS requests > exactly for these reasons. But this is GFP_NOFS|__GFP_NOFAIL which > behaves differently

[PATCH 2/9 v2] xfs: introduce and use KM_NOLOCKDEP to silence reclaim lockdep false positives

2016-12-16 Thread Michal Hocko
Updated patch after Mike noticed a BUG_ON when KM_NOLOCKDEP is used. --- >From 1497e713e11639157aef21cae29052cb3dc7ab44 Mon Sep 17 00:00:00 2001 From: Michal Hocko <mho...@suse.com> Date: Thu, 15 Dec 2016 13:06:43 +0100 Subject: [PATCH] xfs: introduce and use KM_NOLOCKDEP to silenc

Re: [PATCH 0/9 v2] scope GFP_NOFS api

2016-12-16 Thread Michal Hocko
On Fri 16-12-16 16:05:58, Mike Galbraith wrote: > On Thu, 2016-12-15 at 15:07 +0100, Michal Hocko wrote: > > Hi, > > I have posted the previous version here [1]. Since then I have added a > > support to suppress reclaim lockdep warnings (__GFP_NOLOCKDEP) to allow > &

[DEBUG PATCH 2/2] silent warnings which we cannot do anything about

2016-12-16 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE There are some code paths used by all the filesystems which we cannot change to drop the GFP_NOFS, yet they generate a lot of warnings. Provide {disable,enable}_scope_gfp_check to silence

[DEBUG PATCH 1/2] mm, debug: report when GFP_NO{FS,IO} is used explicitly from memalloc_no{fs,io}_{save,restore} context

2016-12-16 Thread Michal Hocko
From: Michal Hocko <mho...@suse.com> THIS PATCH IS FOR TESTING ONLY AND NOT MEANT TO HIT LINUS TREE It is desirable to reduce the direct GFP_NO{FS,IO} usage at minimum and prefer scope usage defined by memalloc_no{fs,io}_{save,restore} API. Let's help this process and add a debuggin

[DEBUG PATCH 0/2] debug explicit GFP_NO{FS,IO} usage from the scope context

2016-12-16 Thread Michal Hocko
Hi, I've forgot to add the following two patches which should help to identify explicit GFP_NO{FS,IO} usage from withing a scope context. Such a usage can be changed to the full GFP_KERNEL because all the calls from within the NO{FS,IO} scope will drop the __GFP_FS resp. __GFP_IO automatically and

Re: OOM: Better, but still there on 4.9

2016-12-15 Thread Michal Hocko
I've started in [1]. I will try to prepare something for testing today for you. Stay tuned. But I would be really happy if somebody from the btrfs camp could check the NOFS aspect of this allocation. We have already seen allocation stalls from this path quite recently [1] http://lkml.kernel

  1   2   >