Re: [PATCH v4 08/13] mm/mempolicy: Create a page allocator for policy
On Wed, Apr 14, 2021 at 03:08:19PM +0200, Michal Hocko wrote:
> On Wed 17-03-21 11:40:05, Feng Tang wrote:
> > From: Ben Widawsky
> >
> > Add a helper function which takes care of handling multiple preferred
> > nodes. It will be called by future patches that need to handle this,
> > specifically VMA based page allocation, and task based page allocation.
> > Huge pages don't quite fit the same pattern because they use different
> > underlying page allocation functions. This consumes the previous
> > interleave policy specific allocation function to make a one stop shop
> > for policy based allocation.
> >
> > With this, MPOL_PREFERRED_MANY's semantic is more like MPOL_PREFERRED
> > that it will first try the preferred node/nodes, and fallback to all
> > other nodes when first try fails. Thanks to Michal Hocko for suggestions
> > on this.
> >
> > For now, only interleaved policy will be used so there should be no
> > functional change yet. However, if bisection points to issues in the
> > next few commits, it was likely the fault of this patch.
>
> I am not sure this is helping much. Let's see in later patches but I
> would keep them separate and rather create a dedicated function for the
> new policy allocation mode.
Thanks for the suggestion, we will rethink the implementations.
- Feng
> > Similar functionality is offered via policy_node() and
> > policy_nodemask(). By themselves however, neither can achieve this
> > fallback style of sets of nodes.
> >
> > [ Feng: for the first try, add NOWARN flag, and skip the direct reclaim
> > to speedup allocation in some case ]
> >
> > Link:
> > https://lore.kernel.org/r/20200630212517.308045-9-ben.widaw...@intel.com
> > Signed-off-by: Ben Widawsky
> > Signed-off-by: Feng Tang
> > ---
> > mm/mempolicy.c | 65
> > ++
> > 1 file changed, 52 insertions(+), 13 deletions(-)
> >
> > diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> > index d945f29..d21105b 100644
> > --- a/mm/mempolicy.c
> > +++ b/mm/mempolicy.c
> > @@ -2187,22 +2187,60 @@ bool mempolicy_nodemask_intersects(struct
> > task_struct *tsk,
> > return ret;
> > }
> >
> > -/* Allocate a page in interleaved policy.
> > - Own path because it needs to do special accounting. */
> > -static struct page *alloc_page_interleave(gfp_t gfp, unsigned order,
> > - unsigned nid)
> > +/* Handle page allocation for all but interleaved policies */
> > +static struct page *alloc_pages_policy(struct mempolicy *pol, gfp_t gfp,
> > + unsigned int order, int preferred_nid)
> > {
> > struct page *page;
> > + gfp_t gfp_mask = gfp;
> >
> > - page = __alloc_pages(gfp, order, nid);
> > - /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */
> > - if (!static_branch_likely(_numa_stat_key))
> > + if (pol->mode == MPOL_INTERLEAVE) {
> > + page = __alloc_pages(gfp, order, preferred_nid);
> > + /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is
> > disabled */
> > + if (!static_branch_likely(_numa_stat_key))
> > + return page;
> > + if (page && page_to_nid(page) == preferred_nid) {
> > + preempt_disable();
> > + __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
> > + preempt_enable();
> > + }
> > return page;
> > - if (page && page_to_nid(page) == nid) {
> > - preempt_disable();
> > - __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
> > - preempt_enable();
> > }
> > +
> > + VM_BUG_ON(preferred_nid != NUMA_NO_NODE);
> > +
> > + preferred_nid = numa_node_id();
> > +
> > + /*
> > +* There is a two pass approach implemented here for
> > +* MPOL_PREFERRED_MANY. In the first pass we try the preferred nodes
> > +* but allow the allocation to fail. The below table explains how
> > +* this is achieved.
> > +*
> > +* | Policy| preferred nid | nodemask |
> > +* |---|---||
> > +* | MPOL_DEFAULT | local | NULL |
> > +* | MPOL_PREFERRED| best | NULL |
> > +* | MPOL_INTERLEAVE | ERR | ERR|
> > +* | MPOL_BIND | local | pol->nodes |
> > +* | MPOL_PREFERRED_MANY | best | pol->nodes |
> > +* | MPOL_PREFERRED_MANY (round 2) | local | NULL |
> > +* +---+---++
> > +*/
> > + if (pol->mode == MPOL_PREFERRED_MANY) {
> > + gfp_mask |= __GFP_NOWARN;
> > +
> > + /* Skip direct reclaim, as there will be a second try */
> > + gfp_mask &= ~__GFP_DIRECT_RECLAIM;
> > + }
> > +
> > + page = __alloc_pages_nodemask(gfp_mask,
Re: [PATCH v4 08/13] mm/mempolicy: Create a page allocator for policy
On Wed 17-03-21 11:40:05, Feng Tang wrote:
> From: Ben Widawsky
>
> Add a helper function which takes care of handling multiple preferred
> nodes. It will be called by future patches that need to handle this,
> specifically VMA based page allocation, and task based page allocation.
> Huge pages don't quite fit the same pattern because they use different
> underlying page allocation functions. This consumes the previous
> interleave policy specific allocation function to make a one stop shop
> for policy based allocation.
>
> With this, MPOL_PREFERRED_MANY's semantic is more like MPOL_PREFERRED
> that it will first try the preferred node/nodes, and fallback to all
> other nodes when first try fails. Thanks to Michal Hocko for suggestions
> on this.
>
> For now, only interleaved policy will be used so there should be no
> functional change yet. However, if bisection points to issues in the
> next few commits, it was likely the fault of this patch.
I am not sure this is helping much. Let's see in later patches but I
would keep them separate and rather create a dedicated function for the
new policy allocation mode.
> Similar functionality is offered via policy_node() and
> policy_nodemask(). By themselves however, neither can achieve this
> fallback style of sets of nodes.
>
> [ Feng: for the first try, add NOWARN flag, and skip the direct reclaim
> to speedup allocation in some case ]
>
> Link: https://lore.kernel.org/r/20200630212517.308045-9-ben.widaw...@intel.com
> Signed-off-by: Ben Widawsky
> Signed-off-by: Feng Tang
> ---
> mm/mempolicy.c | 65
> ++
> 1 file changed, 52 insertions(+), 13 deletions(-)
>
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index d945f29..d21105b 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -2187,22 +2187,60 @@ bool mempolicy_nodemask_intersects(struct task_struct
> *tsk,
> return ret;
> }
>
> -/* Allocate a page in interleaved policy.
> - Own path because it needs to do special accounting. */
> -static struct page *alloc_page_interleave(gfp_t gfp, unsigned order,
> - unsigned nid)
> +/* Handle page allocation for all but interleaved policies */
> +static struct page *alloc_pages_policy(struct mempolicy *pol, gfp_t gfp,
> +unsigned int order, int preferred_nid)
> {
> struct page *page;
> + gfp_t gfp_mask = gfp;
>
> - page = __alloc_pages(gfp, order, nid);
> - /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */
> - if (!static_branch_likely(_numa_stat_key))
> + if (pol->mode == MPOL_INTERLEAVE) {
> + page = __alloc_pages(gfp, order, preferred_nid);
> + /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is
> disabled */
> + if (!static_branch_likely(_numa_stat_key))
> + return page;
> + if (page && page_to_nid(page) == preferred_nid) {
> + preempt_disable();
> + __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
> + preempt_enable();
> + }
> return page;
> - if (page && page_to_nid(page) == nid) {
> - preempt_disable();
> - __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
> - preempt_enable();
> }
> +
> + VM_BUG_ON(preferred_nid != NUMA_NO_NODE);
> +
> + preferred_nid = numa_node_id();
> +
> + /*
> + * There is a two pass approach implemented here for
> + * MPOL_PREFERRED_MANY. In the first pass we try the preferred nodes
> + * but allow the allocation to fail. The below table explains how
> + * this is achieved.
> + *
> + * | Policy| preferred nid | nodemask |
> + * |---|---||
> + * | MPOL_DEFAULT | local | NULL |
> + * | MPOL_PREFERRED| best | NULL |
> + * | MPOL_INTERLEAVE | ERR | ERR|
> + * | MPOL_BIND | local | pol->nodes |
> + * | MPOL_PREFERRED_MANY | best | pol->nodes |
> + * | MPOL_PREFERRED_MANY (round 2) | local | NULL |
> + * +---+---++
> + */
> + if (pol->mode == MPOL_PREFERRED_MANY) {
> + gfp_mask |= __GFP_NOWARN;
> +
> + /* Skip direct reclaim, as there will be a second try */
> + gfp_mask &= ~__GFP_DIRECT_RECLAIM;
> + }
> +
> + page = __alloc_pages_nodemask(gfp_mask, order,
> + policy_node(gfp, pol, preferred_nid),
> + policy_nodemask(gfp, pol));
> +
> + if (unlikely(!page && pol->mode == MPOL_PREFERRED_MANY))
> + page =
[PATCH v4 08/13] mm/mempolicy: Create a page allocator for policy
From: Ben Widawsky
Add a helper function which takes care of handling multiple preferred
nodes. It will be called by future patches that need to handle this,
specifically VMA based page allocation, and task based page allocation.
Huge pages don't quite fit the same pattern because they use different
underlying page allocation functions. This consumes the previous
interleave policy specific allocation function to make a one stop shop
for policy based allocation.
With this, MPOL_PREFERRED_MANY's semantic is more like MPOL_PREFERRED
that it will first try the preferred node/nodes, and fallback to all
other nodes when first try fails. Thanks to Michal Hocko for suggestions
on this.
For now, only interleaved policy will be used so there should be no
functional change yet. However, if bisection points to issues in the
next few commits, it was likely the fault of this patch.
Similar functionality is offered via policy_node() and
policy_nodemask(). By themselves however, neither can achieve this
fallback style of sets of nodes.
[ Feng: for the first try, add NOWARN flag, and skip the direct reclaim
to speedup allocation in some case ]
Link: https://lore.kernel.org/r/20200630212517.308045-9-ben.widaw...@intel.com
Signed-off-by: Ben Widawsky
Signed-off-by: Feng Tang
---
mm/mempolicy.c | 65 ++
1 file changed, 52 insertions(+), 13 deletions(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index d945f29..d21105b 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2187,22 +2187,60 @@ bool mempolicy_nodemask_intersects(struct task_struct
*tsk,
return ret;
}
-/* Allocate a page in interleaved policy.
- Own path because it needs to do special accounting. */
-static struct page *alloc_page_interleave(gfp_t gfp, unsigned order,
- unsigned nid)
+/* Handle page allocation for all but interleaved policies */
+static struct page *alloc_pages_policy(struct mempolicy *pol, gfp_t gfp,
+ unsigned int order, int preferred_nid)
{
struct page *page;
+ gfp_t gfp_mask = gfp;
- page = __alloc_pages(gfp, order, nid);
- /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */
- if (!static_branch_likely(_numa_stat_key))
+ if (pol->mode == MPOL_INTERLEAVE) {
+ page = __alloc_pages(gfp, order, preferred_nid);
+ /* skip NUMA_INTERLEAVE_HIT counter update if numa stats is
disabled */
+ if (!static_branch_likely(_numa_stat_key))
+ return page;
+ if (page && page_to_nid(page) == preferred_nid) {
+ preempt_disable();
+ __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
+ preempt_enable();
+ }
return page;
- if (page && page_to_nid(page) == nid) {
- preempt_disable();
- __inc_numa_state(page_zone(page), NUMA_INTERLEAVE_HIT);
- preempt_enable();
}
+
+ VM_BUG_ON(preferred_nid != NUMA_NO_NODE);
+
+ preferred_nid = numa_node_id();
+
+ /*
+* There is a two pass approach implemented here for
+* MPOL_PREFERRED_MANY. In the first pass we try the preferred nodes
+* but allow the allocation to fail. The below table explains how
+* this is achieved.
+*
+* | Policy| preferred nid | nodemask |
+* |---|---||
+* | MPOL_DEFAULT | local | NULL |
+* | MPOL_PREFERRED| best | NULL |
+* | MPOL_INTERLEAVE | ERR | ERR|
+* | MPOL_BIND | local | pol->nodes |
+* | MPOL_PREFERRED_MANY | best | pol->nodes |
+* | MPOL_PREFERRED_MANY (round 2) | local | NULL |
+* +---+---++
+*/
+ if (pol->mode == MPOL_PREFERRED_MANY) {
+ gfp_mask |= __GFP_NOWARN;
+
+ /* Skip direct reclaim, as there will be a second try */
+ gfp_mask &= ~__GFP_DIRECT_RECLAIM;
+ }
+
+ page = __alloc_pages_nodemask(gfp_mask, order,
+ policy_node(gfp, pol, preferred_nid),
+ policy_nodemask(gfp, pol));
+
+ if (unlikely(!page && pol->mode == MPOL_PREFERRED_MANY))
+ page = __alloc_pages_nodemask(gfp, order, preferred_nid, NULL);
+
return page;
}
@@ -2244,8 +2282,8 @@ alloc_pages_vma(gfp_t gfp, int order, struct
vm_area_struct *vma,
unsigned nid;
nid = interleave_nid(pol, vma, addr, PAGE_SHIFT + order);
+ page = alloc_pages_policy(pol, gfp, order,
