Re: [PATCH 4/9] mm/page_alloc: optimize code layout for __alloc_pages_bulk
On 3/25/21 12:42 PM, Mel Gorman wrote:
> From: Jesper Dangaard Brouer
>
> Looking at perf-report and ASM-code for __alloc_pages_bulk() it is clear
> that the code activated is suboptimal. The compiler guesses wrong and
> places unlikely code at the beginning. Due to the use of WARN_ON_ONCE()
> macro the UD2 asm instruction is added to the code, which confuse the
> I-cache prefetcher in the CPU.
Hm that's weird, WARN_ON_ONCE() uses unlikely() too, so the UD2 should end up in
the out-of-fast-path part?
But anyway.
> [mgorman: Minor changes and rebasing]
> Signed-off-by: Jesper Dangaard Brouer
> Signed-off-by: Mel Gorman
Acked-By: Vlastimil Babka
> ---
> mm/page_alloc.c | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index be1e33a4df39..1ec18121268b 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -5001,7 +5001,7 @@ int __alloc_pages_bulk(gfp_t gfp, int preferred_nid,
> unsigned int alloc_flags;
> int nr_populated = 0;
>
> - if (WARN_ON_ONCE(nr_pages <= 0))
> + if (unlikely(nr_pages <= 0))
> return 0;
>
> /*
> @@ -5048,7 +5048,7 @@ int __alloc_pages_bulk(gfp_t gfp, int preferred_nid,
>* If there are no allowed local zones that meets the watermarks then
>* try to allocate a single page and reclaim if necessary.
>*/
> - if (!zone)
> + if (unlikely(!zone))
> goto failed;
>
> /* Attempt the batch allocation */
> @@ -5066,7 +5066,7 @@ int __alloc_pages_bulk(gfp_t gfp, int preferred_nid,
>
> page = __rmqueue_pcplist(zone, ac.migratetype, alloc_flags,
> pcp, pcp_list);
> - if (!page) {
> + if (unlikely(!page)) {
> /* Try and get at least one page */
> if (!nr_populated)
> goto failed_irq;
>
Re: [PATCH 4/9] mm/page_alloc: optimize code layout for __alloc_pages_bulk
On Thu, Mar 25, 2021 at 12:12:17PM +, Matthew Wilcox wrote: > On Thu, Mar 25, 2021 at 11:42:23AM +, Mel Gorman wrote: > > > > - if (WARN_ON_ONCE(nr_pages <= 0)) > > + if (unlikely(nr_pages <= 0)) > > return 0; > > If we made nr_pages unsigned, we wouldn't need this check at all (ok, > we'd still need to figure out what to do with 0). But then, if a user > inadvertently passes in -ENOMEM, we'll try to allocate 4 billion pages. This is exactly why nr_pages is signed. An error in accounting by the caller potentially puts the system under severe memory pressure. This *should* only be a problem when a new caller of the API is being implemented. The warning goes away in a later patch for reasons explained in the changelog. > So maybe we should check it. Gah, API design is hard. Yep. -- Mel Gorman SUSE Labs
Re: [PATCH 4/9] mm/page_alloc: optimize code layout for __alloc_pages_bulk
On Thu, Mar 25, 2021 at 11:42:23AM +, Mel Gorman wrote: > > - if (WARN_ON_ONCE(nr_pages <= 0)) > + if (unlikely(nr_pages <= 0)) > return 0; If we made nr_pages unsigned, we wouldn't need this check at all (ok, we'd still need to figure out what to do with 0). But then, if a user inadvertently passes in -ENOMEM, we'll try to allocate 4 billion pages. So maybe we should check it. Gah, API design is hard.
[PATCH 4/9] mm/page_alloc: optimize code layout for __alloc_pages_bulk
From: Jesper Dangaard Brouer
Looking at perf-report and ASM-code for __alloc_pages_bulk() it is clear
that the code activated is suboptimal. The compiler guesses wrong and
places unlikely code at the beginning. Due to the use of WARN_ON_ONCE()
macro the UD2 asm instruction is added to the code, which confuse the
I-cache prefetcher in the CPU.
[mgorman: Minor changes and rebasing]
Signed-off-by: Jesper Dangaard Brouer
Signed-off-by: Mel Gorman
---
mm/page_alloc.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index be1e33a4df39..1ec18121268b 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5001,7 +5001,7 @@ int __alloc_pages_bulk(gfp_t gfp, int preferred_nid,
unsigned int alloc_flags;
int nr_populated = 0;
- if (WARN_ON_ONCE(nr_pages <= 0))
+ if (unlikely(nr_pages <= 0))
return 0;
/*
@@ -5048,7 +5048,7 @@ int __alloc_pages_bulk(gfp_t gfp, int preferred_nid,
* If there are no allowed local zones that meets the watermarks then
* try to allocate a single page and reclaim if necessary.
*/
- if (!zone)
+ if (unlikely(!zone))
goto failed;
/* Attempt the batch allocation */
@@ -5066,7 +5066,7 @@ int __alloc_pages_bulk(gfp_t gfp, int preferred_nid,
page = __rmqueue_pcplist(zone, ac.migratetype, alloc_flags,
pcp, pcp_list);
- if (!page) {
+ if (unlikely(!page)) {
/* Try and get at least one page */
if (!nr_populated)
goto failed_irq;
--
2.26.2
