Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-27 Thread Minchan Kim
On Fri, Jan 25, 2013 at 08:53:24PM +1100, paul.sz...@sydney.edu.au wrote:
> Dear Minchan,
> 
> > So what's the effect for user?
> > ...
> > It seems you saw old kernel.
> > ...
> > Current kernel includes ...
> > So I think we don't need this patch.
> 
> As I understand now, my patch is "right" and needed for older kernels;
> for newer kernels, the issue has been fixed in equivalent ways; it was
> an oversight that the change was not backported; and any justification
> you need, you can get from those "later better" patches.

I don't know your problem because you didn't write down your problem in
changelog. Anyway, If you want to apply it into older kernel,
please read Documentation/stable_kernel_rules.txt.

In summary,

1. Define your problem.
2. Apply your fix to see the problem goes away in older kernel.
3. If so, write the problem and effect in changelog
4. Send it to stable maintainers and mm maintainer

That's all.

> 
> I asked:
> 
>   A question: what is the use or significance of vm_highmem_is_dirtyable?
>   It seems odd that it would be used in setting limits or threshholds, but
>   not used in decisions where to put dirty things. Is that so, is that as
>   should be? What is the recommended setting of highmem_is_dirtyable?
> 
> The silence is deafening. I guess highmem_is_dirtyable is an aberration.

I hope this helps you find primary reason of your problem.
git show 195cf453


> 
> Thanks, Paul
> 
> Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
> School of Mathematics and Statistics   University of SydneyAustralia
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majord...@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: mailto:"d...@kvack.org;> em...@kvack.org 

-- 
Kind regards,
Minchan Kim
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-27 Thread Minchan Kim
On Fri, Jan 25, 2013 at 08:53:24PM +1100, paul.sz...@sydney.edu.au wrote:
 Dear Minchan,
 
  So what's the effect for user?
  ...
  It seems you saw old kernel.
  ...
  Current kernel includes ...
  So I think we don't need this patch.
 
 As I understand now, my patch is right and needed for older kernels;
 for newer kernels, the issue has been fixed in equivalent ways; it was
 an oversight that the change was not backported; and any justification
 you need, you can get from those later better patches.

I don't know your problem because you didn't write down your problem in
changelog. Anyway, If you want to apply it into older kernel,
please read Documentation/stable_kernel_rules.txt.

In summary,

1. Define your problem.
2. Apply your fix to see the problem goes away in older kernel.
3. If so, write the problem and effect in changelog
4. Send it to stable maintainers and mm maintainer

That's all.

 
 I asked:
 
   A question: what is the use or significance of vm_highmem_is_dirtyable?
   It seems odd that it would be used in setting limits or threshholds, but
   not used in decisions where to put dirty things. Is that so, is that as
   should be? What is the recommended setting of highmem_is_dirtyable?
 
 The silence is deafening. I guess highmem_is_dirtyable is an aberration.

I hope this helps you find primary reason of your problem.
git show 195cf453


 
 Thanks, Paul
 
 Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
 School of Mathematics and Statistics   University of SydneyAustralia
 
 --
 To unsubscribe, send a message with 'unsubscribe linux-mm' in
 the body to majord...@kvack.org.  For more info on Linux MM,
 see: http://www.linux-mm.org/ .
 Don't email: a href=mailto:d...@kvack.org; em...@kvack.org /a

-- 
Kind regards,
Minchan Kim
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-26 Thread paul . szabo
Dear Jonathan,

>> If you can identify where it was fixed then your patch for older
>> versions should go to stable with a reference to the upstream fix (see
>> Documentation/stable_kernel_rules.txt).
>
> How about this patch?
>
> It was applied in mainline during the 3.3 merge window, so kernels
> newer than 3.2.y shouldn't need it.
>
> ...
> commit ab8fabd46f811d5153d8a0cd2fac9a0d41fb593d upstream.
> ...

Yes, I beleive that is the correct patch, surely better than my simple
subtraction of min_free_kbytes.

Noting, that this does not "solve" all problems, the latest 3.8 kernel
still crashes with OOM:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1098961/comments/18

Thanks, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-26 Thread Theodore Ts'o
(In the teach a person to fish category...)

If you know the file and line number where a bug/regression was
introduced, the "git blame" command is a great tool for identifying
the commit which changed a given line of code.  Then use "git tag
--contains " to see when a particular commit was introduced
into the mainline kernel.

- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-26 Thread Theodore Ts'o
(In the teach a person to fish category...)

If you know the file and line number where a bug/regression was
introduced, the git blame command is a great tool for identifying
the commit which changed a given line of code.  Then use git tag
--contains commit it to see when a particular commit was introduced
into the mainline kernel.

- Ted
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-26 Thread paul . szabo
Dear Jonathan,

 If you can identify where it was fixed then your patch for older
 versions should go to stable with a reference to the upstream fix (see
 Documentation/stable_kernel_rules.txt).

 How about this patch?

 It was applied in mainline during the 3.3 merge window, so kernels
 newer than 3.2.y shouldn't need it.

 ...
 commit ab8fabd46f811d5153d8a0cd2fac9a0d41fb593d upstream.
 ...

Yes, I beleive that is the correct patch, surely better than my simple
subtraction of min_free_kbytes.

Noting, that this does not solve all problems, the latest 3.8 kernel
still crashes with OOM:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1098961/comments/18

Thanks, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Jonathan Nieder
Hi Paul,

Ben Hutchings wrote:

> If you can identify where it was fixed then your patch for older
> versions should go to stable with a reference to the upstream fix (see
> Documentation/stable_kernel_rules.txt).

How about this patch?

It was applied in mainline during the 3.3 merge window, so kernels
newer than 3.2.y shouldn't need it.

-- >8 --
From: Johannes Weiner 
Date: Tue, 10 Jan 2012 15:07:42 -0800
Subject: mm: exclude reserved pages from dirtyable memory

commit ab8fabd46f811d5153d8a0cd2fac9a0d41fb593d upstream.

Per-zone dirty limits try to distribute page cache pages allocated for
writing across zones in proportion to the individual zone sizes, to reduce
the likelihood of reclaim having to write back individual pages from the
LRU lists in order to make progress.

This patch:

The amount of dirtyable pages should not include the full number of free
pages: there is a number of reserved pages that the page allocator and
kswapd always try to keep free.

The closer (reclaimable pages - dirty pages) is to the number of reserved
pages, the more likely it becomes for reclaim to run into dirty pages:

   +--+ ---
   |   anon   |  |
   +--+  |
   |  |  |
   |  |  -- dirty limit new-- flusher new
   |   file   |  | |
   |  |  | |
   |  |  -- dirty limit old-- flusher old
   |  ||
   +--+   --- reclaim
   | reserved |
   +--+
   |  kernel  |
   +--+

This patch introduces a per-zone dirty reserve that takes both the lowmem
reserve as well as the high watermark of the zone into account, and a
global sum of those per-zone values that is subtracted from the global
amount of dirtyable pages.  The lowmem reserve is unavailable to page
cache allocations and kswapd tries to keep the high watermark free.  We
don't want to end up in a situation where reclaim has to clean pages in
order to balance zones.

Not treating reserved pages as dirtyable on a global level is only a
conceptual fix.  In reality, dirty pages are not distributed equally
across zones and reclaim runs into dirty pages on a regular basis.

But it is important to get this right before tackling the problem on a
per-zone level, where the distance between reclaim and the dirty pages is
mostly much smaller in absolute numbers.

[a...@linux-foundation.org: fix highmem build]
Signed-off-by: Johannes Weiner 
Reviewed-by: Rik van Riel 
Reviewed-by: Michal Hocko 
Reviewed-by: Minchan Kim 
Acked-by: Mel Gorman 
Cc: KAMEZAWA Hiroyuki 
Cc: Christoph Hellwig 
Cc: Wu Fengguang 
Cc: Dave Chinner 
Cc: Jan Kara 
Cc: Shaohua Li 
Cc: Chris Mason 
Signed-off-by: Andrew Morton 
Signed-off-by: Linus Torvalds 
Signed-off-by: Jonathan Nieder 
---
 include/linux/mmzone.h |  6 ++
 include/linux/swap.h   |  1 +
 mm/page-writeback.c|  5 +++--
 mm/page_alloc.c| 19 +++
 4 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 25842b6e72e1..a594af3278bc 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -319,6 +319,12 @@ struct zone {
 */
unsigned long   lowmem_reserve[MAX_NR_ZONES];
 
+   /*
+* This is a per-zone reserve of pages that should not be
+* considered dirtyable memory.
+*/
+   unsigned long   dirty_balance_reserve;
+
 #ifdef CONFIG_NUMA
int node;
/*
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 67b3fa308988..3e60228e7299 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -207,6 +207,7 @@ struct swap_list_t {
 /* linux/mm/page_alloc.c */
 extern unsigned long totalram_pages;
 extern unsigned long totalreserve_pages;
+extern unsigned long dirty_balance_reserve;
 extern unsigned int nr_free_buffer_pages(void);
 extern unsigned int nr_free_pagecache_pages(void);
 
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 50f08241f981..f620e7b0dc26 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -320,7 +320,7 @@ static unsigned long highmem_dirtyable_memory(unsigned long 
total)
_DATA(node)->node_zones[ZONE_HIGHMEM];
 
x += zone_page_state(z, NR_FREE_PAGES) +
-zone_reclaimable_pages(z);
+zone_reclaimable_pages(z) - z->dirty_balance_reserve;
}
/*
 * Make sure that the number of highmem pages is never larger
@@ -344,7 +344,8 @@ unsigned long determine_dirtyable_memory(void)
 {
unsigned long x;
 
-   x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
+   x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages() -
+   dirty_balance_reserve;
 
if (!vm_highmem_is_dirtyable)
x -= highmem_dirtyable_memory(x);
diff --git a/mm/page_alloc.c 

Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Ben Hutchings
On Sat, 2013-01-26 at 14:07 +1100, paul.sz...@sydney.edu.au wrote:
> Dear Ben,
> 
> > ... the mm maintainers are probably much better placed ...
> 
> Exactly. Now I wonder: are you one of them?

Hah, no.

Ben.

-- 
Ben Hutchings
Any smoothly functioning technology is indistinguishable from a rigged demo.


signature.asc
Description: This is a digitally signed message part


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread paul . szabo
Dear Ben,

> ... the mm maintainers are probably much better placed ...

Exactly. Now I wonder: are you one of them?

Thanks, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread paul . szabo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Jonathan Nieder
Hi Paul,

paul.sz...@sydney.edu.au wrote:
> Dear Ben,

>> If you can identify where it was fixed then ...
>
> Sorry I cannot do that. I have no idea where kernel changelogs are kept.

Here are some tools.

  # prerequisite:
  apt-get install git; # as root

  # to get the kernel history:
  git clone \
https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git
  cd linux

  # to view the changelog:
  git log v3.2..

  # to grep change descriptions:
  git log --grep=min_free_kbytes v3.2..

  # to view the patches corresponding to changes:
  git log --patch v3.2.. -- mm/

  # graphical interface
  apt-get install gitk; # as root
  gitk v3.2.. -- mm

  # web interface:
  w3m http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git

The "exploring git history" section of the git user manual has more
details:

  http://git-htmldocs.googlecode.com/git/user-manual.html#exploring-git-history

Thanks,
Jonathan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Ben Hutchings
On Sat, 2013-01-26 at 10:49 +1100, paul.sz...@sydney.edu.au wrote:
> Dear Ben,
> 
> > If you can identify where it was fixed then ...
> 
> Sorry I cannot do that. I have no idea where kernel changelogs are kept.
> 
> I am happy to do some work. Please do not call me lazy.

The changelogs are in git repositories.  But the mm maintainers are
probably much better placed to identify which was the upstream fix.

Ben.

-- 
Ben Hutchings
Any smoothly functioning technology is indistinguishable from a rigged demo.


signature.asc
Description: This is a digitally signed message part


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread paul . szabo
Dear Ben,

> If you can identify where it was fixed then ...

Sorry I cannot do that. I have no idea where kernel changelogs are kept.

I am happy to do some work. Please do not call me lazy.

Cheers, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Ben Hutchings
On Fri, 2013-01-25 at 20:53 +1100, paul.sz...@sydney.edu.au wrote:
> Dear Minchan,
> 
> > So what's the effect for user?
> > ...
> > It seems you saw old kernel.
> > ...
> > Current kernel includes ...
> > So I think we don't need this patch.
> 
> As I understand now, my patch is "right" and needed for older kernels;
> for newer kernels, the issue has been fixed in equivalent ways; it was
> an oversight that the change was not backported; and any justification
> you need, you can get from those "later better" patches.
[...]

If you can identify where it was fixed then your patch for older
versions should go to stable with a reference to the upstream fix (see
Documentation/stable_kernel_rules.txt).

Ben.

-- 
Ben Hutchings
Q.  Which is the greater problem in the world today, ignorance or apathy?
A.  I don't know and I couldn't care less.


signature.asc
Description: This is a digitally signed message part


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread paul . szabo
Dear Minchan,

> So what's the effect for user?
> ...
> It seems you saw old kernel.
> ...
> Current kernel includes ...
> So I think we don't need this patch.

As I understand now, my patch is "right" and needed for older kernels;
for newer kernels, the issue has been fixed in equivalent ways; it was
an oversight that the change was not backported; and any justification
you need, you can get from those "later better" patches.

I asked:

  A question: what is the use or significance of vm_highmem_is_dirtyable?
  It seems odd that it would be used in setting limits or threshholds, but
  not used in decisions where to put dirty things. Is that so, is that as
  should be? What is the recommended setting of highmem_is_dirtyable?

The silence is deafening. I guess highmem_is_dirtyable is an aberration.

Thanks, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread paul . szabo
Dear Minchan,

 So what's the effect for user?
 ...
 It seems you saw old kernel.
 ...
 Current kernel includes ...
 So I think we don't need this patch.

As I understand now, my patch is right and needed for older kernels;
for newer kernels, the issue has been fixed in equivalent ways; it was
an oversight that the change was not backported; and any justification
you need, you can get from those later better patches.

I asked:

  A question: what is the use or significance of vm_highmem_is_dirtyable?
  It seems odd that it would be used in setting limits or threshholds, but
  not used in decisions where to put dirty things. Is that so, is that as
  should be? What is the recommended setting of highmem_is_dirtyable?

The silence is deafening. I guess highmem_is_dirtyable is an aberration.

Thanks, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Ben Hutchings
On Fri, 2013-01-25 at 20:53 +1100, paul.sz...@sydney.edu.au wrote:
 Dear Minchan,
 
  So what's the effect for user?
  ...
  It seems you saw old kernel.
  ...
  Current kernel includes ...
  So I think we don't need this patch.
 
 As I understand now, my patch is right and needed for older kernels;
 for newer kernels, the issue has been fixed in equivalent ways; it was
 an oversight that the change was not backported; and any justification
 you need, you can get from those later better patches.
[...]

If you can identify where it was fixed then your patch for older
versions should go to stable with a reference to the upstream fix (see
Documentation/stable_kernel_rules.txt).

Ben.

-- 
Ben Hutchings
Q.  Which is the greater problem in the world today, ignorance or apathy?
A.  I don't know and I couldn't care less.


signature.asc
Description: This is a digitally signed message part


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread paul . szabo
Dear Ben,

 If you can identify where it was fixed then ...

Sorry I cannot do that. I have no idea where kernel changelogs are kept.

I am happy to do some work. Please do not call me lazy.

Cheers, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Ben Hutchings
On Sat, 2013-01-26 at 10:49 +1100, paul.sz...@sydney.edu.au wrote:
 Dear Ben,
 
  If you can identify where it was fixed then ...
 
 Sorry I cannot do that. I have no idea where kernel changelogs are kept.
 
 I am happy to do some work. Please do not call me lazy.

The changelogs are in git repositories.  But the mm maintainers are
probably much better placed to identify which was the upstream fix.

Ben.

-- 
Ben Hutchings
Any smoothly functioning technology is indistinguishable from a rigged demo.


signature.asc
Description: This is a digitally signed message part


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Jonathan Nieder
Hi Paul,

paul.sz...@sydney.edu.au wrote:
 Dear Ben,

 If you can identify where it was fixed then ...

 Sorry I cannot do that. I have no idea where kernel changelogs are kept.

Here are some tools.

  # prerequisite:
  apt-get install git; # as root

  # to get the kernel history:
  git clone \
https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git
  cd linux

  # to view the changelog:
  git log v3.2..

  # to grep change descriptions:
  git log --grep=min_free_kbytes v3.2..

  # to view the patches corresponding to changes:
  git log --patch v3.2.. -- mm/

  # graphical interface
  apt-get install gitk; # as root
  gitk v3.2.. -- mm

  # web interface:
  w3m http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git

The exploring git history section of the git user manual has more
details:

  http://git-htmldocs.googlecode.com/git/user-manual.html#exploring-git-history

Thanks,
Jonathan
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread paul . szabo
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread paul . szabo
Dear Ben,

 ... the mm maintainers are probably much better placed ...

Exactly. Now I wonder: are you one of them?

Thanks, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Ben Hutchings
On Sat, 2013-01-26 at 14:07 +1100, paul.sz...@sydney.edu.au wrote:
 Dear Ben,
 
  ... the mm maintainers are probably much better placed ...
 
 Exactly. Now I wonder: are you one of them?

Hah, no.

Ben.

-- 
Ben Hutchings
Any smoothly functioning technology is indistinguishable from a rigged demo.


signature.asc
Description: This is a digitally signed message part


Re: Bug#695182: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-25 Thread Jonathan Nieder
Hi Paul,

Ben Hutchings wrote:

 If you can identify where it was fixed then your patch for older
 versions should go to stable with a reference to the upstream fix (see
 Documentation/stable_kernel_rules.txt).

How about this patch?

It was applied in mainline during the 3.3 merge window, so kernels
newer than 3.2.y shouldn't need it.

-- 8 --
From: Johannes Weiner jwei...@redhat.com
Date: Tue, 10 Jan 2012 15:07:42 -0800
Subject: mm: exclude reserved pages from dirtyable memory

commit ab8fabd46f811d5153d8a0cd2fac9a0d41fb593d upstream.

Per-zone dirty limits try to distribute page cache pages allocated for
writing across zones in proportion to the individual zone sizes, to reduce
the likelihood of reclaim having to write back individual pages from the
LRU lists in order to make progress.

This patch:

The amount of dirtyable pages should not include the full number of free
pages: there is a number of reserved pages that the page allocator and
kswapd always try to keep free.

The closer (reclaimable pages - dirty pages) is to the number of reserved
pages, the more likely it becomes for reclaim to run into dirty pages:

   +--+ ---
   |   anon   |  |
   +--+  |
   |  |  |
   |  |  -- dirty limit new-- flusher new
   |   file   |  | |
   |  |  | |
   |  |  -- dirty limit old-- flusher old
   |  ||
   +--+   --- reclaim
   | reserved |
   +--+
   |  kernel  |
   +--+

This patch introduces a per-zone dirty reserve that takes both the lowmem
reserve as well as the high watermark of the zone into account, and a
global sum of those per-zone values that is subtracted from the global
amount of dirtyable pages.  The lowmem reserve is unavailable to page
cache allocations and kswapd tries to keep the high watermark free.  We
don't want to end up in a situation where reclaim has to clean pages in
order to balance zones.

Not treating reserved pages as dirtyable on a global level is only a
conceptual fix.  In reality, dirty pages are not distributed equally
across zones and reclaim runs into dirty pages on a regular basis.

But it is important to get this right before tackling the problem on a
per-zone level, where the distance between reclaim and the dirty pages is
mostly much smaller in absolute numbers.

[a...@linux-foundation.org: fix highmem build]
Signed-off-by: Johannes Weiner jwei...@redhat.com
Reviewed-by: Rik van Riel r...@redhat.com
Reviewed-by: Michal Hocko mho...@suse.cz
Reviewed-by: Minchan Kim minchan@gmail.com
Acked-by: Mel Gorman mgor...@suse.de
Cc: KAMEZAWA Hiroyuki kamezawa.hir...@jp.fujitsu.com
Cc: Christoph Hellwig h...@infradead.org
Cc: Wu Fengguang fengguang...@intel.com
Cc: Dave Chinner da...@fromorbit.com
Cc: Jan Kara j...@suse.cz
Cc: Shaohua Li shaohua...@intel.com
Cc: Chris Mason chris.ma...@oracle.com
Signed-off-by: Andrew Morton a...@linux-foundation.org
Signed-off-by: Linus Torvalds torva...@linux-foundation.org
Signed-off-by: Jonathan Nieder jrnie...@gmail.com
---
 include/linux/mmzone.h |  6 ++
 include/linux/swap.h   |  1 +
 mm/page-writeback.c|  5 +++--
 mm/page_alloc.c| 19 +++
 4 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 25842b6e72e1..a594af3278bc 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -319,6 +319,12 @@ struct zone {
 */
unsigned long   lowmem_reserve[MAX_NR_ZONES];
 
+   /*
+* This is a per-zone reserve of pages that should not be
+* considered dirtyable memory.
+*/
+   unsigned long   dirty_balance_reserve;
+
 #ifdef CONFIG_NUMA
int node;
/*
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 67b3fa308988..3e60228e7299 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -207,6 +207,7 @@ struct swap_list_t {
 /* linux/mm/page_alloc.c */
 extern unsigned long totalram_pages;
 extern unsigned long totalreserve_pages;
+extern unsigned long dirty_balance_reserve;
 extern unsigned int nr_free_buffer_pages(void);
 extern unsigned int nr_free_pagecache_pages(void);
 
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 50f08241f981..f620e7b0dc26 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -320,7 +320,7 @@ static unsigned long highmem_dirtyable_memory(unsigned long 
total)
NODE_DATA(node)-node_zones[ZONE_HIGHMEM];
 
x += zone_page_state(z, NR_FREE_PAGES) +
-zone_reclaimable_pages(z);
+zone_reclaimable_pages(z) - z-dirty_balance_reserve;
}
/*
 * Make sure that the number of highmem pages is never larger
@@ -344,7 +344,8 @@ unsigned long determine_dirtyable_memory(void)
 {
unsigned long x;
 
-   

Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-22 Thread paul . szabo
Dear Minchan,

> So what's the effect for user?

Sorry I have no idea.

The kernel seems to work well without this patch; or in fact not so
well, PAE crashing with spurious OOM. In my fruitless efforts of
avoiding OOM by sensible choices of sysctl tunables, I noticed that
maybe the treatment of min_free_kbytes was not right. Getting this
right did not help in avoiding OOM.

> It seems you saw old kernel.

Yes I have Debian on my machines. :-)

> Current kernel includes following logic.
> 
> static unsigned long global_dirtyable_memory(void)
> {
> unsigned long x;
> 
> x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
> x -= min(x, dirty_balance_reserve);
> 
> if (!vm_highmem_is_dirtyable)
> x -= highmem_dirtyable_memory(x);
> 
> return x + 1;   /* Ensure that we never return 0 */
> }
> 
> And dirty_lanace_reserve already includes high_wmark_pages.
> Look at calculate_totalreserve_pages.
> 
> So I think we don't need this patch.
> Thanks.

Presumably, dirty_balance_reserve takes min_free_kbytes into account?
Then I agree that this patch is not needed on those newer kernels.

A question: what is the use or significance of vm_highmem_is_dirtyable?
It seems odd that it would be used in setting limits or threshholds, but
not used in decisions where to put dirty things. Is that so, is that as
should be? What is the recommended setting of highmem_is_dirtyable?

Thanks, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-22 Thread Minchan Kim
On Mon, Jan 21, 2013 at 02:15:49PM +1100, paul.sz...@sydney.edu.au wrote:
> When calculating amount of dirtyable memory, min_free_kbytes should be
> subtracted because it is not intended for dirty pages.

So what's the effect for user?
It would be better to include that in description if possible.

> 
> Using an "extern int" because that is the only interface to some such
> sysctl values.
> 
> (This patch does not solve the PAE OOM issue.)
> 
> Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
> School of Mathematics and Statistics   University of SydneyAustralia
> 
> Reported-by: Paul Szabo 
> Reference: http://bugs.debian.org/695182
> Signed-off-by: Paul Szabo 
> 
> --- mm/page-writeback.c.old   2012-12-06 22:20:40.0 +1100
> +++ mm/page-writeback.c   2013-01-21 13:57:05.0 +1100
> @@ -343,12 +343,16 @@
>  unsigned long determine_dirtyable_memory(void)
>  {
>   unsigned long x;
> + extern int min_free_kbytes;
>  
>   x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
>  
>   if (!vm_highmem_is_dirtyable)
>   x -= highmem_dirtyable_memory(x);
>  
> + /* Subtract min_free_kbytes */
> + x -= min(x, min_free_kbytes >> (PAGE_SHIFT - 10));

It seems you saw old kernel.
Current kernel includes following logic.

static unsigned long global_dirtyable_memory(void)
{
unsigned long x;

x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
x -= min(x, dirty_balance_reserve);

if (!vm_highmem_is_dirtyable)
x -= highmem_dirtyable_memory(x);

return x + 1;   /* Ensure that we never return 0 */
}

And dirty_lanace_reserve already includes high_wmark_pages.
Look at calculate_totalreserve_pages.

So I think we don't need this patch.
Thanks.

> +
>   return x + 1;   /* Ensure that we never return 0 */
>  }
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majord...@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: mailto:"d...@kvack.org;> em...@kvack.org 

-- 
Kind regards,
Minchan Kim
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-22 Thread Andrew Morton
On Mon, 21 Jan 2013 14:15:49 +1100
paul.sz...@sydney.edu.au wrote:

> When calculating amount of dirtyable memory, min_free_kbytes should be
> subtracted because it is not intended for dirty pages.

Makes sense.

> Using an "extern int" because that is the only interface to some such
> sysctl values.

urgh, not that way.  Let's do it properly:

From: Andrew Morton 
Subject: page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix

fix up min_free_kbytes extern declarations

Cc: Paul Szabo 
Cc: Rik van Riel 
Cc: Wu Fengguang 
Signed-off-by: Andrew Morton 
---

 include/linux/mm.h  |3 +++
 kernel/sysctl.c |1 -
 mm/huge_memory.c|1 -
 mm/page-writeback.c |1 -
 4 files changed, 3 insertions(+), 3 deletions(-)

--- 
a/mm/page-writeback.c~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix
+++ a/mm/page-writeback.c
@@ -233,7 +233,6 @@ static unsigned long highmem_dirtyable_m
 static unsigned long global_dirtyable_memory(void)
 {
unsigned long x;
-   extern int min_free_kbytes;
 
x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
x -= min(x, dirty_balance_reserve);
--- 
a/include/linux/mm.h~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix
+++ a/include/linux/mm.h
@@ -1387,6 +1387,9 @@ extern void setup_per_cpu_pageset(void);
 extern void zone_pcp_update(struct zone *zone);
 extern void zone_pcp_reset(struct zone *zone);
 
+/* page_alloc.c */
+extern int min_free_kbytes;
+
 /* nommu.c */
 extern atomic_long_t mmap_pages_allocated;
 extern int nommu_shrink_inode_mappings(struct inode *, size_t, size_t);
--- 
a/mm/huge_memory.c~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix
+++ a/mm/huge_memory.c
@@ -105,7 +105,6 @@ static int set_recommended_min_free_kbyt
struct zone *zone;
int nr_zones = 0;
unsigned long recommended_min;
-   extern int min_free_kbytes;
 
if (!khugepaged_enabled())
return 0;
--- 
a/kernel/sysctl.c~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix
+++ a/kernel/sysctl.c
@@ -104,7 +104,6 @@ extern char core_pattern[];
 extern unsigned int core_pipe_limit;
 #endif
 extern int pid_max;
-extern int min_free_kbytes;
 extern int pid_max_min, pid_max_max;
 extern int sysctl_drop_caches;
 extern int percpu_pagelist_fraction;
_


> (This patch does not solve the PAE OOM issue.)
> 
> Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
> School of Mathematics and Statistics   University of SydneyAustralia
> 
> Reported-by: Paul Szabo 

Reported-by isn't needed in such cases.  It is assumed that finder==fixer.

> Reference: http://bugs.debian.org/695182
> Signed-off-by: Paul Szabo 
> 
> --- mm/page-writeback.c.old   2012-12-06 22:20:40.0 +1100
> +++ mm/page-writeback.c   2013-01-21 13:57:05.0 +1100

Please prepare patches in `patch -p1' form.  This should be covered in
Documentation/SubmittingPatches, but isn't. 
Documentation/applying-patches.txt mentions it.

> @@ -343,12 +343,16 @@
>  unsigned long determine_dirtyable_memory(void)

You appear to be patching an old kernel.  But the change is still
applicable, to global_dirtyable_memory().

>  {
>   unsigned long x;
> + extern int min_free_kbytes;
>  
>   x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
>  
>   if (!vm_highmem_is_dirtyable)
>   x -= highmem_dirtyable_memory(x);
>  
> + /* Subtract min_free_kbytes */
> + x -= min(x, min_free_kbytes >> (PAGE_SHIFT - 10));

Generates

mm/page-writeback.c:244: warning: comparison of distinct pointer types lacks a 
cast

because of the problematic min(int, unsigned long).  min_free_kbytes
should have an unsigned (long?) type, but I can't be bothered fixing
that right now..

From: Andrew Morton 
Subject: page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix-fix

fix min() warning

Cc: Paul Szabo 
Cc: Rik van Riel 
Cc: Wu Fengguang 
Signed-off-by: Andrew Morton 
---

 mm/page-writeback.c |2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- 
a/mm/page-writeback.c~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix-fix
+++ a/mm/page-writeback.c
@@ -241,7 +241,7 @@ static unsigned long global_dirtyable_me
x -= highmem_dirtyable_memory(x);
 
/* Subtract min_free_kbytes */
-   x -= min(x, min_free_kbytes >> (PAGE_SHIFT - 10));
+   x -= min_t(unsigned long, x, min_free_kbytes >> (PAGE_SHIFT - 10));
 
return x + 1;   /* Ensure that we never return 0 */
 }
_

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-22 Thread Andrew Morton
On Mon, 21 Jan 2013 14:15:49 +1100
paul.sz...@sydney.edu.au wrote:

 When calculating amount of dirtyable memory, min_free_kbytes should be
 subtracted because it is not intended for dirty pages.

Makes sense.

 Using an extern int because that is the only interface to some such
 sysctl values.

urgh, not that way.  Let's do it properly:

From: Andrew Morton a...@linux-foundation.org
Subject: page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix

fix up min_free_kbytes extern declarations

Cc: Paul Szabo p...@maths.usyd.edu.au
Cc: Rik van Riel r...@redhat.com
Cc: Wu Fengguang fengguang...@intel.com
Signed-off-by: Andrew Morton a...@linux-foundation.org
---

 include/linux/mm.h  |3 +++
 kernel/sysctl.c |1 -
 mm/huge_memory.c|1 -
 mm/page-writeback.c |1 -
 4 files changed, 3 insertions(+), 3 deletions(-)

--- 
a/mm/page-writeback.c~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix
+++ a/mm/page-writeback.c
@@ -233,7 +233,6 @@ static unsigned long highmem_dirtyable_m
 static unsigned long global_dirtyable_memory(void)
 {
unsigned long x;
-   extern int min_free_kbytes;
 
x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
x -= min(x, dirty_balance_reserve);
--- 
a/include/linux/mm.h~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix
+++ a/include/linux/mm.h
@@ -1387,6 +1387,9 @@ extern void setup_per_cpu_pageset(void);
 extern void zone_pcp_update(struct zone *zone);
 extern void zone_pcp_reset(struct zone *zone);
 
+/* page_alloc.c */
+extern int min_free_kbytes;
+
 /* nommu.c */
 extern atomic_long_t mmap_pages_allocated;
 extern int nommu_shrink_inode_mappings(struct inode *, size_t, size_t);
--- 
a/mm/huge_memory.c~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix
+++ a/mm/huge_memory.c
@@ -105,7 +105,6 @@ static int set_recommended_min_free_kbyt
struct zone *zone;
int nr_zones = 0;
unsigned long recommended_min;
-   extern int min_free_kbytes;
 
if (!khugepaged_enabled())
return 0;
--- 
a/kernel/sysctl.c~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix
+++ a/kernel/sysctl.c
@@ -104,7 +104,6 @@ extern char core_pattern[];
 extern unsigned int core_pipe_limit;
 #endif
 extern int pid_max;
-extern int min_free_kbytes;
 extern int pid_max_min, pid_max_max;
 extern int sysctl_drop_caches;
 extern int percpu_pagelist_fraction;
_


 (This patch does not solve the PAE OOM issue.)
 
 Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
 School of Mathematics and Statistics   University of SydneyAustralia
 
 Reported-by: Paul Szabo p...@maths.usyd.edu.au

Reported-by isn't needed in such cases.  It is assumed that finder==fixer.

 Reference: http://bugs.debian.org/695182
 Signed-off-by: Paul Szabo p...@maths.usyd.edu.au
 
 --- mm/page-writeback.c.old   2012-12-06 22:20:40.0 +1100
 +++ mm/page-writeback.c   2013-01-21 13:57:05.0 +1100

Please prepare patches in `patch -p1' form.  This should be covered in
Documentation/SubmittingPatches, but isn't. 
Documentation/applying-patches.txt mentions it.

 @@ -343,12 +343,16 @@
  unsigned long determine_dirtyable_memory(void)

You appear to be patching an old kernel.  But the change is still
applicable, to global_dirtyable_memory().

  {
   unsigned long x;
 + extern int min_free_kbytes;
  
   x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
  
   if (!vm_highmem_is_dirtyable)
   x -= highmem_dirtyable_memory(x);
  
 + /* Subtract min_free_kbytes */
 + x -= min(x, min_free_kbytes  (PAGE_SHIFT - 10));

Generates

mm/page-writeback.c:244: warning: comparison of distinct pointer types lacks a 
cast

because of the problematic min(int, unsigned long).  min_free_kbytes
should have an unsigned (long?) type, but I can't be bothered fixing
that right now..

From: Andrew Morton a...@linux-foundation.org
Subject: page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix-fix

fix min() warning

Cc: Paul Szabo p...@maths.usyd.edu.au
Cc: Rik van Riel r...@redhat.com
Cc: Wu Fengguang fengguang...@intel.com
Signed-off-by: Andrew Morton a...@linux-foundation.org
---

 mm/page-writeback.c |2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- 
a/mm/page-writeback.c~page-writebackc-subtract-min_free_kbytes-from-dirtyable-memory-fix-fix
+++ a/mm/page-writeback.c
@@ -241,7 +241,7 @@ static unsigned long global_dirtyable_me
x -= highmem_dirtyable_memory(x);
 
/* Subtract min_free_kbytes */
-   x -= min(x, min_free_kbytes  (PAGE_SHIFT - 10));
+   x -= min_t(unsigned long, x, min_free_kbytes  (PAGE_SHIFT - 10));
 
return x + 1;   /* Ensure that we never return 0 */
 }
_

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  

Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-22 Thread Minchan Kim
On Mon, Jan 21, 2013 at 02:15:49PM +1100, paul.sz...@sydney.edu.au wrote:
 When calculating amount of dirtyable memory, min_free_kbytes should be
 subtracted because it is not intended for dirty pages.

So what's the effect for user?
It would be better to include that in description if possible.

 
 Using an extern int because that is the only interface to some such
 sysctl values.
 
 (This patch does not solve the PAE OOM issue.)
 
 Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
 School of Mathematics and Statistics   University of SydneyAustralia
 
 Reported-by: Paul Szabo p...@maths.usyd.edu.au
 Reference: http://bugs.debian.org/695182
 Signed-off-by: Paul Szabo p...@maths.usyd.edu.au
 
 --- mm/page-writeback.c.old   2012-12-06 22:20:40.0 +1100
 +++ mm/page-writeback.c   2013-01-21 13:57:05.0 +1100
 @@ -343,12 +343,16 @@
  unsigned long determine_dirtyable_memory(void)
  {
   unsigned long x;
 + extern int min_free_kbytes;
  
   x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
  
   if (!vm_highmem_is_dirtyable)
   x -= highmem_dirtyable_memory(x);
  
 + /* Subtract min_free_kbytes */
 + x -= min(x, min_free_kbytes  (PAGE_SHIFT - 10));

It seems you saw old kernel.
Current kernel includes following logic.

static unsigned long global_dirtyable_memory(void)
{
unsigned long x;

x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
x -= min(x, dirty_balance_reserve);

if (!vm_highmem_is_dirtyable)
x -= highmem_dirtyable_memory(x);

return x + 1;   /* Ensure that we never return 0 */
}

And dirty_lanace_reserve already includes high_wmark_pages.
Look at calculate_totalreserve_pages.

So I think we don't need this patch.
Thanks.

 +
   return x + 1;   /* Ensure that we never return 0 */
  }
 
 --
 To unsubscribe, send a message with 'unsubscribe linux-mm' in
 the body to majord...@kvack.org.  For more info on Linux MM,
 see: http://www.linux-mm.org/ .
 Don't email: a href=mailto:d...@kvack.org; em...@kvack.org /a

-- 
Kind regards,
Minchan Kim
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-22 Thread paul . szabo
Dear Minchan,

 So what's the effect for user?

Sorry I have no idea.

The kernel seems to work well without this patch; or in fact not so
well, PAE crashing with spurious OOM. In my fruitless efforts of
avoiding OOM by sensible choices of sysctl tunables, I noticed that
maybe the treatment of min_free_kbytes was not right. Getting this
right did not help in avoiding OOM.

 It seems you saw old kernel.

Yes I have Debian on my machines. :-)

 Current kernel includes following logic.
 
 static unsigned long global_dirtyable_memory(void)
 {
 unsigned long x;
 
 x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
 x -= min(x, dirty_balance_reserve);
 
 if (!vm_highmem_is_dirtyable)
 x -= highmem_dirtyable_memory(x);
 
 return x + 1;   /* Ensure that we never return 0 */
 }
 
 And dirty_lanace_reserve already includes high_wmark_pages.
 Look at calculate_totalreserve_pages.
 
 So I think we don't need this patch.
 Thanks.

Presumably, dirty_balance_reserve takes min_free_kbytes into account?
Then I agree that this patch is not needed on those newer kernels.

A question: what is the use or significance of vm_highmem_is_dirtyable?
It seems odd that it would be used in setting limits or threshholds, but
not used in decisions where to put dirty things. Is that so, is that as
should be? What is the recommended setting of highmem_is_dirtyable?

Thanks, Paul

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-21 Thread Rik van Riel

On 01/20/2013 10:15 PM, paul.sz...@sydney.edu.au wrote:

When calculating amount of dirtyable memory, min_free_kbytes should be
subtracted because it is not intended for dirty pages.

Using an "extern int" because that is the only interface to some such
sysctl values.

(This patch does not solve the PAE OOM issue.)

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia

Reported-by: Paul Szabo 
Reference: http://bugs.debian.org/695182
Signed-off-by: Paul Szabo 


Acked-by: Rik van Riel 


--
All rights reversed
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: [PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-21 Thread Rik van Riel

On 01/20/2013 10:15 PM, paul.sz...@sydney.edu.au wrote:

When calculating amount of dirtyable memory, min_free_kbytes should be
subtracted because it is not intended for dirty pages.

Using an extern int because that is the only interface to some such
sysctl values.

(This patch does not solve the PAE OOM issue.)

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia

Reported-by: Paul Szabo p...@maths.usyd.edu.au
Reference: http://bugs.debian.org/695182
Signed-off-by: Paul Szabo p...@maths.usyd.edu.au


Acked-by: Rik van Riel r...@redhat.com


--
All rights reversed
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


[PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-20 Thread paul . szabo
When calculating amount of dirtyable memory, min_free_kbytes should be
subtracted because it is not intended for dirty pages.

Using an "extern int" because that is the only interface to some such
sysctl values.

(This patch does not solve the PAE OOM issue.)

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia

Reported-by: Paul Szabo 
Reference: http://bugs.debian.org/695182
Signed-off-by: Paul Szabo 

--- mm/page-writeback.c.old 2012-12-06 22:20:40.0 +1100
+++ mm/page-writeback.c 2013-01-21 13:57:05.0 +1100
@@ -343,12 +343,16 @@
 unsigned long determine_dirtyable_memory(void)
 {
unsigned long x;
+   extern int min_free_kbytes;
 
x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
 
if (!vm_highmem_is_dirtyable)
x -= highmem_dirtyable_memory(x);
 
+   /* Subtract min_free_kbytes */
+   x -= min(x, min_free_kbytes >> (PAGE_SHIFT - 10));
+
return x + 1;   /* Ensure that we never return 0 */
 }

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


[PATCH] Subtract min_free_kbytes from dirtyable memory

2013-01-20 Thread paul . szabo
When calculating amount of dirtyable memory, min_free_kbytes should be
subtracted because it is not intended for dirty pages.

Using an extern int because that is the only interface to some such
sysctl values.

(This patch does not solve the PAE OOM issue.)

Paul Szabo   p...@maths.usyd.edu.au   http://www.maths.usyd.edu.au/u/psz/
School of Mathematics and Statistics   University of SydneyAustralia

Reported-by: Paul Szabo p...@maths.usyd.edu.au
Reference: http://bugs.debian.org/695182
Signed-off-by: Paul Szabo p...@maths.usyd.edu.au

--- mm/page-writeback.c.old 2012-12-06 22:20:40.0 +1100
+++ mm/page-writeback.c 2013-01-21 13:57:05.0 +1100
@@ -343,12 +343,16 @@
 unsigned long determine_dirtyable_memory(void)
 {
unsigned long x;
+   extern int min_free_kbytes;
 
x = global_page_state(NR_FREE_PAGES) + global_reclaimable_pages();
 
if (!vm_highmem_is_dirtyable)
x -= highmem_dirtyable_memory(x);
 
+   /* Subtract min_free_kbytes */
+   x -= min(x, min_free_kbytes  (PAGE_SHIFT - 10));
+
return x + 1;   /* Ensure that we never return 0 */
 }

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/