Re: [RFC][PATCH 3/8] mem_notify v5: introduce /dev/mem_notify new device (the core of this patch series)
Hi KOSAKI, On 1/24/08, KOSAKI Motohiro <[EMAIL PROTECTED]> wrote: > +#define PROC_WAKEUP_GUARD (10*HZ) [...] > + timeout = info->last_proc_notify + PROC_WAKEUP_GUARD; If only one or a few processes are using the system I think 10 seconds is a little long time to wait before they get the notification again. Can we decrease this value? Or make it configurable under /proc? Or make it lower with fewer users? Something like: timeout = info->last_proc_notify + min(mem_notify_users, PROC_WAKEUP_GUARD); Cheers, Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC][PATCH 4/5] memory_pressure_notify() caller
On 1/17/08, KOSAKI Motohiro <[EMAIL PROTECTED]> wrote:
> Hi Daniel
>
> > > Thank you for good point out!
> > > Could you please post your test program and reproduced method?
> >
> > Sure:
> >
> > 1. Fill almost all available memory with page cache in a system without
> > swap.
> > 2. Run attached alloc-test program.
> > 3. Notification fires when page cache is reclaimed.
>
> Unfortunately, I can't reproduce it.
>
> my machine
> CPU:Pentium4 2.8GHz with HT
> memory: 512M
>
>
> 1. I doubt ZONE_DMA, please shipment ignore zone_dma patch(below).
> 2. Could you please send your .config and /etc/sysctl.conf?
>I hope more reproduce challenge.
>
> thanks.
>
> - kosaki
>
>
>
>
> Signed-off-by: KOSAKI Motohiro <[EMAIL PROTECTED]>
>
> ---
> include/linux/mem_notify.h |3 +++
> mm/page_alloc.c|6 +-
> 2 files changed, 8 insertions(+), 1 deletion(-)
>
> Index: linux-2.6.24-rc6-mm1-memnotify/include/linux/mem_notify.h
> ===
> --- linux-2.6.24-rc6-mm1-memnotify.orig/include/linux/mem_notify.h
> 2008-01-16 21:31:09.0 +0900
> +++ linux-2.6.24-rc6-mm1-memnotify/include/linux/mem_notify.h
> 2008-01-16 21:34:24.0 +0900
> @@ -22,6 +22,9 @@ static inline void memory_pressure_notif
> unsigned long target;
> unsigned long pages_high, pages_free, pages_reserve;
>
> + if (unlikely(zone->mem_notify_status == -1))
> + return;
> +
> if (pressure) {
> target = atomic_long_read(&last_mem_notify) + MEM_NOTIFY_FREQ;
> if (likely(time_before(jiffies, target)))
> Index: linux-2.6.24-rc6-mm1-memnotify/mm/page_alloc.c
> ===
> --- linux-2.6.24-rc6-mm1-memnotify.orig/mm/page_alloc.c 2008-01-13
> 19:50:27.0 +0900
> +++ linux-2.6.24-rc6-mm1-memnotify/mm/page_alloc.c 2008-01-16
> 21:41:58.0 +0900
> @@ -3467,7 +3467,11 @@ static void __meminit free_area_init_cor
> zone->zone_pgdat = pgdat;
>
> zone->prev_priority = DEF_PRIORITY;
> - zone->mem_notify_status = 0;
> +
> + if (zone->present_pages < (pgdat->node_present_pages / 10))
> + zone->mem_notify_status = -1;
> + else
> + zone->mem_notify_status = 0;
>
> zone_pcp_init(zone);
> INIT_LIST_HEAD(&zone->active_list);
Your patch above solves the problem I had with early notification.
Cheers,
Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC][PATCH 3/5] add /dev/mem_notify device
On 1/16/08, Pavel Machek <[EMAIL PROTECTED]> wrote: > On Wed 2008-01-16 02:13:32, Marcelo Tosatti wrote: > > On Wed, Jan 16, 2008 at 10:57:16AM +0900, KOSAKI Motohiro wrote: > > > Hi Pavel > > > > > > > > err = poll(&pollfds, 1, -1); // wake up at low memory > > > > > > > > > > ... > > > > > > > > > > > > > Nice, this is really needed for openmoko, zaurus, etc > > > > > > > > But this changelog needs to go into Documentation/... > > > > > > > > ...and /dev/mem_notify is really a bad name. /dev/memory_low? > > > > /dev/oom? > > > > > > thank you for your kindful advise. > > > > > > but.. > > > > > > to be honest, my english is very limited. > > > I can't make judgments name is good or not. > > > > > > Marcelo, What do you think his idea? > > > > "mem_notify" sounds alright, but I don't really care. > > > > Notify: > > > > To give notice to; inform: notified the citizens of the curfew by > > posting signs. > > I'd read mem_notify as "tell me when new memory is unplugged" or > something. /dev/oom_notify? Plus, /dev/ names usually do not have "_" > in them. I don't think we should use oom in the name, since the notification is sent long before oom. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC][PATCH 4/5] memory_pressure_notify() caller
On 1/16/08, KOSAKI Motohiro <[EMAIL PROTECTED]> wrote: > Hi Daniel > > > > > The notification fires after only ~100 MB allocated, i.e., when page > > > > reclaim is beginning to nag from page cache. Isn't this a bit early? > > > > Repeating the test with swap enabled results in a notification after > > > > ~600 MB allocated, which is more reasonable and just before the system > > > > starts to swap. > > > > > > Your issue may have more to do with the fact that the > > > highmem zone is 128MB in size and some balancing issues > > > between __alloc_pages and try_to_free_pages. > > > > I don't think so. I ran the test again without highmem and noticed the > > same behaviour: > > Thank you for good point out! > Could you please post your test program and reproduced method? Sure: 1. Fill almost all available memory with page cache in a system without swap. 2. Run attached alloc-test program. 3. Notification fires when page cache is reclaimed. Example: $ cat /bigfile > /dev/null $ cat /proc/meminfo MemTotal: 895876 kB MemFree: 94272 kB Buffers: 884 kB Cached: 782868 kB SwapCached: 0 kB Active: 15356 kB Inactive: 778000 kB HighTotal: 0 kB HighFree:0 kB LowTotal: 895876 kB LowFree: 94272 kB SwapTotal: 0 kB SwapFree:0 kB Dirty: 0 kB Writeback: 0 kB AnonPages:9624 kB Mapped: 1352 kB Slab: 4220 kB SReclaimable: 1168 kB SUnreclaim: 3052 kB PageTables:528 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit:447936 kB Committed_AS:28988 kB VmallocTotal: 122872 kB VmallocUsed: 904 kB VmallocChunk: 121864 kB $ ./test-alloc - Got notification, allocated 90 MB $ cat /proc/meminfo MemTotal: 895876 kB MemFree:101960 kB Buffers: 888 kB Cached: 775200 kB SwapCached: 0 kB Active: 15356 kB Inactive: 770336 kB HighTotal: 0 kB HighFree:0 kB LowTotal: 895876 kB LowFree:101960 kB SwapTotal: 0 kB SwapFree:0 kB Dirty: 28 kB Writeback: 0 kB AnonPages:9624 kB Mapped: 1352 kB Slab: 4224 kB SReclaimable: 1168 kB SUnreclaim: 3056 kB PageTables:532 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit:447936 kB Committed_AS:28988 kB VmallocTotal: 122872 kB VmallocUsed: 904 kB VmallocChunk: 121864 kB alloc-test.c Description: Binary data
Re: [RFC][PATCH 4/5] memory_pressure_notify() caller
On 1/15/08, Rik van Riel <[EMAIL PROTECTED]> wrote: > > On Tue, 15 Jan 2008 23:55:17 +0100 > "Daniel Spång" <[EMAIL PROTECTED]> wrote: > > > The notification fires after only ~100 MB allocated, i.e., when page > > reclaim is beginning to nag from page cache. Isn't this a bit early? > > Repeating the test with swap enabled results in a notification after > > ~600 MB allocated, which is more reasonable and just before the system > > starts to swap. > > Your issue may have more to do with the fact that the > highmem zone is 128MB in size and some balancing issues > between __alloc_pages and try_to_free_pages. I don't think so. I ran the test again without highmem and noticed the same behaviour: $ cat /proc/meminfo MemTotal: 895876 kB MemFree:111292 kB Buffers: 924 kB Cached: 768664 kB SwapCached: 0 kB Active: 9196 kB Inactive: 767480 kB HighTotal: 0 kB HighFree:0 kB LowTotal: 895876 kB LowFree:111292 kB SwapTotal: 0 kB SwapFree:0 kB Dirty: 32 kB Writeback: 0 kB AnonPages:7108 kB Mapped: 1224 kB Slab: 4288 kB SReclaimable: 1316 kB SUnreclaim: 2972 kB PageTables:448 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit:447936 kB Committed_AS:19676 kB VmallocTotal: 122872 kB VmallocUsed: 904 kB VmallocChunk: 121864 kB Start to allocate memory, 10 MB every second, exit on notification which happened after 110 MB. $ cat /proc/meminfo #after MemTotal: 895876 kB MemFree:116748 kB Buffers: 904 kB Cached: 762944 kB SwapCached: 0 kB Active: 12864 kB Inactive: 758064 kB HighTotal: 0 kB HighFree:0 kB LowTotal: 895876 kB LowFree:116748 kB SwapTotal: 0 kB SwapFree:0 kB Dirty: 4 kB Writeback: 0 kB AnonPages:7108 kB Mapped: 1224 kB Slab: 4284 kB SReclaimable: 1316 kB SUnreclaim: 2968 kB PageTables:448 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit:447936 kB Committed_AS:19676 kB VmallocTotal: 122872 kB VmallocUsed: 904 kB VmallocChunk: 121864 kB -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC][PATCH 4/5] memory_pressure_notify() caller
Hi, On 1/15/08, KOSAKI Motohiro <[EMAIL PROTECTED]> wrote: > the notification point to happen whenever the VM moves an > anonymous page to the inactive list - this is a pretty good indication > that there are unused anonymous pages present which will be very likely > swapped out soon. > + /* deal with the case where there is no > +* swap but an anonymous page would be > +* moved to the inactive list. > +*/ > + if (!total_swap_pages && reclaim_mapped && > + PageAnon(page)) > + inactivated_anon = 1; As you know I have had some concerns regarding a too early notification in a swapless system. I did a test with a populated page cache in a swapless system: $ cat /bigfile > /dev/null # populate page cache $ cat /proc/meminfo MemTotal: 1037040 kB MemFree:113976 kB Buffers: 1068 kB Cached: 907552 kB SwapCached: 0 kB Active: 6 kB Inactive: 903968 kB HighTotal: 130992 kB HighFree: 252 kB LowTotal: 906048 kB LowFree:113724 kB SwapTotal: 0 kB SwapFree:0 kB Dirty: 36 kB Writeback: 0 kB AnonPages:6484 kB Mapped: 1216 kB Slab: 4024 kB SReclaimable: 864 kB SUnreclaim: 3160 kB PageTables:444 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit:518520 kB Committed_AS:18816 kB VmallocTotal: 114680 kB VmallocUsed: 904 kB VmallocChunk: 113672 kB Start to allocate memory, 10 MB every second, exit on notification. $ cat /proc/meminfo # just after notification MemTotal: 1037040 kB MemFree:123468 kB Buffers: 876 kB Cached: 897976 kB SwapCached: 0 kB Active: 12984 kB Inactive: 892332 kB HighTotal: 130992 kB HighFree: 1064 kB LowTotal: 906048 kB LowFree:122404 kB SwapTotal: 0 kB SwapFree:0 kB Dirty: 0 kB Writeback: 0 kB AnonPages:6484 kB Mapped: 1220 kB Slab: 4012 kB SReclaimable: 864 kB SUnreclaim: 3148 kB PageTables:448 kB NFS_Unstable:0 kB Bounce: 0 kB CommitLimit:518520 kB Committed_AS:18816 kB VmallocTotal: 114680 kB VmallocUsed: 904 kB VmallocChunk: 113672 kB The notification fires after only ~100 MB allocated, i.e., when page reclaim is beginning to nag from page cache. Isn't this a bit early? Repeating the test with swap enabled results in a notification after ~600 MB allocated, which is more reasonable and just before the system starts to swap. Cheers, Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Out of memory management in embedded systems
On 9/28/07, linux-os (Dick Johnson) <[EMAIL PROTECTED]> wrote: > > On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > > > On 9/28/07, linux-os (Dick Johnson) <[EMAIL PROTECTED]> wrote: > >> > >> On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > >> > >>> On 9/28/07, linux-os (Dick Johnson) <[EMAIL PROTECTED]> wrote: > >>>> > >>>> But an embedded system contains all the software that will > >>>> ever be executed on that system! If it is properly designed, > >>>> it can never run out of memory because everything it will > >>>> ever do is known at design time. > >>> > >>> Not if its input is not known beforehand. Take a browser in a mobile > >>> phone as an example, it does not know at design time how big the web > >>> pages are. On the other hand we want to use as much memory as > >>> possible, for cache etc., a method that involves the kernel would > >>> simplify this and avoids setting manual limits. > >>> > >>> Daniel > >>> > >> > >> Any networked appliance can (will) throw data away if there are > >> no resources available. > >> > >> The length of a web-page is not relevent, nor is the length > >> of any external data. Your example will buffer whatever it > >> can and not read anything more from the external source until > >> it has resources available unless it is broken. > > > > And how do you determine when no resources are availabe? We are using > > overcommit here so malloc() will always return non null. > > > > A networked appliance using embedded software is not your daddy's > Chevrolet. Any task that is permanent needs to allocate all its > resources when it starts. That's how it knows how much there are, > and incidentally, it doesn't do it blindly. The system designer > must know how much memory is available in the system and how much > is allocated to the kernel. > > The fact that you can give a fictitious value to malloc() is not > relevant. If you don't provide resources for malloc(), like > (ultimately) a swap file, then you can't assume that it can do > any design work for you. > > An embedded system is NOT an ordinary system that happens to > boot from flash. An embedded system requires intelligent design. We might be talking about slightly different systems. I agree that systems that are really embedded, in the classic meaning often with real time constraints, should be designed as you suggests. But there are a lot of other systems that almost actually are ordinary systems but with limited memory and often without demand paging. This could be a set top box, a video game console or a mobile phone that run dynamic applications. Actually this is not only about applications allocating an unknown amount of dynamic memory. A similar situation could also appear when we run an unknown number of applications at once, each allocating just a static amount of memory and then later starts to touching it. For those systems I think we need a method to dynamically decrease the working set of a process when memory is scarce, and not just accept that we "are screwed" and let the OOM killer solve the problem. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Out of memory management in embedded systems
On 9/28/07, linux-os (Dick Johnson) <[EMAIL PROTECTED]> wrote: > > On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > > > On 9/28/07, linux-os (Dick Johnson) <[EMAIL PROTECTED]> wrote: > >> > >> But an embedded system contains all the software that will > >> ever be executed on that system! If it is properly designed, > >> it can never run out of memory because everything it will > >> ever do is known at design time. > > > > Not if its input is not known beforehand. Take a browser in a mobile > > phone as an example, it does not know at design time how big the web > > pages are. On the other hand we want to use as much memory as > > possible, for cache etc., a method that involves the kernel would > > simplify this and avoids setting manual limits. > > > > Daniel > > > > Any networked appliance can (will) throw data away if there are > no resources available. > > The length of a web-page is not relevent, nor is the length > of any external data. Your example will buffer whatever it > can and not read anything more from the external source until > it has resources available unless it is broken. And how do you determine when no resources are availabe? We are using overcommit here so malloc() will always return non null. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Out of memory management in embedded systems
On 9/28/07, linux-os (Dick Johnson) <[EMAIL PROTECTED]> wrote: > > On Fri, 28 Sep 2007, [iso-8859-1] Daniel Spång wrote: > > > Applications with dynamic input and dynamic memory usage have some > > issues with the current overcommitting kernel. A high memory usage > > situation eventually results in that a process is killed by the OOM > > killer. This is especially evident in swapless embedded systems with > > limited memory and no swap available. > > > > Some kind of notification to the application that the available memory > > is scarce and let the application free up some memory (e.g., by > > flushing caches), could be used to improve the situation and avoid the > > OOM killer. I am currently not aware of any general solution to this > > problem, but I have found some approaches that might (or might not) > > work: > > > > o Turn off overcommit. Results in a waste of memory. > > > > o Nokia uses a lowmem security module to signal on predetermined > > thresholds. Currently available in the -omap tree. But this requires > > manual tuning of the thresholds. > > http://www.linuxjournal.com/article/8502 > > > > o Using madvise() with MADV_FREE to get the kernel to free mmaped > > memory, typically application caches, when the kernel needs the > > memory. > > > > o A OOM handler that the application registers with the kernel, and > > that the kernel executes before the OOM-killer steps in. > > > > Does it exist any other solutions to this problem? > > > > Daniel > > - > > But an embedded system contains all the software that will > ever be executed on that system! If it is properly designed, > it can never run out of memory because everything it will > ever do is known at design time. Not if its input is not known beforehand. Take a browser in a mobile phone as an example, it does not know at design time how big the web pages are. On the other hand we want to use as much memory as possible, for cache etc., a method that involves the kernel would simplify this and avoids setting manual limits. Daniel - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Out of memory management in embedded systems
Applications with dynamic input and dynamic memory usage have some issues with the current overcommitting kernel. A high memory usage situation eventually results in that a process is killed by the OOM killer. This is especially evident in swapless embedded systems with limited memory and no swap available. Some kind of notification to the application that the available memory is scarce and let the application free up some memory (e.g., by flushing caches), could be used to improve the situation and avoid the OOM killer. I am currently not aware of any general solution to this problem, but I have found some approaches that might (or might not) work: o Turn off overcommit. Results in a waste of memory. o Nokia uses a lowmem security module to signal on predetermined thresholds. Currently available in the -omap tree. But this requires manual tuning of the thresholds. http://www.linuxjournal.com/article/8502 o Using madvise() with MADV_FREE to get the kernel to free mmaped memory, typically application caches, when the kernel needs the memory. o A OOM handler that the application registers with the kernel, and that the kernel executes before the OOM-killer steps in. Does it exist any other solutions to this problem? Daniel - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
