Re: [Sugar-devel] poor man's mmap "sliding window" on Python 2.5.x
On Thu, Jul 9, 2009 at 1:45 PM, John Gilmore wrote: >> The process is doing a linear read through the file, and is slow >> enough that it appears only to grow. But if I run another process that >> allocates a lot of memory, then the kernel does discard pages pages. > > So are you saying that two identical processes that mmap large amounts > of memory, (much larger than the DRAM and with no swap space), then > walk sequentially through the memory, will split the memory evenly > among them, and will complete in a reasonable amount of time. But one > process will get very slow and never complete? No, something much simpler. No bug in the mmap implementation but in my testing. I was testing with 1 process walking the file slowly & linearly. Watching `top` it _looked_ like it was exhausting memory and getting slower. The kernel did seem to discard other process' clean pages, because all other processes were idle. However, if instead of just watching top I start other processes, (a Browse.xo instance), the kernel reacts quickly, discards old pages from the test script, and Browse runs nicely. So the problem was in my procedure. mmap works as it should. cheers, m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Sugar-devel] poor man's mmap "sliding window" on Python 2.5.x
> The process is doing a linear read through the file, and is slow > enough that it appears only to grow. But if I run another process that > allocates a lot of memory, then the kernel does discard pages pages. So are you saying that two identical processes that mmap large amounts of memory, (much larger than the DRAM and with no swap space), then walk sequentially through the memory, will split the memory evenly among them, and will complete in a reasonable amount of time. But one process will get very slow and never complete? Sounds like a reproducible test case like that would be worth reporting to the linux-kernel mailing list, to see if anybody recognizes the bug, or wants to go hunting it. John ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Sugar-devel] poor man's mmap "sliding window" on Python 2.5.x
On Wed, 2009-07-08 at 15:23 +1200, Martin Langhoff wrote: > Had some time to retest this on the plane, and I think it was > mis-diagnosis. The original code I was testing is lost. In re-testing > this I find that the problem is more nuanced, and I may have been > wrong: looking at 'top', the kernel does not appear very eager to > discard old mapped pages. You can probably influence this by marking the ranges that you're done with with madvise(). Daniel ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Sugar-devel] poor man's mmap "sliding window" on Python 2.5.x
On Tue, Jul 7, 2009 at 9:23 AM, Martin Langhoff wrote: > Thanks for looking into this. I'll post the (trivial, really) repro > code I have later (I'm on a gruelling >35hr trip at the moment). Had some time to retest this on the plane, and I think it was mis-diagnosis. The original code I was testing is lost. In re-testing this I find that the problem is more nuanced, and I may have been wrong: looking at 'top', the kernel does not appear very eager to discard old mapped pages. The process is doing a linear read through the file, and is slow enough that it appears only to grow. But if I run another process that allocates a lot of memory, then the kernel does discard pages pages. A good way of monitoring this seems to be: watch --differences grep -A8 /proc//smaps So the mmap does the right thing. ACCESS_READ doesn't seem to make any difference. #!/usr/bin/python import mmap import sys def mmap_to_death(fpath): fh = open(fpath, 'r+') mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ) l = len(mm) c = 0 buf = '' while c < l: buf = mm[c] c = c+1 mm.close() fh.close() mmap_to_death(sys.argv[1]) m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Sugar-devel] poor man's mmap "sliding window" on Python 2.5.x
On Tue, Jul 7, 2009 at 7:16 AM, Benjamin M. Schwartz wrote: > huh. I looked through python's mmap implementation [1] and there doesn't > seem to be any caching or funny business going on. > > I wonder if it could be over-aggressive caching somewhere in jffs2, in an > attempt to avoid repeatedly decompressing the same block. Thanks for looking into this. I'll post the (trivial, really) repro code I have later (I'm on a gruelling >35hr trip at the moment). So far, tested on 8.2.x _on vfat partitions mounted from a USB device_ and on the local disk (ext3) on Ubuntu Hardy. Have not tested it (yet) with data file on jffs2 as that's not the use case I'm gunning for. m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Sugar-devel] poor man's mmap "sliding window" on Python 2.5.x
Martin Langhoff wrote: > On Tue, Jul 7, 2009 at 2:06 AM, Benjamin M. > Schwartz wrote: >> Is this (a) a kernel bug, (b) Python layering extra caching over mmap, or >> (c) a misunderstanding of mmap on my part? > > money is b huh. I looked through python's mmap implementation [1] and there doesn't seem to be any caching or funny business going on. I wonder if it could be over-aggressive caching somewhere in jffs2, in an attempt to avoid repeatedly decompressing the same block. --Ben [1] http://svn.python.org/projects/python/trunk/Modules/mmapmodule.c P.S. JFFS2 appears to support read-only mmap, which I presume is what you're using. signature.asc Description: OpenPGP digital signature ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Sugar-devel] poor man's mmap "sliding window" on Python 2.5.x
On Tue, Jul 7, 2009 at 2:06 AM, Benjamin M. Schwartz wrote: > Is this (a) a kernel bug, (b) Python layering extra caching over mmap, or > (c) a misunderstanding of mmap on my part? money is b m -- martin.langh...@gmail.com mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
Re: [Sugar-devel] poor man's mmap "sliding window" on Python 2.5.x
Martin Langhoff wrote: > Along the way, found that Python 2.5.x doesn't support an offset to > mmap(), which at first blush makes re-mapping with a sliding window > problematic. Why is an explicit sliding window necessary? Isn't the point of mmap that you can access as you like, and the kernel will clear old caches if there's memory pressure? > On the XO-1, it's the difference of "churning through it" and slowing > the whole OS to a crawl, and then inching towards a big OOM zap. Is this (a) a kernel bug, (b) Python layering extra caching over mmap, or (c) a misunderstanding of mmap on my part? --Ben signature.asc Description: OpenPGP digital signature ___ Devel mailing list Devel@lists.laptop.org http://lists.laptop.org/listinfo/devel
