Re: remap_file_pages() use
Hi Dave, On 19 May 2014 19:50, Dave Hansen wrote: > We keep the current count as mm->map_count in the kernel, and the limit > is available because it's a sysctl. It wouldn't be hard to dump > mm->map_count out in a /proc file somewhere if it would be useful to > you. Would that work, or is there some other interface that would be > more convenient? We can keep an in-process estimate of this value anyway. The sysctl to get vm.max_map_count is all we need. That is, unless the limit is moved to be per-user instead of per-process, as I think was discussed here --- which would complicate further our job :-/ A bientôt, Armin. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: remap_file_pages() use
Hi Kirill, On 19 May 2014 17:53, Kirill A. Shutemov wrote: > Is it nessesary to remap in 4k chunks for you? > What about 64k chunks? Or something bigger? Good point. We remap chunks of 4k, which is not much, but is already much larger than the typical object size. Suppose we do such a remapping for a single object: then all other neighbouring objects that happen to live in the same page are also copied. Then, if some other thread modifies these other objects, we need extra copies to keep the objects in sync across all of their versions. That's the reason for keeping the size of remappings as small as possible. But we need to measure the actual impact. We can easily argue that if the process is using many GB of memory, then the risk of unrelated copies starts to decrease. It might be fine to increase the remapping unit in this case. If there is an official way to know in advance how many remappings our process is allowed to perform, then we could adapt as the size increases. Or maybe catching ENOMEM and doubling the remapping size (in some process-wide synchronization point). All in all, thanks for the note: it looks like there are solutions (even if less elegant than remap_file_pages from the user's perspective). A bientôt, Armin. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: remap_file_pages() use
Hi Kirill, On 19 May 2014 17:17, Kirill A. Shutemov wrote: > IIUC PyPy uses the syscall in some early prototype and looks like guys are > okay to rework it to mmap() if default sysctl_max_map_count will be high > enough. Yes, we can switch easily if needed. The syscall is not in any "production" version yet. Please note that "high enough" in this context means higher than 2**20. We need it high enough to handle regularly 10-20% of all the RAM used by each program. If I count correctly, at 20%, 2**20 fails above 20GB. In general I would suggest to use a default limit that depends on the amount of RAM (+swap) available. A bientôt, Armin. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 2/2] mm: replace remap_file_pages() syscall with emulation
Hi Sasha, On 12 May 2014 17:11, Sasha Levin wrote: > Since we can't find any actual users, The PyPy project doesn't count as an "actual user"? It's not just an idea in the air. It's beta code that is already released (and open source): http://morepypy.blogspot.ch/2014/04/stm-results-and-second-call-for.html The core library is available from there (see the test suite in c7/test/): https://bitbucket.org/pypy/stmgc I already reacted to the discussion here by making remap_file_pages() optional (#undef USE_REMAP_FILE_PAGES) but didn't measure the performance impact of this, if any (I expect it to be reasonable). Still, if you're looking for a real piece of code using remap_file_pages(), it's one. A bientôt, Armin. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCHv2 0/2] remap_file_pages() decommission
Hi Andi, On 12 May 2014 05:36, Andi Kleen wrote: >> Here is a note from the PyPy project (mentioned earlier in this >> thread, and at https://lwn.net/Articles/587923/ ). > > Your use is completely bogus. remap_file_pages() pins everything > and disables any swapping for the area. ? No. Trying this example: http://bpaste.net/show/fCUTnR9mDzJ2IEKrQLAR/ ...really allocates 4GB of RAM, and on a 4GB machine it causes some swapping. It seems to work fine. I'm not sure to understand you. I'm also not sure that a property as essential as "disables swapping" should be omitted from the man page; if so, that would be a real man page bug. A bientôt, Armin. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCHv2 0/2] remap_file_pages() decommission
Hi everybody, Here is a note from the PyPy project (mentioned earlier in this thread, and at https://lwn.net/Articles/587923/ ). Yes, we use remap_file_pages() heavily on the x86-64 architecture. However, the individual calls to remap_file_pages() are not performance-critical, so it is easy to switch to using multiple mmap()s. We need to perform more measurements to know exactly what the overhead would be, in terms notably of kernel memory. However, an issue with that approach is the upper bound on the number of VMAs. By default, it is not large enough. Right now, it is possible to remap say 10% of the individual pages from an anonymous mmap of multiple GBs in size; but doing so with individual calls to mmap hits this arbitrary limit. I have no particular weight to give for or against keeping remap_file_pages() in the kernel, but if it is removed or emulated, it would be a plus if the programs would run on a machine with the default configuration --- i.e. if you remove or emulate remap_file_pages(), please increase the default limit as well. A bientôt, Armin. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
