Re: remap_file_pages() use

2014-05-26 Thread Kirill A. Shutemov
Jeff Smith wrote: > I've got no real issues at this point, but could you perhaps elaborate > a bit on the rough order of magnitude of "long" time We have never-break-ABI policy in kernel. In practice it means we don't remove an interface if somebody could notice that it disappears. Most likely it

Re: remap_file_pages() use

2014-05-26 Thread Jeff Smith
I've got no real issues at this point, but could you perhaps elaborate a bit on the rough order of magnitude of "long" time and what cases would be slower? I'm pretty sure that some places I've worked that still have some remap_file_pages() logic in place aren't too religious about checking dmesg.

Re: remap_file_pages() use

2014-05-26 Thread Kirill A. Shutemov
Jeff Smith wrote: > OK, I misinterpreted "the overlapped part of the mapping(s) will be > discarded" as discarding the -new- mappings. My objections about > needing a replacement for remap_file_pages() are gone, but my concerns > about existing code still remain. As I said, emulation will be there

Re: remap_file_pages() use

2014-05-26 Thread Jeff Smith
OK, I misinterpreted "the overlapped part of the mapping(s) will be discarded" as discarding the -new- mappings. My objections about needing a replacement for remap_file_pages() are gone, but my concerns about existing code still remain. --Jeff On Mon, May 26, 2014 at 8:35 AM, Paolo Bonzini wrote

Re: remap_file_pages() use

2014-05-26 Thread Kirill A. Shutemov
Jeff Smith wrote: > >> Mirrored mapping is absolutely required by several > >> independent proprietary platforms I'm aware of, and remap_file_pages() > >> has historically been the only sane way to accomplish this. (i.e., > >> shm_open(), mmap(NULL, 2^(n+1) pages), remap_file_pages() on 2nd > >> ha

Re: remap_file_pages() use

2014-05-26 Thread Paolo Bonzini
Il 26/05/2014 15:24, Jeff Smith ha scritto: Your addr2 mmap() call is a bit incorrect semantically and syntactically (you skipped the length arg). The addr2 request will fail because mmap() does not implicitly munmap() occupied virtual address space. With MAP_FIXED it does. It is in the man pa

Re: remap_file_pages() use

2014-05-26 Thread Jeff Smith
>> Mirrored mapping is absolutely required by several >> independent proprietary platforms I'm aware of, and remap_file_pages() >> has historically been the only sane way to accomplish this. (i.e., >> shm_open(), mmap(NULL, 2^(n+1) pages), remap_file_pages() on 2nd >> half). > > Em.. What's wrong w

Re: remap_file_pages() use

2014-05-26 Thread Kirill A. Shutemov
Jeff Smith wrote: > On Mon, May 19, 2014 at 9:38 AM, Christoph Hellwig wrote: > > On Mon, May 19, 2014 at 05:35:40PM +0300, Kirill A. Shutemov wrote: > >> >From functional POV, emulation *should* be identical to original > >> remap_file_pages(), but slower. It would be nice, if you test it early.

Re: remap_file_pages() use

2014-05-25 Thread Jeff Smith
On Mon, May 19, 2014 at 9:38 AM, Christoph Hellwig wrote: > On Mon, May 19, 2014 at 05:35:40PM +0300, Kirill A. Shutemov wrote: >> >From functional POV, emulation *should* be identical to original >> remap_file_pages(), but slower. It would be nice, if you test it early. >> >> It's not clear yet h

Re: remap_file_pages() use

2014-05-20 Thread Hugh Dickins
On Tue, 20 May 2014, Kenny Simpson wrote: > I might need a gentle nudge with a clue stick... > checking against latest git tree it looks as though most common > filesystem types do support remap_file_pages. > > I just wrote a simple test case and it worked on my 3.13-based ubuntu > 14.04 system o

Re: remap_file_pages() use

2014-05-20 Thread Kenny Simpson
I might need a gentle nudge with a clue stick... checking against latest git tree it looks as though most common filesystem types do support remap_file_pages. I just wrote a simple test case and it worked on my 3.13-based ubuntu 14.04 system on an ext4 filesystem. thanks, -Kenny Here was my simp

Re: remap_file_pages() use

2014-05-20 Thread Kenny Simpson
ouch... hope they don't try to run that code on anything newer then :( Will let them know. -Kenny On Mon, May 19, 2014 at 5:24 PM, Kirill A. Shutemov wrote: > On Mon, May 19, 2014 at 01:34:05PM -0400, Kenny Simpson wrote: >> For the other cases I had used the remapping to have more of a slidin

Re: remap_file_pages() use

2014-05-19 Thread Armin Rigo
Hi Dave, On 19 May 2014 19:50, Dave Hansen wrote: > We keep the current count as mm->map_count in the kernel, and the limit > is available because it's a sysctl. It wouldn't be hard to dump > mm->map_count out in a /proc file somewhere if it would be useful to > you. Would that work, or is ther

Re: remap_file_pages() use

2014-05-19 Thread Kirill A. Shutemov
On Mon, May 19, 2014 at 01:34:05PM -0400, Kenny Simpson wrote: > For the other cases I had used the remapping to have more of a sliding > window over a disk-backed file where I also was using aliasing to > eliminate the corner cases of hitting the end of a window and needing > to split records due

Re: remap_file_pages() use

2014-05-19 Thread Richard Weinberger
On Mon, May 19, 2014 at 5:17 PM, Kirill A. Shutemov wrote: > Christoph Hellwig wrote: >> On Mon, May 19, 2014 at 06:02:38PM +0300, Kirill A. Shutemov wrote: >> > > Stop right there. We found out about two real life users of >> > > remap_file_pages() already, without even committing the patches to

Re: remap_file_pages() use

2014-05-19 Thread Dave Hansen
On 05/19/2014 09:42 AM, Armin Rigo wrote: > If there is an official way to know in advance how many remappings our > process is allowed to perform, then we could adapt as the size > increases. Or maybe catching ENOMEM and doubling the remapping size > (in some process-wide synchronization point).

Re: remap_file_pages() use

2014-05-19 Thread Kenny Simpson
For the circular buffer case, yes I could make some temp file under /dev/shm, unlink it, mmap() it multiple times, etc... its just makes it a little more hairy. https://lkml.org/lkml/2011/3/19/94 For the other cases I had used the remapping to have more of a sliding window over a disk-backed file

Re: remap_file_pages() use

2014-05-19 Thread Christoph Hellwig
On Mon, May 19, 2014 at 06:17:58PM +0300, Kirill A. Shutemov wrote: > IIUC PyPy uses the syscall in some early prototype and looks like guys are > okay to rework it to mmap() if default sysctl_max_map_count will be high > enough. My point is that we already found a few users just by discussing the

Re: remap_file_pages() use

2014-05-19 Thread Armin Rigo
Hi Kirill, On 19 May 2014 17:53, Kirill A. Shutemov wrote: > Is it nessesary to remap in 4k chunks for you? > What about 64k chunks? Or something bigger? Good point. We remap chunks of 4k, which is not much, but is already much larger than the typical object size. Suppose we do such a remappin

Re: remap_file_pages() use

2014-05-19 Thread Kirill A. Shutemov
Armin Rigo wrote: > Hi Kirill, > > On 19 May 2014 17:17, Kirill A. Shutemov > wrote: > > IIUC PyPy uses the syscall in some early prototype and looks like guys are > > okay to rework it to mmap() if default sysctl_max_map_count will be high > > enough. > > Yes, we can switch easily if needed. T

Re: remap_file_pages() use

2014-05-19 Thread Armin Rigo
Hi Kirill, On 19 May 2014 17:17, Kirill A. Shutemov wrote: > IIUC PyPy uses the syscall in some early prototype and looks like guys are > okay to rework it to mmap() if default sysctl_max_map_count will be high > enough. Yes, we can switch easily if needed. The syscall is not in any "production

Re: remap_file_pages() use

2014-05-19 Thread Kirill A. Shutemov
Christoph Hellwig wrote: > On Mon, May 19, 2014 at 06:02:38PM +0300, Kirill A. Shutemov wrote: > > > Stop right there. We found out about two real life users of > > > remap_file_pages() already, without even committing the patches to warn > > > about using it to any tree. > > > > Who is the secon

Re: remap_file_pages() use

2014-05-19 Thread Christoph Hellwig
On Mon, May 19, 2014 at 06:02:38PM +0300, Kirill A. Shutemov wrote: > > Stop right there. We found out about two real life users of > > remap_file_pages() already, without even committing the patches to warn > > about using it to any tree. > > Who is the second here? Oracle? PyPy. Oracle

Re: remap_file_pages() use

2014-05-19 Thread Kirill A. Shutemov
Christoph Hellwig wrote: > On Mon, May 19, 2014 at 05:35:40PM +0300, Kirill A. Shutemov wrote: > > >From functional POV, emulation *should* be identical to original > > remap_file_pages(), but slower. It would be nice, if you test it early. > > > > It's not clear yet how long emulation will be the

Re: remap_file_pages() use

2014-05-19 Thread Christoph Hellwig
On Mon, May 19, 2014 at 05:35:40PM +0300, Kirill A. Shutemov wrote: > >From functional POV, emulation *should* be identical to original > remap_file_pages(), but slower. It would be nice, if you test it early. > > It's not clear yet how long emulation will be there. Stop right there. We found ou

Re: remap_file_pages() use

2014-05-19 Thread Kirill A. Shutemov
Michal Hocko wrote: > [CCing Kirill and other people involved] > > On Sun 18-05-14 00:03:28, Kenny Simpson wrote: > > I saw that remap_file_pages() was possibly going away to be replaced > > by some emulation. I've used this call in several projects over the > > years mostly as a way of mapping mu

Re: remap_file_pages() use

2014-05-19 Thread Michal Hocko
[CCing Kirill and other people involved] On Sun 18-05-14 00:03:28, Kenny Simpson wrote: > I saw that remap_file_pages() was possibly going away to be replaced > by some emulation. I've used this call in several projects over the > years mostly as a way of mapping multiple virtual memory pages to >