Re: remap_file_pages() use
I might need a gentle nudge with a clue stick...
checking against latest git tree it looks as though most common
filesystem types do support remap_file_pages.
I just wrote a simple test case and it worked on my 3.13-based ubuntu
14.04 system on an ext4 filesystem.
thanks,
-Kenny
Here was my simple test case: (it doesn't have error handling, but the
case passed, and running under strace shows all system calls as
passing as well)
#include
#include
#include
#include
#include
// make a 16-page file, map page 17 over the first page, write to the
aliasing page, assert that it is seen on the first page
int main()
{
unlink("foo");
int fd = open("foo", O_CREAT | O_RDWR, 0755);
ftruncate(fd, 16*4096);
void* ptr = mmap(0, 17*4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
char* cptr = (char*) ptr;
// remap the last page over the first
remap_file_pages(cptr + 16*4096/*addr*/, 4096/*size*/,
0/*prot*/, 0/*pgoff*/, 0/*flags*/);
cptr[16*4096] = 'a';
return cptr[0] != 'a'; // if this aliases, this will be 'a'
}
On Tue, May 20, 2014 at 9:53 PM, Kenny Simpson
wrote:
> ouch... hope they don't try to run that code on anything newer then :(
> Will let them know.
>
> -Kenny
>
>
> On Mon, May 19, 2014 at 5:24 PM, Kirill A. Shutemov
> wrote:
>> On Mon, May 19, 2014 at 01:34:05PM -0400, Kenny Simpson wrote:
>>> For the other cases I had used the remapping to have more of a sliding
>>> window over a disk-backed file where I also was using aliasing to
>>> eliminate the corner cases of hitting the end of a window and needing
>>> to split records due to crossing boundaries, etc..
>>
>> Disk backed files are not supported by remap_file_pages() since 2007.
>> See commit 3ee6dafc677a.
>>
>> --
>> Kirill A. Shutemov
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: remap_file_pages() use
ouch... hope they don't try to run that code on anything newer then :( Will let them know. -Kenny On Mon, May 19, 2014 at 5:24 PM, Kirill A. Shutemov wrote: > On Mon, May 19, 2014 at 01:34:05PM -0400, Kenny Simpson wrote: >> For the other cases I had used the remapping to have more of a sliding >> window over a disk-backed file where I also was using aliasing to >> eliminate the corner cases of hitting the end of a window and needing >> to split records due to crossing boundaries, etc.. > > Disk backed files are not supported by remap_file_pages() since 2007. > See commit 3ee6dafc677a. > > -- > Kirill A. Shutemov -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: remap_file_pages() use
ouch... hope they don't try to run that code on anything newer then :( Will let them know. -Kenny On Mon, May 19, 2014 at 5:24 PM, Kirill A. Shutemov kir...@shutemov.name wrote: On Mon, May 19, 2014 at 01:34:05PM -0400, Kenny Simpson wrote: For the other cases I had used the remapping to have more of a sliding window over a disk-backed file where I also was using aliasing to eliminate the corner cases of hitting the end of a window and needing to split records due to crossing boundaries, etc.. Disk backed files are not supported by remap_file_pages() since 2007. See commit 3ee6dafc677a. -- Kirill A. Shutemov -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: remap_file_pages() use
I might need a gentle nudge with a clue stick...
checking against latest git tree it looks as though most common
filesystem types do support remap_file_pages.
I just wrote a simple test case and it worked on my 3.13-based ubuntu
14.04 system on an ext4 filesystem.
thanks,
-Kenny
Here was my simple test case: (it doesn't have error handling, but the
case passed, and running under strace shows all system calls as
passing as well)
#include unistd.h
#include fcntl.h
#include sys/mman.h
#include sys/types.h
#include sys/stat.h
// make a 16-page file, map page 17 over the first page, write to the
aliasing page, assert that it is seen on the first page
int main()
{
unlink(foo);
int fd = open(foo, O_CREAT | O_RDWR, 0755);
ftruncate(fd, 16*4096);
void* ptr = mmap(0, 17*4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
char* cptr = (char*) ptr;
// remap the last page over the first
remap_file_pages(cptr + 16*4096/*addr*/, 4096/*size*/,
0/*prot*/, 0/*pgoff*/, 0/*flags*/);
cptr[16*4096] = 'a';
return cptr[0] != 'a'; // if this aliases, this will be 'a'
}
On Tue, May 20, 2014 at 9:53 PM, Kenny Simpson
theonetrueke...@gmail.com wrote:
ouch... hope they don't try to run that code on anything newer then :(
Will let them know.
-Kenny
On Mon, May 19, 2014 at 5:24 PM, Kirill A. Shutemov
kir...@shutemov.name wrote:
On Mon, May 19, 2014 at 01:34:05PM -0400, Kenny Simpson wrote:
For the other cases I had used the remapping to have more of a sliding
window over a disk-backed file where I also was using aliasing to
eliminate the corner cases of hitting the end of a window and needing
to split records due to crossing boundaries, etc..
Disk backed files are not supported by remap_file_pages() since 2007.
See commit 3ee6dafc677a.
--
Kirill A. Shutemov
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: remap_file_pages() use
For the circular buffer case, yes I could make some temp file under /dev/shm, unlink it, mmap() it multiple times, etc... its just makes it a little more hairy. https://lkml.org/lkml/2011/3/19/94 For the other cases I had used the remapping to have more of a sliding window over a disk-backed file where I also was using aliasing to eliminate the corner cases of hitting the end of a window and needing to split records due to crossing boundaries, etc.. These were done for other projects in the past, but I am no longer on those projects (or with some of the companies), so I can't as easily double check the code or make changes. Being able to replace mappings in-place was logically simpler than doing mmap() over the old mappings. I was also under the impression that it should have less pressure on the mm_sem - I'm pretty sure I did profile it many years ago (>5), but many things have changed since then so maybe this is not a big deal any more. This use case was more performance-critical than the one-time setup of the anonymous circular buffer as it was more like a transaction log/journal. All that said, if remap_file_pages were to go away as of a mainline kernel this year, the projects I had worked on would probably not be impacted for many years and would probably have been retired/rewritten before they would ever see the change (unless some Linux vendor in N.C. backported its removal). Since valgrind doesn't support it, I tend to write fallback/debug variants anyway. -Kenny On Mon, May 19, 2014 at 10:35 AM, Kirill A. Shutemov wrote: > Michal Hocko wrote: >> [CCing Kirill and other people involved] >> >> On Sun 18-05-14 00:03:28, Kenny Simpson wrote: >> > I saw that remap_file_pages() was possibly going away to be replaced >> > by some emulation. I've used this call in several projects over the >> > years mostly as a way of mapping multiple virtual memory pages to >> > alias the same private or shared memory region (to do things like >> > circular buffers). mmap() >> > in the case of anonymous memory doesn't work as well since there is >> > not a file descriptor to reference. >> > >> > Would this sort of thing be supported in the emulation, or should I be >> > planning on reimplementing/rewriting some things? > > From functional POV, emulation *should* be identical to original > remap_file_pages(), but slower. It would be nice, if you test it early. > > It's not clear yet how long emulation will be there. > Is there a reason why you can't use fd from shared memory (shm_open() or > direct open() on /dev/shm/xxx)? > > -- > Kirill A. Shutemov -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: remap_file_pages() use
For the circular buffer case, yes I could make some temp file under /dev/shm, unlink it, mmap() it multiple times, etc... its just makes it a little more hairy. https://lkml.org/lkml/2011/3/19/94 For the other cases I had used the remapping to have more of a sliding window over a disk-backed file where I also was using aliasing to eliminate the corner cases of hitting the end of a window and needing to split records due to crossing boundaries, etc.. These were done for other projects in the past, but I am no longer on those projects (or with some of the companies), so I can't as easily double check the code or make changes. Being able to replace mappings in-place was logically simpler than doing mmap() over the old mappings. I was also under the impression that it should have less pressure on the mm_sem - I'm pretty sure I did profile it many years ago (5), but many things have changed since then so maybe this is not a big deal any more. This use case was more performance-critical than the one-time setup of the anonymous circular buffer as it was more like a transaction log/journal. All that said, if remap_file_pages were to go away as of a mainline kernel this year, the projects I had worked on would probably not be impacted for many years and would probably have been retired/rewritten before they would ever see the change (unless some Linux vendor in N.C. backported its removal). Since valgrind doesn't support it, I tend to write fallback/debug variants anyway. -Kenny On Mon, May 19, 2014 at 10:35 AM, Kirill A. Shutemov kirill.shute...@linux.intel.com wrote: Michal Hocko wrote: [CCing Kirill and other people involved] On Sun 18-05-14 00:03:28, Kenny Simpson wrote: I saw that remap_file_pages() was possibly going away to be replaced by some emulation. I've used this call in several projects over the years mostly as a way of mapping multiple virtual memory pages to alias the same private or shared memory region (to do things like circular buffers). mmap() in the case of anonymous memory doesn't work as well since there is not a file descriptor to reference. Would this sort of thing be supported in the emulation, or should I be planning on reimplementing/rewriting some things? From functional POV, emulation *should* be identical to original remap_file_pages(), but slower. It would be nice, if you test it early. It's not clear yet how long emulation will be there. Is there a reason why you can't use fd from shared memory (shm_open() or direct open() on /dev/shm/xxx)? -- Kirill A. Shutemov -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
remap_file_pages() use
I saw that remap_file_pages() was possibly going away to be replaced by some emulation. I've used this call in several projects over the years mostly as a way of mapping multiple virtual memory pages to alias the same private or shared memory region (to do things like circular buffers). mmap() in the case of anonymous memory doesn't work as well since there is not a file descriptor to reference. Would this sort of thing be supported in the emulation, or should I be planning on reimplementing/rewriting some things? thanks, -Kenny -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
remap_file_pages() use
I saw that remap_file_pages() was possibly going away to be replaced by some emulation. I've used this call in several projects over the years mostly as a way of mapping multiple virtual memory pages to alias the same private or shared memory region (to do things like circular buffers). mmap() in the case of anonymous memory doesn't work as well since there is not a file descriptor to reference. Would this sort of thing be supported in the emulation, or should I be planning on reimplementing/rewriting some things? thanks, -Kenny -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
