On Tuesday, 26 July 2016 at 19:30:35 UTC, Charles Hixson wrote:
It looks as if the entire file is stored in memory, which is
not at all what I want, but I also can't really believe that's
what's going on.
It is just mapped to virtual memory without actually being loaded
into physical memory, so when you access the array it returns,
the kernel loads a page of the file into memory, but it doesn't
do that until it actually has to.
Think of it as being like this:
struct MagicFile {
ubyte[] opIndex(size_t idx) {
auto buffer = new ubyte[](some_block_length);
fseek(fp, idx, SEEK_SET);
fread(buffer.ptr, buffer.length, 1);
return buffer;
}
}
And something analogous for writing, but instead of being done
with overloaded operators in D, it is done with the MMU hardware
by the kernel (and the kernel also does smarter buffering than
this little example).
A part of the problem is that I don't want this to be a process
with an arbitrarily high memory use.
The kernel will automatically handle physical memory usage too,
similarly to a page file. If you haven't read a portion of the
file recently, it will discard that page, since it can always
read it again off disk if needed, but if you do have memory to
spare, it will keep the data in memory for faster access later.
So basically the operating system handles a lot of the details
which makes it efficient.
Growing a memory mapped file is a bit tricky though, you need to
unmap and remap. Since it is an OS concept, you can always look
for C or C++ examples too, like herE:
http://stackoverflow.com/questions/4460507/appending-to-a-memory-mapped-file/4461462#4461462
