On 07/26/2016 12:53 PM, Adam D. Ruppe via Digitalmars-d-learn wrote:
On Tuesday, 26 July 2016 at 19:30:35 UTC, Charles Hixson wrote:
It looks as if the entire file is stored in memory, which is not at
all what I want, but I also can't really believe that's what's going on.
It is just mapped to virtual memory without actually being loaded into
physical memory, so when you access the array it returns, the kernel
loads a page of the file into memory, but it doesn't do that until it
actually has to.
Think of it as being like this:
struct MagicFile {
ubyte[] opIndex(size_t idx) {
auto buffer = new ubyte[](some_block_length);
fseek(fp, idx, SEEK_SET);
fread(buffer.ptr, buffer.length, 1);
return buffer;
}
}
And something analogous for writing, but instead of being done with
overloaded operators in D, it is done with the MMU hardware by the
kernel (and the kernel also does smarter buffering than this little
example).
A part of the problem is that I don't want this to be a process with
an arbitrarily high memory use.
The kernel will automatically handle physical memory usage too,
similarly to a page file. If you haven't read a portion of the file
recently, it will discard that page, since it can always read it again
off disk if needed, but if you do have memory to spare, it will keep
the data in memory for faster access later.
So basically the operating system handles a lot of the details which
makes it efficient.
Growing a memory mapped file is a bit tricky though, you need to unmap
and remap. Since it is an OS concept, you can always look for C or C++
examples too, like herE:
http://stackoverflow.com/questions/4460507/appending-to-a-memory-mapped-file/4461462#4461462
O, dear. It was sounding like such an excellent approach until this
last paragraph, but growing the file is going to be one of the common
operations. (Certainly at first.) It sounds as if that means the file
needs to be closed and re-opened for extensions. And I quote from
https://www.gnu.org/software/libc/manual/html_node/Memory_002dmapped-I_002fO.html:
<END
Function: /void */ *mremap* /(void *address, size_t length, size_t
new_length, int flag)/
Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety
Concepts
<https://www.gnu.org/software/libc/manual/html_node/POSIX-Safety-Concepts.html#POSIX-Safety-Concepts>.
This function can be used to change the size of an existing memory
area. address and length must cover a region entirely mapped in the
same |mmap| statement. A new mapping with the same characteristics
will be returned with the length new_length.
...
This function is only available on a few systems. Except for performing
optional optimizations one should not rely on this function.
END
So I'm probably better off sticking to using a seek based i/o system.
