On Wednesday, 21 January 2015 at 13:07:11 UTC, Nordlöw wrote:
On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
My executable throws as
core.exception.InvalidMemoryOperationError@(0)
I've tracked it down to being caused by
foreach (line; File(path).byLine) {}
when path contains a very large text file (392 MB, 1658080
lines).
Do I have any alternatives to this that doesn't trigger
exception?
I get the exact same behaviour when I try using MmFile instead:
import std.mmfile: MmFile;
auto mmf = new MmFile(path, MmFile.Mode.read, 0, null,
pageSize);
const data = cast(char[])mmf[];
foreach (line; data.splitter('\n') {}
Does byLine use splitter? If so, maybe splitter is the
problem...
I tested it with a generated large file, and the (rough) file
size and line count are apparently not enough to trigger the bug.
Could you share the text file? Maybe it can be compressed to a
manageable size? Or maybe you can reduce it to a manageable size?
A 'binary reduction' might work: Take the first half of the file,
and test with that. Error? Proceed with that half. No error? Try
the other half. When both halves produce no error, give up.
Or maybe dustmite can help here?
If the file contains sensitive information, and you cannot reduce
it to a reasonable size, you may be able to programmatically
replace classes of characters with one character. E.g. replace
all alphanumeric characters with 'x'. Be cautious of replacing
multibyte characters with single bytes. And newlines should
probably be left intact.
