What to do with InvalidMemoryOperationError
My executable throws as core.exception.InvalidMemoryOperationError@(0) when compiled with DMD git master. I get no stack trace in GDB. What to do?
Re: What to do with InvalidMemoryOperationError
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
My executable throws as
core.exception.InvalidMemoryOperationError@(0)
I've tracked it down to being caused by
foreach (line; File(path).byLine) {}
when path contains a very large text file (392 MB, 1658080 lines).
Do I have any alternatives to this that doesn't trigger exception?
Re: What to do with InvalidMemoryOperationError
On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
My executable throws as
core.exception.InvalidMemoryOperationError@(0)
I've tracked it down to being caused by
foreach (line; File(path).byLine) {}
when path contains a very large text file (392 MB, 1658080
lines).
Do I have any alternatives to this that doesn't trigger
exception?
I get the exact same behaviour when I try using MmFile instead:
import std.mmfile: MmFile;
auto mmf = new MmFile(path, MmFile.Mode.read, 0, null,
pageSize);
const data = cast(char[])mmf[];
foreach (line; data.splitter('\n') {}
Does byLine use splitter? If so, maybe splitter is the problem...
Re: What to do with InvalidMemoryOperationError
On 01/21/2015 04:10 AM, "Nordlöw" wrote:
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
My executable throws as
core.exception.InvalidMemoryOperationError@(0)
I've tracked it down to being caused by
foreach (line; File(path).byLine) {}
when path contains a very large text file (392 MB, 1658080 lines).
Do I have any alternatives to this that doesn't trigger exception?
Known bug with a pull request:
https://issues.dlang.org/show_bug.cgi?id=13856
Here is the duplicate of it, which I've opened recently:
https://issues.dlang.org/show_bug.cgi?id=14005
Ali
Re: What to do with InvalidMemoryOperationError
On Wednesday, 21 January 2015 at 14:50:06 UTC, Ali Çehreli wrote: Known bug with a pull request: https://issues.dlang.org/show_bug.cgi?id=13856 Here is the duplicate of it, which I've opened recently: https://issues.dlang.org/show_bug.cgi?id=14005 Ali How can this affect Mmfile plus splitter() aswell?
Re: What to do with InvalidMemoryOperationError
On Wednesday, 21 January 2015 at 15:34:35 UTC, Per Nordlöw wrote: On Wednesday, 21 January 2015 at 14:50:06 UTC, Ali Çehreli wrote: Known bug with a pull request: https://issues.dlang.org/show_bug.cgi?id=13856 Here is the duplicate of it, which I've opened recently: https://issues.dlang.org/show_bug.cgi?id=14005 Ali How can this affect Mmfile plus splitter() aswell? I think it can't. The root of 13856/14005 is in std.stdio.readln. Unless MmFile uses readln somehow, this looks like a different issue to me.
Re: What to do with InvalidMemoryOperationError
On Wednesday, 21 January 2015 at 13:07:11 UTC, Nordlöw wrote:
On Wednesday, 21 January 2015 at 12:10:20 UTC, Nordlöw wrote:
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote:
My executable throws as
core.exception.InvalidMemoryOperationError@(0)
I've tracked it down to being caused by
foreach (line; File(path).byLine) {}
when path contains a very large text file (392 MB, 1658080
lines).
Do I have any alternatives to this that doesn't trigger
exception?
I get the exact same behaviour when I try using MmFile instead:
import std.mmfile: MmFile;
auto mmf = new MmFile(path, MmFile.Mode.read, 0, null,
pageSize);
const data = cast(char[])mmf[];
foreach (line; data.splitter('\n') {}
Does byLine use splitter? If so, maybe splitter is the
problem...
I tested it with a generated large file, and the (rough) file
size and line count are apparently not enough to trigger the bug.
Could you share the text file? Maybe it can be compressed to a
manageable size? Or maybe you can reduce it to a manageable size?
A 'binary reduction' might work: Take the first half of the file,
and test with that. Error? Proceed with that half. No error? Try
the other half. When both halves produce no error, give up.
Or maybe dustmite can help here?
If the file contains sensitive information, and you cannot reduce
it to a reasonable size, you may be able to programmatically
replace classes of characters with one character. E.g. replace
all alphanumeric characters with 'x'. Be cautious of replacing
multibyte characters with single bytes. And newlines should
probably be left intact.
Re: What to do with InvalidMemoryOperationError
On Wednesday, 21 January 2015 at 20:50:30 UTC, anonymous wrote: Or maybe dustmite can help here? If the file contains sensitive information, and you cannot reduce it to a reasonable size, you may be able to programmatically replace classes of characters with one character. E.g. replace all alphanumeric characters with 'x'. Be cautious of replacing multibyte characters with single bytes. And newlines should probably be left intact. Breaking out that single code calling File.byLine on the same file didn't trigger the error, unfortunately... I guess dustmite is my only friend here. I'm guessing some other code prior to the call together with the call is causing the bug. I'll do some more testing and see if I can reduce my code...
Re: What to do with InvalidMemoryOperationError
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote: My executable throws as core.exception.InvalidMemoryOperationError@(0) when compiled with DMD git master. I get no stack trace in GDB. What to do? InvalidMemoryOperationError generally means that you are performing certain disallowed memory operations during a full garbage collection, such as allocating while the gc is running. This usually happens when you call a function that allocates in a destructor, which will trigger this error as the destructor is run by the gc. It appears that the gc issues mentioned above can also trigger it. If you're running off git head, maybe you can apply that PR 2794 and see if it helps. Otherwise, maybe you've hit some other gc issue somewhere.
Re: What to do with InvalidMemoryOperationError
On Friday, 23 January 2015 at 10:13:43 UTC, Joakim wrote: InvalidMemoryOperationError generally means that you are performing certain disallowed memory operations during a full garbage collection, such as allocating while the gc is running. If my app is single-threaded this cannot happen then. This usually happens when you call a function that allocates in a destructor, which will trigger this error as the destructor is run by the gc. It appears that the gc issues mentioned above can also trigger it. If you're running off git head, maybe you can apply that PR 2794 and see if it helps. Otherwise, maybe you've hit some other gc issue somewhere. So if GC.disable prevents the error from occurring I will have a clue, right?
Re: What to do with InvalidMemoryOperationError
On Friday, 23 January 2015 at 21:25:01 UTC, Nordlöw wrote: On Friday, 23 January 2015 at 10:13:43 UTC, Joakim wrote: InvalidMemoryOperationError generally means that you are performing certain disallowed memory operations during a full garbage collection, such as allocating while the gc is running. If my app is single-threaded this cannot happen then. Single-/multi-threaded doesn't make a difference. This happens with a single thread, too. The GC calls destructors. When a destructor allocates then, it allocates during a GC run.
Re: What to do with InvalidMemoryOperationError
On Wednesday, 21 January 2015 at 12:00:47 UTC, Nordlöw wrote: My executable throws as core.exception.InvalidMemoryOperationError@(0) when compiled with DMD git master. I get no stack trace in GDB. What to do? Hi, I created a wiki page which I hope will help you solve this problem: http://wiki.dlang.org/InvalidMemoryOperationError Hope this helps.
Re: What to do with InvalidMemoryOperationError
On Sunday, 25 January 2015 at 00:44:07 UTC, Vladimir Panteleev wrote: I created a wiki page which I hope will help you solve this problem: http://wiki.dlang.org/InvalidMemoryOperationError Hope this helps. Great! Thanks!
