Re: Apparent problem with GC not collecting on Windows
On Friday, 30 November 2012 at 18:46:08 UTC, Dmitry Olshansky wrote: I'd just throw in that we have a (almost) precise GC that is used by at least one large project (the VisualD apparently). Though there were some problems with it. Anyway I'd expect to see it in upstream by 2.062 at least. It should help the cases like this tremendously. I expect it wouldn't help in this particular case because stack scanning is still conservative there (correct me if I'm wrong), and here we've got some stack value looking like a pointer to the array.
Re: Apparent problem with GC not collecting on Windows
12/1/2012 2:53 PM, thedeemon пишет: On Friday, 30 November 2012 at 18:46:08 UTC, Dmitry Olshansky wrote: I'd just throw in that we have a (almost) precise GC that is used by at least one large project (the VisualD apparently). Though there were some problems with it. Anyway I'd expect to see it in upstream by 2.062 at least. It should help the cases like this tremendously. I expect it wouldn't help in this particular case because stack scanning is still conservative there (correct me if I'm wrong), and here we've got some stack value looking like a pointer to the array. It should help because: 1) Each allocation happens in a new stack frame thus the pointer is overwritten each time. 2) Precise heap scanning helps this case because it greatly reduces the amount of false pointers (note it's not the stack variables that hold these allocations in place) It doesn't guarantee that it will collect all of these chunks but number of false pointers that could point to them is far lower and the chance of being collected is that much higher. The stack is typically small, unlike the heap. -- Dmitry Olshansky
Re: Apparent problem with GC not collecting on Windows
On Saturday, 1 December 2012 at 12:55:19 UTC, Dmitry Olshansky wrote: It should help because: 1) Each allocation happens in a new stack frame thus the pointer is overwritten each time. To probably the same stinky value (a false pointer). 2) Precise heap scanning helps this case because it greatly reduces the amount of false pointers (note it's not the stack variables that hold these allocations in place) I'm pretty sure in this case the false pointers are indeed in stack or data segment, not in the heap. Data arrays must be allocated with NO_SCAN flag so they're not scanned for pointers.
Re: Apparent problem with GC not collecting on Windows
12/1/2012 10:46 PM, thedeemon пишет: On Saturday, 1 December 2012 at 12:55:19 UTC, Dmitry Olshansky wrote: It should help because: 1) Each allocation happens in a new stack frame thus the pointer is overwritten each time. To probably the same stinky value (a false pointer). Pointer is different every time. So the initial pointer itself is overwritten. Everything else depends on luck. Some other values up the stack might still point to the block. 2) Precise heap scanning helps this case because it greatly reduces the amount of false pointers (note it's not the stack variables that hold these allocations in place) I'm pretty sure in this case the false pointers are indeed in stack or data segment, not in the heap. Data arrays must be allocated with NO_SCAN flag so they're not scanned for pointers. I've meant the actual array variable. Basically it's some other data that looks like a pointer to it elsewhere. Depending on how much runtime does dynamically allocate vs statically it may be the case that it is not in heap but in data segment. Or the used length of stack that should be quite small here. Bottom line is that the bigger array you have the greater chance it won't be collected (because it's a nice target for random values in stack/data segment/heap). The more precise the scanner the better are chances of it being collected. -- Dmitry Olshansky
Re: Apparent problem with GC not collecting on Windows
11/30/2012 12:36 AM, Ali Çehreli пишет: On 11/29/2012 12:06 PM, Michael wrote: Because you used uint instead of ubyte, array is bigger, memory exhausts faster. Oh, I see. 3. Why it helps? GC.free(data.ptr); Initial leak happened because for some reason array allocated in previous iteration was not collected by GC when allocating new one, so the new one was allocated in another space growing the heap. If you place GC.free the array gets removed from heap on each iteration and each new allocation reuses the same memory, heap doesn't grow. If we do this manually it's works, but automatically is broken? Nothing is broken. GC.free would have been applied by the GC only if there have been no more references to the allocated block of memory. The fact is, dmd uses a conservative GC. The issue is due to the combination of conservative GC, 32-bit address space, and a large chunk of memory. When that happens, it is very likely that any other value in the system, including an innocent int, has the risk of looking like a reference into that memory. For that reason the GC keeps that memory block allocated. I'd just throw in that we have a (almost) precise GC that is used by at least one large project (the VisualD apparently). Though there were some problems with it. Anyway I'd expect to see it in upstream by 2.062 at least. It should help the cases like this tremendously. P.S. Either way the manual memory management (and reuse) is the way to go with big allocations. -- Dmitry Olshansky
Re: Apparent problem with GC not collecting on Windows
Because you used uint instead of ubyte, array is bigger, memory exhausts faster. Oh, I see. 3. Why it helps? GC.free(data.ptr); Initial leak happened because for some reason array allocated in previous iteration was not collected by GC when allocating new one, so the new one was allocated in another space growing the heap. If you place GC.free the array gets removed from heap on each iteration and each new allocation reuses the same memory, heap doesn't grow. If we do this manually it's works, but automatically is broken?
Re: Apparent problem with GC not collecting on Windows
On 11/29/2012 12:06 PM, Michael wrote: Because you used uint instead of ubyte, array is bigger, memory exhausts faster. Oh, I see. 3. Why it helps? GC.free(data.ptr); Initial leak happened because for some reason array allocated in previous iteration was not collected by GC when allocating new one, so the new one was allocated in another space growing the heap. If you place GC.free the array gets removed from heap on each iteration and each new allocation reuses the same memory, heap doesn't grow. If we do this manually it's works, but automatically is broken? Nothing is broken. GC.free would have been applied by the GC only if there have been no more references to the allocated block of memory. The fact is, dmd uses a conservative GC. The issue is due to the combination of conservative GC, 32-bit address space, and a large chunk of memory. When that happens, it is very likely that any other value in the system, including an innocent int, has the risk of looking like a reference into that memory. For that reason the GC keeps that memory block allocated. The risk of that happening is grossly reduced in a 64-bit address space. Similarly, allocating a much smaller buffer helps as well. Alternatively, we can use a better GC or a runtime that has a better GC. Ali
Apparent problem with GC not collecting on Windows
A friend of mine reports that the memory usage of the following program
grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1
home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
If you are on Windows, could you please say whether you have the same
problem. (I don't have Windows.) The following is how we reproduce:
Generate a file of about 1G by running the following program:
import std.stdio;
void foo()
{
auto file = File(one_gigabyte_file, w);
auto data = new ubyte[](100 * 1024 * 1024);
foreach (i; 0 .. 10) {
file.rawWrite(data);
}
}
void main()
{
foo();
}
That part is not the problem. The problem is when reading such a large
file by chunks. Could you please run the following program and report
whether the memory consumption of it is increasing continuously:
import std.stdio;
void foo()
{
auto file = File(one_gigabyte_file, r);
auto data = new ubyte[](100 * 1024 * 1024);
foreach (i; 0 .. 10) {
file.rawRead(data);
}
}
void main()
{
for (size_t i; true; ++i) {
writeln(i);
foo();
}
}
The program gets terminated by a core.exception.OutOfMemoryError. (Same
problem with std.stream.BufferedFile.)
I don't see any problem under Linux.
Thank you,
Ali
Re: Apparent problem with GC not collecting on Windows
11/28/2012 10:51 PM, Ali Çehreli пишет: A friend of mine reports that the memory usage of the following program grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]). If you are on Windows, could you please say whether you have the same problem. (I don't have Windows.) The following is how we reproduce: [snip] The program gets terminated by a core.exception.OutOfMemoryError. (Same problem with std.stream.BufferedFile.) Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 running 32-bit app. -- Dmitry Olshansky
Re: Apparent problem with GC not collecting on Windows
On 11/28/2012 12:57 PM, Dmitry Olshansky wrote: 11/28/2012 10:51 PM, Ali Çehreli пишет: A friend of mine reports that the memory usage of the following program grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]). If you are on Windows, could you please say whether you have the same problem. (I don't have Windows.) The following is how we reproduce: [snip] The program gets terminated by a core.exception.OutOfMemoryError. (Same problem with std.stream.BufferedFile.) Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 running 32-bit app. Isn't this a known problem with a conservative GC on 32bit? The GC sees something that *Could* be a reference, and refuses to collect that data.
Re: Apparent problem with GC not collecting on Windows
On 11/28/2012 11:11 AM, 1100110 wrote: On 11/28/2012 12:57 PM, Dmitry Olshansky wrote: 11/28/2012 10:51 PM, Ali Çehreli пишет: A friend of mine reports that the memory usage of the following program grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]). If you are on Windows, could you please say whether you have the same problem. (I don't have Windows.) The following is how we reproduce: [snip] The program gets terminated by a core.exception.OutOfMemoryError. (Same problem with std.stream.BufferedFile.) Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 running 32-bit app. Isn't this a known problem with a conservative GC on 32bit? The GC sees something that *Could* be a reference, and refuses to collect that data. I think it is more like an integer value looking like a pointer to something. I don't see what the GC may be confused with in this case. I've still created a bug: http://d.puremagic.com/issues/show_bug.cgi?id=9094 Ali
Re: Apparent problem with GC not collecting on Windows
On 11/28/2012 02:53 PM, Ali Çehreli wrote: On 11/28/2012 11:11 AM, 1100110 wrote: On 11/28/2012 12:57 PM, Dmitry Olshansky wrote: 11/28/2012 10:51 PM, Ali Çehreli пишет: A friend of mine reports that the memory usage of the following program grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]). If you are on Windows, could you please say whether you have the same problem. (I don't have Windows.) The following is how we reproduce: [snip] The program gets terminated by a core.exception.OutOfMemoryError. (Same problem with std.stream.BufferedFile.) Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 running 32-bit app. Isn't this a known problem with a conservative GC on 32bit? The GC sees something that *Could* be a reference, and refuses to collect that data. I think it is more like an integer value looking like a pointer to something. I don't see what the GC may be confused with in this case. I've still created a bug: http://d.puremagic.com/issues/show_bug.cgi?id=9094 Ali That's what I meant. =P
Re: Apparent problem with GC not collecting on Windows
On Wednesday, 28 November 2012 at 22:02:35 UTC, Michael wrote: Only 3 iterations before out of memory exception? Because you used uint instead of ubyte, array is bigger, memory exhausts faster. 3. Why it helps? GC.free(data.ptr); Initial leak happened because for some reason array allocated in previous iteration was not collected by GC when allocating new one, so the new one was allocated in another space growing the heap. If you place GC.free the array gets removed from heap on each iteration and each new allocation reuses the same memory, heap doesn't grow.
