Re: Who wants to have some fun memory debugging?
Maybe it's time to put together an instrumented GC... Wishlist: - some way to know _when_ a collection happened (or how often) - logging of allocations (module/linenumber of allocator, size of allocation, type of allocation) - per TypeInfo allocation statistics - per module allocation statistics
Re: Who wants to have some fun memory debugging?
Sean Kelly wrote: > ... > > At this point it's quite possible that you still have a reference to > newData in a register. If you want to be sure the collect call below > works as intended for this test, try adding: > > newData = new Data; > > here. Debugging it with ddbg under Win XP, I found that at "data = null", there is what appears to be a pointer to the newest allocation in EDX, and a pointer to the previous one in ECX. >> i = 0; After this line, the ECX reference hasn't been touched. >> GC.collect(); Disassembling the code, there's nothing that touches ECX before the call to GC.collect, so it's quite probable that this is why the memory isn't being freed in a given iteration, but not necessarily why it isn't being collected at all. I modified the code to explicitly zero-out ECX just before the call to GC.collect, and it didn't appear to help. I then instrumented the code with calls to gc_stats. Here is the run of it reporting the stats AFTER each forced collect (output via printf): GCStats: poolsize, usedsize, freeblocks, freelistsize, pageblocks GCStats: 81985536, 640, 16, 7552, GCStats: 16384, 640, 3200, 7552, 18399 GCStats: 232652800, 640, 2030, 7552, 27384 GCStats: 306315264, 640, 5948, 7552, 34417 GCStats: 360251392, 640, 6954, 7552, 40498 GCStats: 410517504, 640, 5502, 7552, 47360 GCStats: 469958656, 640, 8498, 7552, 53118 The program also got progressively slower as the size of the heap increased. Odd thing is that it would pause at a certain heap size for many seconds at a time, increase in size, pause, increase, pause... it was like the allocations were getting slower. What's also weird is that even though the heap usage should be the same each iteration, it's clearly creating more and more blocks. It's like the GC is freeing those blocks, but the allocator isn't using them. Then I ran it so that it would compute the difference of gc_stats before and after the collect. This one is somewhat depressing: GCStats: poolsize, usedsize, freeblocks, freelistsize, pageblocks GCStats: 0, 0, 2, 0, -1 GCStats: 0, 0, 2, 0, -1 GCStats: 0, 0, 2, 0, -1 GCStats: 0, 0, 2, 0, -1 GCStats: 0, 0, 2, 0, -1 GCStats: 0, 0, 2, 0, -1 It appears to free *two* blocks and consume an extra page during a collection, but doesn't affect the pool or used size at all. Maybe it's time to put together an instrumented GC... -- Daniel
Re: Who wants to have some fun memory debugging?
Wild guess: there's a false pointer, that keeps one element in the list from being collected, and because the list-prev pointers are still there, all following elements won't be collected either in consequence. If I had time, I'd try two experiments: 1. before freeing everything, reset the prev field of each element; if this makes the leak go away, my guess would probably be right 2. use the destructor (Data had to be a class) to keep track of how many elements are actually freed by the GC (just a thread safe counter, that's incremented in the ctor and decremented in the dtor); just to find out if this is an internal GC problem, or if you have too many live garbage
Re: Who wants to have some fun memory debugging?
Sean Kelly wrote: At this point it's quite possible that you still have a reference to newData in a register. That argument works for one iteration, but this fool just keeps on growing. Also, this is a pared down test from a much larger program, so this is unlikely. If you want to be sure the collect call below works as intended for this test, try adding: newData = new Data; here. i = 0; GC.collect(); } } Nope, didn't do anything. It still swells forever. Beyond that, you'll just potentially end up with a bunch of false positives, since you have a big static array in the same block as a pointer. But since you aren't actually setting the array contents in this test, that clearly isn't a problem here. The real program is a tree structure that contains class references, so is entirely pointers. The root of the tree is then discarded, and a new tree created, however the old tree isn't being collected.
Re: Who wants to have some fun memory debugging?
Robert Fraser wrote:
Simpler version, sans printf:
module leak;
import tango.core.Memory;
struct Data
{
Data* prev;
char[4092] something;
}
public void main()
{
Data* data;
Data* newData;
int i;
while(true)
{
for(i = 0; i < 10_000; i++)
{
newData = new Data;
newData.prev = data;
data = newData;
}
data = null;
newData = null;
At this point it's quite possible that you still have a reference to
newData in a register. If you want to be sure the collect call below
works as intended for this test, try adding:
newData = new Data;
here.
i = 0;
GC.collect();
}
}
Beyond that, you'll just potentially end up with a bunch of false
positives, since you have a big static array in the same block as a
pointer. But since you aren't actually setting the array contents in
this test, that clearly isn't a problem here.
Re: Who wants to have some fun memory debugging?
Simpler version, sans printf:
module leak;
import tango.core.Memory;
struct Data
{
Data* prev;
char[4092] something;
}
public void main()
{
Data* data;
Data* newData;
int i;
while(true)
{
for(i = 0; i < 10_000; i++)
{
newData = new Data;
newData.prev = data;
data = newData;
}
data = null;
newData = null;
i = 0;
GC.collect();
}
}
Re: A couple of questions
Hello Sam,
Hello,
For the given example below,
E1:
template Chain(R...) if (allSatisfy!(isInputRange, R))
{
static if (R.length > 1)
alias ChainImpl!(R) Chain;
else
alias R[0] Chain;
}
Q1: What's *if* statement doing right after the template definite?I
can guess what the purpose is but I can not find the answer from the
spec.
E2:
template isInputRange(R)
{
enum bool isInputRange = is(typeof(
{
R r; // can define a range object
if (r.empty) {} // can test for empty
r.popFront; // can invoke next
auto h = r.front; // can get the front of the range
}()));
}
Q2:What's the logic inside the enum...{} blocks?
That is an odd construct and not an enum block, breaking it into pieces:
enum bool isInputRange = SomeBoolExp;
SomeBoolExp: // true if SomeExp is a valid expression
is(typeof(SomeExp))
SomeExp: // a function call
SomeFnExp()
SomeFnExp: // a delegate literal
{
R r; // can define a range object
if (r.empty) {} // can test for empty
r.popFront; // can invoke next
auto h = r.front; // can get the front of the range
}
Q3:The *typeof* expression here. is it an anonymous function/delegate
which return a bool value?If not,how come there is a () pairs appeared
right after the enum{}blocks?If yes,shouldn't the () pairs appear
before the last *)* ?
It would be grateful if anybody can help.
see above
Regards,
Sam
Who wants to have some fun memory debugging?
Running this program with Tango SVN + DMD 1.045 or Tango 0.98 + DMD
1.041 on WinXP SP3 32-bit results in a memory leak (the program keeps
increasing in size at every iteration)
leak.d:
---
module leak;
import tango.stdc.stdio;
import tango.core.Memory;
struct Data
{
Data* prev;
char[4092] something;
}
public void main()
{
Data* data;
Data* newData;
int i;
while(true)
{
for(i = 0; i < 10_000; i++)
{
newData = new Data;
newData.prev = data;
data = newData;
}
data = null;
newData = null;
i = 0;
GC.collect();
printf("Iteration...");
fflush(stdout);
fgetc(stdin);
}
}
---
Running it through a debugger shows that at the printf, the entire
contents of the stack frame of main is zeros and there's no global data
referenced by the program.
A couple of questions
Hello,
For the given example below,
E1:
template Chain(R...) if (allSatisfy!(isInputRange, R))
{
static if (R.length > 1)
alias ChainImpl!(R) Chain;
else
alias R[0] Chain;
}
Q1: What's *if* statement doing right after the template definite?I can guess
what the purpose is but I can not find the answer from the spec.
E2:
template isInputRange(R)
{
enum bool isInputRange = is(typeof(
{
R r; // can define a range object
if (r.empty) {} // can test for empty
r.popFront; // can invoke next
auto h = r.front; // can get the front of the range
}()));
}
Q2:What's the logic inside the enum...{} blocks?
Q3:The *typeof* expression here. is it an anonymous function/delegate which
return a bool value?If not,how come there is a () pairs appeared right after
the enum{}blocks?If yes,shouldn't the () pairs appear before the last *)* ?
It would be grateful if anybody can help.
Regards,
Sam
invariant
Hi all, i want to declare an invariant interger, for instance "invariant int x = 3;". But i've got this error while compiling. Error: constant 3 is not an lvalue could someone explain me what's wrong here ? I'm using dmd 2.030
Re: 3 variant questions
> Dear Saaa, these varargs suck badly and you shouldn't use them. It's so
> simple to introduce portability errors or heisenbugs, and it's
heisenbugs :)
> incredibly hard to get it right. You're better off with alternatives.
>
> Alternative 1: Typesafe Variadic Functions
> Useful if the variadic arguments should have only one type. And they're
> very easy to use. It's documented somewhere on
> http://www.digitalmars.com/d/1.0/function.html
It doesn't say much about setting/changing the passed arguments.
How do these differ?
int test1( int i ...) { return i;}
int test2( int i ) { return i;}
test1(3); // returns 3
test2(3); // returns 3
>
> Alternative 2: Tuples
> http://www.digitalmars.com/d/1.0/template.html#TemplateTupleParameter
> The variadic arguments can have any types, but such functions can't really
> be virtual methods or delegates.
Well, I should learn about them templates anyways :)
Templates create the necessary appropriately typed instances of a function
at compile-time.. seems more usefull as my get function should work with
any type.
>
>>> Why is this extra step necessary and why won't simple casting not work?
>>
>> You should never directly work with _argptr. It's not guaranteed to be
>> a simple pointer. For example, I believe that GCC will sometimes use
>> registers (God only knows WHY).
>
> I think it's because GCC tries to keep the usual calling convention, where
> some arguments are passed as registers. Which is utterly pointless and
> stupid (ok, maybe it was easier to implement in gdc), and turns using
> varargs into hell.
>
> It would be so simple if _argptr was just an array of pointers to the
> actual arguments. It'd be so simple it makes me mad.
Re: 3 variant questions
>>
>
> You get an AV because you're passing the argument by value. You need to
> pass its address instead.
>
> Try this:
>
> void main() {
> int i;
> get(file, "i", &i);
> writeln(i);
> }
>
> It will print "7".
Ah of course, thanks.
Re: 3 variant questions
Dear Saaa, these varargs suck badly and you shouldn't use them. It's so simple to introduce portability errors or heisenbugs, and it's incredibly hard to get it right. You're better off with alternatives. Alternative 1: Typesafe Variadic Functions Useful if the variadic arguments should have only one type. And they're very easy to use. It's documented somewhere on http://www.digitalmars.com/d/1.0/function.html Alternative 2: Tuples http://www.digitalmars.com/d/1.0/template.html#TemplateTupleParameter The variadic arguments can have any types, but such functions can't really be virtual methods or delegates. Why is this extra step necessary and why won't simple casting not work? You should never directly work with _argptr. It's not guaranteed to be a simple pointer. For example, I believe that GCC will sometimes use registers (God only knows WHY). I think it's because GCC tries to keep the usual calling convention, where some arguments are passed as registers. Which is utterly pointless and stupid (ok, maybe it was easier to implement in gdc), and turns using varargs into hell. It would be so simple if _argptr was just an array of pointers to the actual arguments. It'd be so simple it makes me mad.
Re: 3 variant questions
Saaa Wrote:
> >
> > import std.stdarg;
> >
> > assert( _arguments[0] is typeid(int*) );
> > auto arg = va_arg!(int*)(_argptr);
> > *arg = 10;
> >
> > Probably.
> >
> > -- Daniel
>
> Calling the following returns an Access Violation Error after
> correctly writing the two lines.
>
> void main()
> {
> int i;
> get( file, `i`, i);
> }
>
> public void get(in char[][] file, in char[] identifier, ...)
> {
> assert( _arguments[0] is typeid(int) );
> writefln(`assert done`);
> auto arg = va_arg!(int*)(_argptr);
> writefln(`assign done`);
> *arg = 7;
> return;
> }
>
>
You get an AV because you're passing the argument by value. You need to pass
its address instead.
Try this:
void main() {
int i;
get(file, "i", &i);
writeln(i);
}
It will print "7".
Re: Threading
"Georg Wrede" wrote in message news:guc6ep$2im...@digitalmars.com... > Saaa wrote: > Steven Schveighoffer wrote: >>> i.e. I want thread 1 to initialize elements 0, 1, and 2, and thread 2 to >>> initialize elements 3, 4, and 5: >>> >>> thread1 = new ProcessingThread(arrayToInit[0..3]); >>> thread2 = new ProcessingThread(arrayToInit[3..6]); >>> thread1.start(); >>> thread2.start(); >>> >>> Should be completely thread safe. >> >> Could become more difficult if the distinction would be odd/even elements >> :P > > Why??? If your threads have no bugs (i.e. the "odd" thread doesn't read or > write the "even" elements, and vice versa), then this should be easy! > >> Or creatures on a grid who can only see the cell they stand on :D > > Then the only thing that has to be thread safe is the code that moves a > creature from cell to cell. And, of course, it has to guarantee that no > two creatures end up in the same cell. > I meant slicing would become more difficult.
Re: 3 variant questions
>
> import std.stdarg;
>
> assert( _arguments[0] is typeid(int*) );
> auto arg = va_arg!(int*)(_argptr);
> *arg = 10;
>
> Probably.
>
> -- Daniel
Calling the following returns an Access Violation Error after
correctly writing the two lines.
void main()
{
int i;
get( file, `i`, i);
}
public void get(in char[][] file, in char[] identifier, ...)
{
assert( _arguments[0] is typeid(int) );
writefln(`assert done`);
auto arg = va_arg!(int*)(_argptr);
writefln(`assign done`);
*arg = 7;
return;
}
Re: Threading
Saaa wrote: Steven Schveighoffer wrote: i.e. I want thread 1 to initialize elements 0, 1, and 2, and thread 2 to initialize elements 3, 4, and 5: thread1 = new ProcessingThread(arrayToInit[0..3]); thread2 = new ProcessingThread(arrayToInit[3..6]); thread1.start(); thread2.start(); Should be completely thread safe. Could become more difficult if the distinction would be odd/even elements :P Why??? If your threads have no bugs (i.e. the "odd" thread doesn't read or write the "even" elements, and vice versa), then this should be easy! Or creatures on a grid who can only see the cell they stand on :D Then the only thing that has to be thread safe is the code that moves a creature from cell to cell. And, of course, it has to guarantee that no two creatures end up in the same cell.
Re: Threading
>>> I probably should have phrased my question better with the array, what I was wondering is if it was safe for say two threads to right to the array at the same time as long as I'm sure they're not writing to the same index of the array? >>> >>> You can do anything you want without thread safety, but you run the risk >>> of deadlocks or corrupted memory. >> That is why the question was whether it was safe. > > If two threads are writing to two different sections of memory, yes it is > always safe :) I think that's one of the fundamental premises of threads > running anyways. If you couldn't do this, you couldn't have threads. I used to think of an array as one thing, thus making it unsafe to write to it from multiple threads at the same time :) I kind of thought he was asking along this conception. > >> >>> >>> The problem isn't writing to two different elements of an array, >> >>> the hard part is *ensuring* that you are writing to two different >>> elements >>> of an array. Multithreading code is tough to write correctly, you may >>> want to read a book on it. >> And sometimes it is extremely easy to ensure you are never writing to the >> same elements. > > If you are doing anything interesting with an array, this is not the > case. Might as well not pass the same array to both threads. > > Maybe the OP doesn't understand that you can slice up an array quite > easily. If you want to ensure two threads don't touch the same memory, > don't give both threads access to the same memory. That's the easiest way > to ensure thread safety. > > i.e. I want thread 1 to initialize elements 0, 1, and 2, and thread 2 to > initialize elements 3, 4, and 5: > > thread1 = new ProcessingThread(arrayToInit[0..3]); > thread2 = new ProcessingThread(arrayToInit[3..6]); > thread1.start(); > thread2.start(); > > Should be completely thread safe. > > -Steve Could become more difficult if the distinction would be odd/even elements :P Or creatures on a grid who can only see the cell they stand on :D But yes, slicing is neat!
Re: Threading
On Tue, 12 May 2009 10:12:48 -0400, Saaa wrote: I probably should have phrased my question better with the array, what I was wondering is if it was safe for say two threads to right to the array at the same time as long as I'm sure they're not writing to the same index of the array? You can do anything you want without thread safety, but you run the risk of deadlocks or corrupted memory. That is why the question was whether it was safe. If two threads are writing to two different sections of memory, yes it is always safe :) I think that's one of the fundamental premises of threads running anyways. If you couldn't do this, you couldn't have threads. The problem isn't writing to two different elements of an array, the hard part is *ensuring* that you are writing to two different elements of an array. Multithreading code is tough to write correctly, you may want to read a book on it. And sometimes it is extremely easy to ensure you are never writing to the same elements. If you are doing anything interesting with an array, this is not the case. Might as well not pass the same array to both threads. Maybe the OP doesn't understand that you can slice up an array quite easily. If you want to ensure two threads don't touch the same memory, don't give both threads access to the same memory. That's the easiest way to ensure thread safety. i.e. I want thread 1 to initialize elements 0, 1, and 2, and thread 2 to initialize elements 3, 4, and 5: thread1 = new ProcessingThread(arrayToInit[0..3]); thread2 = new ProcessingThread(arrayToInit[3..6]); thread1.start(); thread2.start(); Should be completely thread safe. -Steve
Re: Threading
> >> I probably should have phrased my question better with the array, what I >> was >> wondering is if it was safe for say two threads to right to the array at >> the same >> time as long as I'm sure they're not writing to the same index of the >> array? > > You can do anything you want without thread safety, but you run the risk > of deadlocks or corrupted memory. That is why the question was whether it was safe. > > The problem isn't writing to two different elements of an array, >the hard part is *ensuring* that you are writing to two different elements >of an array. Multithreading code is tough to write correctly, you may >want to read a book on it. And sometimes it is extremely easy to ensure you are never writing to the same elements.
Re: Threading
On Tue, 12 May 2009 05:11:49 -0400, Brok3n Halo
wrote:
I probably should have phrased my question better with the array, what I
was
wondering is if it was safe for say two threads to right to the array at
the same
time as long as I'm sure they're not writing to the same index of the
array?
You can do anything you want without thread safety, but you run the risk
of deadlocks or corrupted memory.
The problem isn't writing to two different elements of an array, the hard
part is *ensuring* that you are writing to two different elements of an
array. Multithreading code is tough to write correctly, you may want to
read a book on it.
D2 promises to be a lot better at helping you ensure this.
Also I'm still getting the "Error: Win32 Exception" with my test code
any idea
why? Attached is the latest version.
Just realized from reading your code, you are using D1 with Phobos, I have
no idea what bugs there are, or how to use threads there, so I can't
really help you.
I can tell you that your code ported to Tango runs without throwing an
exception, code below:
import tango.core.Thread;
import tango.io.Stdout;
int count = 0;
void main(){
Thread testthread;
void testfunc(){
while(count<1000){
++count;
}
}
testthread = new Thread(&testfunc);
testthread.start();
int lastCount = 0;
while(testthread.isRunning){
if(count != lastCount){
Stdout.formatln("{}", count);
lastCount = count;
}
}
}
output:
164
16789
23750
29998
36054
4472263
4482283
4488871
4495320
4501356
4507264
4513158
4518987
4524886
4530722
4536557
4542362
4548221
4554051
4559848
4565753
4571592
4577354
4583152
4588942
4594719
4600579
4606375
4612181
4617981
4623686
4629421
-Steve
Re: 3 variant questions
Saaa wrote: > ... >> var_arg!(T) will convert _argptr into the type you specify and it will >> also advance _argptr to the next argument. > What would happen if you'd cast it incorrectly if it wasn't a simple pointer > ? :D Same as would happen if you incorrectly cast anything. i.e. anything. Segfault if you're lucky. If you're unlucky, random crashes. -- Daniel
Re: 3 variant questions
"Daniel Keep" wrote in message news:gubip2$187...@digitalmars.com... > > > Saaa wrote: >> I just noticed D1 does have std.stdarg. >> I shouldn't just search on the website :( >> (where it is missing on the phobos page) >> >>> import std.stdarg; >>> >>> assert( _arguments[0] is typeid(int*) ); >>> auto arg = va_arg!(int*)(_argptr); >>> *arg = 10; >>> >>> Probably. >> :D >>> -- Daniel >> >> So, you make arg point to the same as va_arg. > > No, va_arg is a function. I meant to write _argptr :) > >> Why is this extra step necessary and why won't simple casting not work? > > You should never directly work with _argptr. It's not guaranteed to be > a simple pointer. For example, I believe that GCC will sometimes use > registers (God only knows WHY). ok... > > var_arg!(T) will convert _argptr into the type you specify and it will > also advance _argptr to the next argument. What would happen if you'd cast it incorrectly if it wasn't a simple pointer ? :D > > -- Daniel
Re: 3 variant questions
Saaa wrote: > I just noticed D1 does have std.stdarg. > I shouldn't just search on the website :( > (where it is missing on the phobos page) > >> import std.stdarg; >> >> assert( _arguments[0] is typeid(int*) ); >> auto arg = va_arg!(int*)(_argptr); >> *arg = 10; >> >> Probably. > :D >> -- Daniel > > So, you make arg point to the same as va_arg. No, va_arg is a function. > Why is this extra step necessary and why won't simple casting not work? You should never directly work with _argptr. It's not guaranteed to be a simple pointer. For example, I believe that GCC will sometimes use registers (God only knows WHY). var_arg!(T) will convert _argptr into the type you specify and it will also advance _argptr to the next argument. -- Daniel
Re: 3 variant questions
I just noticed D1 does have std.stdarg. I shouldn't just search on the website :( (where it is missing on the phobos page) > import std.stdarg; > > assert( _arguments[0] is typeid(int*) ); > auto arg = va_arg!(int*)(_argptr); > *arg = 10; > > Probably. :D > > -- Daniel So, you make arg point to the same as va_arg. Why is this extra step necessary and why won't simple casting not work?
Re: 3 variant questions
Saaa wrote: >> Saaa wrote: >>> ... >>> Passing variadic arguments as ref I think is what I am asking for :) >> You can't. You have to explicitly take the address of the arguments. >> >> -- Daniel > > Like this ? > *_argptr = 10; > :D > > I don't know how to tell the compiler I want to write data there. import std.stdarg; assert( _arguments[0] is typeid(int*) ); auto arg = va_arg!(int*)(_argptr); *arg = 10; Probably. -- Daniel
Re: Threading
I have never used threads before, but it sounds safe to me as when the thread gets thrown of the core and later gets back nothing referred by the register has changed. Again, I am a newbie :) "Brok3n Halo" wrote in message news:gubegl$uu...@digitalmars.com... >I probably should have phrased my question better with the array, what I >was > wondering is if it was safe for say two threads to right to the array at > the same > time as long as I'm sure they're not writing to the same index of the > array? > > Also I'm still getting the "Error: Win32 Exception" with my test code any > idea > why? Attached is the latest version. >
Re: 3 variant questions
> Saaa wrote: >> ... >> Passing variadic arguments as ref I think is what I am asking for :) > > You can't. You have to explicitly take the address of the arguments. > > -- Daniel Like this ? *_argptr = 10; :D I don't know how to tell the compiler I want to write data there.
Re: Threading
I probably should have phrased my question better with the array, what I was
wondering is if it was safe for say two threads to right to the array at the
same
time as long as I'm sure they're not writing to the same index of the array?
Also I'm still getting the "Error: Win32 Exception" with my test code any idea
why? Attached is the latest version.
begin 644 test.d
M;6]D=6QE('1EPT*"0EI9BAC;W5N="`A/2!L87-T0V]U;G0I>PT*"0D)=W)I=&5F;&XH
M(B5D(b...@8v]u;G0I.PT*"0D);&%S=$-O=6YT(#...@8v]u;G0[#0H)"7T-"@E]
%#0H-"GT`
`
end
Re: 3 variant questions
Saaa wrote: > ... > Passing variadic arguments as ref I think is what I am asking for :) You can't. You have to explicitly take the address of the arguments. -- Daniel
