Re: Apparent problem with GC not collecting on Windows
On Wednesday, 28 November 2012 at 22:02:35 UTC, Michael wrote: Only 3 iterations before out of memory exception? Because you used uint instead of ubyte, array is bigger, memory exhausts faster. 3. Why it helps? GC.free(data.ptr); Initial leak happened because for some reason array allocated in previous iteration was not collected by GC when allocating new one, so the new one was allocated in another space growing the heap. If you place GC.free the array gets removed from heap on each iteration and each new allocation reuses the same memory, heap doesn't grow.
Re: Is there a portable way to limit memory/cpu usage of a D application?
On Wednesday, 9 November 2011 at 17:13:14 UTC, Dejan Lekic wrote: Sure nothing prevents me from using setrlimit() in my D app, but perhaps it is something to think about a portable way of doing that. I know I'm over a year late coming in on this conversation, but how would you use setrlimit() in your D app at all? I can't seem to figure out what the correct library is to include in my program to be able to call set/getrlimit in any manner. Thanks!
Re: assertion failure with template instantiation of forward reference class
comco: Looks like a bug? This is what DMD 2.061alpha prints, it doesn't crash: ...\dmd2\src\phobos\std\range.d(603): Error: static assert "Cannot put a string into a LockingTextWriter" ...\dmd2\src\phobos\std\format.d(2349):instantiated from here: put!(LockingTextWriter, string) ...\dmd2\src\phobos\std\format.d(2782):instantiated from here: formatValue!(LockingTextWriter, TypeInfo_Class, char) ...\dmd2\src\phobos\std\format.d(415):instantiated from here: formatGeneric!(LockingTextWriter, TypeInfo_Class, char) ...\dmd2\src\phobos\std\stdio.d(712):... (1 instantiations, -v to show) ... ...\dmd2\src\phobos\std\stdio.d(1618):instantiated from here: write!(TypeInfo_Class,char) test.d(5):instantiated from here: writeln!(TypeInfo_Class) Bye, bearophile
assertion failure with template instantiation of forward reference class
When trying to compile this code: import std.stdio; class A; class B(T) : T { } void main() { writeln(typeid(B!A)); } I get this error: Assertion failure: '!scope' on line 358 in file 'toobj.c'. Shouldn't it be more like: Error: class main.A unable to resolve forward reference in definition Looks like a bug?
Re: Segfault with std.container.Array but not regular dynamic array
On Wednesday, 28 November 2012 at 20:30:41 UTC, Maxim Fomin wrote: On Wednesday, 28 November 2012 at 18:08:59 UTC, Dan wrote: This code with version=bug produces garbage because of opAssign. It seems that opAssign is actually called before accessing map: Maxim, thanks for looking more at this. This bug is not affecting my work in any way - I'm just trying to learn more about D and how to debug, and your responses are helpful. I've now built druntime and phobos with debug to see if I can see what is going on more. For me the relevant assembly of your code looks like below. I see the value in the assoc array get created and memset initialized to 0 inside aaA.d function _aaGetX with this line: memset(ptail + aligntsize(keytitsize), 0, valuesize); // zero value Then between +62 and +102 it goes off the rails and upon entry to opAssign *this* is likely garbage. I can't be certain because the values I see are 0, which would be consistent with 0 initialization - but could also just be luck. At this point I wish I knew some assembly. Anyway, I don't know if it is a problem with associative array code, per se, or the code generated by the compiler when opAssign and/or postblit are defined for the value type. I do know that if you have no postblit and no opAssign there are no issues - even with a destructor. This is consistent with the examples you have shown, they have postblit and/or opAssign as well as the RefCount code which has postblit and dtor. In both your example crash and the RefCount crash the actual seg fault is in the dtor accessing bogus data because as you pointed out it was not correctly initialized. Thanks Dan 0x0041a630 <+0>: push %rbp 0x0041a631 <+1>: mov%rsp,%rbp 0x0041a634 <+4>: sub$0x38,%rsp 0x0041a638 <+8>: push %rbx 0x0041a639 <+9>: mov$0x3,%eax 0x0041a63e <+14>: mov%eax,-0x30(%rbp) 0x0041a641 <+17>: lea-0x30(%rbp),%rcx 0x0041a645 <+21>: movabs $0x8,%rdx 0x0041a64f <+31>: movabs $0x43cc10,%rsi 0x0041a659 <+41>: mov%fs:0x0,%rdi 0x0041a662 <+50>: add0x229957(%rip),%rdi# 0x643fc0 0x0041a669 <+57>: callq 0x41ae84 <_aaGetX> 0x0041a66e <+62>: mov%rax,-0x28(%rbp) 0x0041a672 <+66>: test %rax,%rax 0x0041a675 <+69>: jne0x41a681 <_Dmain+81> 0x0041a677 <+71>: mov$0x12,%edi 0x0041a67c <+76>: callq 0x41a710 <_D5again7__arrayZ> => 0x0041a681 <+81>: movabs $0x2a,%rax 0x0041a68b <+91>: mov%rax,-0x18(%rbp) 0x0041a68f <+95>: mov%rax,%rsi 0x0041a692 <+98>: lea-0x20(%rbp),%rdi 0x0041a696 <+102>: callq 0x41a5c8 <_D5again1S8opAssignMFS5again1SZv>
Re: Apparent problem with GC not collecting on Windows
1. Can be solved using allocators http://dlang.org/memory.html#newdelete and http://dlang.org/class.html#ClassAllocator (here deprecated)? 2. Why with class Too { private uint[] pp; this(int s) { pp = new unit[s]; } alias pp this; } Only 3 iterations before out of memory exception? 3. Why it helps? foreach (i; 0 .. 10) { file.rawRead(data); } GC.free(data.ptr); Win 8 Pro, 64 bit.
Re: Apparent problem with GC not collecting on Windows
On 11/28/2012 02:53 PM, Ali Çehreli wrote: On 11/28/2012 11:11 AM, 1100110 wrote: > On 11/28/2012 12:57 PM, Dmitry Olshansky wrote: >> 11/28/2012 10:51 PM, Ali Çehreli пишет: >>> A friend of mine reports that the memory usage of the following program >>> grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 >>> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]). >>> >>> If you are on Windows, could you please say whether you have the same >>> problem. (I don't have Windows.) The following is how we reproduce: >> >> [snip] >> >>> The program gets terminated by a core.exception.OutOfMemoryError. (Same >>> problem with std.stream.BufferedFile.) >> >> Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 >> running 32-bit app. >> >> > > Isn't this a known problem with a conservative GC on 32bit? The GC sees > something that *Could* be a reference, and refuses to collect that data. I think it is more like an integer value looking like a pointer to something. I don't see what the GC may be confused with in this case. I've still created a bug: http://d.puremagic.com/issues/show_bug.cgi?id=9094 Ali That's what I meant. =P
Re: path matching problem
OTOH, I still don't know where "any" is documented. It's clearly some sort of template instantiation, but it doesn't seem to be defined in either std.string or std.object (or anywhere else I've thought to check). And it look as if it would be something very useful to know. It's documented here: http://dlang.org/phobos/std_algorithm.html#any
Re: Apparent problem with GC not collecting on Windows
On 11/28/2012 11:11 AM, 1100110 wrote: > On 11/28/2012 12:57 PM, Dmitry Olshansky wrote: >> 11/28/2012 10:51 PM, Ali Çehreli пишет: >>> A friend of mine reports that the memory usage of the following program >>> grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 >>> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]). >>> >>> If you are on Windows, could you please say whether you have the same >>> problem. (I don't have Windows.) The following is how we reproduce: >> >> [snip] >> >>> The program gets terminated by a core.exception.OutOfMemoryError. (Same >>> problem with std.stream.BufferedFile.) >> >> Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 >> running 32-bit app. >> >> > > Isn't this a known problem with a conservative GC on 32bit? The GC sees > something that *Could* be a reference, and refuses to collect that data. I think it is more like an integer value looking like a pointer to something. I don't see what the GC may be confused with in this case. I've still created a bug: http://d.puremagic.com/issues/show_bug.cgi?id=9094 Ali
Re: Segfault with std.container.Array but not regular dynamic array
On Wednesday, 28 November 2012 at 18:08:59 UTC, Dan wrote: Thanks! I see what you are saying in valgrind. However, the following shows no problem in valgrind. Same code, only using S instead of RefCounted!(int). How could that be explained? Note that both RefCount!() and your posted S have opAssign, whereas the one below does not. Perhaps it is a problem only when there exists a custom opAssign? Thanks Dan Because code does not access memory expected to be corrupted. import core.stdc.stdio : printf; struct S { long x = 42; version(bug) { void opAssign(S rhs) { printf("%d\n", this.x); } } } //alias RefCounted!(int) Foo; alias S Foo; Foo[int] map; void main() { map[3] = Foo(); printf("%d\n", map[3].x); } This code with version=bug produces garbage because of opAssign. It seems that opAssign is actually called before accessing map: push %rbp mov%rsp,%rbp sub$0x28,%rsp push %rbx movabs $0x2a,%rax mov%rax,-0x10(%rbp) mov%rax,%rsi lea-0x18(%rbp),%rdi callq 0x418c70 <_D3aux1S8opAssignMFS3aux1SZv> mov-0x18(%rbp),%rcx mov%rcx,-0x28(%rbp) mov$0x3,%edx mov%edx,-0x20(%rbp) lea-0x20(%rbp),%rcx mov$0x8,%dl movabs $0x430db0,%rsi mov%fs:0x0,%rdi add0x21d2c1(%rip),%rdi# 0x635fb0 callq 0x419048 <_aaGetX>
Re: Apparent problem with GC not collecting on Windows
On 11/28/2012 12:57 PM, Dmitry Olshansky wrote: 11/28/2012 10:51 PM, Ali Çehreli пишет: A friend of mine reports that the memory usage of the following program grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]). If you are on Windows, could you please say whether you have the same problem. (I don't have Windows.) The following is how we reproduce: [snip] The program gets terminated by a core.exception.OutOfMemoryError. (Same problem with std.stream.BufferedFile.) Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 running 32-bit app. Isn't this a known problem with a conservative GC on 32bit? The GC sees something that *Could* be a reference, and refuses to collect that data.
Re: Apparent problem with GC not collecting on Windows
11/28/2012 10:51 PM, Ali Çehreli пишет: A friend of mine reports that the memory usage of the following program grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]). If you are on Windows, could you please say whether you have the same problem. (I don't have Windows.) The following is how we reproduce: [snip] The program gets terminated by a core.exception.OutOfMemoryError. (Same problem with std.stream.BufferedFile.) Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 running 32-bit app. -- Dmitry Olshansky
Apparent problem with GC not collecting on Windows
A friend of mine reports that the memory usage of the following program grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]). If you are on Windows, could you please say whether you have the same problem. (I don't have Windows.) The following is how we reproduce: Generate a file of about 1G by running the following program: import std.stdio; void foo() { auto file = File("one_gigabyte_file", "w"); auto data = new ubyte[](100 * 1024 * 1024); foreach (i; 0 .. 10) { file.rawWrite(data); } } void main() { foo(); } That part is not the problem. The problem is when reading such a large file by chunks. Could you please run the following program and report whether the memory consumption of it is increasing continuously: import std.stdio; void foo() { auto file = File("one_gigabyte_file", "r"); auto data = new ubyte[](100 * 1024 * 1024); foreach (i; 0 .. 10) { file.rawRead(data); } } void main() { for (size_t i; true; ++i) { writeln(i); foo(); } } The program gets terminated by a core.exception.OutOfMemoryError. (Same problem with std.stream.BufferedFile.) I don't see any problem under Linux. Thank you, Ali
Re: path matching problem
On 11/27/2012 06:45 PM, jerro wrote: You could replace the inner loop with somehting like: bool excl = exclude.any!(part => name.canFind(part)); std.algorithm seems to generally be running the match in the opposite direction, if I'm understanding it properly. (Dealing with D template is always confusing to me.) OTOH, I couldn't find the string any method, so I'm not really sure what you're proposing, though it does look attractive. I don't understand what you mean with running the match in the opposite direction, but I'll explain how my line of code works. First of all, it is equivalent to: any!(part => canFind(name, part))(exclude); The feature that that lets you write that in the way I did in my previous post is called uniform function call syntax (often abbreviated to UFCS) and is described at http://www.drdobbs.com/cpp/uniform-function-call-syntax/232700394. canFind(name, part) returns true if name contains part. (part => canFind(name, part)) is a short syntax for (part){ return canFind(name, part); } any!(condition)(range) returns true if condition is true for any element of range So the line of code in my previous post sets excl to true if name contains any of the strings in exclude. If you know all the strings you want to exclude in advance, it is easier to do that with a regex like Joshua did. If you want to learn about D templates, try this tutorial: https://github.com/PhilippeSigaud/D-templates-tutorial/blob/master/dtemplates.pdf?raw=true Still, though your basic approach sounds good, the suggestion of Joshua Niehus would let me filter out the strings that didn't fit before entering the loop. There's probably no real advantage to doing it that way, but it does seem more elegant. I agree, it is more elegant. Thanks for the tutorial link, I'll give it a try. (Whee! A 182 page tutorial!) Those things, though, don't seem to stick in my mind. I learned programming in FORTRAN IV, and I don't seem to be able to force either templates, Scheme, or Haskell into my way of thinking about programming. (Interestingly, classes and structured programming fit without problems.) The link to the Walter article in Dr. Dobbs is interesting. I intend to read it first. OTOH, I still don't know where "any" is documented. It's clearly some sort of template instantiation, but it doesn't seem to be defined in either std.string or std.object (or anywhere else I've thought to check). And it look as if it would be something very useful to know.
Re: Segfault with std.container.Array but not regular dynamic array
On Wednesday, 28 November 2012 at 13:43:04 UTC, Maxim Fomin wrote: On Wednesday, 28 November 2012 at 13:09:36 UTC, Dan wrote: Actually bug is still there - changing unittest to main() does not fix program, even if it seems to run correctly. The problem with memory corruption is that it may happen with no observable segfaults just because erroneous operation was performed on memory which was permitted to be read/written. Valgrind is tool which may show absence of memory corruption errors. Giving your example: Thanks! I see what you are saying in valgrind. However, the following shows no problem in valgrind. Same code, only using S instead of RefCounted!(int). How could that be explained? Note that both RefCount!() and your posted S have opAssign, whereas the one below does not. Perhaps it is a problem only when there exists a custom opAssign? Thanks Dan - import std.typecons; import std.stdio; import pprint.pp; struct S { int x; char[] c; } //alias RefCounted!(int) Foo; alias S Foo; Foo[int] map; Foo f; unittest { int i=3; map[i] = Foo(); }
Re: Segfault with std.container.Array but not regular dynamic array
On 11/28/2012 02:09 PM, Dan wrote: It would be interesting to know if Joseph was doing his "testing" out in unittest or in a main. In main -- the code you saw attached to my earlier email is the code I was running. Thanks very much for your efforts at chasing down the bug!
Re: telnet and D
On 11/27/2012 01:56 PM, maarten van damme wrote: Haven't looked at vibe.d yet because it looked more like a library for writing web apps and normal sockets should be enough. Didn't know about Tango, I'll try deciphering the original d1 module. ... I mean no offense, but that sounds painful. vibe.d is very modular, just import what you need. I hate to push it, but it has very good documentation and examples. Its designed for web apps, but it makes a nice little web library as well. import vibe.vibe; void main() { auto client = new HttpClient; client.connect("www.google.com", 80); auto res = client.request((req){ req.url = "/"; }); logInfo("Response: %d", res.statusCode); foreach( k, v; res.headers ) logInfo("Header: %s: %s", k, v); (new NullOutputStream).write(res.bodyReader); client.disconnect(); }
Re: Error: SIMD vector types not supported on this platform
On Monday, 26 November 2012 at 22:59:49 UTC, jerro wrote: I don't know when dmd on windows will get SIMD support. I have checked windows 64 bit dmd alpha now and it already supports SIMD. Tnx ;)
Re: safety of move
11/28/2012 7:19 AM, Ellery Newcomer пишет: I find myself using [abusing?] move lately: import std.algorithm; import std.stdio; struct A { const(int) i; int j; int k; } void main() { A* a = new A(); // pretend this is malloc or something // *a = A(1) A a2 = A(1); move(a2, *a); A[] arr = new A[](2); //arr[1] = *a; move(*a, arr[1]); } For the first part, I have a A* pointing to uninitialized memory and I need to initialize it somehow. emplace should work for constructing A in a given chunk of memory. move works I guess because it uses memcpy or something. Not complaining, but wondering. Yes it hacks through const/immutable at ease. The only requirement seems that it has to be shallow immutable/cont. The second part violates D's const semantics and maybe shouldn't be permitted. But it is. I agree. -- Dmitry Olshansky
Re: Segfault with std.container.Array but not regular dynamic array
On Wednesday, 28 November 2012 at 13:09:36 UTC, Dan wrote: On Monday, 26 November 2012 at 15:44:42 UTC, Joseph Rushton Wakeling wrote: Hello all, I'm writing some code which is meant to represent a network of linked nodes. [snip] Ok, another follow up. I can reproduce your segfault using your posted code, it is included below. But the interesting thing is, I was doing the "testing" out of your Network!Node2 in a unittest section. By whittling down, the smallest crash case I could come up with is this: import std.typecons; import std.stdio; alias RefCounted!(int) Foo; unittest { Foo[int] map; map[1] = Foo(); } When I change that unittest block to a 'void main()' it works just fine. I tried the same change of unittest to 'void main()' on your code and found the same results - no crash. I think the crash issue might not be with map (even though Maxim found some troubling stuff with uninitialized structs being destructed when inserting a key that is not present). Or maybe it is just a map problem and by switching to main I am just getting lucky in not getting a crash. Actually bug is still there - changing unittest to main() does not fix program, even if it seems to run correctly. The problem with memory corruption is that it may happen with no observable segfaults just because erroneous operation was performed on memory which was permitted to be read/written. Valgrind is tool which may show absence of memory corruption errors. Giving your example: import std.typecons; import std.stdio; alias RefCounted!(int) Foo; void main() { // main instead of unittest Foo[int] map; map[1] = Foo(); } valgrind outputs: ==2562== Conditional jump or move depends on uninitialised value(s) ==2562==at 0x41A424: _D3std8typecons18__T10RefCountedTiZ10RefCounted6__dtorMFZv ==2562==by 0x41A4D5: _D3std8typecons18__T10RefCountedTiZ10RefCounted8opAssignMFS3std8typecons18__T10RefCountedTiZ10RefCountedZv ==2562==by 0x41A238: _Dmain ==2562==by 0x41C42B: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZi7runMainMFZv ==2562==by 0x41BCCD: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZi7tryExecMFMDFZvZv ==2562==by 0x41C472: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZi6runAllMFZv ==2562==by 0x41BCCD: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZi7tryExecMFMDFZvZv ==2562==by 0x41BC89: _d_run_main ==2562==by 0x41BACA: main which is exactly the same problem with unittest version: uninitialized object and bogus this pointer - now "this" just points to some allocated memory. Probably this happens because between running unittest and main function druntime has allocated more memory - and more memory is valid to be read. Actually, I have also found this and considered to be not significant (bug still exists) and thus have not reported - next time I will be more verbose.
Re: Segfault with std.container.Array but not regular dynamic array
On Monday, 26 November 2012 at 15:44:42 UTC, Joseph Rushton Wakeling wrote: Hello all, I'm writing some code which is meant to represent a network of linked nodes. [snip] Ok, another follow up. I can reproduce your segfault using your posted code, it is included below. But the interesting thing is, I was doing the "testing" out of your Network!Node2 in a unittest section. By whittling down, the smallest crash case I could come up with is this: import std.typecons; import std.stdio; alias RefCounted!(int) Foo; unittest { Foo[int] map; map[1] = Foo(); } When I change that unittest block to a 'void main()' it works just fine. I tried the same change of unittest to 'void main()' on your code and found the same results - no crash. I think the crash issue might not be with map (even though Maxim found some troubling stuff with uninitialized structs being destructed when inserting a key that is not present). Or maybe it is just a map problem and by switching to main I am just getting lucky in not getting a crash. It would be interesting to know if Joseph was doing his "testing" out in unittest or in a main. Thanks Dan Here is the code that crashes. Change unittest to 'void main()' and it works?? --- import std.container; import std.stdio; struct Link { int id; this(int i) { id=i; } } struct Node1 { uint id; Link[] links; this(uint id) { this.id = id; } void addLink(uint l) { links ~= Link(l); } } struct Node2 { uint id; Array!(Link) links; this(uint id) { this.id = id; } void addLink(uint l) { links.insert(Link(l)); } } struct Network(Node) { Node[uint] nodes; void add(uint i, uint j) { if((i in nodes) is null) nodes[i] = Node(i); if((j in nodes) is null) nodes[j] = Node(j); nodes[i].addLink(j); nodes[j].addLink(i); } void print() { foreach(k; nodes.keys) { write("[", k, "]"); foreach(l; nodes[k].links) write(" ", l.id); writeln(); } writeln(); } } unittest { Network!Node2 net2; net2.add(1, 7); writeln(net2); }
Re: Segfault with std.container.Array but not regular dynamic array
On Tuesday, 27 November 2012 at 18:04:19 UTC, Maxim Fomin wrote: I think it crashes because of using associative array. Assignment to an absent aa member causes memory allocation without proper object construction, and immediately after compiler issues call to opAssign for not-constructed object - that is why "this" pointer is "bogus" on entering function. Thanks. Good troubleshooting.
Re: auto ref and arrays
On Tuesday, 27 November 2012 at 22:09:21 UTC, Jack Applegame wrote: I don't understand why auto ref doesn't work with arrays. void test1(T)(auto ref const T[] val) {} void test2(T)(auto ref const T val) {} void main() { int b; test2(b); // OK string a; test1(a); // Error: cast(const(char[]))a is not an lvalue } Since a is mutable itself, compiler uses ref storage class. cast(const(char[]))a isn't an lvalue, so it's impossible to pass it by ref. But cast(const int)b isn't an lvalue too. Why it's no errors in this case? I thought with auto ref you dispense with the const since it figures it out. The below seems to work. Thanks, Dan import std.traits; import std.stdio; void test1(T)(auto ref T val) if(!isArray!T) { writeln("Nonarray ", typeid(typeof(val))); } void test1(T)(auto ref T val) if(isArray!T) { writeln("Array ", typeid(typeof(val))); } void main() { string a; const(string) b = "string that never changes"; test1(a); test1(b); int[] x; test1(x); int y; test1(y); }