Memory management
Naive newbie question: Can we have (in theory) in D lang memory management like V lang? Quote: https://github.com/vlang/v/blob/master/doc/docs.md#memory-management "V doesn't use garbage collection or reference counting. The compiler cleans everything up during compilation. If your V program compiles, it's guaranteed that it's going to be leak free."
custom memory management
I am trying to implement a structure with value semantics which uses an internal buffer. The first approach looks like this: struct S { byte[] buf; this(int size) { buf = new byte[size]; } this(this) { buf = buf.dup; } ~this(this) { delete buf; } } This works fine as long as such an object is allocated on the stack (so the destructor is called at the end of the scope). However when the destructor is called by the gc, the buffer might already be collected, and freeing it a second time is obviously invalid. My second approach was to allocate the buffer outside the gc-managed heap, like so: this(int size) { buf = (cast(byte*)core.stdc.stdlib.malloc(size))[0..size]; } ~this(this) { core.stdc.stdlib.free(buf); } Sadly, this is incorrect as well. Because if such an object is collected by the gc, but the gc decides not to run the destructor, the buffer will never be free'd. If the gc would either always or never call struct-destructors, one of my two solutions would work. But the current situation is (in compliance with the language spec), that it is called _sometimes_, which breaks both solutions. One way the first approach could work would be for the destructor to check wether it was called by the gc, and skip the deallocation in that case. But as far as I know, the gc does not provide such a method. It would be trivial to implement, but seems kinda hackish. I know the suggested way in D is to not deallocate the buffer at all, but rely on the gc to collect it eventually. But it still puzzles me that it seems to be impossible to do. Anybody have an idea how I could make it work? thanks, simon
Re: Memory management
On 9/29/20 3:57 AM, novice3 wrote:> Naive newbie question: > > Can we have (in theory) in D lang memory management like V lang? > > Quote: > https://github.com/vlang/v/blob/master/doc/docs.md#memory-management > > "V doesn't use garbage collection or reference counting. The compiler > cleans everything up during compilation. If your V program compiles, > it's guaranteed that it's going to be leak free." I am not a language expert but I can't imagine how the compiler knows whether an event will happen at runtime. Imagine a server program allocates memory for a client. Let's say, that memory will be deallocated when the client logs out or the server times that client out. The compiler cannot know either of that will ever happen, right? Ali
Re: Memory management
On Tuesday, 29 September 2020 at 15:47:09 UTC, Ali Çehreli wrote: I am not a language expert but I can't imagine how the compiler knows whether an event will happen at runtime. Imagine a server program allocates memory for a client. Let's say, that memory will be deallocated when the client logs out or the server times that client out. The compiler cannot know either of that will ever happen, right? Ali It doesn't need to know when a certain event happens but based on ownership and program flow. Let's think Rust for a moment. You get a connection and you allocate some metadata about it. Then you put that metadata on a list and the list owns that object. It will stay there as long there is a connection and the program can go and do other things, the list still owns the metadata. After a while the client disconnects and the program finds the metadata in the list and removes it and puts it in a local variable. Some cleanup is one but the metadata is still owned by the local variable and when that variable goes out of scope (basically end of a {} block), then it will deallocated. It's really a combination of ownership and scopes. That little simple example shows that you don't necessarily need to know things in advance in order to have static lifetimes. However, there are examples where there is no possibility for the compiler to infer when the object goes out of scope. Multiple ownership is an obvious example of that.
Re: Memory management
On Tuesday, 29 September 2020 at 10:57:07 UTC, novice3 wrote: Naive newbie question: Can we have (in theory) in D lang memory management like V lang? Quote: https://github.com/vlang/v/blob/master/doc/docs.md#memory-management "V doesn't use garbage collection or reference counting. The compiler cleans everything up during compilation. If your V program compiles, it's guaranteed that it's going to be leak free." Completely avoiding the question about D, all it says in that section is "The strings don't escape draw_text, so they are cleaned up when the function exits. In fact, the first two calls won't result in any allocations at all. These two strings are small, V will use a preallocated buffer for them." That's a toy example. It would be hard to mess that up in C, and I'd expect it to be easy for the compiler to handle it.
Re: Memory management
On Tuesday, 29 September 2020 at 16:03:39 UTC, IGotD- wrote: That little simple example shows that you don't necessarily need to know things in advance in order to have static lifetimes. However, there are examples where there is no possibility for the compiler to infer when the object goes out of scope. Multiple ownership is an obvious example of that. Seems like a mix of Go and Rust... People will probably end up using array indexes instead of references... Just like in Rust.
Re: Memory management
On Tuesday, 29 September 2020 at 10:57:07 UTC, novice3 wrote: Naive newbie question: Can we have (in theory) in D lang memory management like V lang? I don't know V so can't be sure, but doing it the same way as in the examples sounds possible. The first two calls are easy. D string literals are stored in the read-only memory that is uploaded together with the machine code before execution. So the naive implementation wouldn't allocate any more than the V version. The third one is more difficult. To allocate without using GC or RC means using an owning container, that releases memory when exiting the scope. While they can be checked against leaking referenced using the `-dip1000` feature, it's practical only when the memory has a single clear owner and there is no need to store references to it elsewhere. The GC and RC are better as a general-purpose solution. On the other hand, D could avoid all allocation in the third example by using ranges. That would cause the printing function to figure out the message as it prints.
Memory management documentation
Some of the examples at http://digitalmars.com/d/2.0/memory.html seem to be outdated: - dmd 2.046 does not have std.outofmemory. Where is OutOfMemoryException defined? - no std.gc: (I found out later that core.memory is it) Where can I learn more about the current features on memory management. Thank you, Ali
Re: custom memory management
A struct is a value type. So it is passed by value and is placed on the stack. { S s; } S DTor is called at the end of the scope. So you can rely on RAII as long as you use structs.
Re: custom memory management
On Thursday, 27 February 2014 at 22:04:50 UTC, Namespace wrote: A struct is a value type. So it is passed by value and is placed on the stack. { S s; } S DTor is called at the end of the scope. So you can rely on RAII as long as you use structs. On the stack yes. But not on the heap: S[] s = new S[17]; s = null; the GC will collect the memory eventually, but without calling any destructor. On the other hand: class C { S s; } C c = new c; c = null; in this case, when the gc collects the memory, it will call both destrcutors. The one of C as well as of S.
Re: custom memory management
On Thursday, 27 February 2014 at 22:15:41 UTC, Steven Schveighoffer wrote: On Thu, 27 Feb 2014 16:46:15 -0500, Simon Bürger [...] More and more, I think a thread-local flag of "I'm in the GC collection cycle" would be hugely advantageous -- if it doesn't already exist... I don't think it does, so I actually implemented it myself (not thread-local, but same locking as the rest of the gc): github.com/Krox/druntime/commit/38b718f1dcf08ab8dabb6eed10ff1073e215890f . But now that you mention it, a thread-local flag might be better.
Re: custom memory management
On Thu, 27 Feb 2014 16:46:15 -0500, Simon Bürger wrote: I know the suggested way in D is to not deallocate the buffer at all, but rely on the gc to collect it eventually. But it still puzzles me that it seems to be impossible to do. Anybody have an idea how I could make it work? Unfortunately, nothing is foolproof. The most "correct" solution is likely to use malloc/free. Yes, if you just new one of these, you will have to destroy it. But if you have a destructor that uses GC allocated memory such an object can NEVER be a member of a heap-allocated class. More and more, I think a thread-local flag of "I'm in the GC collection cycle" would be hugely advantageous -- if it doesn't already exist... -Steve
Re: custom memory management
I asked something similar some days ago. Maybe this provides some information tat is helpful to you: http://forum.dlang.org/thread/mekdjoyejtfpafpcd...@forum.dlang.org
Re: custom memory management
On Thursday, 27 February 2014 at 21:46:17 UTC, Simon Bürger wrote: Sadly, this is incorrect as well. Because if such an object is collected by the gc, but the gc decides not to run the destructor, the buffer will never be free'd. I think you misiterpret the spec. If object is collected, destructor is guaranteed to run. But not all objects are guaranteed to be collected. For example, no collection happens at program termination. So it is OK to release resources in destructor that will be reclaimed by OS at program termination anyway. List of such resources is OS-specific but heap memory tends to be there.
Re: custom memory management
On Friday, 28 February 2014 at 10:40:17 UTC, Dicebot wrote: On Thursday, 27 February 2014 at 21:46:17 UTC, Simon Bürger wrote: Sadly, this is incorrect as well. Because if such an object is collected by the gc, but the gc decides not to run the destructor, the buffer will never be free'd. I think you misiterpret the spec. If object is collected, destructor is guaranteed to run. But not all objects are guaranteed to be collected. For example, no collection happens at program termination. So it is OK to release resources in destructor that will be reclaimed by OS at program termination anyway. List of such resources is OS-specific but heap memory tends to be there. If you are right that would mean that the current dmd/runtime does not follow the spec. Curious. The current implementation is not aware of strcut-destructors on the heap, i.e. the GC.BlkAttr.FINALIZE flag is not set for structs (or arrays of structs). In the struct-inside-a-class example, the struct destructor is called by the (automatically generated) class-destructor. The gc only knows about class-destrcutor and calls only that one directly.
Re: custom memory management
On Friday, 28 February 2014 at 12:36:48 UTC, Simon Bürger wrote: If you are right that would mean that the current dmd/runtime does not follow the spec. Curious. The current implementation is not aware of strcut-destructors on the heap, i.e. the GC.BlkAttr.FINALIZE flag is not set for structs (or arrays of structs). Ah, structs are a bit tricky. Spec has override for struct destructors that says explicitly "Destructors are called when an object goes out of scope" so one can argue heap ignorance matches it. But the very next line contradicts it : "Their purpose is to free up resources owned by the struct object". I believe it is a DMD bug though. There is no reason why destructors can't be run here. Most likely it is just defect of current GC implementation that everyone got used to. This bug report discussion confirms it : https://d.puremagic.com/issues/show_bug.cgi?id=2834 Looks like decision was to fix it once precise GC gets added. In the struct-inside-a-class example, the struct destructor is called by the (automatically generated) class-destructor. The gc only knows about class-destructor and calls only that one directly. Yes, that matches my current observations. Did not occure before because of C coding habits :) Lucky me.
Re: custom memory management
On Friday, 28 February 2014 at 13:06:05 UTC, Namespace wrote: What can still take a long time. It annoys me very much that with arrays I can not rely on that the struct DTors are called. Yep, this bug has immediately got my vote :) It does require someone knowledgable of GC to fix in forseeable future unfortunately.
Re: custom memory management
On Friday, 28 February 2014 at 12:54:32 UTC, Dicebot wrote: On Friday, 28 February 2014 at 12:36:48 UTC, Simon Bürger wrote: If you are right that would mean that the current dmd/runtime does not follow the spec. Curious. The current implementation is not aware of strcut-destructors on the heap, i.e. the GC.BlkAttr.FINALIZE flag is not set for structs (or arrays of structs). Ah, structs are a bit tricky. Spec has override for struct destructors that says explicitly "Destructors are called when an object goes out of scope" so one can argue heap ignorance matches it. But the very next line contradicts it : "Their purpose is to free up resources owned by the struct object". I believe it is a DMD bug though. There is no reason why destructors can't be run here. Most likely it is just defect of current GC implementation that everyone got used to. This bug report discussion confirms it : https://d.puremagic.com/issues/show_bug.cgi?id=2834 Looks like decision was to fix it once precise GC gets added. What can still take a long time. It annoys me very much that with arrays I can not rely on that the struct DTors are called.
Re: custom memory management
On Friday, 28 February 2014 at 13:32:33 UTC, Namespace wrote: I will vote, too. It's somewhat strange: Since it works with delete it should also work with the current GC, or? Someone should figure out why and how delete works this way. :) Well, delete is deprecated so it can do any kind of arcane horrors :) More idiomatic destroy + GC.free pair will work because destroy is a template function.
Re: custom memory management
On Friday, 28 February 2014 at 13:16:40 UTC, Dicebot wrote: On Friday, 28 February 2014 at 13:06:05 UTC, Namespace wrote: What can still take a long time. It annoys me very much that with arrays I can not rely on that the struct DTors are called. Yep, this bug has immediately got my vote :) It does require someone knowledgable of GC to fix in forseeable future unfortunately. I will vote, too. It's somewhat strange: Since it works with delete it should also work with the current GC, or? Someone should figure out why and how delete works this way. :)
Re: custom memory management
On Friday, 28 February 2014 at 13:38:59 UTC, Dicebot wrote: On Friday, 28 February 2014 at 13:32:33 UTC, Namespace wrote: I will vote, too. It's somewhat strange: Since it works with delete it should also work with the current GC, or? Someone should figure out why and how delete works this way. :) Well, delete is deprecated so it can do any kind of arcane horrors :) More idiomatic destroy + GC.free pair will work because destroy is a template function. No, currently it is not deprecated. It is suggested to be deprecated. :P And destroy doesn't finalize the data. :/ See: http://forum.dlang.org/thread/bug-1225...@https.d.puremagic.com%2Fissues%2F and http://forum.dlang.org/thread/bug-1227...@https.d.puremagic.com%2Fissues%2F But that is only a workaround. I don't want to call every time "arr.finalize" because the GC is silly... I meant that someone should analyse the internal delete code and implement something like this for the current GC related to struct arrays (and AA's).
Re: custom memory management
On Friday, 28 February 2014 at 14:08:11 UTC, Namespace wrote: No, currently it is not deprecated. It is suggested to be deprecated. :P And destroy doesn't finalize the data. :/ See: http://forum.dlang.org/thread/bug-1225...@https.d.puremagic.com%2Fissues%2F and http://forum.dlang.org/thread/bug-1227...@https.d.puremagic.com%2Fissues%2F But that is only a workaround. I don't want to call every time "arr.finalize" because the GC is silly... "intended to be deprecated" is a better word. There is not a smallest chance it will stay in the long term, better get used to it. Quick solution would have been to merge "finalize" with destroy itself. As I have mentioned, it is a template and has all necessary information for traversal. Proper solution will be to fix the struct destructor bug as it is the root cause for your array issues too and then patch destroy to only do traversal when pointers are not owned by GC. I meant that someone should analyse the internal delete code and implement something like this for the current GC related to struct arrays (and AA's). I am too scared of what I may find :)
Re: custom memory management
On Friday, 28 February 2014 at 14:47:31 UTC, Dicebot wrote: On Friday, 28 February 2014 at 14:08:11 UTC, Namespace wrote: No, currently it is not deprecated. It is suggested to be deprecated. :P And destroy doesn't finalize the data. :/ See: http://forum.dlang.org/thread/bug-1225...@https.d.puremagic.com%2Fissues%2F and http://forum.dlang.org/thread/bug-1227...@https.d.puremagic.com%2Fissues%2F But that is only a workaround. I don't want to call every time "arr.finalize" because the GC is silly... "intended to be deprecated" is a better word. There is not a smallest chance it will stay in the long term, better get used to it. Quick solution would have been to merge "finalize" with destroy itself. As I have mentioned, it is a template and has all necessary information for traversal. Proper solution will be to fix the struct destructor bug as it is the root cause for your array issues too and then patch destroy to only do traversal when pointers are not owned by GC. I'm not sure if that is possible with the current gc. But I hope it! I meant that someone should analyse the internal delete code and implement something like this for the current GC related to struct arrays (and AA's). I am too scared of what I may find :)
Best memory management D idioms
I was going to name this thread "SEX!!" but then I thought "best memory management" would get me more reads ;) Anyway now that I have your attention... What I want to learn (not debate) is the currently available types, idioms etc. whenever one wants deterministic memory management. Please do not derail it debating how often deterministic should be preferred to GC or not. Just, whenever one should happen to require it, what are the available means? And how do they compare in your daily use, if any? If you want to post your code samples using special types for manual memory management, that would be great. AFAIK (from here on please correct me wherever I'm wrong) the original D design was, if you don't want to use GC, then malloc() and free() are available from std.c. Pretty solid. I guess the downside is less nice syntax than new/delete, and having to check the returned value instead of exceptions. I guess these were the original reasons why C++ introduced new/delete but I've never been sure. Then from this nice summary [1] I've learned about the existence of new libraries and Phobos modules: std.typecons, Dlib, and std.experimental.allocator. Sadly in this department D starts to look a bit like C++ in that there are too many possible ways to do one certain thing, and what's worse none of them is the "standard" way, and none of them is deprecated atm either. I've just taken a quick look at them, and I was wondering how many people prefer either, and what are their reasons and their experience. dlib.core.memory and dlib.memory lack documentation, but according to this wiki page [2] I found, dlib defines New/Delete substitutes without GC a-la-C++, with the nice addition of a "memoryDebug" version (how ironclad is this to debug every memory leak?) From std.typecons what caught my eye first is scoped() and Unique. std.experimental.allocator sounded quite, well, experimental or advanced, so I stopped reading before trying to wrap my head around all of it. Should I take another look? scoped() seems to work nicely for auto variables, and if I understood it right, not only it provides deterministic management, but allocates statically/in the stack, so it is like C++ without pointers right? Looking into the implementation, I just hope most of that horribly unsafe casting can be taken care of at compile time. The whole thing looks a bit obscure under the hood and in its usage: auto is mandatory or else allocation doesn't hold, but even reflection cannot tell the different at runtime between T and typeof(scoped!T) //eew. Unfortunately this also makes scoped() extremely unwieldy for member variables: their type has to be explicitly declared as typeof(scoped!T), and they cannot be initialized at the declaration. To me this looks like scoped() could be useful in some cases but it looks hardly recommendable to the same extent as the analogous C++ idiom. Then Unique seems to be analogous to C++ unique_ptr, fair enough... Or are there significant differences? Your experience? And am I right in assuming that scoped() and Unique (and dlib.core.memory) prevent the GC from monitoring the memory they manage (just like malloc?), thus also saving those few cycles? This I haven't seen clearly stated in the documentation. [1] http://forum.dlang.org/post/stohzfatiwjzemqoj...@forum.dlang.org [2] https://github.com/gecko0307/dlib/wiki/Manual-Memory-Management
Block statements and memory management
Does anyone know if when I create a variable inside a scope as in {int a = 10;} it disappears complete from the memory when the scope finishes? Or does it remain in some part of the memory? I am thinking of using scopes to make optimized programs that consume less memory.
Memory management and garbage collectors
In theory, garbage collectors make memory leaks a thing of the past. In practice, garbage collectors don't always work according to theory. This makes me curious: how does one test for memory leaks in a D program? I also don't know how smart or dumb garbage collectors are. How much help does it need? Okay, that is too general a question. So take a for- instance. Consider a circular linked list that we are done with. Each node has a reference to it (from another node in the list). But, being done with the list, we imagine that there are no references from outside the list to any node. Is D's garbage collector smart enough to deal recognize the circle of nodes as garbage? Is it necessary or at least helpful to set all the circle's links to null before being done with it? I assume that reference counting collectors are not smart enough to handle an abandoned circular list. But D's GC is not a reference counter. I don't know how smart GC's get. Is it okay to leave a single node linking to itself? Another for-instance. Consider this time a linear linked list. This time, the list is finite: it terminates with a node pointing to null. Are the issues here any different? Thanks in advance for any insights.
toStringz, and memory management, also fromStringz
Hi, In the std.string document at toStringz it has this note: Important Note: When passing a char* to a C function, and the C function keeps it around for any reason, make sure that you keep a reference to it in your D code. Otherwise, it may go away during a garbage collection cycle and cause a nasty bug when the C code tries to use it. What does it mean, and what is an example of how to do it? It's not done like this is it? char* a = cast(char*)toStringz( "test" ); char* b = cast(char*)toStringz( "word" ); hold ~= a; hold ~= b; char* cstr2 = al_get_config_value( cfg, a, b ); My programs using a C library do crash when exiting some times. Also, how do I go the other way round, some thing like toDString. I've made my own version. I can't seem to find fromStringz. Thanks for any help. - Joelcnz
Rust style memory management in D?
I've been looking into alternatives to C++ and have been following D since back in the D1 Tango/Phobos days, and recently started digging in again. I'm quite impressed with the progress, and I've started a simple toy game project to test out some of the language features. One thing that really bothers me as someone who works on applications where responsiveness is critical and even 50ms delays can cause problems is the GC though - It seems to be that the only really good answer is to to just use malloc & free from the standard library, but that seems really awful for such an elegant language. Every area but memory management is beautiful and can do what you like, but trying to avoid the GC is downright painful and forces you to ditch much of the safety D provides. Given that it is designed to be a systems programming language & in such applications memory management is so important, this seems like a tremendous oversight. In my searching, I ran across Rust, which is relatively new and takes a rather different approach to a lot of things, including memory management - making unique shared pointers(~ is used to denote them) a language feature along side garbage collection so that the compiler can enforce ownership rules. One of the main language developers has noted that the unique pointer with ownership transfer rules is used so much more than the GC option that he's trying to get GC removed from the language and placed in the standard library. Given D's target domain of high performance systems programming, this memory management model seems like a radically better fit than "screw it, we'll GC everything". I've seen a few other people talk about this issue, and the difficulty of avoiding the GC seems to be THE argument against D that I wind up seeing. Has such a model been considered, and is there a reason (besides the fact that the entire standard library would probably have to be rewritten) that it isn't used?
Region-based memory management and GC?
Have there been any investigations into using region-based memory management (aka memory arenas) in D, possibly in conjunction with GC allocated memory? This would be a very speculative idea, but it'd be interesting to know if there have been looks at this area. My own interest is request-response applications, where memory allocated as part of a specific request can be discarded as a single block when the processing of that request completes, without running destructors. I've also seen some papers describing GC systems targeting big data platforms that incorporate this idea. eg. http://www.ics.uci.edu/~khanhtn1/papers/osdi16.pdf --Jon
Classes and Structs, Memory management questions
from what i got Classes are always reference types and structs are value types in D. i am wondering why that is. for me the main difference between classes and structs is not how they are allocated, but that one has inhertiance, and the other hasn't. Supporting inheritance has some overhead, so at least the split between classes and structs makes sense to me. I like garbage collection, but in many cases it's just unnecessary (like in the example below) or hurts performance on a bigger scale. { FileReader reader = new // Annoy the garbage collector for no reason? auto blob = reader.read(); delete reader; } Since garbage collection is a very nice feature that I wouldn't wanna miss for certain scenarios I think D should give us the opportunity to determine how an object is allocated. In the example above putting it on the stack is probably a good idea. Having a self managed reference to the heap can be good too if manual memory management is wanted. Or of course let the GC manage it ( i love it for prototyping code and also as a D beginner it is beneficial that i just dont need to care about memory management). Could somebody explain to me if this is seen as a problem why/whynot and how I should address that kind of issues in my code?
Re: Best memory management D idioms
On Sunday, 5 March 2017 at 20:54:06 UTC, XavierAP wrote: I was going to name this thread "SEX!!" but then I thought "best memory management" would get me more reads ;) Anyway now that I have your attention... What I want to learn (not debate) is the currently available types, idioms etc. whenever one wants deterministic memory management. Please do not derail it debating how often deterministic should be preferred to GC or not. Just, whenever one should happen to require it, what are the available means? And how do they compare in your daily use, if any? If you want to post your code samples using special types for manual memory management, that would be great. AFAIK (from here on please correct me wherever I'm wrong) the original D design was, if you don't want to use GC, then malloc() and free() are available from std.c. Pretty solid. I guess the downside is less nice syntax than new/delete, and having to check the returned value instead of exceptions. I guess these were the original reasons why C++ introduced new/delete but I've never been sure. Then from this nice summary [1] I've learned about the existence of new libraries and Phobos modules: std.typecons, Dlib, and std.experimental.allocator. Sadly in this department D starts to look a bit like C++ in that there are too many possible ways to do one certain thing, and what's worse none of them is the "standard" way, and none of them is deprecated atm either. I've just taken a quick look at them, and I was wondering how many people prefer either, and what are their reasons and their experience. dlib.core.memory and dlib.memory lack documentation, but according to this wiki page [2] I found, dlib defines New/Delete substitutes without GC a-la-C++, with the nice addition of a "memoryDebug" version (how ironclad is this to debug every memory leak?) From std.typecons what caught my eye first is scoped() and Unique. std.experimental.allocator sounded quite, well, experimental or advanced, so I stopped reading before trying to wrap my head around all of it. Should I take another look? scoped() seems to work nicely for auto variables, and if I understood it right, not only it provides deterministic management, but allocates statically/in the stack, so it is like C++ without pointers right? Looking into the implementation, I just hope most of that horribly unsafe casting can be taken care of at compile time. The whole thing looks a bit obscure under the hood and in its usage: auto is mandatory or else allocation doesn't hold, but even reflection cannot tell the different at runtime between T and typeof(scoped!T) //eew. Unfortunately this also makes scoped() extremely unwieldy for member variables: their type has to be explicitly declared as typeof(scoped!T), and they cannot be initialized at the declaration. To me this looks like scoped() could be useful in some cases but it looks hardly recommendable to the same extent as the analogous C++ idiom. Then Unique seems to be analogous to C++ unique_ptr, fair enough... Or are there significant differences? Your experience? And am I right in assuming that scoped() and Unique (and dlib.core.memory) prevent the GC from monitoring the memory they manage (just like malloc?), thus also saving those few cycles? This I haven't seen clearly stated in the documentation. [1] http://forum.dlang.org/post/stohzfatiwjzemqoj...@forum.dlang.org [2] https://github.com/gecko0307/dlib/wiki/Manual-Memory-Management The memory management in D is becoming a mess. Yes, D was developed with the GC in mind and the attempts to make it usable without GC came later. Now std has functions that allocate with GC, there're containers that use malloc/free directly or reference counting for the internal storage, and there is std.experimental.allocator. And it doesn't really get better. There is also some effort to add reference counting directly into the language. I really fear we will have soon signatures like "void myfunc() @safe @nogc @norc..". Stuff like RefCounted or Unique are similar to C++ analogues, but not the same. They throw exceptions allocated with GC, factory methods (like Unique.create) use GC to create the object. Also dlib's memory management is a nightmare: some stuff uses "new" and GC, some "New" and "Delete". Some functions allocate memory and returns it and you never know if it will be collected or you should free it, you have to look into the source code each time to see what the function does internally, otherwise you will end up with memory leaks or segmentation faults. dlib has a lot of outdated code that isn't easy to update.
Re: Best memory management D idioms
On Monday, 6 March 2017 at 08:26:53 UTC, Eugene Wissner wrote: The memory management in D is becoming a mess. I am aware this is a hot topic, hence the opening joke. But I just want to learn what toolboxes are currently available, not really discuss how adequate they are, or how often GC is adequate enough. Neither how much % GC-haram Phobos or other libraries are internally atm. There are several threads already discussing this, which I've browsed. My impression so far is that dlib's New/Delete is the most straightforward or efficient tool, and it can kind of cover all the bases (and is GC-halal), as far as I can tell in advance. Plus it has printMemoryLog() as a bonus, which is already better than C++ new/delete. Just an observation that it doesn't provide an option to allocate an uninitialized array, considering that this module is already targeting performance applications. Not sure if I would use any other tool (besides GC itself). I'm curious about Unique but it's not fully clear to me what happens (regarding lifetime, deterministic destruction, GC monitoring) when you need to call "new" to use it.
Re: Best memory management D idioms
On Sunday, 5 March 2017 at 20:54:06 UTC, XavierAP wrote: What I want to learn (not debate) is the currently available types, idioms etc. whenever one wants deterministic memory management. There's nothing like that of C++. Currently you have Unique, RefCounted, scoped and individual people efforts on this. BTW, do you want to manage non-memory resources with these memory management mechanisms too?
Re: Best memory management D idioms
On Tuesday, 7 March 2017 at 16:51:23 UTC, Kagamin wrote: There's nothing like that of C++. Don't you think New/Delete from dlib.core.memory fills the bill? for C++ style manual dynamic memory management? It looks quite nice to me, being no more than a simple malloc wrapper with constructor/destructor calling and type safety. Plus printMemoryLog() for debugging, much easier than valgrind. do you want to manage non-memory resources with these memory management mechanisms too? I wasn't thinking about this now, but I'm sure the need will come up.
Re: Best memory management D idioms
On Tuesday, 7 March 2017 at 17:37:43 UTC, XavierAP wrote: On Tuesday, 7 March 2017 at 16:51:23 UTC, Kagamin wrote: There's nothing like that of C++. Don't you think New/Delete from dlib.core.memory fills the bill? for C++ style manual dynamic memory management? It looks quite nice to me, being no more than a simple malloc wrapper with constructor/destructor calling and type safety. Plus printMemoryLog() for debugging, much easier than valgrind. do you want to manage non-memory resources with these memory management mechanisms too? I wasn't thinking about this now, but I'm sure the need will come up. Yes. For simple memory management New/Delete would be enough. But you depend on your libc in this case, that is mostly not a problem. From experience it wasn't enough for some code bases, so the C-world invented some work arounds: 1) Link to an another libc providing a different malloc/free implementations 2) Use macros that default to the libc's malloc/free, but can be set at compile time to an alternative implementation (mbedtls uses for example mbedtls_malloc, mbedtls_calloc and mbedtls_free macros) To avoid this from the beginning, it may be better to use allocators. You can use "make" and "dispose" from std.experimental.allocator the same way as New/Delete. I tried to introduce the allocators in dlib but it failed, because dlib is difficult to modify because of other projects based on it (although to be honest it was mostly a communication problem as it often happens), so I started a similar lib from scratch.
Re: Best memory management D idioms
On Tuesday, 7 March 2017 at 18:21:43 UTC, Eugene Wissner wrote: To avoid this from the beginning, it may be better to use allocators. You can use "make" and "dispose" from std.experimental.allocator the same way as New/Delete. Thanks! looking into it. Does std.experimental.allocator have a leak debugging tool like dlib's printMemoryLog()?
Re: Best memory management D idioms
On Tuesday, 7 March 2017 at 20:15:37 UTC, XavierAP wrote: On Tuesday, 7 March 2017 at 18:21:43 UTC, Eugene Wissner wrote: To avoid this from the beginning, it may be better to use allocators. You can use "make" and "dispose" from std.experimental.allocator the same way as New/Delete. Thanks! looking into it. Does std.experimental.allocator have a leak debugging tool like dlib's printMemoryLog()? Yes, but printMemoryLog is anyway useful only for simple searching for memory leaks. For the advanced debugging it is anyway better to learn some memory debugger or profiler.
Re: Best memory management D idioms
On Tuesday, 7 March 2017 at 18:21:43 UTC, Eugene Wissner wrote: To avoid this from the beginning, it may be better to use allocators. You can use "make" and "dispose" from std.experimental.allocator the same way as New/Delete. OK I've been reading on std.experimental.allocator; it looks really powerful and general, more than I need. I see the potential but I don't really have the knowledge to tweak memory management, and the details of the "building blocks" are well beyond me. But even if I don't go there, I guess it's a good thing that I can change my program's allocator by changing one single line or version assigning theAllocator, and benchmark the results among different possibilities. I see the default allocator is the same GC heap used by 'new'. Just for my learning curiosity, does this mean that if I theAllocator.make() something and then forget to dispose() it, it will be garbage collected the same once no longer referenced? And so are these objects traversed by the GC? I've also looked at mallocator, [2] can it be used in some way to provide an allocator instead of the default theAllocator? As far as I can tell mallocator is not enough to implement an IAllocator, is there a reason, or where's the rest, am I missing it? [1] https://dlang.org/phobos/std_experimental_allocator.html [2] https://dlang.org/phobos/std_experimental_allocator_mallocator.html
Re: Best memory management D idioms
On 03/08/2017 02:15 AM, XavierAP wrote: I see the default allocator is the same GC heap used by 'new'. Just for my learning curiosity, does this mean that if I theAllocator.make() something and then forget to dispose() it, it will be garbage collected the same once no longer referenced? And so are these objects traversed by the GC? Yes and yes. GCAllocator.allocate calls core.memory.GC.malloc with does pretty much the same thing as the builtin `new`. One difference might be with preciseness. `new` can take the type into account and automatically mark the allocation as NO_SCAN. I'm not sure if GCAllocator can do that. Maybe if you use `make`/`makeArray` to make the allocations. I've also looked at mallocator, [2] can it be used in some way to provide an allocator instead of the default theAllocator? As far as I can tell mallocator is not enough to implement an IAllocator, is there a reason, or where's the rest, am I missing it? To make an IAllocator, use std.experimental.allocator.allocatorObject. The example in the documentation shows how it's done. https://dlang.org/phobos/std_experimental_allocator.html#.allocatorObject
Re: Best memory management D idioms
On Wednesday, 8 March 2017 at 06:42:40 UTC, ag0aep6g wrote: [...] Yes and yes. GCAllocator.allocate calls core.memory.GC.malloc with does pretty much the same thing as the builtin `new`. Nitpicking: `new` is typed (i.e. allocation+construction), `malloc` and `allocate` are not (only allocation). If you want allocation *and* construction with the new Allocator interface, you'll want to use the make[1] (and dispose[2] for the reverse path) template function; and they are a superset of `new`: You cannot, e.g., construct a delegate with new, but you can with `make`. [1] https://dlang.org/phobos/std_experimental_allocator.html#.make [2] https://dlang.org/phobos/std_experimental_allocator.html#.dispose
D Beginner Trying Manual Memory Management
I'm new to D. I have some modest knowledge of C++, but am more familiar with scripting languages (Matlab, Python, R). D seems so much easier than C++ in a lot of ways (and I just learned about rdmd today, which is pretty cool). I am concerned about performance of D vs. C++, so I wanted to learn a little bit more about manual memory management, in case I might ever need it (not for any particular application). The D Language book Section 6.3.4-5 covers the topic. I basically copied below and made some small changes. import core.stdc.stdlib; import std.stdio; class Buffer { private void* data; // Constructor this() { data = malloc(1024); } // Destructor ~this() { free(data); } } unittest { auto b = new Buffer; auto b1 = b; destroy(b); //changed from clear in book example assert(b1.data is null); writeln("Unit Test Finished"); } I was thinking that it might be cool to use scope(exit) to handle the memory management. It turns out the below unit test works. unittest { auto b = new Buffer; scope(exit) destroy(b); writeln("Unittest Finished"); } However, if you leave in the auto b1 and assert, then it fails. I suspect this is for the same reason that shared pointers are a thing in C++ (it can't handle copies of the pointer). Alternately, you can use some other scope and something like this unittest { { Buffer b = new Buffer; scope(exit) destroy(b); } destroy(b); writeln("Unittest Finished"); } will fail because b has already been destroyed. I thought this behavior was pretty cool. If you followed this approach, then you wouldn't have to wait until the end of the program to delete the pointers. The downside would be if you need to write a lot of pointers and there would be a lot of nesting. (likely the motivation for the reference counting approach to smart pointers). I wasn't sure how to figure out a way to combine these two components so that I only have to write one line. I thought one approach might be to put a scope(exit) within the constructor, but that doesn't work. Also, if I try to do it within a template function, then the scope(exit) is limited to the function scope, which isn't the same thing. Outside of writing a unique_ptr template class (which I have tried to do, but couldn't get it to work) I don't really know how to combine them into one line. Any ideas?
Re: Block statements and memory management
On Saturday, 16 March 2019 at 03:47:43 UTC, Murilo wrote: Does anyone know if when I create a variable inside a scope as in {int a = 10;} it disappears complete from the memory when the scope finishes? Or does it remain in some part of the memory? I am thinking of using scopes to make optimized programs that consume less memory. It depends on how the compiler translates and optimizes the code. An integer variable like `a` in your example might never exist in memory at all, if the compiler can allocate a register for it until it goes out of scope. The easiest way to find out is to look at a disassembly of the compiled code.
Re: Block statements and memory management
On Saturday, 16 March 2019 at 03:47:43 UTC, Murilo wrote: Does anyone know if when I create a variable inside a scope as in {int a = 10;} it disappears complete from the memory when the scope finishes? Or does it remain in some part of the memory? I am thinking of using scopes to make optimized programs that consume less memory. I'd recommend against these sorts of micro-optimizations. Compilers are every good at doing this kind of thing manually so you don't have to worry about it and can concentrate on the actual logic of your program.
Re: Block statements and memory management
On Saturday, 16 March 2019 at 03:47:43 UTC, Murilo wrote: Does anyone know if when I create a variable inside a scope as in {int a = 10;} it disappears complete from the memory when the scope finishes? Or does it remain in some part of the memory? I am thinking of using scopes to make optimized programs that consume less memory. In general, you want variables to have no larger scope than needed, so in large functions reducing the scope may be useful. When it comes to efficiency however, doing that is neither necessary nor sufficient for the compiler to re-use registers / stack space. I looked at the assembly output of DMD for this: ``` void func(int a); void main() { { int a = 2; func(a); } { int b = 3; func(b); } } ``` Without optimizations (the -O flag), it stores a and b on different places in the stack. With optimizations, the values of a and b (2 and 3) are simply loaded in the EDI register before the call. Removing the braces doesn't change anything about that. The compiler does live variable analysis [1] as well as data-flow analysis [2] to figure out that it's only needed to load the values 2 and 3 just before the function call. This is just a trivial example, but the same applies to larger functions. In any case, for better memory efficiency I'd consider looking at reducing dynamic allocations such as new or malloc. Memory on the stack is basically free compared to that, so even if adding lots of braces to your code reduces stack memory, chances are it's a low leverage point. [1] https://en.wikipedia.org/wiki/Live_variable_analysis [2] https://en.wikipedia.org/wiki/Data-flow_analysis
Re: Block statements and memory management
On 16/03/2019 11:19, Dennis via Digitalmars-d-learn wrote: On Saturday, 16 March 2019 at 03:47:43 UTC, Murilo wrote: Does anyone know if when I create a variable inside a scope as in {int a = 10;} it disappears complete from the memory when the scope finishes? Or does it remain in some part of the memory? I am thinking of using scopes to make optimized programs that consume less memory. In general, you want variables to have no larger scope than needed, so in large functions reducing the scope may be useful. When it comes to efficiency however, doing that is neither necessary nor sufficient for the compiler to re-use registers / stack space. I looked at the assembly output of DMD for this: ``` void func(int a); void main() { { int a = 2; func(a); } { int b = 3; func(b); } } ``` Without optimizations (the -O flag), it stores a and b on different places in the stack. With optimizations, the values of a and b (2 and 3) are simply loaded in the EDI register before the call. Removing the braces doesn't change anything about that. The compiler does live variable analysis [1] as well as data-flow analysis [2] to figure out that it's only needed to load the values 2 and 3 just before the function call. This is just a trivial example, but the same applies to larger functions. In any case, for better memory efficiency I'd consider looking at reducing dynamic allocations such as new or malloc. Memory on the stack is basically free compared to that, so even if adding lots of braces to your code reduces stack memory, chances are it's a low leverage point. [1] https://en.wikipedia.org/wiki/Live_variable_analysis [2] https://en.wikipedia.org/wiki/Data-flow_analysis Just to add a bit on what has been said: * Register allocation (see wikipedia) is a well-researched area. * By coding that way, you force the compiler to optimise *a certain way* which may prevent it to perform other, more relevant optimisations. * You cannot beat the knowledge in that domain, it is simply too big and complex, just be confident. diniz
Re: Block statements and memory management
On Sat, Mar 16, 2019 at 01:21:02PM +0100, spir via Digitalmars-d-learn wrote: > On 16/03/2019 11:19, Dennis via Digitalmars-d-learn wrote: [...] > > In any case, for better memory efficiency I'd consider looking at > > reducing dynamic allocations such as new or malloc. Memory on the > > stack is basically free compared to that, so even if adding lots of > > braces to your code reduces stack memory, chances are it's a low > > leverage point. > > > > [1] https://en.wikipedia.org/wiki/Live_variable_analysis > > [2] https://en.wikipedia.org/wiki/Data-flow_analysis > > Just to add a bit on what has been said: > * Register allocation (see wikipedia) is a well-researched area. > * By coding that way, you force the compiler to optimise *a certain way* > which may prevent it to perform other, more relevant optimisations. > * You cannot beat the knowledge in that domain, it is simply too big > and complex, just be confident. [...] And to add even more to that: before embarking on micro-optimizations of this sort, always check with a profiler whether or not the bottleneck is even in that part of the code. Often I find myself very surprised at where the real bottleneck is, which is often nowhere near where I thought it should be. Also, check with a memory profiler to find out where the real heavy memory usage points are. It may not be where you thought it was. Generally speaking, in this day and age of highly-optimizing compilers, premature optimization is the root of all evils, because it uglifies your code and makes it hard to maintain for little or no gain, and sometimes for *negative* gain, because by writing code in an unusual way, you confuse the optimizer as to your real intent, thereby reducing its effectiveness at producing optimized code. Don't optimize until you have verified with a profiler where your bottlenecks are. It takes a lot of time and effort to write code this way, so make it count by applying it where it actually matters. Of course, this assumes you use a compiler with a powerful-enough optimizer. I recommend ldc/gdc if performance is important to you. Dmd compiles somewhat faster, but at the cost of poorer codegen. T -- The early bird gets the worm. Moral: ewww...
Re: Block statements and memory management
On Saturday, 16 March 2019 at 03:47:43 UTC, Murilo wrote: Does anyone know if when I create a variable inside a scope as in {int a = 10;} it disappears complete from the memory when the scope finishes? Or does it remain in some part of the memory? I am thinking of using scopes to make optimized programs that consume less memory. Others have made good points in this thread, but what is missing is that indeed scopes _can_ be used beneficially to reduce memory footprint. I recommend playing with this code on d.godbolt.org: ``` void func(ref int[10] a); // important detail: pointer void foo() { { int[10] a; func(a); } { int[10] b; func(b); } } ``` Because the variable is passed by reference (pointer), the optimizer cannot merge the storage space of `a` and `b` _unless_ scope information is taken into account. Without taking scope into account, the first `func` call could store the pointer to `a` somewhere for later use in the second `func` call for example. However, because of scope, using `a` after its scope has ended is UB, and thus variables `a` and `b` can be used. GDC uses scope information for variable lifetime optimization, but LDC and DMD both do not. For anyone interested in working on compilers: adding variable scope lifetime to LDC (not impossibly hard) would be a nice project and be very valuable. -Johan
Re: Block statements and memory management
On Saturday, 16 March 2019 at 15:53:26 UTC, Johan Engelen wrote: On Saturday, 16 March 2019 at 03:47:43 UTC, Murilo wrote: Does anyone know if when I create a variable inside a scope as in {int a = 10;} it disappears complete from the memory when the scope finishes? Or does it remain in some part of the memory? I am thinking of using scopes to make optimized programs that consume less memory. Others have made good points in this thread, but what is missing is that indeed scopes _can_ be used beneficially to reduce memory footprint. -Johan I would like to thank everyone for your help, those informations were very helpful.
Memory management by interfacing C/C++
Hi, I am wrapping some C++ code for my personal project (opencvd), and I am creating so many array pointers at cpp side and containing them in structs. I want to learn if I am leaking memory like crazy, although I am not facing crashes so far. Is GC of D handling things for me? Here is an example: ``` //declaration in d struct IntVector { int* val; int length; } // in cpp typedef struct IntVector { int* val; int length; } IntVector; // cpp function returning a struct containing an array pointer allocated with "new" op. IntVector Subdiv2D_GetLeadingEdgeList(Subdiv2D sd){ std::vector iv; sd->getLeadingEdgeList(iv); int *cintv = new int[iv.size()]; // I don't call delete anywhere? for(size_t i=0; i < iv.size(); i++){ cintv[i] = iv[i]; } IntVector ret = {cintv, (int)iv.size()}; return ret; }; // call extern c function in d: extern (C) IntVector Subdiv2D_GetLeadingEdgeList(Subdiv2d sd); int[] getLeadingEdgeList(){ IntVector intv = Subdiv2D_GetLeadingEdgeList(this); int[] ret = intv.val[0..intv.length]; // just D magic. Still no delete anywhere! return ret; } ``` The question is now: what will happen to "int *cintv" which is allocated with new operator in cpp code? I have many code similar in the project, but I have not encounter any problem so far even in looped video processings. Is GC of D doing deallocation automagically? https://github.com/aferust/opencvd
Re: Memory management and garbage collectors
JMRyan: > In theory, garbage collectors make memory leaks a thing of the past. Even with a perfect GC you may leave around references that keep alive some data that you will never need to use. This is a kind of memory leak. And the current D GC is not fully precise, this means that sometimes it sees as possible pointers even stuff that is not a pointer, so noise may keep alive memory blocks that are not really referenced. > So take a for- > instance. Consider a circular linked list that we are done with. Each > node has a reference to it (from another node in the list). But, being > done with the list, we imagine that there are no references from outside > the list to any node. Is D's garbage collector smart enough to deal > recognize the circle of nodes as garbage? It's not a matter of being smart, it's a matter of how its algorithm works. In a GC based on reference counts, you keep an object alive if its count is nonzero, this means if one or more references point to it. In a GC like the D one (well, in a basic implementation of its idea), you have some starting pointers, the roots, and starting form them you explore the graph of the references, and along the way you mark the visited objects. At the end you scan the list of all the objects and delete the ones with no mark. So the presence of a cycle is not a problem. A cycle is recycled if no external pointers point to it. CPython GC is a reference counter, but they have added extra logic to recognize and recycle cycles too. I leave other of your questions to other people. Bye, bearophile
Re: Memory management and garbage collectors
Thank you for your reply. It was helpful.
Re: toStringz, and memory management, also fromStringz
On Saturday, September 10, 2011 13:00:02 Joel Christensen wrote: > Hi, > > In the std.string document at toStringz it has this note: > > Important Note: When passing a char* to a C function, and the C function > keeps it around for any reason, make sure that you keep a reference to > it in your D code. Otherwise, it may go away during a garbage collection > cycle and cause a nasty bug when the C code tries to use it. > > What does it mean, and what is an example of how to do it? > > It's not done like this is it? > > char* a = cast(char*)toStringz( "test" ); > char* b = cast(char*)toStringz( "word" ); > hold ~= a; > hold ~= b; > > char* cstr2 = al_get_config_value( cfg, a, b ); > > My programs using a C library do crash when exiting some times. It means that you need to keep a pointer to the result of toStringz around if the C function that you pass it to keeps a pointer to it. As long as your D code still has a copy of the pointer, the GC won't collect it. But if your D code doesn't have a copy of the pointer, then the GC may choose to collect it at some point. And once it collects it, the C code which kept the pointer has a pointer to freed memory and will have memory corruption problems (which could result in crash among other things). If the C code doesn't keep a pointer to the data that you pass to it, then it's not an issue. It's just a problem if the C code retains a pointer to it. > Also, how do I go the other way round, some thing like toDString. I've > made my own version. I can't seem to find fromStringz. If you want to convert from a pointer to a string, use std.conv.to. It should do it. If it doesn't, it's a bug. - Jonathan M Davis
Re: toStringz, and memory management, also fromStringz
On 10-Sep-11 3:09 PM, Jonathan M Davis wrote: Jonathan M Davis Ok, now I have a better idea with char pointers. And the std.conv.to worked too. Thanks. - Joelcnz
non-determinant object lifetime and memory management
I need to pass around some objects(specifically int[]) that may be used by several other objects at the same time. While I could clone these and free them when the parent object is done this wastes memory for no real reason except ease of use. Since many objects may contain a ptr to the array, what would be the best way to deal with deallocating them? I could wrap the array in a collection an use ARC but is there a better way? Is there something in std.allocators that can help? (Should be obvious that I'm trying to avoid the GC)
Re: Rust style memory management in D?
Please post on[0] regarding better memory management. As currently work is being done on rewriting the GC (which really was needed). [0] http://forum.dlang.org/post/lao9fn$1d70$1...@digitalmars.com
Re: Region-based memory management and GC?
On Friday, 29 September 2017 at 22:13:01 UTC, Jon Degenhardt wrote: Have there been any investigations into using region-based memory management (aka memory arenas) in D, possibly in conjunction with GC allocated memory? This would be a very speculative idea, but it'd be interesting to know if there have been looks at this area. My own interest is request-response applications, where memory allocated as part of a specific request can be discarded as a single block when the processing of that request completes, without running destructors. I've also seen some papers describing GC systems targeting big data platforms that incorporate this idea. eg. http://www.ics.uci.edu/~khanhtn1/papers/osdi16.pdf --Jon Sounds like just want to use https://dlang.org/phobos/std_experimental_allocator_building_blocks_region.html.
Re: Region-based memory management and GC?
On Saturday, 30 September 2017 at 07:41:21 UTC, Igor wrote: On Friday, 29 September 2017 at 22:13:01 UTC, Jon Degenhardt wrote: Have there been any investigations into using region-based memory management (aka memory arenas) in D, possibly in conjunction with GC allocated memory? Sounds like just want to use https://dlang.org/phobos/std_experimental_allocator_building_blocks_region.html. Wow, thanks, I did not know about this. Will check it out.
Re: Classes and Structs, Memory management questions
i haven't read it fully yet, but i think this DIP contains some or all of my concerns https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md
Re: Classes and Structs, Memory management questions
On Friday, 2 September 2016 at 08:54:33 UTC, dom wrote: i haven't read it fully yet, but i think this DIP contains some or all of my concerns https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md Check this: https://dlang.org/phobos/std_experimental_allocator.html
Re: Classes and Structs, Memory management questions
On Friday, 2 September 2016 at 08:59:38 UTC, Andrea Fontana wrote: On Friday, 2 September 2016 at 08:54:33 UTC, dom wrote: i haven't read it fully yet, but i think this DIP contains some or all of my concerns https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md Check this: https://dlang.org/phobos/std_experimental_allocator.html thx that is very interesting. it seems like that can cover very complex allocation schemes with a general interface! i have also found this to allocate a class on the stack http://dlang.org/phobos/std_typecons.html#.scoped class A { ... } auto instance = scoped!A(); that has even quite a nice syntax!
Re: Classes and Structs, Memory management questions
On Friday, 2 September 2016 at 08:43:45 UTC, dom wrote: from what i got Classes are always reference types and structs are value types in D. i am wondering why that is. for me the main difference between classes and structs is not how they are allocated, but that one has inhertiance, and the other hasn't. Supporting inheritance has some overhead, so at least the split between classes and structs makes sense to me. How instances in an inheritance tree are allocated is actually an important consideration, particularly when it comes to object slicing. In C++, this can be a problem: ``` class Foo {}; class Bar : public Foo {}; Bar bar; Foo foo = bar; ``` All of the information about the type Bar is lost in the assignment to foo. The same thing is going to happen when passing bar to a function that takes a Foo by value as a parameter. The only way to avoid the problem is to pass by reference (or pointer). In Modern C++, with move semantics being a thing, passing by value is much more common than it used to be, but this is the sort of thing it's easy either not to know or to forget about. In D, you don't have to worry about it. I read somewhere (in old forum discussions or an old article) that object slicing is one of the motivations behind the distinction in D.
Re: Classes and Structs, Memory management questions
On Friday, 2 September 2016 at 08:43:45 UTC, dom wrote: from what i got Classes are always reference types and structs are value types in D. i am wondering why that is. for me the main difference between classes and structs is not how they are allocated, but that one has inhertiance, and the other hasn't. It depends by language you're using. In C++, for example you can inherit both! The only difference AFAIK is that c++ structs has public default inheritance vs private for classes. Andrea
Re: Classes and Structs, Memory management questions
On Friday, 2 September 2016 at 08:43:45 UTC, dom wrote: Since garbage collection is a very nice feature that I wouldn't wanna miss for certain scenarios I think D should give us the opportunity to determine how an object is allocated. In the example above putting it on the stack is probably a good idea. Having a self managed reference to the heap can be good too if manual memory management is wanted. Or of course let the GC manage it ( i love it for prototyping code and also as a D beginner it is beneficial that i just dont need to care about memory management). Could somebody explain to me if this is seen as a problem why/whynot and how I should address that kind of issues in my code? You can allocate class instance on stack: https://dlang.org/phobos/std_typecons.html#.scoped
Re: D Beginner Trying Manual Memory Management
On Mon, 12 Jan 2015 19:29:53 + jmh530 via Digitalmars-d-learn wrote: the proper answer is too long to write (it will be more an article that a forum answer ;-), so i'll just give you some directions: import std.typecons; { auto b = scoped!B(); // `auto` is important here! ... } `scoped!` allocating class instance *on* *stack*, and automatically calls destructor when object goes out of scope. but you'd better consider using struct for such things, as struct are stack-allocated by default (unlike classes, which are reference type and should be allocated manually). there is a big difference between `class` and `struct` in D, much bigger that in C++ (where it's only about default protection, actually). signature.asc Description: PGP signature
Re: D Beginner Trying Manual Memory Management
Thanks for the reply, I wasn't familiar with scoped. I was aware that structs are on the stack and classes are on the heap in D, but I didn't know it was possible to put a class on the stack. Might be interesting to see how this is implemented. After looking up some more C++, I think what I was trying to do is more like make_unique than unique_ptr. On Monday, 12 January 2015 at 19:42:14 UTC, ketmar via Digitalmars-d-learn wrote: On Mon, 12 Jan 2015 19:29:53 + jmh530 via Digitalmars-d-learn wrote: the proper answer is too long to write (it will be more an article that a forum answer ;-), so i'll just give you some directions: import std.typecons; { auto b = scoped!B(); // `auto` is important here! ... } `scoped!` allocating class instance *on* *stack*, and automatically calls destructor when object goes out of scope. but you'd better consider using struct for such things, as struct are stack-allocated by default (unlike classes, which are reference type and should be allocated manually). there is a big difference between `class` and `struct` in D, much bigger that in C++ (where it's only about default protection, actually).
Re: D Beginner Trying Manual Memory Management
On Mon, 12 Jan 2015 20:14:19 + jmh530 via Digitalmars-d-learn wrote: > Thanks for the reply, I wasn't familiar with scoped. I was aware > that structs are on the stack and classes are on the heap in D, > but I didn't know it was possible to put a class on the stack. > Might be interesting to see how this is implemented. actually, there is nothing complicated there (if you don't want to write an universal thing like `emplace!` ;-). it builds a wrapper struct big enough to hold class instance, copies class `.init` there and calls class' constructor. the rest of the magic is done by the compiler: when struct goes out of scope, compiler calls struct destructor, which in turn calls class destructor. ah, and it forwards all other requests with `alias this` trick. > After looking up some more C++, I think what I was trying to do > is more like make_unique than unique_ptr. i don't remember C++ well, but nevertheless i encouraging you to take a look at `std.typecons`. there are some handy things there, like `Rebindable!` or `Nullable!`. and some funny things like `BlackHole!` and `WhiteHole!`. ;-) it even has `RefCounted!`, but it doesn't play well with classes yet (AFAIR). signature.asc Description: PGP signature
Re: D Beginner Trying Manual Memory Management
On Monday, 12 January 2015 at 20:30:45 UTC, ketmar via Digitalmars-d-learn wrote: it even has `RefCounted!`, but it doesn't play well with classes yet (AFAIR). I wonder if it's possible to somehow make a version of refcounted that would work with classes (even if limited/restricted in some certain ways), or is it just technically impossible because of reference semantics?
Re: D Beginner Trying Manual Memory Management
On Mon, 12 Jan 2015 21:37:27 + aldanor via Digitalmars-d-learn wrote: > On Monday, 12 January 2015 at 20:30:45 UTC, ketmar via > Digitalmars-d-learn wrote: > > it even has `RefCounted!`, but it doesn't play well with > > classes yet > > (AFAIR). > I wonder if it's possible to somehow make a version of refcounted > that would work with classes (even if limited/restricted in some > certain ways), or is it just technically impossible because of > reference semantics? it's hard. especially hard when you considering inheritance (which is not playing well with templates) and yes, ref semantics. on the other side i found myself rarely using classes at all. i mostly writing templates that checks if a passed "thing" has all the neccessary methods and properties in place and just using that. with D metaprogramming abilities (and `alias this` trick ;-) inheritance becomes not so important. and so classes too. sometimes i'm using structs with delegate fields to simulate some sort of "virtual methods" 'cause i tend to constantly forgetting about that `class` thingy. ;-) OOP is overrated. at least c++-like (should i say "simula-like"?) OOP. ;-) signature.asc Description: PGP signature
Re: D Beginner Trying Manual Memory Management
On Monday, 12 January 2015 at 21:54:51 UTC, ketmar via Digitalmars-d-learn wrote: On Mon, 12 Jan 2015 21:37:27 + aldanor via Digitalmars-d-learn wrote: On Monday, 12 January 2015 at 20:30:45 UTC, ketmar via Digitalmars-d-learn wrote: > it even has `RefCounted!`, but it doesn't play well with > classes yet > (AFAIR). I wonder if it's possible to somehow make a version of refcounted that would work with classes (even if limited/restricted in some certain ways), or is it just technically impossible because of reference semantics? it's hard. especially hard when you considering inheritance (which is not playing well with templates) and yes, ref semantics. on the other side i found myself rarely using classes at all. i mostly writing templates that checks if a passed "thing" has all the neccessary methods and properties in place and just using that. with D metaprogramming abilities (and `alias this` trick ;-) inheritance becomes not so important. and so classes too. sometimes i'm using structs with delegate fields to simulate some sort of "virtual methods" 'cause i tend to constantly forgetting about that `class` thingy. ;-) OOP is overrated. at least c++-like (should i say "simula-like"?) OOP. ;-) I see, thanks! :) I've started liking structs more and more recently as well and been pondering on how to convert a class-based code that looks like this (only the base class has any data): class Base { T m_variable; } class Common : Base { /* tons of methods; uses m_variable */ } class Extra : Base { /* another ton of methods; uses m_variable */ } class A : Extra, Common { ... } class B : Common { ... } class C : Extra { ... } to refcounted structs with alias this but couldn't quite figure out how to do it (other than use mixin templates...). Even if the multiple alias this DIP was implemented, I don't think it would help much here :/
Re: D Beginner Trying Manual Memory Management
On Monday, 12 January 2015 at 19:29:54 UTC, jmh530 wrote: I'm new to D. I have some modest knowledge of C++, but am more familiar with scripting languages (Matlab, Python, R). D seems so much easier than C++ in a lot of ways (and I just learned about rdmd today, which is pretty cool). I am concerned about performance of D vs. C++, so I wanted to learn a little bit more about manual memory management, in case I might ever need it (not for any particular application). There is a good article on the D Wiki that covers this topic with several different patterns and working examples: http://wiki.dlang.org/Memory_Management I hope you'll find it helpful. Mike
Re: D Beginner Trying Manual Memory Management
I had seen some stuff on alias thing, but I hadn't bothered to try to understand it until now. If I'm understanding the first example http://dlang.org/class.html#AliasThis";>here, alias this let's you refer to x in s by writing either s.x (as normally) or just s. That didn't seem that interesting, but then I found http://3d.benjamin-thaut.de/?p=90";>example where they alias this'ed a struct method. That's pretty interesting. OOP seems like a good idea, but every time I've written a bunch of classes in C++ or Python, I inevitably wonder to myself why I just spent 5 times as long doing something I could do with functions. Then there's endless discussion about pimpl. On Monday, 12 January 2015 at 21:54:51 UTC, ketmar via Digitalmars-d-learn wrote: On Mon, 12 Jan 2015 21:37:27 + aldanor via Digitalmars-d-learn wrote: On Monday, 12 January 2015 at 20:30:45 UTC, ketmar via Digitalmars-d-learn wrote: > it even has `RefCounted!`, but it doesn't play well with > classes yet > (AFAIR). I wonder if it's possible to somehow make a version of refcounted that would work with classes (even if limited/restricted in some certain ways), or is it just technically impossible because of reference semantics? it's hard. especially hard when you considering inheritance (which is not playing well with templates) and yes, ref semantics. on the other side i found myself rarely using classes at all. i mostly writing templates that checks if a passed "thing" has all the neccessary methods and properties in place and just using that. with D metaprogramming abilities (and `alias this` trick ;-) inheritance becomes not so important. and so classes too. sometimes i'm using structs with delegate fields to simulate some sort of "virtual methods" 'cause i tend to constantly forgetting about that `class` thingy. ;-) OOP is overrated. at least c++-like (should i say "simula-like"?) OOP. ;-)
Re: D Beginner Trying Manual Memory Management
On Mon, 12 Jan 2015 23:06:16 + jmh530 via Digitalmars-d-learn wrote: > I had seen some stuff on alias thing, but I hadn't bothered to > try to understand it until now. If I'm understanding the first > example http://dlang.org/class.html#AliasThis";>here, > alias this let's you refer to x in s by writing either s.x (as > normally) or just s. That didn't seem that interesting, but then > I found http://3d.benjamin-thaut.de/?p=90";>example > where they alias this'ed a struct method. That's pretty > interesting. there is nice page by p0nce and it shows nice and simple `alias this` trick usage: http://p0nce.github.io/d-idioms/#Extending-a-struct-with-alias-this and i have "stream.d" module in my IV package ( http://repo.or.cz/w/iv.d.git/tree ), which works with "i/o streams" by testing if passed struct/class has necessary methods (`isReadable!`, `isWriteable!`, `isSeekable!`, etc. signature.asc Description: PGP signature
Re: D Beginner Trying Manual Memory Management
On Mon, 12 Jan 2015 22:07:13 + aldanor via Digitalmars-d-learn wrote: > I see, thanks! :) I've started liking structs more and more > recently as well and been pondering on how to convert a > class-based code that looks like this (only the base class has > any data): it's hard to tell by brief description. but having multiple inheritance immediately rings an alarm ring for me. something is very-very-very wrong if you need to have a winged whale. ;-) signature.asc Description: PGP signature
Re: D Beginner Trying Manual Memory Management
On Mon, 12 Jan 2015 22:07:13 + aldanor via Digitalmars-d-learn wrote: > I see, thanks! :) I've started liking structs more and more > recently as well and been pondering on how to convert a > class-based code that looks like this (only the base class has > any data): p.s. can't you convert most of that to free functions? thanks to UFCS you'll be able to use them with `obj.func` notation. and by either defining `package` protection for class fields, or simply writing that functions in the same module they will have access to internal object stuff. and if you can return `obj` from each function, you can go with templates and chaining. ;-) signature.asc Description: PGP signature
Re: D Beginner Trying Manual Memory Management
On Tuesday, 13 January 2015 at 08:33:57 UTC, ketmar via Digitalmars-d-learn wrote: On Mon, 12 Jan 2015 22:07:13 + aldanor via Digitalmars-d-learn wrote: I see, thanks! :) I've started liking structs more and more recently as well and been pondering on how to convert a class-based code that looks like this (only the base class has any data): it's hard to tell by brief description. but having multiple inheritance immediately rings an alarm ring for me. something is very-very-very wrong if you need to have a winged whale. ;-) A real-world example: http://www.hdfgroup.org/HDF5/doc/cpplus_RM/hierarchy.html H5::File is both an H5::Location and H5::CommonFG (but not an H5::Object) H5::Group is both an H5::Object (subclass of H5::Location) and H5::CommonFG H5::Dataset is an H5::Object
Re: D Beginner Trying Manual Memory Management
On Tue, 13 Jan 2015 16:08:15 + aldanor via Digitalmars-d-learn wrote: > On Tuesday, 13 January 2015 at 08:33:57 UTC, ketmar via > Digitalmars-d-learn wrote: > > On Mon, 12 Jan 2015 22:07:13 + > > aldanor via Digitalmars-d-learn > > > > wrote: > > > >> I see, thanks! :) I've started liking structs more and more > >> recently as well and been pondering on how to convert a > >> class-based code that looks like this (only the base class has > >> any data): > > it's hard to tell by brief description. but having multiple > > inheritance > > immediately rings an alarm ring for me. something is > > very-very-very > > wrong if you need to have a winged whale. ;-) > A real-world example: > http://www.hdfgroup.org/HDF5/doc/cpplus_RM/hierarchy.html > > H5::File is both an H5::Location and H5::CommonFG (but not an > H5::Object) > H5::Group is both an H5::Object (subclass of H5::Location) and > H5::CommonFG > H5::Dataset is an H5::Object i see something named "CommonFG" here, which seems to good thing to move out of hierarchy altogether. bwah, i don't even sure that given hierarchy is good for D. C++ has no UFCS, and it's incredibly hard to check if some entity has some methods/properties in C++, so they have no other choice than to work around that limitations. it may be worthful to redesign the whole thing for D, exploiting D shiny UFCS and metaprogramming features. and, maybe, moving some things to interfaces too. signature.asc Description: PGP signature
Re: D Beginner Trying Manual Memory Management
>> I see, thanks! :) I've started liking structs more and more >> recently as well and been pondering on how to convert a >> class-based code that looks like this (only the base class >> has any data): > it's hard to tell by brief description. but having multiple > inheritance > immediately rings an alarm ring for me. something is > very-very-very > wrong if you need to have a winged whale. ;-) A real-world example: http://www.hdfgroup.org/HDF5/doc/cpplus_RM/hierarchy.html H5::File is both an H5::Location and H5::CommonFG (but not an H5::Object) H5::Group is both an H5::Object (subclass of H5::Location) and H5::CommonFG H5::Dataset is an H5::Object i see something named "CommonFG" here, which seems to good thing to move out of hierarchy altogether. bwah, i don't even sure that given hierarchy is good for D. C++ has no UFCS, and it's incredibly hard to check if some entity has some methods/properties in C++, so they have no other choice than to work around that limitations. it may be worthful to redesign the whole thing for D, exploiting D shiny UFCS and metaprogramming features. and, maybe, moving some things to interfaces too. I just finished reading aldanor's blog, so I know he is slightly allergic to naked functions and prefers classes ;) With Ketmar, I very much agree (predominantly as a user of HDF5 and less so as an inexperienced D programmr writing a wrapper for it). It's a pain to figure out just how to do simple things until you know the H5 library. You have to create an object for file permissions before you even get started, then similarly for the data series (datasets) within, another for the dimensions of the array, etc etc - that doesn't fit with the intrinsic nature of the domain. There is a more general question of bindings/wrappers - preserve the original structure and naming so existing code can be ported, or write a wrapper that makes it easy for the user to accomplish his objectives. It seems like for the bindings preserving the library structure is fine, but for the wrapper one might as well make things easy. Eg here https://gist.github.com/Laeeth/9637233db41a11a9d1f4 line 146. (sorry for duplication and messiness of code, which I don't claim to be perfectly written - I wanted to try something quickly and have not yet tidied up). So rather than navigate the Byzantine hierarchy, one can just do something like this (which will take a struct of PriceBar - date,open,high,low,close - and put it in your desired dataset and file, appending or overwriting as you prefer). dumpDataSpaceVector!PriceBar(file,ticker,array(priceBars[ticker]),DumpMode.truncate); which is closer to h5py in Python. (It uses reflection to figure out the contents of a non-nested struct, but won't yet cope with arrays and nested structs inside). And of course a full wrapper might be a bit more complicated, but I truly think one can do better than mapping the HDF5 hierarchy one for one. Laeeth.
Re: D Beginner Trying Manual Memory Management
On Tue, 13 Jan 2015 17:08:37 + Laeeth Isharc via Digitalmars-d-learn wrote: > I just finished reading aldanor's blog, so I know he is slightly > allergic to naked functions and prefers classes ;) that's due to absense of modules in C/C++. and namespaces aren't of big help here too. and, of course, due to missing UFCS, that prevents nice `obj.func()` for free functions. ;-) signature.asc Description: PGP signature
Re: D Beginner Trying Manual Memory Management
On Tuesday, 13 January 2015 at 17:08:38 UTC, Laeeth Isharc wrote: >> I see, thanks! :) I've started liking structs more and >> more recently as well and been pondering on how to convert >> a class-based code that looks like this (only the base >> class has any data): > it's hard to tell by brief description. but having multiple > inheritance > immediately rings an alarm ring for me. something is > very-very-very > wrong if you need to have a winged whale. ;-) A real-world example: http://www.hdfgroup.org/HDF5/doc/cpplus_RM/hierarchy.html H5::File is both an H5::Location and H5::CommonFG (but not an H5::Object) H5::Group is both an H5::Object (subclass of H5::Location) and H5::CommonFG H5::Dataset is an H5::Object i see something named "CommonFG" here, which seems to good thing to move out of hierarchy altogether. bwah, i don't even sure that given hierarchy is good for D. C++ has no UFCS, and it's incredibly hard to check if some entity has some methods/properties in C++, so they have no other choice than to work around that limitations. it may be worthful to redesign the whole thing for D, exploiting D shiny UFCS and metaprogramming features. and, maybe, moving some things to interfaces too. I just finished reading aldanor's blog, so I know he is slightly allergic to naked functions and prefers classes ;) With Ketmar, I very much agree (predominantly as a user of HDF5 and less so as an inexperienced D programmr writing a wrapper for it). It's a pain to figure out just how to do simple things until you know the H5 library. You have to create an object for file permissions before you even get started, then similarly for the data series (datasets) within, another for the dimensions of the array, etc etc - that doesn't fit with the intrinsic nature of the domain. There is a more general question of bindings/wrappers - preserve the original structure and naming so existing code can be ported, or write a wrapper that makes it easy for the user to accomplish his objectives. It seems like for the bindings preserving the library structure is fine, but for the wrapper one might as well make things easy. Eg here https://gist.github.com/Laeeth/9637233db41a11a9d1f4 line 146. (sorry for duplication and messiness of code, which I don't claim to be perfectly written - I wanted to try something quickly and have not yet tidied up). So rather than navigate the Byzantine hierarchy, one can just do something like this (which will take a struct of PriceBar - date,open,high,low,close - and put it in your desired dataset and file, appending or overwriting as you prefer). dumpDataSpaceVector!PriceBar(file,ticker,array(priceBars[ticker]),DumpMode.truncate); which is closer to h5py in Python. (It uses reflection to figure out the contents of a non-nested struct, but won't yet cope with arrays and nested structs inside). And of course a full wrapper might be a bit more complicated, but I truly think one can do better than mapping the HDF5 hierarchy one for one. Laeeth. In the hierarchy example above (c++ hdf hierarchy link), by using UFCS to implement the shared methods (which are achieved by multiple inheritance in the c++ counterpart) did you mean something like this? // id.d struct ID { int id; ... } // location.d struct Location { ID _id; alias _id this; ... } // file.d public import commonfg; // ugh struct File { Location _location; alias _location this; ... } // group.d public import commonfg; struct File { Location _location; alias _location this; ... } // commonfg.d { ... } enum isContainer(T) = is(T: File) || is(T : Group); auto method1(T)(T obj, args) if (isContainer!T) { ... } auto method2(T)(T obj, args) if (isContainer!T) { ... } I guess two of my gripes with UFCS is (a) you really have to use public imports in the modules where the target types are defined so you bring all the symbols in whether you want it or not (b) you lose access to private members because it's not the same module anymore (correct me if I'm wrong?). Plus, you need to decorate every single free function with a template constraint. // another hdf-specific thing here but a good example in general is that some functions return you an id for an object which is one of the location subtypes (e.g. it could be a File or could be a Group depending on run-time conditions), so it kind of feels natural to use polymorphism and classes for that, but what would you do with the struct approach? The only thing that comes to mind is Variant, but it's quite meh to use in practice.
Re: D Beginner Trying Manual Memory Management
On Tue, 13 Jan 2015 18:35:15 + aldanor via Digitalmars-d-learn wrote: > I guess two of my gripes with UFCS is (a) you really have to use > public imports in the modules where the target types are defined > so you bring all the symbols in whether you want it or not (b) > you lose access to private members because it's not the same > module anymore (correct me if I'm wrong?). Plus, you need to > decorate every single free function with a template constraint. you can make a package and set protection to `package` instead of `private`, so your function will still be able to access internal fields, but package users will not. this feature is often missed by the people who are used to `public`/`protected`/`private` triad. signature.asc Description: PGP signature
Re: D Beginner Trying Manual Memory Management
In the hierarchy example above (c++ hdf hierarchy link), by using UFCS to implement the shared methods (which are achieved by multiple inheritance in the c++ counterpart) did you mean something like this? // id.d struct ID { int id; ... } // location.d struct Location { ID _id; alias _id this; ... } // file.d public import commonfg; // ugh struct File { Location _location; alias _location this; ... } // group.d public import commonfg; struct File { Location _location; alias _location this; ... } // commonfg.d { ... } enum isContainer(T) = is(T: File) || is(T : Group); auto method1(T)(T obj, args) if (isContainer!T) { ... } auto method2(T)(T obj, args) if (isContainer!T) { ... } I guess two of my gripes with UFCS is (a) you really have to // another hdf-specific thing here but a good example in general is that some functions return you an id for an object which is one of the location subtypes (e.g. it could be a File or could be a Group depending on run-time conditions), so it kind of feels natural to use polymorphism and classes for that, but what would you do with the struct approach? The only thing that comes to mind is Variant, but it's quite meh to use in practice. Void unlink(File f){} Void unlink(Group g){} For simple cases maybe one can keep it simple, and despite the Byzantine interface what one is trying to do when using HDF5 is not intrinsically so complex.
Re: D Beginner Trying Manual Memory Management
On Wednesday, 14 January 2015 at 14:54:09 UTC, Laeeth Isharc wrote: In the hierarchy example above (c++ hdf hierarchy link), by using UFCS to implement the shared methods (which are achieved by multiple inheritance in the c++ counterpart) did you mean something like this? // id.d struct ID { int id; ... } // location.d struct Location { ID _id; alias _id this; ... } // file.d public import commonfg; // ugh struct File { Location _location; alias _location this; ... } // group.d public import commonfg; struct File { Location _location; alias _location this; ... } // commonfg.d { ... } enum isContainer(T) = is(T: File) || is(T : Group); auto method1(T)(T obj, args) if (isContainer!T) { ... } auto method2(T)(T obj, args) if (isContainer!T) { ... } I guess two of my gripes with UFCS is (a) you really have to // another hdf-specific thing here but a good example in general is that some functions return you an id for an object which is one of the location subtypes (e.g. it could be a File or could be a Group depending on run-time conditions), so it kind of feels natural to use polymorphism and classes for that, but what would you do with the struct approach? The only thing that comes to mind is Variant, but it's quite meh to use in practice. Void unlink(File f){} Void unlink(Group g){} For simple cases maybe one can keep it simple, and despite the Byzantine interface what one is trying to do when using HDF5 is not intrinsically so complex. So your solution is copying and pasting the code? But now repeat that for 200 other functions and a dozen more types that can be polymorphic in weirdest ways possible...
Re: D Beginner Trying Manual Memory Management
struct File { Location _location; alias _location this; ... } // group.d public import commonfg; struct File { Location _location; alias _location this; ... } // commonfg.d { ... } enum isContainer(T) = is(T: File) || is(T : Group); auto method1(T)(T obj, args) if (isContainer!T) { ... } auto method2(T)(T obj, args) if (isContainer!T) { ... } I guess two of my gripes with UFCS is (a) you really have to // another hdf-specific thing here but a good example in general is that some functions return you an id for an object which is one of the location subtypes (e.g. it could be a File or could be a Group depending on run-time conditions), so it kind of feels natural to use polymorphism and classes for that, but what would you do with the struct approach? The only thing that comes to mind is Variant, but it's quite meh to use in practice. Void unlink(File f){} Void unlink(Group g){} For simple cases maybe one can keep it simple, and despite the Byzantine interface what one is trying to do when using HDF5 is not intrinsically so complex. So your solution is copying and pasting the code? But now repeat that for 200 other functions and a dozen more types that can be polymorphic in weirdest ways possible... If you are simply have a few lines calling the API and the validation is different enough for file and group (I haven't written unlink yet) then why not (and move proper shared code out into helper functions). The alternative is a long method with lots of conditions, which may be the best in some cases but may be harder to follow. I do like the h5py and pytables approaches. One doesn't need to bother too much with the implementation when using their library. However, what I am doing is quite simple from a data perspective - a decent amount of it, but it is not an interesting problem from a theoretical perspective - just execution. Now if you are higher octane as a user you may be able to see what I cannot. But on the other hand, the Pareto principle applies, and in my view a library should make it simple to do simple things. One can't get there if the primary interface is a direct mapping of the HDF5 hierarchy, and I also think that is unnecessary with D. But I very much appreciate your work as the final result is better for everyone that way, and you are evidently a much longer running user of D than me. I never used C++ as it just seemed too ugly! and I suspect the difference in backgrounds is shaping perspectives. What do you think the trickiest parts are with HDF5? (You mention weird polymorphism). Laeeth
Re: D Beginner Trying Manual Memory Management
On Wednesday, 14 January 2015 at 16:27:17 UTC, Laeeth Isharc wrote: struct File { Location _location; alias _location this; ... } // group.d public import commonfg; struct File { Location _location; alias _location this; ... } // commonfg.d { ... } enum isContainer(T) = is(T: File) || is(T : Group); auto method1(T)(T obj, args) if (isContainer!T) { ... } auto method2(T)(T obj, args) if (isContainer!T) { ... } I guess two of my gripes with UFCS is (a) you really have to // another hdf-specific thing here but a good example in general is that some functions return you an id for an object which is one of the location subtypes (e.g. it could be a File or could be a Group depending on run-time conditions), so it kind of feels natural to use polymorphism and classes for that, but what would you do with the struct approach? The only thing that comes to mind is Variant, but it's quite meh to use in practice. Void unlink(File f){} Void unlink(Group g){} For simple cases maybe one can keep it simple, and despite the Byzantine interface what one is trying to do when using HDF5 is not intrinsically so complex. So your solution is copying and pasting the code? But now repeat that for 200 other functions and a dozen more types that can be polymorphic in weirdest ways possible... If you are simply have a few lines calling the API and the validation is different enough for file and group (I haven't written unlink yet) then why not (and move proper shared code out into helper functions). The alternative is a long method with lots of conditions, which may be the best in some cases but may be harder to follow. I do like the h5py and pytables approaches. One doesn't need to bother too much with the implementation when using their library. However, what I am doing is quite simple from a data perspective - a decent amount of it, but it is not an interesting problem from a theoretical perspective - just execution. Now if you are higher octane as a user you may be able to see what I cannot. But on the other hand, the Pareto principle applies, and in my view a library should make it simple to do simple things. One can't get there if the primary interface is a direct mapping of the HDF5 hierarchy, and I also think that is unnecessary with D. But I very much appreciate your work as the final result is better for everyone that way, and you are evidently a much longer running user of D than me. I never used C++ as it just seemed too ugly! and I suspect the difference in backgrounds is shaping perspectives. What do you think the trickiest parts are with HDF5? (You mention weird polymorphism). Laeeth I don't think you've read h5py source in enough detail :) It's based HEAVILY on duck typing. In addition, it has way MORE classes than the C++ hierarchy does. E.g., the high-level File object actually has these parents: File : Group, Group : HLObject, MutableMappingWithLock, HLObject : CommonStateObject and internally the File also keeps a reference to file id which is an instance of FileID which inherits from GroupID which inherits from ObjectID, do I need to continue? :) PyTables, on the contrary is quite badly written (although it works quite well and there are brilliant folks on the dev team like francesc alted) and looks like a dump of C code interweaved with hackish Python code. In h5py you can do things like file["/dataset"].write(...) --> this just wouldn't work as is in a strictly typed language since the indexing operator generally returns you something of a Location type (or an interface, rather) which can be a group/datatype/dataset which is only known at runtime. Out of all of them, only the dataset supports the write method but you don't know it's going to be a dataset. See the problem? I don't want the user code to deal with any of the HDF5 C API and/or have a bunch of if conditions or explicit casts which is outright ugly. Ideally, it would work kind of like H5PY, abstracting the user away from refcounting, error code checking after each operation, object type checking and all that stuff.
Re: D Beginner Trying Manual Memory Management
I don't think you've read h5py source in enough detail :) You're right - I haven't done more than browsed it. It's based HEAVILY on duck typing. There is a question here about what to do in D. On the one hand, the flexibility of being able to open a foreign HDF5 file where you don't know beforehand the dataset type is very nice. On the other, the adaptations needed to handle this flexibly get in the way when you are dealing with your own data that has a set format and where recompilation is acceptable if it changes. Looking at the 'ease' of processing JSON, even using vibed, I think that one will need to implement both eventually, but perhaps starting with static typing. In addition, it has way MORE classes than the C++ hierarchy does. E.g., the high-level File object actually has these parents: File : Group, Group : HLObject, MutableMappingWithLock, HLObject : CommonStateObject and internally the File also keeps a reference to file id which is an instance of FileID which inherits from GroupID which inherits from ObjectID, do I need to continue? Okay - I guess there is a distinction between the interface to the outside world (where I think the h5py etc way is superior for most uses) and the implementation. Is not the reason h5py has lots of classes primarily because that is how you write good code in python, whereas in many cases this is not true in D (not that you should ban classes, but often structs + free floating functions are more suitable). PyTables, on the contrary is quite badly written (although it works quite well and there are brilliant folks on the dev team like francesc alted) and looks like a dump of C code interweaved with hackish Python code. Interesting. What do you think is low quality about the design? In h5py you can do things like file["/dataset"].write(...) --> this just wouldn't work as is in a strictly typed language since the indexing operator generally returns you something of a Location type (or an interface, rather) which can be a group/datatype/dataset which is only known at runtime. Well, if you don't mind recompiling your code when the data set type changes (or you encounter a new data set) then you can do that (which is what I posted a link to earlier). It depends on your use case. It's hard to think of an application more dynamic than web sites, and yet people seem happy enough with vibed's use of compiled diet templates as the primary implementation. They would like the option of dynamic ones too, and I think this would be useful in this domain too, since one does look at foreign data on occasion. One could of course use the quick compilation of D to regenerate parts of the code when this happens. Whether or not this is acceptable depends on your use case - for some it might be okay, but obviously it is no good if you are writing a generic H5 browser/charting tool. So I think if you don't allow static dataset typing it means the flexibility of dynamic typing gets in the way for some uses (which might be most of them), but you need to add dynamic typing too. Shall we move this to a different thread and/or email, as I am afraid I have hijacked the poor original poster's request. On the refcounting question, I confess that I do not fully understand your concern, which may well reflect a lack of deep experience with D on my part. Adam Ruppe suggests that it's generally okay to rely on a struct destructor to call C cleanup code. I can appreciate this may not be true with h5 and, if you can spare the time, I would love to understand more precisely why not. Out of all of them, only the dataset supports the write method but you don't know it's going to be a dataset. See the problem? In this case I didn't quite follow. Where does this fall down ? void h5write(T)(Dataset x, T data) I have your email somewhere and will drop you a line. Or you can email me laeeth at laeeth.com. And let's create a new thread. Laeeth.
Re: Memory management by interfacing C/C++
On Saturday, 27 April 2019 at 22:25:58 UTC, Ferhat Kurtulmuş wrote: Hi, I am wrapping some C++ code for my personal project (opencvd), and I am creating so many array pointers at cpp side and containing them in structs. I want to learn if I am leaking memory like crazy, although I am not facing crashes so far. Is GC of D handling things for me? Here is an example: [...] The question is now: what will happen to "int *cintv" which is allocated with new operator in cpp code? I have many code similar in the project, but I have not encounter any problem so far even in looped video processings. Is GC of D doing deallocation automagically? https://github.com/aferust/opencvd D's GC only collects memory allocated with D's `new` operator. Memory allocated by C++'s `new` operator must be freed by C++'s `delete` operator.
Re: Memory management by interfacing C/C++
On Sunday, 28 April 2019 at 03:54:17 UTC, Paul Backus wrote: On Saturday, 27 April 2019 at 22:25:58 UTC, Ferhat Kurtulmuş wrote: Hi, I am wrapping some C++ code for my personal project (opencvd), and I am creating so many array pointers at cpp side and containing them in structs. I want to learn if I am leaking memory like crazy, although I am not facing crashes so far. Is GC of D handling things for me? Here is an example: [...] The question is now: what will happen to "int *cintv" which is allocated with new operator in cpp code? I have many code similar in the project, but I have not encounter any problem so far even in looped video processings. Is GC of D doing deallocation automagically? https://github.com/aferust/opencvd D's GC only collects memory allocated with D's `new` operator. Memory allocated by C++'s `new` operator must be freed by C++'s `delete` operator. You are right. I am rewriting the things using mallocs, and will use core.stdc.stdlib.free on d side. I am not sure if I can use core.stdc.stdlib.free to destroy arrays allocated with new op.
Re: Memory management by interfacing C/C++
On Sunday, 28 April 2019 at 23:10:24 UTC, Ferhat Kurtulmuş wrote: You are right. I am rewriting the things using mallocs, and will use core.stdc.stdlib.free on d side. I am not sure if I can use core.stdc.stdlib.free to destroy arrays allocated with new op. core.stdc.stdlib.free is (as the name suggests) the standard C `free` function. As such, it can only be used to free memory allocated by the standard C functions `malloc`, `calloc`, and `realloc`. This is the same in D as it is in C and C++.
Re: Memory management by interfacing C/C++
On Monday, 29 April 2019 at 00:53:34 UTC, Paul Backus wrote: On Sunday, 28 April 2019 at 23:10:24 UTC, Ferhat Kurtulmuş wrote: You are right. I am rewriting the things using mallocs, and will use core.stdc.stdlib.free on d side. I am not sure if I can use core.stdc.stdlib.free to destroy arrays allocated with new op. core.stdc.stdlib.free is (as the name suggests) the standard C `free` function. As such, it can only be used to free memory allocated by the standard C functions `malloc`, `calloc`, and `realloc`. This is the same in D as it is in C and C++. Thank you. It is now like: /* c/cpp side */ extern (C) void deleteArr(void* arr); void deleteArr(void* arr){ delete[] arr; } struct IntVector Subdiv2D_GetLeadingEdgeList(Subdiv2D sd){ std::vector iv; sd->getLeadingEdgeList(iv); int *cintv = new int[iv.size()]; for(size_t i=0; i < iv.size(); i++){ cintv[i] = iv[i]; } IntVector ret = {cintv, (int)iv.size()}; return ret; }; /* c/cpp side */ ... int[] getLeadingEdgeList(){ // d function IntVector intv = Subdiv2D_GetLeadingEdgeList(this); int[] ret = intv.val[0..intv.length].dup; deleteArr(intv.val); return ret; } ...
Re: Memory management by interfacing C/C++
On Saturday, 27 April 2019 at 22:25:58 UTC, Ferhat Kurtulmuş wrote: Hi, I am wrapping some C++ code for my personal project (opencvd), and I am creating so many array pointers at cpp side and containing them in structs. I want to learn if I am leaking memory like crazy, although I am not facing crashes so far. Is GC of D handling things for me? Here is an example: ``` //declaration in d struct IntVector { int* val; int length; } // in cpp typedef struct IntVector { int* val; int length; } IntVector; // cpp function returning a struct containing an array pointer allocated with "new" op. IntVector Subdiv2D_GetLeadingEdgeList(Subdiv2D sd){ std::vector iv; sd->getLeadingEdgeList(iv); int *cintv = new int[iv.size()]; // I don't call delete anywhere? for(size_t i=0; i < iv.size(); i++){ cintv[i] = iv[i]; } IntVector ret = {cintv, (int)iv.size()}; return ret; }; // call extern c function in d: extern (C) IntVector Subdiv2D_GetLeadingEdgeList(Subdiv2d sd); int[] getLeadingEdgeList(){ IntVector intv = Subdiv2D_GetLeadingEdgeList(this); int[] ret = intv.val[0..intv.length]; // just D magic. Still no delete anywhere! return ret; } ``` The question is now: what will happen to "int *cintv" which is allocated with new operator in cpp code? I have many code similar in the project, but I have not encounter any problem so far even in looped video processings. Is GC of D doing deallocation automagically? https://github.com/aferust/opencvd Hello Ferhat, You can use RCArray!T or Slice!(RCI!T) [1, 2] as common thread safe @nogc types for D and C++ code. See also integration C++ example [3] and C++ headers [4]. RCArray (fixed length) [1] http://mir-algorithm.libmir.org/mir_rc_array.html RCSlice (allows to get subslices) [2] http://mir-algorithm.libmir.org/mir_ndslice_allocation.html#rcslice C++ integration example [3] https://github.com/libmir/mir-algorithm/tree/master/cpp_example C++ headers [4] https://github.com/libmir/mir-algorithm/tree/master/include/mir
Re: Memory management by interfacing C/C++
On Monday, 29 April 2019 at 14:38:54 UTC, 9il wrote: On Saturday, 27 April 2019 at 22:25:58 UTC, Ferhat Kurtulmuş wrote: [...] Hello Ferhat, You can use RCArray!T or Slice!(RCI!T) [1, 2] as common thread safe @nogc types for D and C++ code. See also integration C++ example [3] and C++ headers [4]. RCArray (fixed length) [1] http://mir-algorithm.libmir.org/mir_rc_array.html RCSlice (allows to get subslices) [2] http://mir-algorithm.libmir.org/mir_ndslice_allocation.html#rcslice C++ integration example [3] https://github.com/libmir/mir-algorithm/tree/master/cpp_example C++ headers [4] https://github.com/libmir/mir-algorithm/tree/master/include/mir An opencv d binding using ndslice substituting cv::Mat would be useful like opencv-python using numpy. However, I started opencvd about a month ago, and I am very new to d. For now, I am the only contributor with zero mir.ndslice experience. When I gain more experience with ndslice, I would try it as substitution for cv::Mat. Thank you for links 9il. I will take a look at them.
Example of code with manual memory management
Hi, I am looking for example of types where memory management is manual, and the type supports operator overloading, etc. Grateful if someone could point me to sample example code. Thanks and Regards Dibyendu
Re: non-determinant object lifetime and memory management
Frustrated: I need to pass around some objects(specifically int[]) that may be used by several other objects at the same time. While I could clone these and free them when the parent object is done this wastes memory for no real reason except ease of use. Since many objects may contain a ptr to the array, what would be the best way to deal with deallocating them? I could wrap the array in a collection an use ARC but is there a better way? Is there something in std.allocators that can help? (Should be obvious that I'm trying to avoid the GC) Your use case seems fit for using the GC. Otherwise take a look at std.typecons.RefCounted, to be used in a wrapper that uses "alias this". Bye, bearophile
Re: non-determinant object lifetime and memory management
On Saturday, 30 November 2013 at 08:35:23 UTC, Frustrated wrote: I need to pass around some objects(specifically int[]) that may be used by several other objects at the same time. While I could clone these and free them when the parent object is done this wastes memory for no real reason except ease of use. Since many objects may contain a ptr to the array, what would be the best way to deal with deallocating them? I could wrap the array in a collection an use ARC but is there a better way? Is there something in std.allocators that can help? (Should be obvious that I'm trying to avoid the GC) You can use the Array type in std.container [1]. It uses ref counting and the C heap internally. [1] http://dlang.org/phobos/std_container.html#.Array
Re: non-determinant object lifetime and memory management
On Saturday, 30 November 2013 at 12:51:46 UTC, Rene Zwanenburg wrote: On Saturday, 30 November 2013 at 08:35:23 UTC, Frustrated wrote: I need to pass around some objects(specifically int[]) that may be used by several other objects at the same time. While I could clone these and free them when the parent object is done this wastes memory for no real reason except ease of use. Since many objects may contain a ptr to the array, what would be the best way to deal with deallocating them? I could wrap the array in a collection an use ARC but is there a better way? Is there something in std.allocators that can help? (Should be obvious that I'm trying to avoid the GC) You can use the Array type in std.container [1]. It uses ref counting and the C heap internally. [1] http://dlang.org/phobos/std_container.html#.Array How does it work? When you call clear it decrements the reference count and at 0 it free's the memory? That is the only self managed container in std.container?
Re: non-determinant object lifetime and memory management
On Saturday, 30 November 2013 at 08:35:23 UTC, Frustrated wrote: I need to pass around some objects(specifically int[]) that may be used by several other objects at the same time. While I could clone these and free them when the parent object is done this wastes memory for no real reason except ease of use. Since many objects may contain a ptr to the array, what would be the best way to deal with deallocating them? I could wrap the array in a collection an use ARC but is there a better way? Is there something in std.allocators that can help? (Should be obvious that I'm trying to avoid the GC) Why you do not use one of this way: - const ref int[]… into function parameter - using shared/synchronized and ref to array
Re: non-determinant object lifetime and memory management
On Sunday, 1 December 2013 at 02:29:42 UTC, bioinfornatics wrote: On Saturday, 30 November 2013 at 08:35:23 UTC, Frustrated wrote: I need to pass around some objects(specifically int[]) that may be used by several other objects at the same time. While I could clone these and free them when the parent object is done this wastes memory for no real reason except ease of use. Since many objects may contain a ptr to the array, what would be the best way to deal with deallocating them? I could wrap the array in a collection an use ARC but is there a better way? Is there something in std.allocators that can help? (Should be obvious that I'm trying to avoid the GC) Why you do not use one of this way: - const ref int[]… into function parameter - using shared/synchronized and ref to array It would seem if I am going to use some way it needs to be consistent. The first case would require creating []'s outside of the function which then doesn't solve the original problem. I'm not sure how the second case solves anything?
Re: Example of code with manual memory management
https://dlang.org/phobos/std_container.html and corresponding code in phobos. Though recently allocators were introduced and containers are going to be written with support for allocators.
Just another question about memory management in d from a newbie
Hello! First my background: C++ and Java ages ago. Since then only PLSQL. Now learning D just for fun and personal education on time to time and very pleased about it :-) Now I have to ask a question here, because I could not find a corresponding answer for it. Or I am unable to find it :-) I was wondering about the memory system in D (and other C like languages) about the handling of the memory allocation overhead. Just an example that I have seen many times: int main() { foreach(i;0 .. 1) { int x; // do something with x } return 0; } Do I understand it right that the variable x will be created 1 times and destroyed at the end of the scope in each loop ? Or will it be 1 overwritten by creation ? I mean does it not cost the CPU some time to allocate (I know were talking here of nsec) but work is work. As far I know from my school days in assembler, allocation of memory is one of the most expensive instructions on the cpu. Activate memory block, find some free place, reserve, return pointer to caller, and so on.. In my opinion this version should perform a little better: int main() { int x; foreach(i;0 .. 1) { x = 0; // reinitialize // do something with x } return 0; } Or do I miss something and there is an optimization by the compiler to avoid recreating/destroying the variable x 1 times ? I know that version 1 is more secure because there will be no value waste after each loop we could stumble onto, but that is not my question here. And I know that were talking about really small cpu usage compared to what an app should do. But when it is to look on performance like games or gui (and there are a lot of examples like version 1 out there) then I have to ask myself if it is not just a waste of cpu time ? Or is it a styling of code thing ? Thank you for your time! Greetings from Austria Thomas
Preferred method of creating objects, structs, and arrays with deterministic memory management
I've recently been trying to convince a friend of mine that D has at least the functionality of C++, and have been learning the language over C++ for a few months. Memory management is pretty important to him, and a subject I'm honestly curious about as well. I was wondering: what's the preferred method for deterministic memory management? I don't know much about this topic relative to most (Java background), but this is what I've found: - Classes/Structs have constructors and destructors. I am unconfident with my knowledge as to how this works with malloc and free. - core.memory can be used to call malloc and its buddies. It allows determined management, but will still use the GC if something isn't collected (presumably for leaks). - Many features and types use the GC such as exceptions, the new keyword, and all arrays except statics. - Allocators in "std.experimental.allocators" still use the GC, but give more control for class/struct objects. - std.container.array can be used for deterministic arrays, but has the issue of dangling pointers. There is probably a good solution to this, but I don't know it. - There seems to be no way to completely turn off the GC. That is, never have the runtime allocate the memory used by the GC. - Andreau had a discussion in 2009 about how this issue might be handled, but there didn't seem to be an obvious consensus (http://forum.dlang.org/thread/hafpjn$1cu8$1...@digitalmars.com?page=1) This are the pieces I've gathered, but I don't really know what's true or how to use this knowledge. Some ideas I've gleaned may also be outdated. Does anyone know the "correct" way one would go about a non-GC program, and if a program can be ran without ever instantiating a GC? Has there been any progress on reducing the std library's usage of the GC?