Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 15:55:41 UTC, John Burton wrote: On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote: I'm coming from a C++ background so I'm not too used to garbage collection and it's implications. I have a function that creates a std.socket.Socket using new and connects to a tcp server, and writes some stuff to it. I then explicitly close the socket, and the socket object goes out of scope. Am I doing this right? Or is there a better way to do this in D? Thanks. For my use case here, I'm increasingly thinking that just calling the underlying 'C' socket and send calls is better. No need for anything complicated at all for my actual program :) One final piece of advice as this thread seemed to have gone off the rails a bit. You can always put a `scope(exit) socket.close();` after you create the socket. This will ensure that the socket will be closed once the scope is exited no matter what... almost, anyway. If an Error is thrown no stack unwinding is done but at this point your program is in an unrecoverable state anyway and you have a lot more to worry about than a socket that hasn't been closed.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote: I'm coming from a C++ background so I'm not too used to garbage collection and it's implications. I have a function that creates a std.socket.Socket using new and connects to a tcp server, and writes some stuff to it. I then explicitly close the socket, and the socket object goes out of scope. Am I doing this right? Or is there a better way to do this in D? Thanks. For my use case here, I'm increasingly thinking that just calling the underlying 'C' socket and send calls is better. No need for anything complicated at all for my actual program :)
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 13:19:37 UTC, Guillaume Piolat wrote: https://forum.dlang.org/post/pmulowxpikjjffkrs...@forum.dlang.org Not an issue with DerelictUtil, an issue with the strategy of closing resources in GC with this case. Derelict loaders work-around this by not unloading shared libraries so the GC won't unload shared libs before the resources related to the shared library are freed. https://github.com/DerelictOrg/DerelictAL/blob/master/source/derelict/openal/dynload.d#L366 Yeah, the loaders all used to needlessly unload the shared libraries in a static destructor. The fact that they don't anymore isn't to avoid any GC/destructor issues, but because there's no point in unloading the libraries when the system is going to do it anyway.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 12:33:24 UTC, Guillaume Piolat
wrote:
On Wednesday, 28 June 2017 at 11:34:17 UTC, Moritz Maxeiner
wrote:
Requirement: Do not allocate using the GC
Option 1) Use structs with `@disable this`, `@disable
this(this)`, and a destructor that checks whether the resource
reference is != invalid resource reference before trying to
release.
Option 2) Use classes with simple constructor/destructor
layout.
If you want to integrate a check that the requirement holds
(with either class or struct), put `if (gc_inFinalizer) throw
SomeError` into the class/struct destructor
I don't get it.
You asked for a "_simple_ story about resource release" and the
above two options provide that IMHO.
It's completely possible to use the full power of GC and be
deterministic.
Sure, but it's not "simple". With D's GC as the memory allocator
you can currently have
(1) non-deterministic memory management and non-deterministic
object lifetimes:
Use `new` or
`std.experimental.allocator.make!(std.experimental.allocator.gc_allocator.GCAllocator)` for allocation&construction.
The GC will finalize objects with no pointers to them
non-deterministically and deallocate the memory after their
respective finalization.
(2) non-deterministic memory management and deterministic object
lifetimes:
Construct the objects as with (1), but destruct them by
calling `destroy` on them outside of a collection cycle (i.e.
when they, and all their members are still considered "live" by
the GC).
The GC will collect the memory non-deterministically after
there are no more pointers to it.
Warning: You are responsible for ensuring that the GC will
never see an undestroyed object in the collection cycle, because
it might try to finalize (call its `~this`) it.
You can protect yourself against such finalization
attempts by putting `if (gc_inFinalizer) throw Error("Bug! I
forgot to destruct!")` in the objects' respective destructors.
(3) deterministic memory management and deterministic object
lifetimes:
Construct the objects as with (1), but destruct&deallocate
them by calling
`std.experimental.allocator.dispose!(std.experimental.allocator.gc_allocator.GCAllocator)`on them outside of a collection cycle.
The GC will not do anything for those objects.
Same warning as (2) applies.
I'm out of this very _confused_ discussion.
I'm honestly not sure what's confusing about it.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 13:02:04 UTC, Mike Parker wrote: On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat wrote: So far everyone is ignoring my example when A needs B to be destroyed. This happens as soon as you use DerelictUtil for example. What's the issue with DerelictUtil? https://forum.dlang.org/post/pmulowxpikjjffkrs...@forum.dlang.org Not an issue with DerelictUtil, an issue with the strategy of closing resources in GC with this case. Derelict loaders work-around this by not unloading shared libraries so the GC won't unload shared libs before the resources related to the shared library are freed. https://github.com/DerelictOrg/DerelictAL/blob/master/source/derelict/openal/dynload.d#L366
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat wrote: So far everyone is ignoring my example when A needs B to be destroyed. This happens as soon as you use DerelictUtil for example. What's the issue with DerelictUtil?
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 12:28:28 UTC, Guillaume Piolat wrote: On Wednesday, 28 June 2017 at 11:21:07 UTC, Moritz Maxeiner wrote: On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat wrote: So far everyone is ignoring my example when A needs B to be destroyed. This happens as soon as you use DerelictUtil for example. I thought I had (implicitly): B needs to be `@disable finalize`. So in the current language, doesn't exist? I thought that was the premise of the discussion? That the GC *currently* invokes the "destructors" as finalizers and that *currently* the only way to avoid that is calling `destroy` on them manually beforehand (since destructors won't be called twice on a single object).
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 11:34:17 UTC, Moritz Maxeiner wrote: Requirement: Do not allocate using the GC Option 1) Use structs with `@disable this`, `@disable this(this)`, and a destructor that checks whether the resource reference is != invalid resource reference before trying to release. Option 2) Use classes with simple constructor/destructor layout. If you want to integrate a check that the requirement holds (with either class or struct), put `if (gc_inFinalizer) throw SomeError` into the class/struct destructor I don't get it. It's completely possible to use the full power of GC and be deterministic. I'm out of this very _confused_ discussion.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 11:21:07 UTC, Moritz Maxeiner wrote: On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat wrote: So far everyone is ignoring my example when A needs B to be destroyed. This happens as soon as you use DerelictUtil for example. I thought I had (implicitly): B needs to be `@disable finalize`. So in the current language, doesn't exist?
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 09:22:07 UTC, Guillaume Piolat wrote: Deterministic destruction is a _solved_ problem in C++, and a number of users to convert are now coming from C++. It is also in D, as long as you don't use the GC (which is inherently non-deterministic). 3. Suggest a systematic, always working, workaround (or language change such as "GC doesn't call ~this). C++ users have no difficulty having an object graph with detailed ownership schemes, that's what they do day in day out. It's important to have a _simple_ story about resource release, else we will talk about "GC" every day, and hearing about "GC" is bad for marketing. Requirement: Do not allocate using the GC Option 1) Use structs with `@disable this`, `@disable this(this)`, and a destructor that checks whether the resource reference is != invalid resource reference before trying to release. Option 2) Use classes with simple constructor/destructor layout. If you want to integrate a check that the requirement holds (with either class or struct), put `if (gc_inFinalizer) throw SomeError` into the class/struct destructor
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat wrote: So far everyone is ignoring my example when A needs B to be destroyed. This happens as soon as you use DerelictUtil for example. I thought I had (implicitly): B needs to be `@disable finalize`.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 09:16:22 UTC, Guillaume Piolat wrote: On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis wrote: There are definitely cases where finalizers make sense. Case in point: if you have a socket class, it makes perfect sense for it to have a finalizer. Yes, it's better to close it manually, but it will work just fine for the GC to close it when finalizing the class object so long as you don't use so many sockets that you run out before the GC collects them. So, having a finalizer is a good backup to ensure that the socket resource doesn't leak. - Jonathan M Davis So far everyone is ignoring my example when A needs B to be destroyed. This happens as soon as you use DerelictUtil for example. And I'm gonna rant a little bit more. Deterministic destruction is a _solved_ problem in C++, and a number of users to convert are now coming from C++. We should: 1. be honest and tell things as they are: it's more complicated than in C++, but also liberating when you know to juggle between GC and deterministic 2. Avoid making bogus suggestions which don't always work, such as close() methods, releasing resource with GC, . They only work for _some_ resources. 3. Suggest a systematic, always working, workaround (or language change such as "GC doesn't call ~this). C++ users have no difficulty having an object graph with detailed ownership schemes, that's what they do day in day out. It's important to have a _simple_ story about resource release, else we will talk about "GC" every day, and hearing about "GC" is bad for marketing.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis wrote: There are definitely cases where finalizers make sense. Case in point: if you have a socket class, it makes perfect sense for it to have a finalizer. Yes, it's better to close it manually, but it will work just fine for the GC to close it when finalizing the class object so long as you don't use so many sockets that you run out before the GC collects them. So, having a finalizer is a good backup to ensure that the socket resource doesn't leak. - Jonathan M Davis So far everyone is ignoring my example when A needs B to be destroyed. This happens as soon as you use DerelictUtil for example.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On 2017-06-27 11:54, John Burton wrote: I'm coming from a C++ background so I'm not too used to garbage collection and it's implications. I have a function that creates a std.socket.Socket using new and connects to a tcp server, and writes some stuff to it. I then explicitly close the socket, and the socket object goes out of scope. Yes. You can use "scope (exit)", unless you're already doing so. Now the issue is that I now need to call this function more than once every second. I worry that it will create large amounts of uncollected "garbage" which will eventually lead to problems. Sounds like you need a connection pool. The vibe.d [1] framework contains connection pools. It's also possible to manually disable the GC for a while and then enable it again. [1] http://vibed.org -- /Jacob Carlborg
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On 2017-06-27 17:24, Steven Schveighoffer wrote: Yes, Tango solved this by having a separate "finalize()" method. I wish we had something like this. Not sure if this is the same, but I remember that Tango had a separate method called "dispose" that was called if a class was allocated on the stack, i.e. with the "scope" keyword. -- /Jacob Carlborg
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote: I'm coming from a C++ background so I'm not too used to garbage collection and it's implications. I have a function that creates a std.socket.Socket using new and connects to a tcp server, and writes some stuff to it. I then explicitly close the socket, and the socket object goes out of scope. [...] May be you can see yu: https://github.com/dushibaiyu/yu
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 02:13:10 UTC, Jonathan M Davis wrote: On Wednesday, June 28, 2017 01:11:35 Moritz Maxeiner via Digitalmars-d-learn wrote: Not every class can't be finalized, so it might make sense for finalization to remain an available option. There are definitely cases where finalizers make sense. Case in point: if you have a socket class, it makes perfect sense for it to have a finalizer. Yes, it's better to close it manually, but it will work just fine for the GC to close it when finalizing the class object so long as you don't use so many sockets that you run out before the GC collects them. So, having a finalizer is a good backup to ensure that the socket resource doesn't leak. Yes, I think that's like the File class I presented (at least on Posix). But on the other hand, since there are also cases in which finalization of an alive (==undestroyed) object are unacceptable (e.g. when it requires other GC managed objects to be alive), splitting them off into two separate methods and allowing to forbid finalization seems sensible to me.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, June 28, 2017 01:11:35 Moritz Maxeiner via Digitalmars-d-learn wrote: > On Wednesday, 28 June 2017 at 00:05:20 UTC, Guillaume Piolat > > wrote: > > On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote: > >> - Replace calls by the GC to `~this` with calls to `finalize` > >> (or invent some cool other shortened name for the latter) > > > > My point is that in such a "finalize()" function the only sane > > things to do is to crash if the resource wasn't freed already. > > Why so? > > > > [...] > > Not every class can't be finalized, so it might make sense for > finalization to remain an available option. There are definitely cases where finalizers make sense. Case in point: if you have a socket class, it makes perfect sense for it to have a finalizer. Yes, it's better to close it manually, but it will work just fine for the GC to close it when finalizing the class object so long as you don't use so many sockets that you run out before the GC collects them. So, having a finalizer is a good backup to ensure that the socket resource doesn't leak. - Jonathan M Davis
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Wednesday, 28 June 2017 at 00:05:20 UTC, Guillaume Piolat
wrote:
On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote:
- Replace calls by the GC to `~this` with calls to `finalize`
(or invent some cool other shortened name for the latter)
My point is that in such a "finalize()" function the only sane
things to do is to crash if the resource wasn't freed already.
Why so?
[...]
Not every class can't be finalized, so it might make sense for
finalization to remain an available option.
What I think reasonable is treating destruction and finalization
as two distinct things. This could look like the following:
---
// Does not refer to other GC memory, can be finalized
class File
{
private:
int fd;
public:
this() { fd = open(...); }
~this() { close(fd); }
// If the GC collects a File object that hasn't been
destroyed yet, it will call this finalizer,
// which *manually* calls the destructor (because it's safe
to do so in this case)
// A File object can still be manually `destroy`ed
beforehand, in which case this particular finalizer is still
safe, since an object won't be destroyed twice
finalize() { destroy(this); }
// Or maybe prettier: alias finalize = ~this;
}
class Foo
{
private:
File f;
public:
this() { f = new File(); }
// Make the process crash on finalization of an undestroyed
Foo object
@disable finalize();
}
---
- Reserve `~this` for being called by deterministic lifetime
management (std.experimental.allocator.dispose,
object.destroy, RefCounted, Unique, etc.)
That's precisely what the GC-proof-resource-class allows, by
preventing defective, non-composable usages of ~this.
The GC proof resource class idiom is a necessary hack, but it
*is* a hack with its own limitations (e.g. depending on Error
being catchable).
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 23:54:50 UTC, Moritz Maxeiner wrote: Do you mean destructors? Yes. - Replace calls by the GC to `~this` with calls to `finalize` (or invent some cool other shortened name for the latter) My point is that in such a "finalize()" function the only sane things to do is to crash if the resource wasn't freed already. Why so? See my texture example in: https://forum.dlang.org/post/kliasvljzwjhzuzib...@forum.dlang.org More simply: If A needs an alive B to be destroyed, then neither A-owns-B or B-owns-A can guarantee the ordering under GC destruction. Furthermore, this is viral. Whoever owns A needs an alive B to be destroyed. - Reserve `~this` for being called by deterministic lifetime management (std.experimental.allocator.dispose, object.destroy, RefCounted, Unique, etc.) That's precisely what the GC-proof-resource-class allows, by preventing defective, non-composable usages of ~this.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 23:42:38 UTC, Guillaume Piolat wrote: On Tuesday, 27 June 2017 at 18:04:36 UTC, Moritz Maxeiner wrote: Well, technically speaking the `~this` for D classes *is* a finalizer that you may optionally manually call (e.g. via destroy). It would be nice, though, to change class `~this` into a destructor and move the finalization into an extra method like `finalize`. Write a DIP? The only sane way out is to prevent the GC from calling constructors. Do you mean destructors? If so, that's what I meant with the "technically". If the may GC call it (automatically), it's a finalizer, not a destructor (in the usual sense, anyway). - Replace calls by the GC to `~this` with calls to `finalize` (or invent some cool other shortened name for the latter) - Reserve `~this` for being called by deterministic lifetime management (std.experimental.allocator.dispose, object.destroy, RefCounted, Unique, etc.)
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 18:04:36 UTC, Moritz Maxeiner wrote: Well, technically speaking the `~this` for D classes *is* a finalizer that you may optionally manually call (e.g. via destroy). It would be nice, though, to change class `~this` into a destructor and move the finalization into an extra method like `finalize`. Write a DIP? The only sane way out is to prevent the GC from calling constructors.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer wrote: The GC still needs to call something to clean up non-memory resources, Well, I'd much prefer if the GC would simply not call destructors. Yes, destructor can work for _some_ resources, but because of ordering it also create accidental correctness, or just doesn't work with other resources. but having a call to use when cleaning up deterministically can make resource management more efficient (even memory resource management). Yes. My point is that the destructor is uniquely positionned to be that call. (This would also enables back failing destructors through the use of exceptions.) In current D the problem with allowing the GC to close resources is that there is no ordering of destructors. A sub-tree of the ownership graph is freed by the GC together, but sometime an order is required for releasing. Typically: you want to release a SDL_Texture but the Derelict SharedLib object has been freed already, so you can't call sdl_releaseTexture. It doesn't matter that the texture object held a reference to the ShareLib object since the sub-tree is destroyed together. close() methods don't help with that, the problem is with the GC calling destructors. And if _some_ resource needs an ordering, then this property leak on their owners, so little by little the whole ownership graph need determinism.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis wrote: On Tuesday, June 27, 2017 09:54:19 John Burton via Digitalmars-d-learn wrote: [...] Arguably, std.socket should have used structs instead of classes for sockets for precisely this reason (though there are some advantages in using inheritance with sockets). But yes, calling close manually is the correct thing to do. Relying on the GC to call a destructor/finalizer is error-prone. There is no guarantee that the memory will ever be freed (e.g. the runtime could choose to not bother doing cleanup on shutdown), and even if the GC does collect it, there are no guarantees about how soon it will do so. However, if you keep allocating memory with the GC, then over time, the GC will collect GC-allocated memory that isn't currently being used so that it can reuse the memory. So, you really don't need to worry about the memory unless it becomes a bottleneck. It will be collected and reused, not leaked. [...] I agree with that it should have been structs. The inheritance issue could be fixed by having a private member of the struct in a class, that way there could still be a class wrapper around it.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On 6/27/17 2:04 PM, Moritz Maxeiner wrote: On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer wrote: On 6/27/17 9:25 AM, Guillaume Piolat wrote: On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote: But I would use a close method, and not destroy(obj). The reason is because often times, you have wrapper types around your socket type, and just one extra level of indirection means the destructor cannot be used to clean up the socket (you are not allowed to access GC-allocated resources in a destructor). All destructor restrictions do not apply when it's not called by the GC. There really are two categories of destructors: called by the GC and called deterministically. Their usage should not overlap. Yes, Tango solved this by having a separate "finalize()" method. I wish we had something like this. Well, technically speaking the `~this` for D classes *is* a finalizer that you may optionally manually call (e.g. via destroy). It would be nice, though, to change class `~this` into a destructor and move the finalization into an extra method like `finalize`. Write a DIP? I don't have enough motivation to do this. But if we do anything, we should look at what Tango has done. It may not work exactly for druntime because of existing code, but it obviously didn't need compiler changes to work. -Steve
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 15:24:00 UTC, Steven Schveighoffer wrote: On 6/27/17 9:25 AM, Guillaume Piolat wrote: On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote: But I would use a close method, and not destroy(obj). The reason is because often times, you have wrapper types around your socket type, and just one extra level of indirection means the destructor cannot be used to clean up the socket (you are not allowed to access GC-allocated resources in a destructor). All destructor restrictions do not apply when it's not called by the GC. There really are two categories of destructors: called by the GC and called deterministically. Their usage should not overlap. Yes, Tango solved this by having a separate "finalize()" method. I wish we had something like this. Well, technically speaking the `~this` for D classes *is* a finalizer that you may optionally manually call (e.g. via destroy). It would be nice, though, to change class `~this` into a destructor and move the finalization into an extra method like `finalize`. Write a DIP?
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On 6/27/17 12:16 PM, Steven Schveighoffer wrote: On 6/27/17 11:24 AM, Steven Schveighoffer wrote: On 6/27/17 9:25 AM, Guillaume Piolat wrote: That's how the GC-proof resource class came to existence, after many destruction bugs, and it let's you use the GC as a detector for non-deterministic destruction. I miss it in @nogc :) https://p0nce.github.io/d-idioms/#GC-proof-resource-class There are definitely better ways to do this than trying to allocate and catching an error. The GC knows the GC is running, use that mechanism instead :) https://github.com/dlang/druntime/blob/master/src/gc/gcinterface.d#L189 https://github.com/dlang/druntime/blob/master/src/core/memory.d#L150 Apparently that info is limited to the core.memory package. But it's extern(C), so declaring the prototype should work. Should be much more efficient than allocating a byte (which I see you don't delete if it works), and catching an error. Just added this enhancement to publicize this function: https://issues.dlang.org/show_bug.cgi?id=17563 -Steve
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On 6/27/17 11:24 AM, Steven Schveighoffer wrote: On 6/27/17 9:25 AM, Guillaume Piolat wrote: That's how the GC-proof resource class came to existence, after many destruction bugs, and it let's you use the GC as a detector for non-deterministic destruction. I miss it in @nogc :) https://p0nce.github.io/d-idioms/#GC-proof-resource-class There are definitely better ways to do this than trying to allocate and catching an error. The GC knows the GC is running, use that mechanism instead :) https://github.com/dlang/druntime/blob/master/src/gc/gcinterface.d#L189 https://github.com/dlang/druntime/blob/master/src/core/memory.d#L150 Apparently that info is limited to the core.memory package. But it's extern(C), so declaring the prototype should work. Should be much more efficient than allocating a byte (which I see you don't delete if it works), and catching an error. -Steve
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On 6/27/17 9:25 AM, Guillaume Piolat wrote: On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote: But I would use a close method, and not destroy(obj). The reason is because often times, you have wrapper types around your socket type, and just one extra level of indirection means the destructor cannot be used to clean up the socket (you are not allowed to access GC-allocated resources in a destructor). All destructor restrictions do not apply when it's not called by the GC. There really are two categories of destructors: called by the GC and called deterministically. Their usage should not overlap. Yes, Tango solved this by having a separate "finalize()" method. I wish we had something like this. My reasoning went with the following: 1 - "I should have close() methods so that I don't rely on the GC, and the GC is calling ~this from the wrong thread etc, not everything can be released in this context (eg: OpenGL objects should be released from one thread only). Close methods will call close methods of "owned" objects." I hadn't realized this, that is a good point. However, not all resources are restricted this much. That's how the GC-proof resource class came to existence, after many destruction bugs, and it let's you use the GC as a detector for non-deterministic destruction. I miss it in @nogc :) https://p0nce.github.io/d-idioms/#GC-proof-resource-class There are definitely better ways to do this than trying to allocate and catching an error. The GC knows the GC is running, use that mechanism instead :) Remember years ago when Alexandrescu suggested the GC shouldn't call heap destructors? That's what we get for having said no at the time. No, the issue is that there isn't a separate call for destruction and GC. I think we should add this. The GC still needs to call something to clean up non-memory resources, but having a call to use when cleaning up deterministically can make resource management more efficient (even memory resource management). -Steve -Steve
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 13:11:10 UTC, Steven Schveighoffer wrote: But I would use a close method, and not destroy(obj). The reason is because often times, you have wrapper types around your socket type, and just one extra level of indirection means the destructor cannot be used to clean up the socket (you are not allowed to access GC-allocated resources in a destructor). All destructor restrictions do not apply when it's not called by the GC. There really are two categories of destructors: called by the GC and called deterministically. Their usage should not overlap. My reasoning went with the following: 1 - "I should have close() methods so that I don't rely on the GC, and the GC is calling ~this from the wrong thread etc, not everything can be released in this context (eg: OpenGL objects should be released from one thread only). Close methods will call close methods of "owned" objects." 2 - "Uh, oh. Refcounted and Unique and Scoped all use .destroy, so destructors should call close() methods. To avoid double release, close should handle being called several times. The GC will close what I forgot!" 3 - "Actually there is no order of destruction when called by the GC. So I can't release owned objects when called by the GC. Better do nothing when close() is called by the GC, let's detect this. 4 - close() is now identical with ~this (at least for classes). Let's remove the close() method. Crash when a resource is freed by the GC. That's how the GC-proof resource class came to existence, after many destruction bugs, and it let's you use the GC as a detector for non-deterministic destruction. I miss it in @nogc :) https://p0nce.github.io/d-idioms/#GC-proof-resource-class Remember years ago when Alexandrescu suggested the GC shouldn't call heap destructors? That's what we get for having said no at the time.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On 6/27/17 8:29 AM, Guillaume Piolat wrote: On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote: Am I doing this right with GC? In C++ I'd ensure that the Socket class had a destructor that closed the socket and I'd also assume that once it went out of scope there was no memory left allocated. In D am I right to assume I need to manually close the socket but there is no need to worry about the memory? Yes. You can also call a destructor manually with destroy(obj); This avoids having a forest of 'close' methods, who are duplicates of the destructor in spirit, and let's you use destructors like you are accustomed in C++. Generally, it is dangerous to let the GC handle resource release: https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors This is the best to read, it's very tricky for people coming from other languages I think to understand the rules around the GC destructor. I think you have gotten sound advice from many people, but I wanted to chime in as someone who used the GC to clean up sockets and file handles in a long-running program. Even with all these warnings, it does seem to work. My advice is to close the socket in the destructor if still valid, and then manually close it also. This way you don't leak resources if you miss a close call. The fundamental problem with closing sockets and other resources with the GC is that the resource management issues and rules for memory are vastly different than those of other resources. The OS can give you vast sums of memory, but will likely limit your ability to keep sockets open. It also keeps the other end potentially locked up. If you try to allocate a new socket and run out of sockets, there is no "socket collection cycle". This is why closing manually is advised. But I would use a close method, and not destroy(obj). The reason is because often times, you have wrapper types around your socket type, and just one extra level of indirection means the destructor cannot be used to clean up the socket (you are not allowed to access GC-allocated resources in a destructor). Using std.typecons.RefCounted also can help to ensure destruction of a socket when it's no longer used. But like I said, it's not always possible to use a destructor of a higher level object to clean up a socket. -Steve
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 12:29:03 UTC, Guillaume Piolat wrote: [...] Hmm... Isn't it possible to just allocate the object/a pool of objects outside the loop and reuse it? Everybody's proposing other means of allocations which is nice, but I wonder why there is such a need for reallocations in the first place. Of course the specifics depend on the implementation, maybe it isn't as straightforward. It's just that no allocation is faster than reallocation no matter what reallocation method is used.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote: I assume that I do need to explicitly call close on the socket as there is no deterministic destructor for class objects. Yes. I further assume that the runtime will garbage collect any memory allocated to the socket object at a later time. Most probably. Am I doing this right with GC? In C++ I'd ensure that the Socket class had a destructor that closed the socket and I'd also assume that once it went out of scope there was no memory left allocated. In D am I right to assume I need to manually close the socket but there is no need to worry about the memory? Yes. You can also call a destructor manually with destroy(obj); This avoids having a forest of 'close' methods, who are duplicates of the destructor in spirit, and let's you use destructors like you are accustomed in C++. Generally, it is dangerous to let the GC handle resource release: https://p0nce.github.io/d-idioms/#The-trouble-with-class-destructors Now the issue is that I now need to call this function more than once every second. I worry that it will create large amounts of uncollected "garbage" which will eventually lead to problems. If this is a problem you can create it on the stack with std.typecons.scoped Am I doing this right? Or is there a better way to do this in D? What you are doing it OK.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote:
Now the issue is that I now need to call this function more
than once every second. I worry that it will create large
amounts of uncollected "garbage" which will eventually lead to
problems.
Since nobody has mentioned Allocator, yet:
As you seem to know the lifetime of the socket statically, you
can just use std.experimental.allocator.{make,dispose} [1]. With
regards to reusing the memory: Simply use a freelist allocator
[2].
[1] https://dlang.org/phobos/std_experimental_allocator.html#.make
[2]
https://dlang.org/phobos/std_experimental_allocator_building_blocks_free_list.html
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 11:43:27 UTC, John Burton wrote: On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis wrote: On Tuesday, June 27, 2017 09:54:19 John Burton via Digitalmars-d-learn wrote: I'm coming from a C++ background so I'm not too used to garbage collection and it's implications. I have a function that creates a std.socket.Socket using new and connects to a tcp server, and writes some stuff to it. I then explicitly close the socket, and the socket object goes out of scope. Thank you for the advice everyone. The hardest part about learning D isn't the language, or how to program, it's unlearning what you know from C++ and learning the proper way to do things in D. I've tried D several times before and eventually stopped when I get to the stage of "how do I do this c++ thing in d" proves to be hard. Instead this time, I've started writing D programs as "better C" and then slowly started adding in higher level d features. It's going much better as I'm no longer trying so hard to write bad C++ in D :) Heh, there's no reason you can't, you'll just cause a few head scratches when you post code while we figure out what it does. Idiomatic D is rather different to C++ and is centered around ranges + algorithms, but you can write code in pretty much any style. Some of the advanced metaprogramming takes a bit to get your head around. If you do get stuck do post a question here, we're more than happy to help.
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 10:14:16 UTC, Jonathan M Davis wrote: On Tuesday, June 27, 2017 09:54:19 John Burton via Digitalmars-d-learn wrote: I'm coming from a C++ background so I'm not too used to garbage collection and it's implications. I have a function that creates a std.socket.Socket using new and connects to a tcp server, and writes some stuff to it. I then explicitly close the socket, and the socket object goes out of scope. Thank you for the advice everyone. The hardest part about learning D isn't the language, or how to program, it's unlearning what you know from C++ and learning the proper way to do things in D. I've tried D several times before and eventually stopped when I get to the stage of "how do I do this c++ thing in d" proves to be hard. Instead this time, I've started writing D programs as "better C" and then slowly started adding in higher level d features. It's going much better as I'm no longer trying so hard to write bad C++ in D :)
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, June 27, 2017 09:54:19 John Burton via Digitalmars-d-learn wrote: > I'm coming from a C++ background so I'm not too used to garbage > collection and it's implications. I have a function that creates > a std.socket.Socket using new and connects to a tcp server, and > writes some stuff to it. I then explicitly close the socket, and > the socket object goes out of scope. > > I assume that I do need to explicitly call close on the socket as > there is no deterministic destructor for class objects. I further > assume that the runtime will garbage collect any memory allocated > to the socket object at a later time. > > Am I doing this right with GC? In C++ I'd ensure that the Socket > class had a destructor that closed the socket and I'd also assume > that once it went out of scope there was no memory left > allocated. In D am I right to assume I need to manually close the > socket but there is no need to worry about the memory? > > Now the issue is that I now need to call this function more than > once every second. I worry that it will create large amounts of > uncollected "garbage" which will eventually lead to problems. > > Am I doing this right? Or is there a better way to do this in D? Arguably, std.socket should have used structs instead of classes for sockets for precisely this reason (though there are some advantages in using inheritance with sockets). But yes, calling close manually is the correct thing to do. Relying on the GC to call a destructor/finalizer is error-prone. There is no guarantee that the memory will ever be freed (e.g. the runtime could choose to not bother doing cleanup on shutdown), and even if the GC does collect it, there are no guarantees about how soon it will do so. However, if you keep allocating memory with the GC, then over time, the GC will collect GC-allocated memory that isn't currently being used so that it can reuse the memory. So, you really don't need to worry about the memory unless it becomes a bottleneck. It will be collected and reused, not leaked. And if in C++, you would be newing up a socket object each time and then deleting it rather than having the socket on the stack, then that performance could actually be worse than using the GC and having it collect the memory when it determines that it needs more memory. If you do find that it becomes a bottleneck to be allocating all of these sockets, then you can look into using std.typecons.scoped, which would allocate a class on the stack, but that really only works if you're just dealing with a local variable, and in general, I wouldn't worry about anything like that unless the heap allocations are proving to be a performance problem - particularly because you need to be very careful when using it to avoid screwing up and having memory corruption, because you used an object after it was freed. - Jonathan M Davis
Re: Advice wanted on garbage collection of sockets for c++ programmer using D
On Tuesday, 27 June 2017 at 09:54:19 UTC, John Burton wrote: I'm coming from a C++ background so I'm not too used to garbage collection and it's implications. I have a function that creates a std.socket.Socket using new and connects to a tcp server, and writes some stuff to it. I then explicitly close the socket, and the socket object goes out of scope. I assume that I do need to explicitly call close on the socket as there is no deterministic destructor for class objects. I further assume that the runtime will garbage collect any memory allocated to the socket object at a later time. Am I doing this right with GC? In C++ I'd ensure that the Socket class had a destructor that closed the socket and I'd also assume that once it went out of scope there was no memory left allocated. In D am I right to assume I need to manually close the socket but there is no need to worry about the memory? Yes. from the documentation Immediately drop any connections and release socket resources. Calling shutdown before close is recommended for connection-oriented sockets. The Socket object is no longer usable after close. Calling shutdown() before this is recommended for connection-oriented sockets. The GC (if you use it) will pick up the garbage for you. Now the issue is that I now need to call this function more than once every second. I worry that it will create large amounts of uncollected "garbage" which will eventually lead to problems. Have a look at automem (https://github.com/atilaneves/automem/) Am I doing this right? Or is there a better way to do this in D? Thanks. There is also vibe.d if you are looking for more networking stuff. https://github.com/rejectedsoftware/vibe.d Best of luck Nic
Advice wanted on garbage collection of sockets for c++ programmer using D
I'm coming from a C++ background so I'm not too used to garbage collection and it's implications. I have a function that creates a std.socket.Socket using new and connects to a tcp server, and writes some stuff to it. I then explicitly close the socket, and the socket object goes out of scope. I assume that I do need to explicitly call close on the socket as there is no deterministic destructor for class objects. I further assume that the runtime will garbage collect any memory allocated to the socket object at a later time. Am I doing this right with GC? In C++ I'd ensure that the Socket class had a destructor that closed the socket and I'd also assume that once it went out of scope there was no memory left allocated. In D am I right to assume I need to manually close the socket but there is no need to worry about the memory? Now the issue is that I now need to call this function more than once every second. I worry that it will create large amounts of uncollected "garbage" which will eventually lead to problems. Am I doing this right? Or is there a better way to do this in D? Thanks.
