Re: The no gc crowd
On Friday, 11 October 2013 at 17:58:07 UTC, Sean Kelly wrote: On Friday, 11 October 2013 at 09:56:10 UTC, Leandro Lucarella wrote: This is not really what's stopping the porting, is a problem, but an independent one. My idea was to port the GC as it is in Tango, and then see how to overcome its limitations. I tried this a while back, but the GC in Druntime has changed a bunch since it diverged from Tango, and some or all of those changed need to be applied to your concurrent collector before it can be used. Like you, I ended up not having the time for this. I think you'd need to do a diff of the current GC code vs. the code as it was originally checked into SVN on DSource. My plan was actually to redo all the patches I did to Tango (which were A LOT) on druntime master, adapting them as I go (I even started doing that but got busy after just 2 or 3 :S). I hope I can resume the work soon!
Re: The no gc crowd
On Tuesday, 8 October 2013 at 15:43:46 UTC, ponce wrote: At least on Internet forums, there seems to be an entire category of people dismissing D immediately because it has a GC. Whatever rational rebutal we have it's never heard. The long answer is that it's not a real problem. But it seems people want a short answer. It's also an annoying fight to have since so much of it is based on zero data. Just stumbled upon this paper A Study of the Scalability of Stop-the-world Garbage Collectors on Multicores. I have not read it in detail, but the conclusion says: Our evaluation suggests that today, there is no conceptual reason to believe that the pause time of a stop-the-world GC will increase with the increasing number of cores and memory size of multicore hardware. http://pagesperso-systeme.lip6.fr/Gael.Thomas/research/biblio/2013/gidra13asplos-naps.pdf
Re: The no gc crowd
Am 13.10.2013 16:21, schrieb qznc: On Tuesday, 8 October 2013 at 15:43:46 UTC, ponce wrote: At least on Internet forums, there seems to be an entire category of people dismissing D immediately because it has a GC. Whatever rational rebutal we have it's never heard. The long answer is that it's not a real problem. But it seems people want a short answer. It's also an annoying fight to have since so much of it is based on zero data. Just stumbled upon this paper A Study of the Scalability of Stop-the-world Garbage Collectors on Multicores. I have not read it in detail, but the conclusion says: Our evaluation suggests that today, there is no conceptual reason to believe that the pause time of a stop-the-world GC will increase with the increasing number of cores and memory size of multicore hardware. http://pagesperso-systeme.lip6.fr/Gael.Thomas/research/biblio/2013/gidra13asplos-naps.pdf Thanks for the paper, as language geek I love to read them. -- Paulo
Re: The no gc crowd
On 2013-10-11 03:05, Jonathan M Davis wrote:
I'm not disagreeing with how shared works. I'm disagreeing with the idea that
it's not supposed to be normal to cast shared away when operating on shared
objects. I expect that the most common idiom for dealing with shared is to
protect it with a lock, cast it to thread-local, do whatever you're going to
do with it, make sure that there are no thread-local references to it once
you're done operating on it, and then release the lock. e.g.
synchronized
{
auto tc = cast(T)mySharedT;
tc.memberFunc();
doStuff(tc);
//no thread-local references to tc other than tc should
//exist at this point.
}
With Michel Fortin's proposal I think the above could work without a
cast, if doStuff is pure function.
http://michelf.ca/blog/2012/mutex-synchonization-in-d/
--
/Jacob Carlborg
Re: The no gc crowd
On Wednesday, 9 October 2013 at 09:01:12 UTC, Walter Bright wrote: On 10/9/2013 1:59 AM, JR wrote: On Wednesday, 9 October 2013 at 02:22:35 UTC, Andrei Alexandrescu wrote: * Get Robert Schadek's precise GC in. Walter and I have become 101% convinced a precise GC is the one way to go about GC. An orthogonal question, but is Lucarella's CDGC (still) being ported? There's nothing mutually exclusive between a precise and a concurrent gc, no? I thought that got stuck on the problem that the linux system libraries had some sort of threading problem with them. This is not really what's stopping the porting, is a problem, but an independent one. My idea was to port the GC as it is in Tango, and then see how to overcome its limitations. The problem is it's very hard for me to dedicate time to this porting effort. The decision to make the port happen is there, I just how to figure out how and when. I'll keep you posted when I have news.
Re: The no gc crowd
On 2013-10-11 02:51, Jonathan M Davis wrote:
At this point, I don't see how we can have thread-local pools unless casting
to and from shared has hooks for managing that. Otherwise, it's far too likely
that an object is going to be in the wrong pool, because it's being used as
shared when it was constructed as thread-local or vice versa. And we may need
some sort of hook with std.concurrency.send which understands that the object
being sent is being transferred from one thread to another and would tell the
GC to migrate the object from one pool to another (though to do that, it would
probably have to not be typed as shared but rather as thread-local, which
would jive better with what you're talking about doing with std.concurrency).
Certainly, with how shared currently works, it's hard to see how we could get
away with having thread-local GC pools as great as that would be. So, if we
want that, something about how shared works is going to have to change.
A simple solution to the hook would be to pass a dummy type indicating
the object should be transferred:
struct Transfer { }
send(tid, foo, Transfer());
Transfer would be defined in std.concurrency.
--
/Jacob Carlborg
Re: The no gc crowd
Am Thu, 10 Oct 2013 22:04:16 -0400 schrieb Jonathan M Davis jmdavisp...@gmx.com: most D programmers seem to describe when talking about shared is simply using __gshared with normal types, not even using shared, let alone using it with types specifically designed to function as shared. So, the most common approach at this point in D seems to be to avoid shared entirely. One important reason for this is that the types in core.sync still aren't shared. Mutex myMutex; //WRONG, myMutex is in TLS shared Mutex myMutex; //WRONG, can't call .lock, new __gshared Mutex myMutex; //Can't be used in @safe code... //shared Mutex + casting to unshared when accessing: Can't be used in //@safe code See also: http://forum.dlang.org/thread/mailman.2017.1353214033.5162.digitalmar...@puremagic.com?page=2#post-mailman.2037.1353278884.5162.digitalmars-d:40puremagic.com Sean Kelly: I tried this once and it cascaded to requiring modifications of various definitions on core.sys.posix to add a shared qualifier, and since I wasn't ready to do that I rolled back the changes. I guess the alternative would be to have a shared equivalent for every operation that basically just casts away shared and then calls the non-shared function, but that's such a terrible design I've been resisting it.
Re: The no gc crowd
On Tuesday, 8 October 2013 at 22:37:28 UTC, Walter Bright wrote: On 10/8/2013 9:22 AM, Dicebot wrote: It is simply @nogc which is lacking but absolutely mandatory. Adding @nogc is fairly simple. The trouble, though, is (like purity) it is transitive. Every function an @nogc function calls will also have to be @nogc. This will entail a great deal of work updating phobos/druntime to add those annotations. A very naive question but is there no way of analysing the subfunctions to check their purity or lack of GC use rather than having to annotate everything? D does need to be a little wary of becoming too heavily annotated.
Re: The no gc crowd
11-Oct-2013 05:21, Andrei Alexandrescu пишет: On 10/10/13 5:36 PM, Jonathan M Davis wrote: On Thursday, October 10, 2013 10:55:49 Andrei Alexandrescu wrote: On 10/10/13 12:33 AM, Jonathan M Davis wrote: I honestly don't think we can solve it a different way without completely redesigning shared. shared is specifically designed such that you have to either cast it way to do anything with it no or write all of your code to explicitly work with shared, which is not something that generally makes sense to do unless you're creating a type whose only value is in being shared across threads. yes Really? Do you honestly expect the average use of shared to involve creating structs or classes which are designed specifically to be used as shared? Yes. Data structures that can be shared are ALWAYS designed specifically for sharing, unless of course it's a trivial type like int. This. And exactly the same for immutable. It's interesting how folks totally expect complex types (like containers) to meaningfully work with all 3 qualifiers. Sharing means careful interlocking and atomic operations and barriers and stuff. You can't EVER expect to obtain all of that magic by plastering shared on top of your type. Yup. Andrei -- Dmitry Olshansky
Re: The no gc crowd
On 11/10/13 16:32, Dmitry Olshansky wrote: This. And exactly the same for immutable. It's interesting how folks totally expect complex types (like containers) to meaningfully work with all 3 qualifiers. It's not so much that we expect it, as that we might expect that standard library types would _have the appropriate design work put in_ so that they would just work with these qualifiers. (Admittedly shared is a bit of a special case right now that probably needs more work before support is rolled out.) If you tell me that's an unreasonable expectation then fair enough, but it feels pretty bad if e.g. library-implemented number types (big integers or floats, rationals, complex numbers, ...) can't from a user perspective behave exactly like their built-in counterparts.
Re: The no gc crowd
11-Oct-2013 18:46, Joseph Rushton Wakeling пишет: On 11/10/13 16:32, Dmitry Olshansky wrote: This. And exactly the same for immutable. It's interesting how folks totally expect complex types (like containers) to meaningfully work with all 3 qualifiers. It's not so much that we expect it, as that we might expect that standard library types would _have the appropriate design work put in_ so that they would just work with these qualifiers. (Admittedly shared is a bit of a special case right now that probably needs more work before support is rolled out.) It can't - it's like expecting 37 to modify an become 76. Simply put built-ins are special. Imagine a ref-counted type - how would you copy it (and increment a count) with bit-wise immutability? It simply doesn't make sense. (Yes, one can keep count elsewhere e.g. in a global hash-table). More importantly there is little incentive to make immutable stuff ref-counted, especially COW-types. In this sense BigInt simply doesn't work with immutable by design, if need be one can make FixedBigInt that doesn't include COW, doesn't support read-modify-write and mixes well with BigInt. Immutable works best as static data and/or as snapshot style data structures. It's tables, strings and some unique stuff that gets frozen and published at a certain point (usually at start up). It makes a lot less sense at local scope aside from aesthetic beauty as there is plenty of invariants to check, and bitwise immutability is a minority of that. I would even suggest to adopt a convention for a pair of freeze/thaw methods for any UDT that give you a deep copy of object this is made for mutation (thaw on immutable object) or immutable (freeze mutable). If you tell me that's an unreasonable expectation then fair enough, but it feels pretty bad if e.g. library-implemented number types (big integers or floats, rationals, complex numbers, ...) can't from a user perspective behave exactly like their built-in counterparts. No magic paint would automatically expel reference count from the struct's body. With shared it's even more obvious. In general user defined type has to be designed with one of 3 major use cases in mind: local, immutable, shared. -- Dmitry Olshansky
Re: The no gc crowd
On Friday, 11 October 2013 at 02:07:57 UTC, Andrei Alexandrescu wrote: TDPL describes how synchronized automatically peels off the shared off of direct members of the object. Unfortunately that feature is not yet implemented. This would help a ton. I'm still not super happy about having to label an entire method as synchronized for this to work though. I'd prefer to label it shared and synchronize only the part(s) inside that need to hold the lock.
Re: The no gc crowd
On Friday, 11 October 2013 at 01:05:19 UTC, Jonathan M Davis wrote: On Friday, October 11, 2013 02:08:16 Sean Kelly wrote: Shared data needs to be treated differently, explicitly, or things go downhill fast. I'm not disagreeing with how shared works. I'm disagreeing with the idea that it's not supposed to be normal to cast shared away when operating on shared objects. I expect that the most common idiom for dealing with shared is to protect it with a lock, cast it to thread-local, do whatever you're going to do with it, make sure that there are no thread-local references to it once you're done operating on it, and then release the lock. The thing with locks is that you need to use the same lock for all accesses to a set of mutated data or atomicity isn't guaranteed. And if you're locking externally you don't know what might change inside a class during a method call, so you have to use the same lock for all operations on that object, regardless of what you're doing. At that point you may as well just synchronize on the class itself and be done with it. So sure, it saves you from having to define shared or synchronized methods, but I don't think this should be how we want to write concurrent code in D.
Re: The no gc crowd
On Friday, 11 October 2013 at 17:46:01 UTC, Sean Kelly wrote: The thing with locks is that you need to use the same lock for all accesses to a set of mutated data or atomicity isn't guaranteed. And if you're locking externally you don't know what might change inside a class during a method call, so you have to use the same lock for all operations on that object, regardless of what you're doing. At that point you may as well just synchronize on the class itself and be done with it. So sure, it saves you from having to define shared or synchronized methods, but I don't think this should be how we want to write concurrent code in D. How can one possibly used synchronized for this in absence of classes if desire behavior is to lock an entity, not statement block?
Re: The no gc crowd
On Friday, 11 October 2013 at 17:49:11 UTC, Sean Kelly wrote:
On Friday, 11 October 2013 at 02:07:57 UTC, Andrei Alexandrescu
wrote:
TDPL describes how synchronized automatically peels off the
shared off of direct members of the object. Unfortunately
that feature is not yet implemented.
This would help a ton. I'm still not super happy about having
to label an entire method as synchronized for this to work
though. I'd prefer to label it shared and synchronize only the
part(s) inside that need to hold the lock.
It should work as well with
synchronized(stuff) {
// Stuff get its first level sharing removed.
}
Re: The no gc crowd
On Friday, 11 October 2013 at 17:54:12 UTC, deadalnix wrote:
It should work as well with
synchronized(stuff) {
// Stuff get its first level sharing removed.
}
I still stand by the point that for guaranteed safety it must be
not simply removed but replaced with `scope` (assuming it is
finally implemented).
Re: The no gc crowd
On Friday, 11 October 2013 at 09:56:10 UTC, Leandro Lucarella wrote: This is not really what's stopping the porting, is a problem, but an independent one. My idea was to port the GC as it is in Tango, and then see how to overcome its limitations. I tried this a while back, but the GC in Druntime has changed a bunch since it diverged from Tango, and some or all of those changed need to be applied to your concurrent collector before it can be used. Like you, I ended up not having the time for this. I think you'd need to do a diff of the current GC code vs. the code as it was originally checked into SVN on DSource.
Re: The no gc crowd
On Friday, 11 October 2013 at 17:50:26 UTC, Dicebot wrote:
How can one possibly used synchronized for this in absence of
classes if desire behavior is to lock an entity, not statement
block?
I'm not sure I follow. But I was in part objecting to the use of
synchronized without a related object:
synchronized {
// do stuff
}
This statement should be illegal. You must always specify a
synchronization context:
synchronized(myMutex) {
// do stuff
}
For the rest, it seemed like the suggestion was that you could
just wrap a statement in any old synchronized block and all your
problems would be solved, which absolutely isn't the case.
Re: The no gc crowd
On Friday, 11 October 2013 at 18:10:27 UTC, Dicebot wrote:
I was reading this :
http://dlang.org/statement.html#SynchronizedStatement
It says that Expression in sync statement must evaluate to
Object or interface and mutex get created specifically for it.
But what if I want to use struct in that block? Or array?
Synchronize on a dummy object or use core.sync.mutex:
auto m = new Mutex;
synchronized(m) {
}
It's effectively the same as in C++ except that synchronized
saves you the trouble of using an RAII scoped_lock variable.
Re: The no gc crowd
On Friday, 11 October 2013 at 18:05:00 UTC, Sean Kelly wrote:
On Friday, 11 October 2013 at 17:50:26 UTC, Dicebot wrote:
How can one possibly used synchronized for this in absence
of classes if desire behavior is to lock an entity, not
statement block?
I'm not sure I follow. But I was in part objecting to the use
of synchronized without a related object:
synchronized {
// do stuff
}
This statement should be illegal. You must always specify a
synchronization context:
synchronized(myMutex) {
// do stuff
}
For the rest, it seemed like the suggestion was that you could
just wrap a statement in any old synchronized block and all
your problems would be solved, which absolutely isn't the case.
I was reading this :
http://dlang.org/statement.html#SynchronizedStatement
It says that Expression in sync statement must evaluate to Object
or interface and mutex get created specifically for it. But what
if I want to use struct in that block? Or array?
Re: The no gc crowd
On Friday, 11 October 2013 at 18:18:45 UTC, Sean Kelly wrote:
On Friday, 11 October 2013 at 18:10:27 UTC, Dicebot wrote:
I was reading this :
http://dlang.org/statement.html#SynchronizedStatement
It says that Expression in sync statement must evaluate to
Object or interface and mutex get created specifically for it.
But what if I want to use struct in that block? Or array?
Synchronize on a dummy object or use core.sync.mutex:
auto m = new Mutex;
synchronized(m) {
}
It's effectively the same as in C++ except that synchronized
saves you the trouble of using an RAII scoped_lock variable.
Yeah, but it can't possibly work in conjunction with proposed
shared stripping inside the block, can it?
Re: The no gc crowd
On Friday, 11 October 2013 at 18:10:27 UTC, Dicebot wrote:
I was reading this :
http://dlang.org/statement.html#SynchronizedStatement
It says that Expression in sync statement must evaluate to
Object or interface and mutex get created specifically for it.
But what if I want to use struct in that block? Or array?
Synchronize on a dummy object or use core.sync.mutex:
auto m = new Mutex;
synchronized(m) {
}
It's effectively the same as in C++ except that synchronized
saves you the trouble of using an RAII scoped_lock variable.
Re: The no gc crowd
On Friday, 11 October 2013 at 18:19:59 UTC, Dicebot wrote:
On Friday, 11 October 2013 at 18:18:45 UTC, Sean Kelly wrote:
Synchronize on a dummy object or use core.sync.mutex:
auto m = new Mutex;
synchronized(m) {
}
It's effectively the same as in C++ except that synchronized
saves you the trouble of using an RAII scoped_lock variable.
Yeah, but it can't possibly work in conjunction with proposed
shared stripping inside the block, can it?
It should. Stripping shared just means that you'll be able to
call any function available on the struct as opposed to only
explicitly shared functions. And the mutex gives you atomic
behavior (assuming you use the mutex properly anywhere else you
access the struct).
Re: The no gc crowd
On Friday, 11 October 2013 at 18:22:46 UTC, Sean Kelly wrote: It should. Stripping shared just means that you'll be able to call any function available on the struct as opposed to only explicitly shared functions. And the mutex gives you atomic behavior (assuming you use the mutex properly anywhere else you access the struct). How would it know which entity is associated with that mutex? (== which to strip shared from)
Re: The no gc crowd
On Friday, October 11, 2013 16:27:53 ixid wrote: On Tuesday, 8 October 2013 at 22:37:28 UTC, Walter Bright wrote: On 10/8/2013 9:22 AM, Dicebot wrote: It is simply @nogc which is lacking but absolutely mandatory. Adding @nogc is fairly simple. The trouble, though, is (like purity) it is transitive. Every function an @nogc function calls will also have to be @nogc. This will entail a great deal of work updating phobos/druntime to add those annotations. A very naive question but is there no way of analysing the subfunctions to check their purity or lack of GC use rather than having to annotate everything? D does need to be a little wary of becoming too heavily annotated. Attribute inferrence can only work with templates thanks to separate compilation. There's no guarantee that you have the source for the functions that you're using (unless a function is templated). So, there's no way to do the inferrence in the general case. - Jonathan M Davis
Re: The no gc crowd
On Friday, October 11, 2013 18:32:15 Dmitry Olshansky wrote: 11-Oct-2013 05:21, Andrei Alexandrescu пишет: Yes. Data structures that can be shared are ALWAYS designed specifically for sharing, unless of course it's a trivial type like int. This. And exactly the same for immutable. It's interesting how folks totally expect complex types (like containers) to meaningfully work with all 3 qualifiers. That's part of the point. Most stuff can't work with shared. That's why you're forced to cast away shared in many cases. Yes, designing a class specifically to function as shared makes sense some of the time (e.g. concurrent containers), but should I have to create a synchronized class just to wrap a normal type that I happen to want to use as shared in some of my code? That seems like overkill to me, and it forces you to put everything in member functions (if you want to avoid casting away shared anywhere), because it's only inside the member functions that top level of shared is removed for you (and simply removing the top level shared doens't work for more complex objects anyway, thereby still forcing a cast). That makes using shared a royal pain. It's just far cleaner IMHO to protect the shared variable with a lock and cast away shared to operate on it. - Jonathan M Davis
Re: The no gc crowd
On Friday, October 11, 2013 20:04:57 Sean Kelly wrote:
On Friday, 11 October 2013 at 17:50:26 UTC, Dicebot wrote:
How can one possibly used synchronized for this in absence of
classes if desire behavior is to lock an entity, not statement
block?
I'm not sure I follow. But I was in part objecting to the use of
synchronized without a related object:
synchronized {
// do stuff
}
This statement should be illegal. You must always specify a
synchronization context:
synchronized(myMutex) {
// do stuff
}
I agree with that. I was just in too much of a hurry when I threw that code
snippet together and left out the mutex. I was more concerned with what was in
the synchronized block than how the lock was done. It could have been done
with guard/autolock and RAII as far as I was concerned with regards to what I
was trying to show.
For the rest, it seemed like the suggestion was that you could
just wrap a statement in any old synchronized block and all your
problems would be solved, which absolutely isn't the case.
I certainly wasn't suggesting that all problems would be solved by a
synchronized block. I was simply trying to show that in order to actually use
a shared object, you have to cast away shared, and that means protecting the
object with a lock of some kind. You then have the problem of making sure that
no thread-local references to the object escape the lock, but at least shared
then becomes useable.
- Jonathan M Davis
Re: The no gc crowd
On Friday, 11 October 2013 at 18:26:52 UTC, Dicebot wrote: On Friday, 11 October 2013 at 18:22:46 UTC, Sean Kelly wrote: It should. Stripping shared just means that you'll be able to call any function available on the struct as opposed to only explicitly shared functions. And the mutex gives you atomic behavior (assuming you use the mutex properly anywhere else you access the struct). How would it know which entity is associated with that mutex? (== which to strip shared from) It wouldn't. I'm guessing it would just cast away shared.
Re: The no gc crowd
12-Oct-2013 00:14, Jonathan M Davis пишет: On Friday, October 11, 2013 18:32:15 Dmitry Olshansky wrote: 11-Oct-2013 05:21, Andrei Alexandrescu пишет: Yes. Data structures that can be shared are ALWAYS designed specifically for sharing, unless of course it's a trivial type like int. This. And exactly the same for immutable. It's interesting how folks totally expect complex types (like containers) to meaningfully work with all 3 qualifiers. That's part of the point. Most stuff can't work with shared. That's why you're forced to cast away shared in many cases. Yes, designing a class specifically to function as shared makes sense some of the time (e.g. concurrent containers), but should I have to create a synchronized class just to wrap a normal type that I happen to want to use as shared in some of my code? There is not much stuff that needs to be shared. And piles of casts do not inspire confidence at all. If anything having centralized point (e.g. wrapper class as you mention) for code that deals with concurrency and lock is almost always a plus. That seems like overkill to me, and it forces you to put everything in member functions (if you want to avoid casting away shared anywhere), because it's only inside the member functions that top level of shared is removed for you (and simply removing the top level shared doens't work for more complex objects anyway, thereby still forcing a cast). That makes using shared a royal pain. shared needs some library-side help, that's true. It's just far cleaner IMHO to protect the shared variable with a lock and cast away shared to operate on it. Then I respectfully disagree. -- Dmitry Olshansky
Re: The no gc crowd
On 10/11/13 7:46 AM, Joseph Rushton Wakeling wrote: On 11/10/13 16:32, Dmitry Olshansky wrote: This. And exactly the same for immutable. It's interesting how folks totally expect complex types (like containers) to meaningfully work with all 3 qualifiers. It's not so much that we expect it, as that we might expect that standard library types would _have the appropriate design work put in_ so that they would just work with these qualifiers. (Admittedly shared is a bit of a special case right now that probably needs more work before support is rolled out.) If you tell me that's an unreasonable expectation then fair enough, but it feels pretty bad if e.g. library-implemented number types (big integers or floats, rationals, complex numbers, ...) can't from a user perspective behave exactly like their built-in counterparts. I think that's reasonable. Andrei
Re: The no gc crowd
On 11/10/13 23:02, Andrei Alexandrescu wrote: On 10/11/13 7:46 AM, Joseph Rushton Wakeling wrote: It's not so much that we expect it, as that we might expect that standard library types would _have the appropriate design work put in_ so that they would just work with these qualifiers. (Admittedly shared is a bit of a special case right now that probably needs more work before support is rolled out.) If you tell me that's an unreasonable expectation then fair enough, but it feels pretty bad if e.g. library-implemented number types (big integers or floats, rationals, complex numbers, ...) can't from a user perspective behave exactly like their built-in counterparts. I think that's reasonable. Good :-) It's probably clear from discussion that I don't have a sufficient theoretical overview to immediately address what needs to be done here without help, but if anyone is willing to provide some guidance and instruction, I'm happy to try and do the legwork on std.bigint to bring it up to speed in this respect.
Re: The no gc crowd
On 2013-10-10 02:22, Sean Kelly wrote: Only that this would have to be communicated to the user, since moving data later is problematic. Today, I think it's common to construct an object as unshared and then cast it. What is the reason to not create it as shared in the first place? -- /Jacob Carlborg
Re: The no gc crowd
On 2013-10-10 05:55, Jonathan M Davis wrote: That depends. It works with objects I think (for both shared and immutable), but you definitely have to cast to immutable if you want an immutable array or AA. Also, casting _away_ shared is going to be a very common operation due to how shared works. In order to use shared, you basically have to protect the variable with a mutex or synchronized block, cast away shared, and then use it as thread-local for whatever you're doing until you release the lock (in which case, you have to be sure that there are no more thread-local references to the shared object). As such, while it might be possible to construct stuff directly as shared, it's going to have to be cast to thread-local just to use it in any meaningful way. So, at this point, I don't think that it even vaguely flies to try and make it so that casting away shared is something that isn't typically done. It's going to be done about as often as shared is used for anything other than very basic stuff. What's the reason for casting away shared, is it to pass it to a function that doesn't accept shared? The it should be safe if you synchronize around the call? But that function could put away, the now , unshared data, which is actually shared and cause problem? -- /Jacob Carlborg
Re: The no gc crowd
On 2013-10-10 06:24, Jonathan M Davis wrote: And given that std.concurrency requires casting to and from shared or immutable in order to pass objects across threads, it seems ilke most of D's concurrency model requires casting to and/or from shared or immutable. The major exception is structs or classes which are shared or synchronized rather than a normal object which is used as shared, and I suspect that that's done fairly rarely at this point. In fact, it seems like the most common solution is to ignore shared altogether and use __gshared, which is far worse than casting to and from shared IMHO. Isn't the whole point of std.concurrency that is should only accept shared for reference types? If you want to use std.concurrency create a shared object in the first place? So, it's my impression that being able to consider casting to or from shared as abnormal in code which uses shared is a bit of a pipe dream at this point. The current language design pretty much requires casting when doing much of anything with concurrency. There must be a better way to solve this. -- /Jacob Carlborg
Re: The no gc crowd
On 10/9/2013 11:34 PM, Jacob Carlborg wrote: On 2013-10-10 02:22, Sean Kelly wrote: Only that this would have to be communicated to the user, since moving data later is problematic. Today, I think it's common to construct an object as unshared and then cast it. What is the reason to not create it as shared in the first place? 1. Shared data cannot be passed to regular functions. 2. Functions that create data structures would have to know in advance that they'll be creating a shared object. I'm not so sure this would not be an invasive change. 3. Immutable data is implicitly shared. But it is not created immutable - it is created as mutable data, then set to some state, then cast to immutable.
Re: The no gc crowd
On Thursday, October 10, 2013 08:38:31 Jacob Carlborg wrote:
On 2013-10-10 05:55, Jonathan M Davis wrote:
That depends. It works with objects I think (for both shared and
immutable), but you definitely have to cast to immutable if you want an
immutable array or AA.
Also, casting _away_ shared is going to be a very common operation due to
how shared works. In order to use shared, you basically have to protect
the variable with a mutex or synchronized block, cast away shared, and
then use it as thread-local for whatever you're doing until you release
the lock (in which case, you have to be sure that there are no more
thread-local references to the shared object). As such, while it might be
possible to construct stuff directly as shared, it's going to have to be
cast to thread-local just to use it in any meaningful way. So, at this
point, I don't think that it even vaguely flies to try and make it so
that casting away shared is something that isn't typically done. It's
going to be done about as often as shared is used for anything other than
very basic stuff.
What's the reason for casting away shared, is it to pass it to a
function that doesn't accept shared? The it should be safe if you
synchronize around the call? But that function could put away, the now ,
unshared data, which is actually shared and cause problem?
Pretty much nothing accepts shared. At best, templated functions accept
shared. Certainly, shared doesn't work at all with classes and structs unless
the type is specifically intended to be used as shared, because you have to
mark all of its member functions shared to be able to call them. And if you
want to use that class or struct as both shared and unshared, you have to
duplicate all of its member functions.
That being the case, the only way in general to use a shared object is to
protect it with a lock, cast it to thread-local (so that it can actually use
its member functions or be passed to other functions to be used), and then use
it. e.g.
synchronized
{
auto tl = cast(T)mySharedT;
auto result = tl.foo();
auto result2 = bar(tl);
}
Obviously, you then have to make sure that there are no thread-local
references to the shared object when the lock is released, but without casting
away shared like that, you can't do much of anything with it. So, similar to
when you cast away const, it's up to you to guarantee that the code doesn't
violate the type system's guarantees - i.e. that a thread-local variable is
not accessed by multiple threads. So, you use a lock of some kind to protect
the shared variable while it's treated as a thread-local variable in order to
ensure that that guarantee holds. Like with casting away const or with
@trusted, there's obviously risk in doing this, but there's really no other
way to use shared at this point - certainly not without it being incredibly
invasive to your code and forcing code duplication.
- Jonathan M Davis
Re: The no gc crowd
On Thursday, October 10, 2013 08:41:19 Jacob Carlborg wrote: On 2013-10-10 06:24, Jonathan M Davis wrote: And given that std.concurrency requires casting to and from shared or immutable in order to pass objects across threads, it seems ilke most of D's concurrency model requires casting to and/or from shared or immutable. The major exception is structs or classes which are shared or synchronized rather than a normal object which is used as shared, and I suspect that that's done fairly rarely at this point. In fact, it seems like the most common solution is to ignore shared altogether and use __gshared, which is far worse than casting to and from shared IMHO. Isn't the whole point of std.concurrency that is should only accept shared for reference types? If you want to use std.concurrency create a shared object in the first place? You might do that if you're creating the object simply to send it across, but it's frequently the case that the object was created well before it was sent across, and it frequently had to have operations done it other than simply creating it (which wouldn't work if it were shared). So, it often wouldn't make sense for the object being passed to be shared except when being passed. And once it's been passed, it's rarely the case that you want it to be shared. You're usually passing ownership. You're essentially taking a thread-local variable from one thread and making it a thread-local variable on another thread. Unfortunately, the type system does not support the concept of thread ownership (beyond thread-local vs shared), so it's up to the programmer to make sure that no references to the object are kept on the original thread, but there's really no way around that unless you're always creating a new object when you pass it across, which would result in which would usually be a unnecessary copy. So, it becomes like @trusted in that sense. So, it's my impression that being able to consider casting to or from shared as abnormal in code which uses shared is a bit of a pipe dream at this point. The current language design pretty much requires casting when doing much of anything with concurrency. There must be a better way to solve this. I honestly don't think we can solve it a different way without completely redesigning shared. shared is specifically designed such that you have to either cast it way to do anything with it or write all of your code to explicitly work with shared, which is not something that generally makes sense to do unless you're creating a type whose only value is in being shared across threads. Far more frequently, you want to share a type which you would also use normally as a thread-local variable, and that means casting. - Jonathan M Davis
Re: The no gc crowd
On 10/9/2013 9:45 PM, Manu wrote: The are a few problems with mangling the type; I don't understand that. It breaks when you need to interact with libraries. That's true if the library persists copies of the data. But I think it's doable if the library API is stateless, i.e. 'pure'. It's incompatible with struct alignment, and changes the struct size. These are very carefully managed properties of structures. Nobody says there can be only one variant of RefCounted. It obscures/complicates generic code. It seems to not be a problem in C++ with shared_ptrT. It doesn't deal with circular references, which people keep bringing up as a very important problem. ARC doesn't deal with it automatically, either, it requires the user to insert weak pointers at the right places. But, if the RefCounted data is actually allocated on the GC heap, an eventual GC sweep will delete them. What happens when a library receives a T* arg? Micro managing the ref-count at library boundaries sounds like a lot more trouble than manual memory management. Aside from purity mentioned above, another way to deal with that is to encapsulate uses of a RefCounted data structure so that raw pointers into it are unnecessary.
Re: The no gc crowd
On Thursday, October 10, 2013 00:30:55 Walter Bright wrote: It doesn't deal with circular references, which people keep bringing up as a very important problem. ARC doesn't deal with it automatically, either, it requires the user to insert weak pointers at the right places. But, if the RefCounted data is actually allocated on the GC heap, an eventual GC sweep will delete them. That may be true, but if you're using RefCounted because you can't afford the GC, then using the GC heap with them is not an option, because that could trigger a sweep, which is precisely what you're trying to avoid. More normal code may be fine with it but not the folks who can't afford the interruption of stop the world or any of the other costs that come with the GC. So, if RefCounted (or a similar type) is going to be used without the GC, it's going to need some type of weak-ref, even if it's just a normal pointer - though as you've pointed out, that pretty much throws @safety out the window as no GC is involved. But since you've arguably already done that by using malloc instead of the GC anyway, I think that it's debatable how much that matters. However, the GC would allow for more normal code to not worry about circular references with RefCounted. - Jonathan M Davis
Re: The no gc crowd
On 2013-10-10 09:18, Walter Bright wrote: 1. Shared data cannot be passed to regular functions. That I understand. 2. Functions that create data structures would have to know in advance that they'll be creating a shared object. I'm not so sure this would not be an invasive change. If the function doesn't know it creates shared data it will assume it's not and it won't use any synchronization. Then suddenly someone casts it to shared and you're in trouble. 3. Immutable data is implicitly shared. But it is not created immutable - it is created as mutable data, then set to some state, then cast to immutable. It should be possible to create immutable data in the first place. No cast should be required. -- /Jacob Carlborg
Re: The no gc crowd
On 10/10/2013 09:33 AM, Jonathan M Davis wrote: I honestly don't think we can solve it a different way without completely redesigning shared. shared is specifically designed such that you have to either cast it way to do anything with it or write all of your code to explicitly work with shared, which is not something that generally makes sense to do unless you're creating a type whose only value is in being shared across threads. Far more frequently, you want to share a type which you would also use normally as a thread-local variable, and that means casting. - Jonathan M Davis +1
Re: The no gc crowd
On 2013-10-10 09:33, Jonathan M Davis wrote: You might do that if you're creating the object simply to send it across, but it's frequently the case that the object was created well before it was sent across, and it frequently had to have operations done it other than simply creating it (which wouldn't work if it were shared). So, it often wouldn't make sense for the object being passed to be shared except when being passed. I guess if you're not creating it as shared to being with there's not way to tell that the given object now is shared an no thread local references are allowed. And once it's been passed, it's rarely the case that you want it to be shared. You're usually passing ownership. You're essentially taking a thread-local variable from one thread and making it a thread-local variable on another thread. Unfortunately, the type system does not support the concept of thread ownership (beyond thread-local vs shared), so it's up to the programmer to make sure that no references to the object are kept on the original thread, but there's really no way around that unless you're always creating a new object when you pass it across, which would result in which would usually be a unnecessary copy. So, it becomes like @trusted in that sense. It sounds like we need a way to transfer ownership of an object to a different thread. I honestly don't think we can solve it a different way without completely redesigning shared. shared is specifically designed such that you have to either cast it way to do anything with it or write all of your code to explicitly work with shared, which is not something that generally makes sense to do unless you're creating a type whose only value is in being shared across threads. Far more frequently, you want to share a type which you would also use normally as a thread-local variable, and that means casting. I guess it wouldn't be possible to solve it without changing the type system. -- /Jacob Carlborg
Re: The no gc crowd
On 2013-10-10 09:24, Jonathan M Davis wrote:
Pretty much nothing accepts shared. At best, templated functions accept
shared. Certainly, shared doesn't work at all with classes and structs unless
the type is specifically intended to be used as shared, because you have to
mark all of its member functions shared to be able to call them. And if you
want to use that class or struct as both shared and unshared, you have to
duplicate all of its member functions.
That being the case, the only way in general to use a shared object is to
protect it with a lock, cast it to thread-local (so that it can actually use
its member functions or be passed to other functions to be used), and then use
it. e.g.
synchronized
{
auto tl = cast(T)mySharedT;
auto result = tl.foo();
auto result2 = bar(tl);
}
Obviously, you then have to make sure that there are no thread-local
references to the shared object when the lock is released, but without casting
away shared like that, you can't do much of anything with it. So, similar to
when you cast away const, it's up to you to guarantee that the code doesn't
violate the type system's guarantees - i.e. that a thread-local variable is
not accessed by multiple threads. So, you use a lock of some kind to protect
the shared variable while it's treated as a thread-local variable in order to
ensure that that guarantee holds. Like with casting away const or with
@trusted, there's obviously risk in doing this, but there's really no other
way to use shared at this point - certainly not without it being incredibly
invasive to your code and forcing code duplication.
Sounds like we need a way to tell that a parameter is thread local but
not allowed to escape a reference to it.
Object foo;
void bar (shared_tls Object o)
{
foo = o; // Compile error, cannot escape a shared thread local
}
void main ()
{
auto o = new shared(Object);
synchronized { bar(o); }
}
Both shared can thread local be passed to shared_tls. If shared is
passed it assumes to be synchronized during the call to bar.
This will still have the problem of annotating all code with this
attribute. Or this needs to be default, which would cause a lot of code
breakage.
--
/Jacob Carlborg
Re: The no gc crowd
On 2013-10-10 01:21:25 +, deadalnix deadal...@gmail.com said: On Wednesday, 9 October 2013 at 23:37:53 UTC, Michel Fortin wrote: In an ideal world, we'd be able to choose between using a GC or using ARC when building our program. A compiler flag could do the trick. But that becomes messy when libraries (static and dynamic) get involved as they all have to agree on the same codegen to work together. Adding something to mangling that would cause link errors in case of mismatch might be good enough to prevent accidents though. ObjC guys used to think that. It turns out it is a really bad idea. Things were much worse with Objective-C because at the time there was no ARC, reference-counting was manual and supporting both required a lot of manual work. Supporting the GC wasn't always a easy either, as the GC only tracked pointers inside of Objective-C objects and on the stack, not in structs on the heap. The GC had an implementation problem for pointers inside static segments, and keeping code working on both a GC and reference-counted had many perils. I think it can be done better in D. We'd basically just be changing the GC algorithm so it uses reference counting. The differences are: 1. unpredictable lifetimes - predictable lifetime 2. no bother about cyclic references - need to break them with weak The later is probably the most problematic, but if someone has leaks because he uses a library missing weak annotations he can still run the GC to collect them while most memory is reclaimed through ARC, or he can fix the problematic library by adding weak at the right places. -- Michel Fortin michel.for...@michelf.ca http://michelf.ca
Re: The no gc crowd
On 2013-10-10 06:41:19 +, Jacob Carlborg d...@me.com said: On 2013-10-10 06:24, Jonathan M Davis wrote: So, it's my impression that being able to consider casting to or from shared as abnormal in code which uses shared is a bit of a pipe dream at this point. The current language design pretty much requires casting when doing much of anything with concurrency. There must be a better way to solve this. http://michelf.ca/blog/2012/mutex-synchonization-in-d/ -- Michel Fortin michel.for...@michelf.ca http://michelf.ca
Re: The no gc crowd
On Thursday, 10 October 2013 at 04:24:31 UTC, Jonathan M Davis wrote: Also, casting _away_ shared is going to be a very common operation due to how shared works. It is yet another use case for `scope` storage class. Locking `shared` variable via mutex should return same variable but casted to non-shared `scope` (somewhere inside the locking library function). Then it is safe to pass it to functions accepting scope parameters as reference won't possibly escape.
Re: The no gc crowd
On 2013-10-10 13:17, Michel Fortin wrote: http://michelf.ca/blog/2012/mutex-synchonization-in-d/ This looks similar to what I described here: http://forum.dlang.org/thread/bsqqfmhgzntryyaqr...@forum.dlang.org?page=19#post-l35rql:24og2:241:40digitalmars.com -- /Jacob Carlborg
Re: The no gc crowd
On 2013-10-10 13:17, Michel Fortin wrote: http://michelf.ca/blog/2012/mutex-synchonization-in-d/ I think I like the idea, but won't you have the same problem as Jonathan described? You can't pass these variables to another function that doesn't expect it to be passed a synchronized variable. You can pass it to pure functions which mean you can probably pass it to a couple of more functions compared to using shared. -- /Jacob Carlborg
Re: The no gc crowd
On 2013-10-10 13:02:14 +, Jacob Carlborg d...@me.com said: On 2013-10-10 13:17, Michel Fortin wrote: http://michelf.ca/blog/2012/mutex-synchonization-in-d/ I think I like the idea, but won't you have the same problem as Jonathan described? You can't pass these variables to another function that doesn't expect it to be passed a synchronized variable. You can pass it to pure functions which mean you can probably pass it to a couple of more functions compared to using shared. Well, it's one piece of a puzzle. In itself it already is better than having to cast every time. Combined with a way to pass those variables to other functions safely, it should solve practically all the remaining problems that currently require a cast. But I don't have nice a solution for the later problem (short of adding more attributes). -- Michel Fortin michel.for...@michelf.ca http://michelf.ca
Re: The no gc crowd
On 2013-10-10 13:03:37 +, Jacob Carlborg d...@me.com said: On 2013-10-10 13:17, Michel Fortin wrote: http://michelf.ca/blog/2012/mutex-synchonization-in-d/ This looks similar to what I described here: http://forum.dlang.org/thread/bsqqfmhgzntryyaqr...@forum.dlang.org?page=19#post-l35rql:24og2:241:40digitalmars.com Somewhat similar. I don't think it's a good practice to make a mutex part of the public interface of an object (or any public interface for that matter), which is why they're kept private inside the class in my examples. Public mutexes can be locked from anywhere in your program, they lack encapsulation and this makes them prone to deadlocks. -- Michel Fortin michel.for...@michelf.ca http://michelf.ca
Re: The no gc crowd
On 09/10/13 06:25, Andrei Alexandrescu wrote: The way I see it we must devise a robust solution to that, NOT consider the state of the art immutable (heh, a pun). Must say I have had a miserable experience with immutability and any kind of complex data structure, particularly when concurrency is involved. As a practical fact I've often found it necessary to convert to immutable (not always via a cast or std.conv.to; sometimes via assumeUnique) to pass a complex data structure to a thread, but then to convert _away_ from immutable inside the thread in order for that data (which is never actually mutated) to be practically usable. I'm sure there are things that I could do better, but I did not find a superior solution that was also performance-friendly.
Re: The no gc crowd
On 10/10/2013 12:51 AM, Jacob Carlborg wrote: On 2013-10-10 09:18, Walter Bright wrote: 1. Shared data cannot be passed to regular functions. That I understand. 2. Functions that create data structures would have to know in advance that they'll be creating a shared object. I'm not so sure this would not be an invasive change. If the function doesn't know it creates shared data it will assume it's not and it won't use any synchronization. Then suddenly someone casts it to shared and you're in trouble. Same comment as for immutable data - create the data structure as thread local, because of (1), and then cast to shared and hand it to another thread. 3. Immutable data is implicitly shared. But it is not created immutable - it is created as mutable data, then set to some state, then cast to immutable. It should be possible to create immutable data in the first place. No cast should be required.
Re: The no gc crowd
On Oct 9, 2013, at 9:24 PM, Jonathan M Davis jmdavisp...@gmx.com wrote: And given that std.concurrency requires casting to and from shared or immutable in order to pass objects across threads, it seems ilke most of D's concurrency model requires casting to and/or from shared or immutable. std.concurrency won't be this way forever though. We could fake move semantics with something like assumeUnique!T, so send could be modified to accept a non-shared class that's marked as Unique. The other option would be deep copying or serialization.
Re: The no gc crowd
On Oct 9, 2013, at 11:34 PM, Jacob Carlborg d...@me.com wrote: On 2013-10-10 02:22, Sean Kelly wrote: Only that this would have to be communicated to the user, since moving data later is problematic. Today, I think it's common to construct an object as unshared and then cast it. What is the reason to not create it as shared in the first place? The same as immutable--you may not have all the shared functions available to establish the desired state. But I'll grant that this is obviously way more common with immutable than shared.
Re: The no gc crowd
On Thursday, October 10, 2013 19:23:14 Joseph Rushton Wakeling wrote: On 09/10/13 06:25, Andrei Alexandrescu wrote: The way I see it we must devise a robust solution to that, NOT consider the state of the art immutable (heh, a pun). Must say I have had a miserable experience with immutability and any kind of complex data structure, particularly when concurrency is involved. As a practical fact I've often found it necessary to convert to immutable (not always via a cast or std.conv.to; sometimes via assumeUnique) to pass a complex data structure to a thread, but then to convert _away_ from immutable inside the thread in order for that data (which is never actually mutated) to be practically usable. I'm sure there are things that I could do better, but I did not find a superior solution that was also performance-friendly. std.concurrency's design basically requires that you cast objects to shared or immutable in order to pass them across threads (and using assumeUnique is still doing the cast, just internally). And then you have to cast them back to thread-local mutable on the other side to complete the pass and make the object useable again. There's really no way around that at this point, not without completely redesigning shared. Arguably, it's better to use shared when doing that rather than immutable, but at least in the past, shared hasn't worked right with std.concurrency even though it's supposed to (though that's an implementation issue rather than a design one, and it might be fixed by now - I haven't tried recently). And whether you're using shared or immutable, you're still having to cast. I'm honestly surprised that Andrei is rejecting the idea of casting to/from shared or immutable being normal given how it's required by our current concurrency model. And changing that would be a _big_ change. - Jonathan M Davis
Re: The no gc crowd
On Oct 10, 2013, at 4:17 AM, Michel Fortin michel.for...@michelf.ca wrote: On 2013-10-10 06:41:19 +, Jacob Carlborg d...@me.com said: On 2013-10-10 06:24, Jonathan M Davis wrote: So, it's my impression that being able to consider casting to or from shared as abnormal in code which uses shared is a bit of a pipe dream at this point. The current language design pretty much requires casting when doing much of anything with concurrency. There must be a better way to solve this. http://michelf.ca/blog/2012/mutex-synchonization-in-d/ Good article. But why didn't you mention core.sync? It has both a Mutex and a ReadWriteMutex (ie. shared_mutex).
Re: The no gc crowd
On 10/10/13 19:31, Jonathan M Davis wrote: I'm honestly surprised that Andrei is rejecting the idea of casting to/from shared or immutable being normal given how it's required by our current concurrency model. And changing that would be a _big_ change. I'm starting to incline towards the view that type qualifications of _any_ kind become problematic once you start working with any types other than built-in, and not just in the context of concurrency. See e.g.: http://d.puremagic.com/issues/show_bug.cgi?id=11148 http://d.puremagic.com/issues/show_bug.cgi?id=11188 I'd really appreciate advice on how to handle issues like these, because it's becoming a serious obstacle to my work on std.rational.
Re: The no gc crowd
On Oct 10, 2013, at 10:23 AM, Joseph Rushton Wakeling joseph.wakel...@webdrake.net wrote: On 09/10/13 06:25, Andrei Alexandrescu wrote: The way I see it we must devise a robust solution to that, NOT consider the state of the art immutable (heh, a pun). Must say I have had a miserable experience with immutability and any kind of complex data structure, particularly when concurrency is involved. As long as the reference itself can be reassigned (tail-immutable, I suppose) I think immutable is occasionally quite useful for complex data structures. It basically formalizes the RCU (read-copy-update) approach to wait-free concurrency. I'd tend to use this most often for global data structures built up on app start, and updated rarely to never as the program runs.
Re: The no gc crowd
On 10/10/13 19:39, Sean Kelly wrote: As long as the reference itself can be reassigned (tail-immutable, I suppose) I think immutable is occasionally quite useful for complex data structures. It basically formalizes the RCU (read-copy-update) approach to wait-free concurrency. I'd tend to use this most often for global data structures built up on app start, and updated rarely to never as the program runs. This kind of stuff is outside my experience, so if you'd like to offer a more detailed explanation/example, I'd be very grateful :-)
Re: The no gc crowd
On Oct 10, 2013, at 10:36 AM, Joseph Rushton Wakeling joseph.wakel...@webdrake.net wrote: On 10/10/13 19:31, Jonathan M Davis wrote: I'm honestly surprised that Andrei is rejecting the idea of casting to/from shared or immutable being normal given how it's required by our current concurrency model. And changing that would be a _big_ change. I'm starting to incline towards the view that type qualifications of _any_ kind become problematic once you start working with any types other than built-in, and not just in the context of concurrency. See e.g.: http://d.puremagic.com/issues/show_bug.cgi?id=11148 http://d.puremagic.com/issues/show_bug.cgi?id=11188 I'm inclined to agree about shared. But I see this largely as more encouragement to keep data thread-local in D. If we can clean up move semantics via std.concurrency, I would be reasonably happy with data sharing in D. As for const / immutable, I guess I don't see this as such an issue because I've been dealing with it in C++ for so long. You either have to commit 100% to using const attributes or not use them at all. Anything in between is fraught with problems.
Re: The no gc crowd
On Thursday, October 10, 2013 10:27:24 Sean Kelly wrote: On Oct 9, 2013, at 9:24 PM, Jonathan M Davis jmdavisp...@gmx.com wrote: And given that std.concurrency requires casting to and from shared or immutable in order to pass objects across threads, it seems ilke most of D's concurrency model requires casting to and/or from shared or immutable. std.concurrency won't be this way forever though. We could fake move semantics with something like assumeUnique!T, so send could be modified to accept a non-shared class that's marked as Unique. I take it that you mean something other than std.exception.assumeUnique which simply casts to immutable? All that std.exception.assumeUnique does for you over casting is document why the cast is happening. If you're talking about creating a wrapper type which indicates that the object is unique, I'd still expect that the casting would have to be happening underneath the hood when the object was passed (though then for better or worse, it would be encapsulated). And unless the objecte were always in that Unique wrapper, the programmer would still have to be promising that the object was actually unique and not being shared across threads rather than the type system doing it, in which case, I don't see much gain over simply casting. And if it's always in the wrapper, then you're in a similar boat to shared or immutable in that it's not the correct type. I expect that there are nuances in what you're suggesting that I don't grasp at the moment, but as far as I can tell, the type system fundamentally requires a cast when passing objects across threads. It's just a question of whether that cast is hidden or not, and depending on how you hide it, I think that there's a real risk of the situation being worse than if you require explicit casting, because then what you're doing and what you have to be careful about are less obvious, since what's going on is hidden. The other option would be deep copying or serialization. That would be far too costly IMHO. In the vast majority of cases (in my experience at least and from what I've seen others do), what you really want to do is pass ownership of the object from one thread to the other, and while deep copying would allow you to avoid type system issues, it's completely unnecessary otherwise. So, we'd be introducing overhead just to satisfy our very restrictive type system. The only way that I can think of to fix that would be for objects to all have a concept of what thread owns them (so that the type system would be able to understand the concept of an object's ownership being passed from one thread to another), but that would be a _big_ change and likely way too complicated in general. - Jonathan M Davis
Re: The no gc crowd
On Oct 10, 2013, at 10:43 AM, Joseph Rushton Wakeling joseph.wakel...@webdrake.net wrote: On 10/10/13 19:39, Sean Kelly wrote: As long as the reference itself can be reassigned (tail-immutable, I suppose) I think immutable is occasionally quite useful for complex data structures. It basically formalizes the RCU (read-copy-update) approach to wait-free concurrency. I'd tend to use this most often for global data structures built up on app start, and updated rarely to never as the program runs. This kind of stuff is outside my experience, so if you'd like to offer a more detailed explanation/example, I'd be very grateful :-) Configuration data, for example. On app start you might load a config file, generate information about the user, and so on, before real processing begins. This data needs to be visible everywhere and it rarely if ever changes as the program runs, so you fill the data structures and then make them immutable. Assuming, of course, that the data structures have immutable versions of all the necessary functions (which is unfortunately a pretty big assumption).
Re: The no gc crowd
On 10/10/13 12:18 AM, Walter Bright wrote: On 10/9/2013 11:34 PM, Jacob Carlborg wrote: On 2013-10-10 02:22, Sean Kelly wrote: Only that this would have to be communicated to the user, since moving data later is problematic. Today, I think it's common to construct an object as unshared and then cast it. What is the reason to not create it as shared in the first place? 1. Shared data cannot be passed to regular functions. I don't understand this. If a function/method accepts shared, then it can be passed shared data. 2. Functions that create data structures would have to know in advance that they'll be creating a shared object. I'm not so sure this would not be an invasive change. There is no other way around it. And this is not a change - it's fixing something. 3. Immutable data is implicitly shared. But it is not created immutable - it is created as mutable data, then set to some state, then cast to immutable. That all must happen in the runtime, NOT in user code. Andrei
Re: The no gc crowd
On Oct 10, 2013, at 10:50 AM, Jonathan M Davis jmdavisp...@gmx.com wrote: On Thursday, October 10, 2013 10:27:24 Sean Kelly wrote: On Oct 9, 2013, at 9:24 PM, Jonathan M Davis jmdavisp...@gmx.com wrote: And given that std.concurrency requires casting to and from shared or immutable in order to pass objects across threads, it seems ilke most of D's concurrency model requires casting to and/or from shared or immutable. std.concurrency won't be this way forever though. We could fake move semantics with something like assumeUnique!T, so send could be modified to accept a non-shared class that's marked as Unique. I take it that you mean something other than std.exception.assumeUnique which simply casts to immutable? All that std.exception.assumeUnique does for you over casting is document why the cast is happening. If you're talking about creating a wrapper type which indicates that the object is unique, I'd still expect that the casting would have to be happening underneath the hood when the object was passed (though then for better or worse, it would be encapsulated). And unless the objecte were always in that Unique wrapper, the programmer would still have to be promising that the object was actually unique and not being shared across threads rather than the type system doing it, in which case, I don't see much gain over simply casting. And if it's always in the wrapper, then you're in a similar boat to shared or immutable in that it's not the correct type. I expect that there are nuances in what you're suggesting that I don't grasp at the moment, but as far as I can tell, the type system fundamentally requires a cast when passing objects across threads. It's just a question of whether that cast is hidden or not, and depending on how you hide it, I think that there's a real risk of the situation being worse than if you require explicit casting, because then what you're doing and what you have to be careful about are less obvious, since what's going on is hidden. Yes, we couldn't use assumeUnique as-is because then the object would land on the other side as immutable. It would have to wrap the object to tell send() that the object, while not shared or immutable, is safe to put in a message. Then send() would discard the wrapper while constructing the message.
Re: The no gc crowd
On 10/10/13 12:33 AM, Jonathan M Davis wrote: I honestly don't think we can solve it a different way without completely redesigning shared. shared is specifically designed such that you have to either cast it way to do anything with it no or write all of your code to explicitly work with shared, which is not something that generally makes sense to do unless you're creating a type whose only value is in being shared across threads. yes Far more frequently, you want to share a type which you would also use normally as a thread-local variable, and that means casting. no Andrei
Re: The no gc crowd
On Oct 10, 2013, at 10:55 AM, Andrei Alexandrescu seewebsiteforem...@erdani.org wrote: On 10/10/13 12:33 AM, Jonathan M Davis wrote: Far more frequently, you want to share a type which you would also use normally as a thread-local variable, and that means casting. no Yeah, I'd want to see this claim backed up by some examples. The only data I share globally in my own apps is the occasional container. Configuration data, a user database, whatever. I'll also frequently move data between threads while dispatching tasks, but otherwise everything is thread-local. I imagine there are other reasonable methods for using shared data, but I don't know what they are.
Re: The no gc crowd
On Wed, 09 Oct 2013 22:30:05 +0200, Adam D. Ruppe wrote:
On Wednesday, 9 October 2013 at 20:10:40 UTC, Justin Whear wrote:
Related to the latter, it would be really nice to be able to prove that
a section of code makes no heap allocations/GC collections.
As a quick temporary thing, how about gc_throw_on_next(); ?
That'd just set a thread local flag that gc_malloc checks and if it is
set, immediately resets it and throws an AllocAssertError. My thought is
this could be quickly and easily implemented pending a better solution
and in the mean time can be used in unit tests to help check this stuff.
So user-code would look like this?
// Set up code, GC is fine here
...
// Entering critical loop (which may run for months at a time)
debug GC.throw_on_next(true);
while (true)
{
...
}
// Tear-down code, GC is fine here
// (though unnecessary as the process is about to exit)
debug GC.throw_on_next(false);
...
Something like this would make testing simpler and is probably much more
feasible than deep static analysis.
Re: The no gc crowd
On 10/10/13 19:50, Sean Kelly wrote: Configuration data, for example. On app start you might load a config file, generate information about the user, and so on, before real processing begins. This data needs to be visible everywhere and it rarely if ever changes as the program runs, so you fill the data structures and then make them immutable. Assuming, of course, that the data structures have immutable versions of all the necessary functions (which is unfortunately a pretty big assumption). Yup, you're right, it's a big assumption. In my case I was interested in loading a graph (network) and running many simulations on it in parallel. The graph itself was static, so could readily be made immutable. However, I found that it was difficult to write code that would accept both immutable and mutable graphs as input, without impacting performance. So, I opted for threads to receive an immutable graph and cast it to mutable, even though it was never actually altered. My experience was no doubt partially due to issues with the overall design I chose, and maybe I could have found a way around it, but it just seemed easier to use this flawed approach than to re-work everything.
Re: The no gc crowd
On 10/10/13 19:46, Sean Kelly wrote:
As for const / immutable, I guess I don't see this as such an issue because
I've been dealing with it in C++ for so long. You either have to commit 100%
to using const attributes or not use them at all. Anything in between is
fraught with problems.
Well, the problem is essentially that you can have a function like:
void foo(int i) { ... }
... and if you pass it an immutable or const int, this is not a problem, because
you're passing by value.
But now try
void foo(BigInt i) { ... }
... and it won't work when passed a const/immutable variable, even though again
you're passing by value. That's not nice, not intuitive, and generally speaking
makes working with complex data types annoying.
It's why, for example, std.math.abs currently works with BigInt but not with
const or immutable BigInt -- which is very irritating indeed.
Re: The no gc crowd
On Oct 10, 2013, at 11:17 AM, Joseph Rushton Wakeling joseph.wakel...@webdrake.net wrote: On 10/10/13 19:50, Sean Kelly wrote: Configuration data, for example. On app start you might load a config file, generate information about the user, and so on, before real processing begins. This data needs to be visible everywhere and it rarely if ever changes as the program runs, so you fill the data structures and then make them immutable. Assuming, of course, that the data structures have immutable versions of all the necessary functions (which is unfortunately a pretty big assumption). Yup, you're right, it's a big assumption. In my case I was interested in loading a graph (network) and running many simulations on it in parallel. The graph itself was static, so could readily be made immutable. However, I found that it was difficult to write code that would accept both immutable and mutable graphs as input, without impacting performance. So, I opted for threads to receive an immutable graph and cast it to mutable, even though it was never actually altered. My experience was no doubt partially due to issues with the overall design I chose, and maybe I could have found a way around it, but it just seemed easier to use this flawed approach than to re-work everything. That's kind of the issue I ran into with shared in Druntime. It seemed like what I had to do was have a shared method that internally cast this to unshared and then called the real function, which I knew was safe but the type system hated. But this seemed like a horrible approach and so I didn't ever qualify anything as shared.
Re: The no gc crowd
On Oct 10, 2013, at 11:21 AM, Joseph Rushton Wakeling
joseph.wakel...@webdrake.net wrote:
On 10/10/13 19:46, Sean Kelly wrote:
As for const / immutable, I guess I don't see this as such an issue because
I've been dealing with it in C++ for so long. You either have to commit
100% to using const attributes or not use them at all. Anything in between
is fraught with problems.
Well, the problem is essentially that you can have a function like:
void foo(int i) { ... }
... and if you pass it an immutable or const int, this is not a problem,
because you're passing by value.
But now try
void foo(BigInt i) { ... }
... and it won't work when passed a const/immutable variable, even though
again you're passing by value. That's not nice, not intuitive, and generally
speaking makes working with complex data types annoying.
It's why, for example, std.math.abs currently works with BigInt but not with
const or immutable BigInt -- which is very irritating indeed.
Isn't BigInt a struct? I'd expect it to work via copying just like concrete
types.
Re: The no gc crowd
On Thursday, 10 October 2013 at 17:23:20 UTC, Joseph Rushton Wakeling wrote: On 09/10/13 06:25, Andrei Alexandrescu wrote: The way I see it we must devise a robust solution to that, NOT consider the state of the art immutable (heh, a pun). Must say I have had a miserable experience with immutability and any kind of complex data structure, particularly when concurrency is involved. I feel your pain. See also this thread in D.learn: http://forum.dlang.org/post/sdefkajobwcfikkel...@forum.dlang.org
Re: The no gc crowd
On 2013-10-10 17:34:47 +, Sean Kelly s...@invisibleduck.org said: On Oct 10, 2013, at 4:17 AM, Michel Fortin michel.for...@michelf.ca wrote: http://michelf.ca/blog/2012/mutex-synchonization-in-d/ Good article. But why didn't you mention core.sync? It has both a Mutex and a ReadWriteMutex (ie. shared_mutex). Because that would have required a ton of explanations about why you need casts everywhere to remove shared, and I don't even know where to begin to explain shared semantics. Shared just doesn't make sense to me the way it works right now. The examples in C++ are much clearer than anything I could have done in D2. I don't want to have to explain why I have to bypass the type system every time I need to access a variable. I'll add that I'm coding in C++ right now so it's much easier to come up with C++ examples. That said, it might be a good idea to add a note at the end about core.sync in case someone wants to try that technique in D. -- Michel Fortin michel.for...@michelf.ca http://michelf.ca
Re: The no gc crowd
On Thu, Oct 10, 2013 at 07:36:06PM +0200, Joseph Rushton Wakeling wrote:
On 10/10/13 19:31, Jonathan M Davis wrote:
I'm honestly surprised that Andrei is rejecting the idea of casting
to/from shared or immutable being normal given how it's required by
our current concurrency model. And changing that would be a _big_
change.
I'm starting to incline towards the view that type qualifications of
_any_ kind become problematic once you start working with any types
other than built-in, and not just in the context of concurrency.
See e.g.:
http://d.puremagic.com/issues/show_bug.cgi?id=11148
http://d.puremagic.com/issues/show_bug.cgi?id=11188
I'd really appreciate advice on how to handle issues like these,
because it's becoming a serious obstacle to my work on std.rational.
I left some comments on these bugs. Basically, BigInt should not be
implicitly castable from const/immutable to unqual, because unlike the
built-in types, it's *not* a value type:
BigInt x = 123;
BigInt y = x; // creates an alias to x's data.
Allowing implicit conversion to unqual would break immutability:
immutable(BigInt) x = 123;
const(BigInt) sneaky = x; // sneaky aliases x
BigInt y = sneaky; // now y aliases sneaky, which aliases x (oops)
Of course, the way BigInt is implemented, any operation on it causes new
data to be created (essentially it behaves like a copy-on-write type),
so it's not as though you can directly modify immutable this way, but
it breaks the type system and opens up possible loopholes.
What you need to do is to use inout for functions that need to handle
both built-in ints and BigInts, e.g.:
inout(Num) abs(Num)(inout(Num) x) {
return (x = 0) ? x : -x;
}
This *should* work (I think -- I didn't check :-P).
Arguably, a *lot* of generic code involving numerical operations is
broken, because they assume built-in types' behaviour of being
implicitly convertible to/from immutable (due to being value types).
I don't know about shared, though. Last I heard, shared was one big mess
so I'm not even going to touch it.
T
--
If the comments and the code disagree, it's likely that *both* are wrong. --
Christopher
Re: The no gc crowd
On 10/10/2013 10:54 AM, Andrei Alexandrescu wrote: On 10/10/13 12:18 AM, Walter Bright wrote: On 10/9/2013 11:34 PM, Jacob Carlborg wrote: On 2013-10-10 02:22, Sean Kelly wrote: Only that this would have to be communicated to the user, since moving data later is problematic. Today, I think it's common to construct an object as unshared and then cast it. What is the reason to not create it as shared in the first place? 1. Shared data cannot be passed to regular functions. I don't understand this. If a function/method accepts shared, then it can be passed shared data. I meant regular functions as in they are not typed as taking shared arguments. Shared cannot be implicitly cast to unshared. I say regular because very, very few functions are typed as accepting shared arguments. 2. Functions that create data structures would have to know in advance that they'll be creating a shared object. I'm not so sure this would not be an invasive change. There is no other way around it. And this is not a change - it's fixing something. I'm not convinced of that at all. 3. Immutable data is implicitly shared. But it is not created immutable - it is created as mutable data, then set to some state, then cast to immutable. That all must happen in the runtime, NOT in user code. Andrei
Re: The no gc crowd
On 10/10/13 20:28, Sean Kelly wrote: Isn't BigInt a struct? I'd expect it to work via copying just like concrete types. Yes, it's a struct, but somewhere inside its internals I think it contains arrays. I'm not sure how that affects copying etc., but suffice to say that if you try the following: BigInt a = 2; BigInt b = a; b = 3; assert(a != b); assert(a !is b); ... then it passes. So it behaves at least in this extent like a value type. But suffice to say that it was an unpleasant surprise that I couldn't just take it and pass to a function accepting an unqualified BigInt argument.
Re: The no gc crowd
On 10/10/13 20:28, H. S. Teoh wrote: I don't know about shared, though. Last I heard, shared was one big mess so I'm not even going to touch it. Yes, that seems to be the consensus.
Re: The no gc crowd
On 2013-10-10 16:14, Michel Fortin wrote: similar. I don't think it's a good practice to make a mutex part of the public interface of an object (or any public interface for that matter), which is why they're kept private inside the class in my examples. Public mutexes can be locked from anywhere in your program, they lack encapsulation and this makes them prone to deadlocks. Right, it's better to keep them private. -- /Jacob Carlborg
Re: The no gc crowd
On 10/10/13 20:28, H. S. Teoh wrote:
I left some comments on these bugs. Basically, BigInt should not be
implicitly castable from const/immutable to unqual, because unlike the
built-in types, it's *not* a value type:
BigInt x = 123;
BigInt y = x; // creates an alias to x's data.
BigInt a = 2;
BigInt b = a;
b = 3;
writeln(a);
writeln(b);
... gives you:
2
3
So, even though there's an array hidden away inside std.BigInt, it still seems
to copy via value.
Of course, the way BigInt is implemented, any operation on it causes new
data to be created (essentially it behaves like a copy-on-write type),
so it's not as though you can directly modify immutable this way, but
it breaks the type system and opens up possible loopholes.
I guess that explains my result above ?
What you need to do is to use inout for functions that need to handle
both built-in ints and BigInts, e.g.:
inout(Num) abs(Num)(inout(Num) x) {
return (x = 0) ? x : -x;
}
This *should* work (I think -- I didn't check :-P).
I did, and it results in issues with BigInt's opCmp. But that may say more
about BigInt's opCmp than about your solution.
Arguably, a *lot* of generic code involving numerical operations is
broken, because they assume built-in types' behaviour of being
implicitly convertible to/from immutable (due to being value types).
How would you suggest correcting that?
Re: The no gc crowd
On Thu, Oct 10, 2013 at 08:39:52PM +0200, Joseph Rushton Wakeling wrote: On 10/10/13 20:28, Sean Kelly wrote: Isn't BigInt a struct? I'd expect it to work via copying just like concrete types. Yes, it's a struct, but somewhere inside its internals I think it contains arrays. I'm not sure how that affects copying etc., but suffice to say that if you try the following: BigInt a = 2; BigInt b = a; b = 3; assert(a != b); assert(a !is b); ... then it passes. So it behaves at least in this extent like a value type. I took a glance over the BigInt code, and it appears to have some kind of copy-on-write semantics. For example, in your code above, when you wrote b=a, b actually *aliases* a, but when you assign 3 to b, a new data array is created and b is updated to point to the new data instead. So it's not really a true value type, but more like a COW reference type. But suffice to say that it was an unpleasant surprise that I couldn't just take it and pass to a function accepting an unqualified BigInt argument. That only works with true value types, but BigInt isn't really one of them. :) T -- Everybody talks about it, but nobody does anything about it! -- Mark Twain
Re: The no gc crowd
On 10/10/2013 10:27 AM, Sean Kelly wrote: [...] Sean - whatever means you're using to reply breaks the thread.
Re: The no gc crowd
On Oct 10, 2013, at 12:11 PM, Walter Bright newshou...@digitalmars.com wrote: On 10/10/2013 10:27 AM, Sean Kelly wrote: [...] Sean - whatever means you're using to reply breaks the thread. The mailing list Brad set up--I can't do NNTP from most locations. I guess I'll use the website.
Re: The no gc crowd
On 10/10/2013 12:38 PM, Sean Kelly wrote: On Oct 10, 2013, at 12:11 PM, Walter Bright newshou...@digitalmars.com wrote: On 10/10/2013 10:27 AM, Sean Kelly wrote: [...] Sean - whatever means you're using to reply breaks the thread. The mailing list Brad set up--I can't do NNTP from most locations. I guess I'll use the website. I'm curious why NNTP would be blocked. I've been able to access it from any wifi hotspots I've tried it from.
Re: The no gc crowd
On Thursday, 10 October 2013 at 20:50:10 UTC, Walter Bright wrote: I'm curious why NNTP would be blocked. I've been able to access it from any wifi hotspots I've tried it from. My only guess is that usenet may be perceived as a illegal file sharing resource. But it's been a while since I've tried, so I'll give it another shot.
Re: The no gc crowd
On Thursday, 10 October 2013 at 17:36:11 UTC, Joseph Rushton
Wakeling wrote:
On 10/10/13 19:31, Jonathan M Davis wrote:
I'm honestly surprised that Andrei is rejecting the idea of
casting to/from
shared or immutable being normal given how it's required by
our current
concurrency model. And changing that would be a _big_ change.
I'm starting to incline towards the view that type
qualifications of _any_ kind become problematic once you start
working with any types other than built-in, and not just in the
context of concurrency. See e.g.:
http://d.puremagic.com/issues/show_bug.cgi?id=11148
http://d.puremagic.com/issues/show_bug.cgi?id=11188
I'd really appreciate advice on how to handle issues like
these, because it's becoming a serious obstacle to my work on
std.rational.
As qnzc pointed out - check out this thread:
http://forum.dlang.org/post/sdefkajobwcfikkel...@forum.dlang.org
Your problems with BigInt occur because the language has a
special optimization for assignment of structs with no mutable
aliasing. Fundamental math types have no aliasing so that
assignment from any to all is fine and efficient via a data copy.
For types like BigInt with mutable aliasing crossing from mutable
to immutable and back is a no-go because of the reasons pointed
out in the response to bug 11148. You can not and should not be
able to do what you are asking - pass mutable with aliasing into
immutable because then immutable would not be guaranteed.
Two options are copy BigInt beforehand if you want to keep by
value semantics on your function signatures or maybe pass by
reference (they are big after all so why copy)?
Passing by ref won't really be the full solution to your problem
on comment 6 of bug 11148. What you really want to do is take a
const(BigInt) or ref to it and make a mutable copy. So do that!
But wait, how can you do that? You need to have a dup type
function. Ideally there would be a generic way to do this. I have
one that works for most cases and including yours:
https://github.com/patefacio/d-help/blob/master/d-help/opmix/dup.d
This support could easily be put in the standard.
import std.bigint, std.stdio;
import opmix.mix;
void foo(const(BigInt) n) {
// make mutable copy
auto bi = n.gdup;
bi *= 2;
writeln(bi);
}
void main() {
const cbi = BigInt(1234567890987654321);
foo(cbi);
writeln(cbi);
}
--
2469135781975308642
1234567890987654321
Thanks,
Dan
Re: The no gc crowd
On Thursday, 10 October 2013 at 17:39:55 UTC, Sean Kelly wrote: On Oct 10, 2013, at 10:23 AM, Joseph Rushton Wakeling joseph.wakel...@webdrake.net wrote: On 09/10/13 06:25, Andrei Alexandrescu wrote: The way I see it we must devise a robust solution to that, NOT consider the state of the art immutable (heh, a pun). Must say I have had a miserable experience with immutability and any kind of complex data structure, particularly when concurrency is involved. As long as the reference itself can be reassigned (tail-immutable, I suppose) I think immutable is occasionally quite useful for complex data structures. It basically formalizes the RCU (read-copy-update) approach to wait-free concurrency. I'd tend to use this most often for global data structures built up on app start, and updated rarely to never as the program runs. Nice. Please show an example that includes complex data with associative arrays. Thanks Dan
Re: The no gc crowd
On Thursday, 10 October 2013 at 21:15:39 UTC, Sean Kelly wrote: On Thursday, 10 October 2013 at 20:50:10 UTC, Walter Bright wrote: I'm curious why NNTP would be blocked. I've been able to access it from any wifi hotspots I've tried it from. My only guess is that usenet may be perceived as a illegal file sharing resource. But it's been a while since I've tried, so I'll give it another shot. No luck. I can get to the SSL port (563) on usenet hosts that offer it, but not port 119 from anywhere I care to check news from.
Re: The no gc crowd
On Thursday, October 10, 2013 11:28:02 Walter Bright wrote: On 10/10/2013 10:54 AM, Andrei Alexandrescu wrote: On 10/10/13 12:18 AM, Walter Bright wrote: 1. Shared data cannot be passed to regular functions. I don't understand this. If a function/method accepts shared, then it can be passed shared data. I meant regular functions as in they are not typed as taking shared arguments. Shared cannot be implicitly cast to unshared. I say regular because very, very few functions are typed as accepting shared arguments. Yeah. The only times that something is going to accept shared is when it was specifically designed to work as shared (which most code isn't), or if it's templated and the template happens to work with shared. Regular functions just aren't going to work with shared without casting away shared, because that would usually mean either templating everything or duplicating functions all over the place. - Jonathan M Davis
Re: The no gc crowd
On Thursday, October 10, 2013 11:11:12 Sean Kelly wrote: On Oct 10, 2013, at 10:55 AM, Andrei Alexandrescu seewebsiteforem...@erdani.org wrote: On 10/10/13 12:33 AM, Jonathan M Davis wrote: Far more frequently, you want to share a type which you would also use normally as a thread-local variable, and that means casting. no Yeah, I'd want to see this claim backed up by some examples. The only data I share globally in my own apps is the occasional container. Configuration data, a user database, whatever. I'll also frequently move data between threads while dispatching tasks, but otherwise everything is thread-local. I imagine there are other reasonable methods for using shared data, but I don't know what they are. Yeah, but it's that moving data between threads while dispatching tasks which requires casting. Pretty much anything which isn't a value type has to be cast to either immutable or shared in order to pass it across threads, and then it needs to then be cast back to thread-local mutable on the other side for to be useable. Unless you're only passing really basic stuff like int, or the types that your passing are only used for being passed across threads (and thus are designed to work as shared), you end up having to cast. The fact that you can only pass shared or immutable objects across combined with the fact that shared objects are generally unusable makes it so that you're at minimum going to have to either cast the object once it gets to the other thread, even if it was constructed as shared. And since shared is so useless means that if you need to do anything more than simply construct the object before passing it across, you're going have to have it as thread-local in the originating thread as well. I just don't see how you could avoid casting when passing ownership of an object from one thread to another without having a way to pass an object across threads without having to make it shared or immutable to pass it. - Jonathan M Davis
Re: The no gc crowd
On Thursday, 10 October 2013 at 23:33:27 UTC, Jonathan M Davis wrote: Yeah. The only times that something is going to accept shared is when it was specifically designed to work as shared (which most code isn't), or if it's templated and the template happens to work with shared. Regular functions just aren't going to work with shared without casting away shared, because that would usually mean either templating everything or duplicating functions all over the place. I think that's pretty reasonable though. Shared data needs to be treated differently, explicitly, or things go downhill fast.
Re: The no gc crowd
On Thursday, 10 October 2013 at 23:33:17 UTC, Jonathan M Davis wrote: I just don't see how you could avoid casting when passing ownership of an object from one thread to another without having a way to pass an object across threads without having to make it shared or immutable to pass it. Well, the restriction to only pass immutable and shared data is simply enforced statically by the API. So if there were an assumeUnique analog, the check could be modified to accept that as well, and then the class would arrive as unshared. This could be accomplished pretty easily. It would be yet another step towards not having thread-local pools though. I was initially pretty conservative in what was an acceptable type to send, because it's always easier to loosen restrictions than tighten them.
Re: The no gc crowd
On 2013-10-10, 20:28, H. S. Teoh wrote:
On Thu, Oct 10, 2013 at 07:36:06PM +0200, Joseph Rushton Wakeling wrote:
On 10/10/13 19:31, Jonathan M Davis wrote:
I'm honestly surprised that Andrei is rejecting the idea of casting
to/from shared or immutable being normal given how it's required by
our current concurrency model. And changing that would be a _big_
change.
I'm starting to incline towards the view that type qualifications of
_any_ kind become problematic once you start working with any types
other than built-in, and not just in the context of concurrency.
See e.g.:
http://d.puremagic.com/issues/show_bug.cgi?id=11148
http://d.puremagic.com/issues/show_bug.cgi?id=11188
I'd really appreciate advice on how to handle issues like these,
because it's becoming a serious obstacle to my work on std.rational.
I left some comments on these bugs. Basically, BigInt should not be
implicitly castable from const/immutable to unqual, because unlike the
built-in types, it's *not* a value type:
[snip]
What you need to do is to use inout for functions that need to handle
both built-in ints and BigInts, e.g.:
[snip]
Here's a COW reference type that I can easily pass to a function
requiring a mutable version of the type:
struct S {
immutable(int)[] arr;
}
And usage:
void foo(S s) {}
void main() {
const S s;
foo(s);
}
This compiles and works beautifully. Of course, no actual COW is
happening here, but COW is what the type system says has to happen.
Another example COW type:
string;
Now, my point here is that BigInt could easily use an immutable
buffer internally, as long as it's purely COW. It could, and it should.
If it did, we would not be having this discussion, as bugs #11148 and
#11188 would not exist. Inventing rules like 'you should use inout'
does not help - it's obscuring the problem.
TLDR: Do not use inout(T). Fix BigInt.
--
Simen
Re: The no gc crowd
On Thursday, October 10, 2013 10:55:49 Andrei Alexandrescu wrote: On 10/10/13 12:33 AM, Jonathan M Davis wrote: I honestly don't think we can solve it a different way without completely redesigning shared. shared is specifically designed such that you have to either cast it way to do anything with it no or write all of your code to explicitly work with shared, which is not something that generally makes sense to do unless you're creating a type whose only value is in being shared across threads. yes Really? Do you honestly expect the average use of shared to involve creating structs or classes which are designed specifically to be used as shared? That definitely has its use, but I would expect it to be far more common that someone would want to share the exact same types that they're using in their thread-local code. In fact, if anything, the normal responses to discussions on shared go in the complete opposite direction of creating classes which are designed to work as shared. It seems like the normal thing to do is simply avoid shared altogether and use __gshared so that you don't have to deal with any of the problems that shared causes. Granted, I obviously haven't seen everyone's code, but I don't believe that I have ever seen anyone create a type designed to be used as shared, and that's certainly not what people discuss doing when shared comes up. TDPL discusses that - and again, I do think that that has its place - but I've never seen it done, and I've never run into any place in my own code where I would have even considered it. Usually, you want to share an object of the same type that you're using in your thread-local code. And even if a struct or class is set up so that its member functions work great as shared, very little code seems to be written with shared in mind (since thread-local is the default), so the only functions which will work with it are its member functions, functions written specifically to work with that type, and templated functions that happen to work with shared. As such, I fully expect casting away shared to be a very common idiom. Without that, the number of things you can do with a shared object is very limited. Far more frequently, you want to share a type which you would also use normally as a thread-local variable, and that means casting. no What else do you expect to be doing with std.concurrency? That's what it's _for_. Unless all of the stuff that you're passing across threads are value types or are designed to work as immutable or shared (which most types aren't), the objects which get passed across need to be cast to thread-local mutable on the target thread in order to be used there, and if you have to do much of anything with the object other than constructing it before passing it across, then you're going to have to have it as thread-local on the originating thread as well, because most functions are going to be unusable with shared. - Jonathan M Davis
Re: The no gc crowd
On 10/10/13 10:36 AM, Joseph Rushton Wakeling wrote: On 10/10/13 19:31, Jonathan M Davis wrote: I'm honestly surprised that Andrei is rejecting the idea of casting to/from shared or immutable being normal given how it's required by our current concurrency model. And changing that would be a _big_ change. I'm starting to incline towards the view that type qualifications of _any_ kind become problematic once you start working with any types other than built-in, and not just in the context of concurrency. See e.g.: http://d.puremagic.com/issues/show_bug.cgi?id=11148 http://d.puremagic.com/issues/show_bug.cgi?id=11188 I'd really appreciate advice on how to handle issues like these, because it's becoming a serious obstacle to my work on std.rational. I'll look into this soon. Andrei
Re: The no gc crowd
On Friday, 11 October 2013 at 00:30:35 UTC, Simen Kjaeraas wrote:
Here's a COW reference type that I can easily pass to a function
requiring a mutable version of the type:
struct S {
immutable(int)[] arr;
}
And usage:
void foo(S s) {}
void main() {
const S s;
foo(s);
}
This compiles and works beautifully. Of course, no actual COW is
happening here, but COW is what the type system says has to
happen.
Another example COW type:
string;
Now, my point here is that BigInt could easily use an immutable
buffer internally, as long as it's purely COW. It could, and it
should.
If it did, we would not be having this discussion, as bugs
#11148 and
#11188 would not exist. Inventing rules like 'you should use
inout'
does not help - it's obscuring the problem.
TLDR: Do not use inout(T). Fix BigInt.
Good catch. immutable(T)[] is special.
Do the same with a contained associative array and you'll be my
hero.
Re: The no gc crowd
On Friday, October 11, 2013 02:05:19 Sean Kelly wrote: It would be yet another step towards not having thread-local pools though. At this point, I don't see how we can have thread-local pools unless casting to and from shared has hooks for managing that. Otherwise, it's far too likely that an object is going to be in the wrong pool, because it's being used as shared when it was constructed as thread-local or vice versa. And we may need some sort of hook with std.concurrency.send which understands that the object being sent is being transferred from one thread to another and would tell the GC to migrate the object from one pool to another (though to do that, it would probably have to not be typed as shared but rather as thread-local, which would jive better with what you're talking about doing with std.concurrency). Certainly, with how shared currently works, it's hard to see how we could get away with having thread-local GC pools as great as that would be. So, if we want that, something about how shared works is going to have to change. - Jonathan M Davis
Re: The no gc crowd
On Friday, October 11, 2013 02:08:16 Sean Kelly wrote:
On Thursday, 10 October 2013 at 23:33:27 UTC, Jonathan M Davis
wrote:
Yeah. The only times that something is going to accept shared
is when it was
specifically designed to work as shared (which most code
isn't), or if it's
templated and the template happens to work with shared. Regular
functions just
aren't going to work with shared without casting away shared,
because that
would usually mean either templating everything or duplicating
functions all over the place.
I think that's pretty reasonable though. Shared data needs to be
treated differently, explicitly, or things go downhill fast.
I'm not disagreeing with how shared works. I'm disagreeing with the idea that
it's not supposed to be normal to cast shared away when operating on shared
objects. I expect that the most common idiom for dealing with shared is to
protect it with a lock, cast it to thread-local, do whatever you're going to
do with it, make sure that there are no thread-local references to it once
you're done operating on it, and then release the lock. e.g.
synchronized
{
auto tc = cast(T)mySharedT;
tc.memberFunc();
doStuff(tc);
//no thread-local references to tc other than tc should
//exist at this point.
}
That works perfectly fine and makes it so that shared objects are clearly
delineated from thread-local ones by the type system, but it does require
casting, and it requires that you make sure that the object is not misused
while it's not being treated as shared.
The only real alternative to that is to create types which are designed to be
operated on as shared, but I would expect that to be the rare case rather than
the norm, because that requires creating new types just for sharing across
threads rather than using the same types that you use in your thread-local
code, and I don't expect programmers to want to do that in the average case.
However, from what I've seen, at the moment, the most typical reaction is to
simply use __gshared, which is the least safe of the various options. So,
people need to be better educated and/or we need figure out a different design
for shared.
- Jonathan M Davis
Re: The no gc crowd
On 11/10/13 02:30, Simen Kjaeraas wrote: TLDR: Do not use inout(T). Fix BigInt. Yes, that's my feeling too. Even if you do add inout (or in, which also works in terms of allowing qualified BigInts) to std.math.abs, you immediately run back into problems ... with BigInt.
Re: The no gc crowd
On 10/10/13 4:33 PM, Jonathan M Davis wrote: I just don't see how you could avoid casting when passing ownership of an object from one thread to another without having a way to pass an object across threads without having to make it shared or immutable to pass it. By using restricted library types. Andrei
Re: The no gc crowd
On 11/10/13 02:44, Andrei Alexandrescu wrote: I'll look into this soon. That would be fantastic, thank you very much. Any chance you could ask Don Clugston to get in touch with me about these issues? std.bigint is his, and I know he was in touch with David Simcha about std.rational. I don't have his email address, as he (understandably) has a fake email address set up as his reply-to here.
