Re: shared adventures in the realm of thread-safety.
Graham St Jack Wrote: So, what is the design of shared supposed to be then? Its time for Walter to buy in and tell us where this is all going - I for one am very confused right now. Here's what I know: Bartosz's ownership scheme is delayed until at least D3 Shared code will be sequentially consistent Walter likes the idea of optimizing away memory barriers that the compiler can prove are unneeded (some barriers in synchronized sections) Bartosz is rewriting how threads are done similar to what his blogs hint at Issues that Bartosz hits with shared are fixed immediately Here's what I suspect from a number of emails: Because every class contains a monitor, Walter/dmd will treat every class as its own monitor for the purposes of optimization. I too wish Walter would advertise the design, but I think the simple fact is that he doesn't know what the design is!
Re: shared adventures in the realm of thread-safety.
On Wed, 16 Sep 2009 08:00:40 -0400, Jason House wrote: Graham St Jack Wrote: So, what is the design of shared supposed to be then? Its time for Walter to buy in and tell us where this is all going - I for one am very confused right now. Thanks for that. Its good to know that there is a plan in there somewhere, even if the details are still very fuzzy. I agree that the lofty goal of improving thread-safety for mere mortals is worthwhile, and that it won't be easy to pull off. What I was really after though is what the plan is for D2 right now. The whole shared situation in D2 looks like a mess to me, and I would like some reassurance that something simple and tidy will be happening soon. Here's what I know: Bartosz's ownership scheme is delayed until at least D3 Shared code will be sequentially consistent Walter likes the idea of optimizing away memory barriers that the compiler can prove are unneeded (some barriers in synchronized sections) Bartosz is rewriting how threads are done similar to what his blogs hint at Issues that Bartosz hits with shared are fixed immediately Here's what I suspect from a number of emails: Because every class contains a monitor, Walter/dmd will treat every class as its own monitor for the purposes of optimization. I too wish Walter would advertise the design, but I think the simple fact is that he doesn't know what the design is!
Re: shared adventures in the realm of thread-safety.
So, what is the design of shared supposed to be then? Its time for Walter to buy in and tell us where this is all going - I for one am very confused right now. Currently I am working around it by not using synchronized methods (I put synchronized blocks inside the methods), which is very bad form, but what else can I do?
Re: shared adventures in the realm of thread-safety.
Graham St Jack Wrote: So, what is the design of shared supposed to be then? Its time for Walter to buy in and tell us where this is all going - I for one am very confused right now. Currently I am working around it by not using synchronized methods (I put synchronized blocks inside the methods), which is very bad form, but what else can I do? You're already more adventurous than I am, my current workaround has been to drop all threading support in my runtime, I'm focusing on different parts of my project that do not require threading for now and I'll piece it together when shared gets closer to a final concept.
Re: shared adventures in the realm of thread-safety.
Robert Jacques Wrote:
On Sun, 13 Sep 2009 18:08:57 -0400, Jeremie Pelletier jerem...@gmail.com
wrote:
Robert Jacques Wrote:
On Sun, 13 Sep 2009 15:04:57 -0400, Jeremie Pelletier
jerem...@gmail.com
wrote:
[snip]
Unique data could only be used for aggregate properties,
const/immutable
data would also be implicitly unique. This qualifier alone would
simplify shared quite a lot, allowing the use of unshared objects in
shared contexts safely.
Neither const nor immutable data can be considered unique. First, any
const data may be being mutated by another routine, so it can't be
safely
accessed without synchronization. Second, unique data is mutable while
const/immutable data is not. Third, most implementations of unique allow
for deterministic memory reclamation, which isn't possible if the unique
data might actually be const/immutable.
Good points, I can only agree with you here. However I still believe
immutable data should be able to be used in shared contexts without
being 'shared' or protected by a monitor.
One of the purposes behind immutable was lock-free access. As far as I
know you can use immutable data in shared contexts today without any other
modifiers. A quick test seems to indicate this works today, but if you've
got a test case where it doesn't, I'd recommend filing it as a bug.
Oh yeah, I'm confusing it with 'shared' methods not able to call 'const shared'
methods, which is a pain in the ass :(
The compiler should make the distinction between shared code and
shared
data and allow both shared and unshared instances to use shared
methods,
just like both const and mutable instances may call const methods. An
error should also be triggered when calling a shared method of a
shared
object without synchronization, and maybe have a __sync keyword to
override this. If a synchronized method is called from a non-shared
object, no synchronization takes place.
I think you have the wrong paradigm in mind. Shared and non-shared
aren't
mutable and const. They're mutable and immutable. From a technical
perspective, synchronization of shared methods are handled by the
callee,
so there is no way not to call them and non-shared objects don't have a
monitor that can be synchronized. Now you can have the compiler use the
same code to generate two different object types (vtables, object
layouts,
etc) with have the same interface, but that doesn't sound like what
you're
suggesting.
I know that shared/unshared is not const/mutable. What I meant is that
right now in D if a method is 'shared' it cannot be called from a
non-shared object, which makes unshared instance of the class unusable
without plenty of dirty casts. Take the following objects:
class Foo { void foo() const; }
class Bar { void bar() shared; }
Foo foo; foo.foo(); // ok, mutable object can call const method
Bar bar; bar.bar(); // error, unshared object may not call shared method
I had only presented the concept, your idea of using two virtual tables
for shared/unshared instances is also what I had in mind for the
implementation, and it would give exactly the behavior I had in mind.
Bartosz took the concept one step further: when declared as shared, all
methods are implicitly wrapped in synchronize blocks. He then added a
keyword for more manual, lock-free style programming. But this syntactic
sugar isn't implemented yet.
The current D keywords (synchronized and shared) are already designed for that,
since synchronized implies shared. I don't want implicit synchronization, I'd
much rather have a shared class marking all its members/properties as shared
and letting me explicitely decide where the synchronization takes place.
Allow me to illustrate my point with some code:
class Foo {
int bar() shared { return a; }
__sync bar2() { synchronized(this) return a; }
synchronized void foo() { a = 1; }
int a;
}
auto foo1 = new shared(Foo)();
auto foo2 = new Foo;
foo1.foo(); // ok, synchronized call
synchronized(foo1) foo1.foo(); // warning: recursive synchronization
Why a warning? Monitors are designed to handle recursive
synchronization.
Its a performance issue that can easily be avoided, but still generates
valid code.
Really? Every public method that calls another public method (of the same
object) results in recursive synchronization. And if your example was
longer than a one liner, you'd also have to have recursive
synchronization. There are ways to reduce recursive synchronization, like
public wrappers of protected/private methods, but they are not always
appropriate or feasible for the use case. BTW, in general the threshold
for what's a warning in DMD is generally a lot higher than other compilers
(on the theory that if warnings are generated for every build
Re: shared adventures in the realm of thread-safety.
Robert Jacques Wrote: On Sun, 13 Sep 2009 18:08:57 -0400, Jeremie Pelletier jerem...@gmail.com wrote: . Bartosz took the concept one step further: when declared as shared, all methods are implicitly wrapped in synchronize blocks. He then added a keyword for more manual, lock-free style programming. But this syntactic sugar isn't implemented yet. That is not the design for D2. shared means shared. It is neither meant to mean synchronized nor lockfree. I worry about optimization opportunities for shared in D2. There may be way too many memory fences in synchronized code. Without a mapping of a monitor to what's protected under a monitor, the compiler/optimizer's hands are tied. At best, every object will be its own monitor, but that hardly makes any sense...
Re: shared adventures in the realm of thread-safety.
On Mon, 14 Sep 2009 07:44:44 -0400, Jason House jason.james.ho...@gmail.com wrote: Robert Jacques Wrote: On Sun, 13 Sep 2009 18:08:57 -0400, Jeremie Pelletier jerem...@gmail.com wrote: . Bartosz took the concept one step further: when declared as shared, all methods are implicitly wrapped in synchronize blocks. He then added a keyword for more manual, lock-free style programming. But this syntactic sugar isn't implemented yet. That is not the design for D2. shared means shared. It is neither meant to mean synchronized nor lockfree. I worry about optimization opportunities for shared in D2. There may be way too many memory fences in synchronized code. Without a mapping of a monitor to what's protected under a monitor, the compiler/optimizer's hands are tied. At best, every object will be its own monitor, but that hardly makes any sense... That is the Java model by the way. And really, no one knows what the shared design is for D2. The keyword has been added, but it has yet to be fleshed out. And once you start talking about how to flesh it out and what kinds of syntactic sugar are wanted/needed you need to start looking at previous solutions, which is what Bartosz has done in his blog posts. The specific issue you raise, that of excessive monitors, was addressed using the concept of unique/mobile objects which are both thread-safe and don't require locking. However, it appears that this won't make it into D2, which I feel is a shame.
Re: shared adventures in the realm of thread-safety.
Jason House Wrote:
I'm glad to see I'm not the only one trying to use shared. I tried to use it
with 2.031 and rapidly hit bug after bug... I submitted several bug reports
for basic functionality, and none of it appeared in the changelog.
http://d.puremagic.com/issues/show_bug.cgi?id=3089
shared methods no longer trigger errors in 2.032, this issue should be marked
as fixed, it must be a side effect from fixing another issue.
http://d.puremagic.com/issues/show_bug.cgi?id=3090
I just made a quick template this seems to work in 2.032:
immutable template isShared(T) {
static if(is(T U : shared U))
bool isShared = true;
else
bool isShared = false;
}
http://d.puremagic.com/issues/show_bug.cgi?id=3091
This one still isn't solved, I too found it annoying that you cant use 'new
shared Foo()'. You can however declare Foo as 'shared class Foo', this works
just like 'immutable class' or 'const class' by marking all properties and
members with the qualifier. As a side note, I would like the same behavior for
'static class'.
However using 'shared class' is not always the wanted behavior, you may only
want a subset of the members and properties to be shared.
We also miss a unique qualifier to allow unshared objects to be used in shared
contexts without the need for shared methods.
It's also awkward to use, const members may be called from either const or
mutable objects. Shared members must be called from shared objects, so why
allow a class to have shared and unshared members, if all instances are going
to be shared anyways? It makes it much harder to draw the line between shared
and unshared; you may have only a few objects actually shared, but you are
required to make all the other objects they may use shared, even if they are
synchronized or unique. This makes all the other contexts these objects are
used in as shared, and soon your entire program is shared.
Re: shared adventures in the realm of thread-safety.
Graham St Jack Wrote:
I'm also having the same problems.
As Jeremie said, as soon as you start introducing shared methods (via
synchronized for example), you rapidly get into trouble that can only be
overcome by excessive casting.
It may be possible to contain the problem by refactoring multi-threaded
code so that the shared objects are very small and simple, but even then
the casting required is too much. This approach might be ok if you could
define classes as being shared or immutable, and ALL instance of them
were then implicitly shared or immutable. Also, immutable objects should
be implicitly shareable.
I agree, this is also one of my main concerns with shared in its current state.
It's an amazing and powerful concept and has the potential to make multi-thread
code much easier and safer to write. But all the required casting is killing
the safety, and makes it harder to use than it would be not having shared at
all. The lack of an unique qualifier certainly doesn't help either.
Unique data could only be used for aggregate properties, const/immutable data
would also be implicitly unique. This qualifier alone would simplify shared
quite a lot, allowing the use of unshared objects in shared contexts safely.
The compiler should make the distinction between shared code and shared data
and allow both shared and unshared instances to use shared methods, just like
both const and mutable instances may call const methods. An error should also
be triggered when calling a shared method of a shared object without
synchronization, and maybe have a __sync keyword to override this. If a
synchronized method is called from a non-shared object, no synchronization
takes place.
Allow me to illustrate my point with some code:
class Foo {
int bar() shared { return a; }
__sync bar2() { synchronized(this) return a; }
synchronized void foo() { a = 1; }
int a;
}
auto foo1 = new shared(Foo)();
auto foo2 = new Foo;
foo1.foo(); // ok, synchronized call
synchronized(foo1) foo1.foo(); // warning: recursive synchronization
foo2.foo(); // ok, unsynchronized call
synchronized(foo2) foo2.foo(); // ok synchronized call
foo1.bar(); // error, unsynchronized call to bar() shared
synchronized(foo1) foo1.bar(); // ok, synchronized call
foo2.bar(); // ok, unsynchronized call
synchronized(foo1) foo1.bar(); // ok, synchronized call
foo1.bar2(); // ok, method handles synchronized
synchronized(foo1) foo1.bar2(); // warning, recursive synchronization
foo2.bar2(); // ok, method handles synchronized, even on unshared object
synchronized(foo2) foo2.bar2(); // warning, recursive synchronization, even on
unshared object
That's about it, I believe this behavior would allow quite a number of
multi-threaded techniques to be easily implemented and documented. It would
only be the most natural thing since its quite similar to how const works.
Re: shared adventures in the realm of thread-safety.
Jeremie Pelletier Wrote: Jason House Wrote: I'm glad to see I'm not the only one trying to use shared. I tried to use it with 2.031 and rapidly hit bug after bug... I submitted several bug reports for basic functionality, and none of it appeared in the changelog. http://d.puremagic.com/issues/show_bug.cgi?id=3089 shared methods no longer trigger errors in 2.032, this issue should be marked as fixed, it must be a side effect from fixing another issue. *sigh*
Re: shared adventures in the realm of thread-safety.
Robert Jacques Wrote:
On Sun, 13 Sep 2009 15:04:57 -0400, Jeremie Pelletier jerem...@gmail.com
wrote:
[snip]
Unique data could only be used for aggregate properties, const/immutable
data would also be implicitly unique. This qualifier alone would
simplify shared quite a lot, allowing the use of unshared objects in
shared contexts safely.
Neither const nor immutable data can be considered unique. First, any
const data may be being mutated by another routine, so it can't be safely
accessed without synchronization. Second, unique data is mutable while
const/immutable data is not. Third, most implementations of unique allow
for deterministic memory reclamation, which isn't possible if the unique
data might actually be const/immutable.
Good points, I can only agree with you here. However I still believe immutable
data should be able to be used in shared contexts without being 'shared' or
protected by a monitor.
The compiler should make the distinction between shared code and shared
data and allow both shared and unshared instances to use shared methods,
just like both const and mutable instances may call const methods. An
error should also be triggered when calling a shared method of a shared
object without synchronization, and maybe have a __sync keyword to
override this. If a synchronized method is called from a non-shared
object, no synchronization takes place.
I think you have the wrong paradigm in mind. Shared and non-shared aren't
mutable and const. They're mutable and immutable. From a technical
perspective, synchronization of shared methods are handled by the callee,
so there is no way not to call them and non-shared objects don't have a
monitor that can be synchronized. Now you can have the compiler use the
same code to generate two different object types (vtables, object layouts,
etc) with have the same interface, but that doesn't sound like what you're
suggesting.
I know that shared/unshared is not const/mutable. What I meant is that right
now in D if a method is 'shared' it cannot be called from a non-shared object,
which makes unshared instance of the class unusable without plenty of dirty
casts. Take the following objects:
class Foo { void foo() const; }
class Bar { void bar() shared; }
Foo foo; foo.foo(); // ok, mutable object can call const method
Bar bar; bar.bar(); // error, unshared object may not call shared method
I had only presented the concept, your idea of using two virtual tables for
shared/unshared instances is also what I had in mind for the implementation,
and it would give exactly the behavior I had in mind.
Allow me to illustrate my point with some code:
class Foo {
int bar() shared { return a; }
__sync bar2() { synchronized(this) return a; }
synchronized void foo() { a = 1; }
int a;
}
auto foo1 = new shared(Foo)();
auto foo2 = new Foo;
foo1.foo(); // ok, synchronized call
synchronized(foo1) foo1.foo(); // warning: recursive synchronization
Why a warning? Monitors are designed to handle recursive synchronization.
Its a performance issue that can easily be avoided, but still generates valid
code.
foo2.foo(); // ok, unsynchronized call
synchronized(foo2) foo2.foo(); // ok synchronized call
foo1.bar(); // error, unsynchronized call to bar() shared
synchronized(foo1) foo1.bar(); // ok, synchronized call
foo2.bar(); // ok, unsynchronized call
synchronized(foo1) foo1.bar(); // ok, synchronized call
foo1.bar2(); // ok, method handles synchronized
synchronized(foo1) foo1.bar2(); // warning, recursive synchronization
foo2.bar2(); // ok, method handles synchronized, even on unshared object
synchronized(foo2) foo2.bar2(); // warning, recursive synchronization,
even on unshared object
That's about it, I believe this behavior would allow quite a number of
multi-threaded techniques to be easily implemented and documented. It
would only be the most natural thing since its quite similar to how
const works.
The major benefit of const isn't method declaration, but object use: i.e.
only having to declare func(const T var) and not func(immutable T var) and
func(T var). Currently, there's no planned type to fill this role though
there have been some proposals.
I disagree, I think const methods are just as useful as const objects, since
they are the only methods that can be called on such objects. They do not
however prevent you from calling them on a mutable object. This is the behavior
I want with shared too; unshared objects should be able to call shared methods,
but shared objects should only be able to call shared methods.
P.S. Shouldn't 'a' be either private or protected?
It should, but this was just an example ;)
P.S.S. Bartosz Milewski has a good series of blogs on multi-threading
(with an eye on how to do it well in D).
I know, this is what sparked my interest for shared in
Re: shared adventures in the realm of thread-safety.
On 2009-09-13 18:08:57 -0400, Jeremie Pelletier jerem...@gmail.com said: foo1.foo(); // ok, synchronized call synchronized(foo1) foo1.foo(); // warning: recursive synchronization Why a warning? Monitors are designed to handle recursive synchronization. Its a performance issue that can easily be avoided, but still generates valid code. Also, some people consider recursive locks as potential design flaws. http://landheer-cieslak.com/wordpress/?p=57 -- Michel Fortin michel.for...@michelf.com http://michelf.com/
Re: shared adventures in the realm of thread-safety.
On Sun, 13 Sep 2009 18:08:57 -0400, Jeremie Pelletier jerem...@gmail.com
wrote:
Robert Jacques Wrote:
On Sun, 13 Sep 2009 15:04:57 -0400, Jeremie Pelletier
jerem...@gmail.com
wrote:
[snip]
Unique data could only be used for aggregate properties,
const/immutable
data would also be implicitly unique. This qualifier alone would
simplify shared quite a lot, allowing the use of unshared objects in
shared contexts safely.
Neither const nor immutable data can be considered unique. First, any
const data may be being mutated by another routine, so it can't be
safely
accessed without synchronization. Second, unique data is mutable while
const/immutable data is not. Third, most implementations of unique allow
for deterministic memory reclamation, which isn't possible if the unique
data might actually be const/immutable.
Good points, I can only agree with you here. However I still believe
immutable data should be able to be used in shared contexts without
being 'shared' or protected by a monitor.
One of the purposes behind immutable was lock-free access. As far as I
know you can use immutable data in shared contexts today without any other
modifiers. A quick test seems to indicate this works today, but if you've
got a test case where it doesn't, I'd recommend filing it as a bug.
The compiler should make the distinction between shared code and
shared
data and allow both shared and unshared instances to use shared
methods,
just like both const and mutable instances may call const methods. An
error should also be triggered when calling a shared method of a
shared
object without synchronization, and maybe have a __sync keyword to
override this. If a synchronized method is called from a non-shared
object, no synchronization takes place.
I think you have the wrong paradigm in mind. Shared and non-shared
aren't
mutable and const. They're mutable and immutable. From a technical
perspective, synchronization of shared methods are handled by the
callee,
so there is no way not to call them and non-shared objects don't have a
monitor that can be synchronized. Now you can have the compiler use the
same code to generate two different object types (vtables, object
layouts,
etc) with have the same interface, but that doesn't sound like what
you're
suggesting.
I know that shared/unshared is not const/mutable. What I meant is that
right now in D if a method is 'shared' it cannot be called from a
non-shared object, which makes unshared instance of the class unusable
without plenty of dirty casts. Take the following objects:
class Foo { void foo() const; }
class Bar { void bar() shared; }
Foo foo; foo.foo(); // ok, mutable object can call const method
Bar bar; bar.bar(); // error, unshared object may not call shared method
I had only presented the concept, your idea of using two virtual tables
for shared/unshared instances is also what I had in mind for the
implementation, and it would give exactly the behavior I had in mind.
Bartosz took the concept one step further: when declared as shared, all
methods are implicitly wrapped in synchronize blocks. He then added a
keyword for more manual, lock-free style programming. But this syntactic
sugar isn't implemented yet.
Allow me to illustrate my point with some code:
class Foo {
int bar() shared { return a; }
__sync bar2() { synchronized(this) return a; }
synchronized void foo() { a = 1; }
int a;
}
auto foo1 = new shared(Foo)();
auto foo2 = new Foo;
foo1.foo(); // ok, synchronized call
synchronized(foo1) foo1.foo(); // warning: recursive synchronization
Why a warning? Monitors are designed to handle recursive
synchronization.
Its a performance issue that can easily be avoided, but still generates
valid code.
Really? Every public method that calls another public method (of the same
object) results in recursive synchronization. And if your example was
longer than a one liner, you'd also have to have recursive
synchronization. There are ways to reduce recursive synchronization, like
public wrappers of protected/private methods, but they are not always
appropriate or feasible for the use case. BTW, in general the threshold
for what's a warning in DMD is generally a lot higher than other compilers
(on the theory that if warnings are generated for every build you'll never
read them)
[snip]
Bike-shed: I've always preferred the CSP/pi-calculas term 'mobile' for
the
concept of 'unique'. I think mobile better expresses the concept with
regard to multi-threading, where mobile is used to cheaply transfer data
between threads (i.e. it moves around/can move between threads, but
isn't
shared between them). I find 'unique' to mainly convey the memory
storage
aspect of the concept, which is less important outside of C/C++.
Maybe this is where 'volatile' could come back, from what I know it's
still a reserved keyword in D and would fit nicely this
Re: shared adventures in the realm of thread-safety.
I'm glad to see I'm not the only one trying to use shared. I tried to use it
with 2.031 and rapidly hit bug after bug... I submitted several bug reports for
basic functionality, and none of it appeared in the changelog.
http://d.puremagic.com/issues/show_bug.cgi?id=3089
http://d.puremagic.com/issues/show_bug.cgi?id=3090
http://d.puremagic.com/issues/show_bug.cgi?id=3091
Jeremie Pelletier Wrote:
I decided to play once again with shared and see what 2.032 is capable of.
Turns out a lot of the previous issues I was having last time are gone,
however, there are still a few things left which prevent me from rewriting my
code.
The first issue that jumped to my face straight away was how 'shared const'
methods are not callable from 'shared' objects.
shared class Foo {
void bar() const;
}
auto foo = new Foo; // foo is of type shared(Foo)
foo.bar; // Error: function Foo.bar () shared const is not callable using
argument types () shared
Considering how 'const' methods can be called from mutable objects, this
looks like either a bug or a really awkward feature to me. Sending a
shared(Foo) to a method expecting a shared(const(Foo)) also triggers a
similar error from the compiler.
The other issue may be an intended feature, but it doesn't sound practical to
me. Marking a method as shared assumes all used properties in the method's
scope are also shared. Here is an example to illustrate my point:
class SimpleReader {
this(LocalFile file) { _stream = new FileInputStream(file); }
...
private:
synchronized void read(ubyte[] buf, long offset) {
_stream.seek(offset);
_stream.read(buf);
}
FileInputStream _stream;
}
The FileInputStream here is a generic blocking binary stream which is not
thread-safe by design. The reader is a composite class where every instance
has its own unique stream instance and use it to implement asynchronous reads
over the file format it abstracts, which in my case is a specialized
read-only archive using a lot of random accesses from different threads.
This is where the issue shows its ugly head. The 'synchronized' keyword tags
the read method as shared, which in itself is quite neat, what is annoying
however is that it also changes the type of _stream in the method's scope to
shared(FileInputStream) and therefore triggers compiler errors because
_stream.seek and _stream.read are not shared:
Error: function FileInputStream.read (ubyte[]) is not callable using argument
types (ubyte[]) shared
While it may be an attempt to keep shared usage safe, it isn't very
practical. The stream object here is not shared because it is not
thread-safe. While it may be used by different threads, it is unique to the
reader's context and its accesses are synchronized by the reader, the stream
should therefore be completely oblivious to the fact it is being used by
different threads.
Maybe this could be the time to implement an unique qualifier; this is a
context where having _stream be of type unique(FileInputStream) would solve
the problem and allow further compiler optimizations. I don't know if it can
be done with templates, and without any overhead whatsoever. I know I would
much rather see unique(Foo) than Unique!Foo, and it would allow the use of
'is(foo : unique)'.
Furthermore, tagging a method with shared does not make it thread-safe, it
may however use synchronized within its scope to protect its shared or unique
data. This may be confusing when calling shared methods vs calling
synchronized methods; one may think the shared one is not thread-safe and
optionally synchronize the call, resulting in another monitor being used for
nothing, or no monitor being used at all:
class Foo {
shared void bar() {
// Do stuff with local or immutable data
synchronized(this) { /* do stuff with shared data */ }
}
shared void bar2() {
// Do stuff on shared data
}
}
Someone seeing only the prototype of Foo.bar may assume the method is not
thread-safe and call it as 'synchronized(foo) foo.bar()'. Just like they
could see the prototype of bar2 and assume it is thread-safe, calling it as
'foo.bar2()'.
What could be a good design against this sort of misleading behavior?
Phew, that's about enough issues and questions for now :)
Re: shared adventures in the realm of thread-safety.
I'm also having the same problems.
As Jeremie said, as soon as you start introducing shared methods (via
synchronized for example), you rapidly get into trouble that can only be
overcome by excessive casting.
It may be possible to contain the problem by refactoring multi-threaded
code so that the shared objects are very small and simple, but even then
the casting required is too much. This approach might be ok if you could
define classes as being shared or immutable, and ALL instance of them
were then implicitly shared or immutable. Also, immutable objects should
be implicitly shareable.
On Sat, 12 Sep 2009 15:32:05 -0400, Jason House wrote:
I'm glad to see I'm not the only one trying to use shared. I tried to
use it with 2.031 and rapidly hit bug after bug... I submitted several
bug reports for basic functionality, and none of it appeared in the
changelog.
http://d.puremagic.com/issues/show_bug.cgi?id=3089
http://d.puremagic.com/issues/show_bug.cgi?id=3090
http://d.puremagic.com/issues/show_bug.cgi?id=3091
Jeremie Pelletier Wrote:
I decided to play once again with shared and see what 2.032 is capable
of. Turns out a lot of the previous issues I was having last time are
gone, however, there are still a few things left which prevent me from
rewriting my code.
The first issue that jumped to my face straight away was how 'shared
const' methods are not callable from 'shared' objects.
shared class Foo {
void bar() const;
}
auto foo = new Foo; // foo is of type shared(Foo) foo.bar; // Error:
function Foo.bar () shared const is not callable using argument types
() shared
Considering how 'const' methods can be called from mutable objects,
this looks like either a bug or a really awkward feature to me. Sending
a shared(Foo) to a method expecting a shared(const(Foo)) also triggers
a similar error from the compiler.
The other issue may be an intended feature, but it doesn't sound
practical to me. Marking a method as shared assumes all used properties
in the method's scope are also shared. Here is an example to illustrate
my point:
class SimpleReader {
this(LocalFile file) { _stream = new FileInputStream(file); } ...
private:
synchronized void read(ubyte[] buf, long offset) {
_stream.seek(offset);
_stream.read(buf);
}
FileInputStream _stream;
}
The FileInputStream here is a generic blocking binary stream which is
not thread-safe by design. The reader is a composite class where every
instance has its own unique stream instance and use it to implement
asynchronous reads over the file format it abstracts, which in my case
is a specialized read-only archive using a lot of random accesses from
different threads.
This is where the issue shows its ugly head. The 'synchronized' keyword
tags the read method as shared, which in itself is quite neat, what is
annoying however is that it also changes the type of _stream in the
method's scope to shared(FileInputStream) and therefore triggers
compiler errors because _stream.seek and _stream.read are not shared:
Error: function FileInputStream.read (ubyte[]) is not callable using
argument types (ubyte[]) shared
While it may be an attempt to keep shared usage safe, it isn't very
practical. The stream object here is not shared because it is not
thread-safe. While it may be used by different threads, it is unique to
the reader's context and its accesses are synchronized by the reader,
the stream should therefore be completely oblivious to the fact it is
being used by different threads.
Maybe this could be the time to implement an unique qualifier; this is
a context where having _stream be of type unique(FileInputStream) would
solve the problem and allow further compiler optimizations. I don't
know if it can be done with templates, and without any overhead
whatsoever. I know I would much rather see unique(Foo) than Unique!Foo,
and it would allow the use of 'is(foo : unique)'.
Furthermore, tagging a method with shared does not make it thread-safe,
it may however use synchronized within its scope to protect its shared
or unique data. This may be confusing when calling shared methods vs
calling synchronized methods; one may think the shared one is not
thread-safe and optionally synchronize the call, resulting in another
monitor being used for nothing, or no monitor being used at all:
class Foo {
shared void bar() {
// Do stuff with local or immutable data synchronized(this) {
/* do stuff with shared data */ }
}
shared void bar2() {
// Do stuff on shared data
}
}
Someone seeing only the prototype of Foo.bar may assume the method is
not thread-safe and call it as 'synchronized(foo) foo.bar()'. Just like
they could see the prototype of bar2 and assume it is thread-safe,
calling it as 'foo.bar2()'.
What could be a good design against this sort of misleading behavior?
Phew, that's about
shared adventures in the realm of thread-safety.
I decided to play once again with shared and see what 2.032 is capable of.
Turns out a lot of the previous issues I was having last time are gone,
however, there are still a few things left which prevent me from rewriting my
code.
The first issue that jumped to my face straight away was how 'shared const'
methods are not callable from 'shared' objects.
shared class Foo {
void bar() const;
}
auto foo = new Foo; // foo is of type shared(Foo)
foo.bar; // Error: function Foo.bar () shared const is not callable using
argument types () shared
Considering how 'const' methods can be called from mutable objects, this looks
like either a bug or a really awkward feature to me. Sending a shared(Foo) to a
method expecting a shared(const(Foo)) also triggers a similar error from the
compiler.
The other issue may be an intended feature, but it doesn't sound practical to
me. Marking a method as shared assumes all used properties in the method's
scope are also shared. Here is an example to illustrate my point:
class SimpleReader {
this(LocalFile file) { _stream = new FileInputStream(file); }
...
private:
synchronized void read(ubyte[] buf, long offset) {
_stream.seek(offset);
_stream.read(buf);
}
FileInputStream _stream;
}
The FileInputStream here is a generic blocking binary stream which is not
thread-safe by design. The reader is a composite class where every instance has
its own unique stream instance and use it to implement asynchronous reads over
the file format it abstracts, which in my case is a specialized read-only
archive using a lot of random accesses from different threads.
This is where the issue shows its ugly head. The 'synchronized' keyword tags
the read method as shared, which in itself is quite neat, what is annoying
however is that it also changes the type of _stream in the method's scope to
shared(FileInputStream) and therefore triggers compiler errors because
_stream.seek and _stream.read are not shared:
Error: function FileInputStream.read (ubyte[]) is not callable using argument
types (ubyte[]) shared
While it may be an attempt to keep shared usage safe, it isn't very practical.
The stream object here is not shared because it is not thread-safe. While it
may be used by different threads, it is unique to the reader's context and its
accesses are synchronized by the reader, the stream should therefore be
completely oblivious to the fact it is being used by different threads.
Maybe this could be the time to implement an unique qualifier; this is a
context where having _stream be of type unique(FileInputStream) would solve the
problem and allow further compiler optimizations. I don't know if it can be
done with templates, and without any overhead whatsoever. I know I would much
rather see unique(Foo) than Unique!Foo, and it would allow the use of 'is(foo :
unique)'.
Furthermore, tagging a method with shared does not make it thread-safe, it may
however use synchronized within its scope to protect its shared or unique data.
This may be confusing when calling shared methods vs calling synchronized
methods; one may think the shared one is not thread-safe and optionally
synchronize the call, resulting in another monitor being used for nothing, or
no monitor being used at all:
class Foo {
shared void bar() {
// Do stuff with local or immutable data
synchronized(this) { /* do stuff with shared data */ }
}
shared void bar2() {
// Do stuff on shared data
}
}
Someone seeing only the prototype of Foo.bar may assume the method is not
thread-safe and call it as 'synchronized(foo) foo.bar()'. Just like they could
see the prototype of bar2 and assume it is thread-safe, calling it as
'foo.bar2()'.
What could be a good design against this sort of misleading behavior?
Phew, that's about enough issues and questions for now :)
