shared - i need it to be useful
Okay, so I've been thinking on this for a while... I think I have a pretty good feel for how shared is meant to be. 1. shared should behave exactly like const, except in addition to inhibiting write access, it also inhibits read access. I think this is the foundation for a useful definition for shared, and it's REALLY easy to understand and explain. Current situation where you can arbitrarily access shared members undermines any value it has. Shared must assure you don't access members unsafely, and the only way to do that with respect to data members, is to inhibit access completely. I think shared is just const without read access. Assuming this world... how do you use shared? 1. traditional; assert that the object become thread-local by acquiring a lock, cast shared away 2. object may have shared methods; such methods CAN be called on shared instances. such methods may internally implement synchronisation to perform their function. perhaps methods of a lock-free queue structure for instance, or operator overloads on `Atomic!int`, etc. In practise, there is no functional change in usage from the current implementation, except we disallow unsafe accesses (which will make the thing useful). >From there, it opens up another critical opportunity; T* -> shared(T)* promotion. Const would be useless without T* -> const(T)* promotion. Shared suffers a similar problem. If you write a lock-free queue for instance, and all the methods are `shared` (ie, threadsafe), then under the current rules, you can't interact with the object when it's not shared, and that's fairly useless. Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion? All the risks that I think have been identified previously assume that you can arbitrarily modify the data. That's insanity... assume we fix that... I think the promotion actually becomes safe now...? Destroy...
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: 2. object may have shared methods; such methods CAN be called on shared instances. such methods may internally implement synchronisation to perform their function. perhaps methods of a lock-free queue structure for instance, or operator overloads on `Atomic!int`, etc. Just checking my understanding: are you saying here that shared methods can effectively do anything and the burden of correctness is on the author? Or do you still have to cast the shared away first?
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 12:15 PM Peter Alexander via Digitalmars-d wrote: > > On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: > > 2. object may have shared methods; such methods CAN be called on > > shared instances. such methods may internally implement > > synchronisation to perform their function. perhaps methods of a > > lock-free queue structure for instance, or operator overloads on > > `Atomic!int`, etc. > > Just checking my understanding: are you saying here that shared > methods can effectively do anything and the burden of correctness > is on the author? Or do you still have to cast the shared away > first? Burden of correctness that a `shared` method is indeed threadsafe is absolutely on the author; there's nothing we can do to guarantee this (multithreading is hard!), but in this new world, a user of an API will be able to see shared methods and assume that they're threadsafe, since that would be (and should be!) the whole point. Since `shared` has no read or write access, at the bottom of the stack, it may be necessary that the function cast shared away to implement its magic. If you lock a mutex for instance, then you naturally need to cast it away; but that's solving the problem with a sledge-hammer. In my experience, most such functions are implemented with atomics; and the atomic API already receives shared args, ie: https://github.com/dlang/druntime/blob/master/src/core/atomic.d#L379 So, if you do your work with atomics, then you don't need any casts. Also, if the object is a composite, then a `shared` method is able to call `shared` methods on its members, and in that case, you are able to do useful aggregate work without casting. I think the interactions I describe above are all correct.
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 12:15 PM Peter Alexander via Digitalmars-d wrote: > > On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: > > 2. object may have shared methods; such methods CAN be called on > > shared instances. such methods may internally implement > > synchronisation to perform their function. perhaps methods of a > > lock-free queue structure for instance, or operator overloads on > > `Atomic!int`, etc. > > Just checking my understanding: are you saying here that shared > methods can effectively do anything and the burden of correctness > is on the author? Or do you still have to cast the shared away > first? Just to be clear: > are you saying here that shared methods can effectively do anything No, quite the opposite. I am saying that shared objects have neither read nor write access to members. Shared means "no read or write access". You can only call shared methods.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 19:14:58 UTC, Peter Alexander wrote: On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: 2. object may have shared methods; such methods CAN be called on shared instances. such methods may internally implement synchronisation to perform their function. perhaps methods of a lock-free queue structure for instance, or operator overloads on `Atomic!int`, etc. Just checking my understanding: are you saying here that shared methods can effectively do anything and the burden of correctness is on the author? Or do you still have to cast the shared away first? Well `this` is still shared so you would be prevented from doing anything non-atomically or lock cast away shared, but that will be enforced by the type system. So the burden of correctness is on the author, but the complier enforces correct behaviour.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: Okay, so I've been thinking on this for a while... I think I have a pretty good feel for how shared is meant to be. 1. shared should behave exactly like const, except in addition to inhibiting write access, it also inhibits read access. Are you familiar with reference capabilities[1] in the pony language? They describe many of them in terms of read/write uniqueness. Another way they describe them [2] is in denying aliases, like deny global read alias. [1] https://tutorial.ponylang.io/capabilities/reference-capabilities.html [2] See page 12-14: http://www.doc.ic.ac.uk/~scd/Pony-WG2.16.pdf
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: Destroy... Keep in mind immutable is implicitly shared (i.e. not in tls) because nobody can change it. It should stay readable for this reason.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
Destroy...
What you describe sounds better than what we currently have.
I have at least two concerns:
1. A single producer, single consumer (SPSC) queue is necessarily
shared, but is only safe if there is one writing thread and one
reading thread. Is it ok if shared also requires user discipline
and/or runtime checks to ensure correct usage?
2. In your scheme (as I understand), a struct composed entirely
of atomics would be able to implement shared methods without any
casts, but also be completely thread *unsafe*. Is this okay?
Example of #2:
struct TwoInts {
Atomic!int x, y;
void swap() shared {
int z = x.load;
x.store(y.load);
y.store(z);
}
}
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion? All the risks that I think have been identified previously assume that you can arbitrarily modify the data. That's insanity... assume we fix that... I think the promotion actually becomes safe now...? You're still talking about implicit promotion? No, it does not become safe no matter what restrictions you put on `shared` instances, because the caller of any function that takes `shared` arguments remains blissfully unaware of this promotion.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: If you write a lock-free queue for instance, and all the methods are `shared` (ie, threadsafe), then under the current rules, you can't interact with the object when it's not shared, and that's fairly useless. Unless the compiler can show that it is ok to implicit/explicity convert the object to share without any unintended consequences. It should reject it. It seems that the better solution would to implement a implicit/explict covertion system similar to c# conversion Operators https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/statements-expressions-operators/using-conversion-operators But that itself requires an DIP itself. -Alex
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 12:45 PM Nicholas Wilson via Digitalmars-d wrote: > > On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: > > Destroy... > > Keep in mind immutable is implicitly shared (i.e. not in tls) > because nobody can change it. It should stay readable for this > reason. Are you saying `is(immutable(int) == shared) == true)` ??
Re: shared - i need it to be useful
On 10/15/2018 08:46 PM, Manu wrote:
1. traditional; assert that the object become thread-local by
acquiring a lock, cast shared away
2. object may have shared methods; such methods CAN be called on
shared instances. such methods may internally implement
synchronisation to perform their function. perhaps methods of a
lock-free queue structure for instance, or operator overloads on
`Atomic!int`, etc.
[...]
Assuming the rules above: "can't read or write to members", and the
understanding that `shared` methods are expected to have threadsafe
implementations (because that's the whole point), what are the risks
from allowing T* -> shared(T)* conversion?
As far as I understand, the rule "can't read or write to members" is for
the compiler, right? I can still read and write members, but I have to
cast `shared` away and ensure thread-safety myself?
If that's so, then my example from the last thread might still apply:
struct Bob
{
int* p;
void doThing() shared
{
p = &s; /* Might need a cast or two here, and a lock or an atomic
store or whatever. */
}
}
shared int s;
When the needed casts etc. are added, is `doThing` allowed? If not, I
think you have to specify more precisely what a method can and can't do.
If `doThing` is ok, you can't allow T* -> shared(T)*. You'd be allowing
aliasing an unqualified int* with a shared(int*):
void main()
{
Bob* b = new Bob;
shared(Bob)* sb = b; /* You'd allow this line. */
sb.doThing();
/* Now the unqualified int* b.p points to the shared int s. */
}
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 1:05 PM Peter Alexander via Digitalmars-d
wrote:
>
> On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote:
> > Destroy...
>
> What you describe sounds better than what we currently have.
>
> I have at least two concerns:
>
> 1. A single producer, single consumer (SPSC) queue is necessarily
> shared, but is only safe if there is one writing thread and one
> reading thread. Is it ok if shared also requires user discipline
> and/or runtime checks to ensure correct usage?
I think you can model this differently... perhaps rather than a single
object, it's a coupled pair.
For instance, a thread-local enque object, and a separate thread-local
deque object. Those would express the 2 thread-local points of access
to queue and dequeue, and the implementation shunts between them in a
threadsafe way. I mean, it's not *really* a threadsafe primitive, so
for the whole object to be 'shared', I think that might be a
design-fail.
> 2. In your scheme (as I understand), a struct composed entirely
> of atomics would be able to implement shared methods without any
> casts, but also be completely thread *unsafe*. Is this okay?
It would be as safe as the design intends.
A struct with a bunch of public atomics effectively presents a set of
distinct atomics, and each one is thread-safe relative to eachother.
If the members are actually coupled into aggregate state, then you
make then private and methods implement the state transitions in such
a way guaranteeing atomicity.
Like I say before, the language can't "make it threadsafe" for you...
be producing a shared method, you have a responsibility to make sure
it works right.
> Example of #2:
>
> struct TwoInts {
>Atomic!int x, y;
>
>void swap() shared {
> int z = x.load;
> x.store(y.load);
> y.store(z);
>}
> }
Your swap function is plain broken; it doesn't do what the API promises.
You can write all sorts of broken code, and this is a good example of
just plain broken code.
Also, `x` and `y` probably shouldn't be public, or it effectively
communicates that they're de-coupled state.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 20:44:35 UTC, Manu wrote:
snip
Are you saying `is(immutable(int) == shared) == true)` ??
From the spec:
"Applying any qualifier to immutable T results in immutable T.
This makes immutable a fixed point of qualifier combinations and
makes types such as const(immutable(shared T)) impossible to
create."
Example:
import std.stdio : writeln;
void main()
{
writeln(is(immutable(int) == shared immutable(int)) == true);
//prints true
}
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 1:15 PM Stanislav Blinov via Digitalmars-d wrote: > > On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: > > > Assuming the rules above: "can't read or write to members", and > > the understanding that `shared` methods are expected to have > > threadsafe implementations (because that's the whole point), > > what are the risks from allowing T* -> shared(T)* conversion? > > > > All the risks that I think have been identified previously > > assume that you can arbitrarily modify the data. That's > > insanity... assume we fix that... I think the promotion > > actually becomes safe now...? > > You're still talking about implicit promotion? Absolutely. This is critical to make shared useful, and I think there's a path to make it work. > No, it does not > become safe no matter what restrictions you put on `shared` > instances, because the caller of any function that takes `shared` > arguments remains blissfully unaware of this promotion. It doesn't matter... what danger is there of distributing a shared pointer? Are you concerned that someone with a shared reference might call a threadsafe method? If that's an invalid operation, then the method has broken it's basic agreed contract. And I agree that it's conceivable that you could contrive a bad program, but you can contrive a bad program with literally any language feature!
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 20:54:12 UTC, jmh530 wrote:
On Monday, 15 October 2018 at 20:44:35 UTC, Manu wrote:
snip
Are you saying `is(immutable(int) == shared) == true)` ??
From the spec:
"Applying any qualifier to immutable T results in immutable T.
This makes immutable a fixed point of qualifier combinations
and makes types such as const(immutable(shared T)) impossible
to create."
Example:
import std.stdio : writeln;
void main()
{
writeln(is(immutable(int) == shared immutable(int)) ==
true); //prints true
}
The philosophy of this is that: the value never changes,
therefore only one copy of the variable needs to exist (i.e.
immutable variables declared at module scope are _not_ thread
local) and can be shared between threads with no race conditions.
I'm saying that while what you propose sounds very reasonable, we
should make sure that reading immutable variables is still good ;)
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 2:05 PM Nicholas Wilson via Digitalmars-d
wrote:
>
> On Monday, 15 October 2018 at 20:54:12 UTC, jmh530 wrote:
> > On Monday, 15 October 2018 at 20:44:35 UTC, Manu wrote:
> >> snip
> >>
> >> Are you saying `is(immutable(int) == shared) == true)` ??
> >
> > From the spec:
> > "Applying any qualifier to immutable T results in immutable T.
> > This makes immutable a fixed point of qualifier combinations
> > and makes types such as const(immutable(shared T)) impossible
> > to create."
> >
> > Example:
> >
> > import std.stdio : writeln;
> >
> > void main()
> > {
> > writeln(is(immutable(int) == shared immutable(int)) ==
> > true); //prints true
> > }
>
> The philosophy of this is that: the value never changes,
> therefore only one copy of the variable needs to exist (i.e.
> immutable variables declared at module scope are _not_ thread
> local) and can be shared between threads with no race conditions.
>
> I'm saying that while what you propose sounds very reasonable, we
> should make sure that reading immutable variables is still good ;)
I don't think what I describe affects immutable in any way;
`is(immutable(int) == shared) == false`, as it should be. Immutable
wouldn't have its rules affected by any change to shared.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 21:08:38 UTC, Manu wrote: On Mon, Oct 15, 2018 at 2:05 PM Nicholas Wilson via Digitalmars-d wrote: I'm saying that while what you propose sounds very reasonable, we should make sure that reading immutable variables is still good ;) I don't think what I describe affects immutable in any way; `is(immutable(int) == shared) == false`, as it should be. Immutable wouldn't have its rules affected by any change to shared. OK, just making sure you've got this covered.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 20:57:46 UTC, Manu wrote: On Mon, Oct 15, 2018 at 1:15 PM Stanislav Blinov via Digitalmars-d wrote: On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: > Assuming the rules above: "can't read or write to members", > and the understanding that `shared` methods are expected to > have threadsafe implementations (because that's the whole > point), what are the risks from allowing T* -> shared(T)* > conversion? > > All the risks that I think have been identified previously > assume that you can arbitrarily modify the data. That's > insanity... assume we fix that... I think the promotion > actually becomes safe now...? You're still talking about implicit promotion? Absolutely. This is critical to make shared useful, and I think there's a path to make it work. No, it does not become safe no matter what restrictions you put on `shared` instances, because the caller of any function that takes `shared` arguments remains blissfully unaware of this promotion. It doesn't matter... what danger is there of distributing a shared pointer? Are you concerned that someone with a shared reference might call a threadsafe method? If that's an invalid operation, then the method has broken it's basic agreed contract. No, on the contrary. Someone with an unshared pointer may call unshared method or read/write data while someone else accesses it via `shared` interface precisely because you allow T to escape to shared(T). You *need* an explicit cast for this. And I agree that it's conceivable that you could contrive a bad program, but you can contrive a bad program with literally any language feature!
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 2:25 PM Stanislav Blinov via Digitalmars-d wrote: > > On Monday, 15 October 2018 at 20:57:46 UTC, Manu wrote: > > On Mon, Oct 15, 2018 at 1:15 PM Stanislav Blinov via > > Digitalmars-d wrote: > >> > >> On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: > >> > >> > Assuming the rules above: "can't read or write to members", > >> > and the understanding that `shared` methods are expected to > >> > have threadsafe implementations (because that's the whole > >> > point), what are the risks from allowing T* -> shared(T)* > >> > conversion? > >> > > >> > All the risks that I think have been identified previously > >> > assume that you can arbitrarily modify the data. That's > >> > insanity... assume we fix that... I think the promotion > >> > actually becomes safe now...? > >> > >> You're still talking about implicit promotion? > > > > Absolutely. This is critical to make shared useful, and I think > > there's a path to make it work. > > > >> No, it does not > >> become safe no matter what restrictions you put on `shared` > >> instances, because the caller of any function that takes > >> `shared` > >> arguments remains blissfully unaware of this promotion. > > > > It doesn't matter... what danger is there of distributing a > > shared pointer? Are you concerned that someone with a shared > > reference might call a threadsafe method? If that's an invalid > > operation, then the method has broken it's basic agreed > > contract. > > No, on the contrary. Someone with an unshared pointer may call > unshared method or read/write data while someone else accesses it > via `shared` interface precisely because you allow T to escape to > shared(T). You *need* an explicit cast for this. If a shared method is incompatible with an unshared method, your class is broken. Explicit casting doesn't magically implement thread-safety, it basically just guarantees failure. What I suggest are rules that lead to proper behaviour with respect to writing a thread-safe API. You can write bad code with any feature in any number of ways. I see it this way: If your object has shared methods, then it is distinctly and *deliberately* involved in thread-safety. You have deliberately opted-in to writing a thread-safe object, and you must deliver on your promise. The un-shared API of an object that supports `shared` are not exempt from the thread-safety commitment, they are simply the subset of the API that may not be called from a shared context. If your shared method is incompatible with other methods, your class is broken, and you violate your promise. Nobody writes methods of an object such that they don't work with each other... methods are part of a deliberately crafted and packaged entity. If you write a shared object, you do so deliberately, and you buy responsibility of making sure your objects API is thread-safe. If your object is not thread-safe, don't write shared methods.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 20:53:32 UTC, Manu wrote: On Mon, Oct 15, 2018 at 1:05 PM Peter Alexander via Digitalmars-d wrote: On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: 1. A single producer, single consumer (SPSC) queue is necessarily shared, but is only safe if there is one writing thread and one reading thread. Is it ok if shared also requires user discipline and/or runtime checks to ensure correct usage? I think you can model this differently... perhaps rather than a single object, it's a coupled pair. That's a nice design. Your swap function is plain broken; it doesn't do what the API promises. You can write all sorts of broken code, and this is a good example of just plain broken code. If it is broken then why allow it? Why do we need to cast shared away if they weren't atomic and why do we allow it if they are atomic? I understand that shared can't magically tell you when code is thread safe or not. It does make sense to disallow almost everything and require casts. I'm just not seeing the value of allowing shared methods to access shared members if it isn't thread safe. Make it require casts.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 21:51:43 UTC, Manu wrote:
If a shared method is incompatible with an unshared method,
your class is broken.
What?!? So... my unshared methods should also perform all that's
necessary for `shared` methods?
Explicit casting doesn't magically implement thread-safety, it
basically just guarantees failure.
It doesn't indeed. It does, however, at least help prevent silent
bugs. Via that same guaranteed failure. That failure is about all
the help we can get from the compiler anyway.
What I suggest are rules that lead to proper behaviour with
respect to writing a thread-safe API.
You can write bad code with any feature in any number of ways.
Yup. For example, passing an int* to a function expecting shared
int*.
I see it this way:
If your object has shared methods, then it is distinctly and
*deliberately* involved in thread-safety. You have deliberately
opted-in to writing a thread-safe object, and you must deliver
on your promise.
The un-shared API of an object that supports `shared` are not
exempt from the thread-safety commitment, they are simply the
subset of the API that may not be called from a shared context.
And therefore they lack any synchronization. So I don't see how
they *can* be "compatible" with `shared` methods.
If your shared method is incompatible with other methods, your
class is broken, and you violate your promise.
Nope.
class BigCounter {
this() { /* don't even need the mutex if I'm not sharing this
*/ }
this(Mutex m = null) shared {
this.m = m ? m : new Mutex;
}
void increment() { value += 1; }
void increment() shared { synchronized(m)
*value.assumeUnshared += 1; }
private:
Mutex m;
BigInt value;
}
They're not "compatible" in any shape or form. Or would you have
the unshared ctor also create the mutex and unshared increment
also take the lock? What's the point of having them then? Better
disallow mixed implementations altogether (which is actually not
that bad of an idea).
Nobody writes methods of an object such that they don't work
with each other... methods are part of a deliberately crafted
and packaged
entity. If you write a shared object, you do so deliberately,
and you buy responsibility of making sure your objects API is
thread-safe.
If your object is not thread-safe, don't write shared methods.
Ahem... Okay...
import std.concurrency;
import core.atomic;
void thread(shared int* x) {
(*x).atomicOp!"+="(1);
}
shared int c;
void main() {
int x;
auto tid = spawn(&thread, &x); // "just" a typo
}
You're saying that's ok, it should "just" compile. It shouldn't.
It should produce an error and a mild electric discharge into the
developer's chair.
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 4:25 PM Peter Alexander via Digitalmars-d wrote: > > On Monday, 15 October 2018 at 20:53:32 UTC, Manu wrote: > > On Mon, Oct 15, 2018 at 1:05 PM Peter Alexander via > > Digitalmars-d wrote: > >> > >> On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: > >> 1. A single producer, single consumer (SPSC) queue is > >> necessarily shared, but is only safe if there is one writing > >> thread and one reading thread. Is it ok if shared also > >> requires user discipline and/or runtime checks to ensure > >> correct usage? > > > > I think you can model this differently... perhaps rather than a > > single > > object, it's a coupled pair. > > That's a nice design. > > > > Your swap function is plain broken; it doesn't do what the API > > promises. > > You can write all sorts of broken code, and this is a good > > example of > > just plain broken code. > > If it is broken then why allow it? Why do we need to cast shared > away if they weren't atomic and why do we allow it if they are > atomic? It's not that it's 'allowed' other than, yes, access to atomic int's is allowed in a threadsafe way because atomic access to individual int's is threadsafe, so the primitive operation is perfectly acceptable. Your function models a higher-level concept; which is an atomic swap. You need to make sure that a threadsafe API you're authoring does actually deliver on the promise it makes. > I understand that shared can't magically tell you when code is > thread safe or not. It does make sense to disallow almost > everything and require casts. I'm just not seeing the value of > allowing shared methods to access shared members if it isn't > thread safe. Make it require casts. Because in an awful lot of cases, it is threadsafe. Implementing low-level machinery, and consuming such machinery should have a 1:many relationship. You're talking about adding friction to the 'many' such that the '1' knows that they need to implement their function right? I mean, they already know that, because they wrote 'shared' after their function declaration. In the common case, perhaps my object might aggregate a threadsafe queue, and my shared method prepares an item and then adds it to the queue. I think the vast majority case will be making use of utility functionality and that shouldn't present undue friction. It's not like atomic int's that are class members are going to be accidentally twiddled; you added Atomic!int's to your class, and that wasn't an accident... and if you're making use of the lowest-level atomic primitives, it's fair to presume you know how to use them. You're writing a shared method, which means you already encapsulate the promise that you are implementing a function that deals with thread-safety. I don't know what the cast in this case would add.
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 23:30:43 UTC, Stanislav Blinov
wrote:
On Monday, 15 October 2018 at 21:51:43 UTC, Manu wrote:
If a shared method is incompatible with an unshared method,
your class is broken.
What?!? So... my unshared methods should also perform all
that's necessary for `shared` methods?
No, its the other way around: a shared method that does extra
synchronisation should work irrespective of wether or not the
object needs that synchronisation. e.g. atomic loading a TLS
variable is fine.
Explicit casting doesn't magically implement thread-safety, it
basically just guarantees failure.
It doesn't indeed. It does, however, at least help prevent
silent bugs. Via that same guaranteed failure. That failure is
about all the help we can get from the compiler anyway.
What I suggest are rules that lead to proper behaviour with
respect to writing a thread-safe API.
You can write bad code with any feature in any number of ways.
Yup. For example, passing an int* to a function expecting
shared int*.
That is a reasonable thing to do if shared is const + no
unsynched reads.
I see it this way:
If your object has shared methods, then it is distinctly and
*deliberately* involved in thread-safety. You have deliberately
opted-in to writing a thread-safe object, and you must deliver
on your promise.
The un-shared API of an object that supports `shared` are not
exempt from the thread-safety commitment, they are simply the
subset of the API that may not be called from a shared context.
And therefore they lack any synchronization. So I don't see how
they *can* be "compatible" with `shared` methods.
I think Manu means you have a shared object with some shared
methods and some unshared methods. The shared methods deal with
synchronisation and can therefore be call from anywhere by
anyone, whereas the unshared methods must be called on a locked
object.
snip
Nobody writes methods of an object such that they don't work
with each other... methods are part of a deliberately crafted
and packaged
entity. If you write a shared object, you do so deliberately,
and you buy responsibility of making sure your objects API is
thread-safe.
If your object is not thread-safe, don't write shared methods.
Ahem... Okay...
import std.concurrency;
import core.atomic;
void thread(shared int* x) {
(*x).atomicOp!"+="(1);
}
shared int c;
void main() {
int x;
auto tid = spawn(&thread, &x); // "just" a typo
}
You're saying that's ok, it should "just" compile. It
shouldn't. It should produce an error and a mild electric
discharge into the developer's chair.
Indeed that is just a typo, just as that is a contrived example.
You'd notice that pretty quick in a debugger.
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 4:35 PM Stanislav Blinov via Digitalmars-d
wrote:
>
> On Monday, 15 October 2018 at 21:51:43 UTC, Manu wrote:
>
> > If a shared method is incompatible with an unshared method,
> > your class is broken.
>
> What?!? So... my unshared methods should also perform all that's
> necessary for `shared` methods?
Of course! You're writing a threadsafe object... how could you expect otherwise?
> > Explicit casting doesn't magically implement thread-safety, it
> > basically just guarantees failure.
>
> It doesn't indeed. It does, however, at least help prevent silent
> bugs. Via that same guaranteed failure. That failure is about all
> the help we can get from the compiler anyway.
Just to be clear, what I'm suggesting is a significant *restriction*
to what shared already does... there will be a whole lot more safety
under my proposal.
The cast gives exactly nothing that attributing a method as shared
doesn't give you, except that attributing a method shared is so much
more sanitary and clearly communicates intent at the API level.
> > What I suggest are rules that lead to proper behaviour with
> > respect to writing a thread-safe API.
> > You can write bad code with any feature in any number of ways.
>
> Yup. For example, passing an int* to a function expecting shared
> int*.
I don't understand your example. What's the problem you're suggesting?
> > I see it this way:
> > If your object has shared methods, then it is distinctly and
> > *deliberately* involved in thread-safety. You have deliberately
> > opted-in to writing a thread-safe object, and you must deliver
> > on your promise.
>
> > The un-shared API of an object that supports `shared` are not
> > exempt from the thread-safety commitment, they are simply the
> > subset of the API that may not be called from a shared context.
>
> And therefore they lack any synchronization. So I don't see how
> they *can* be "compatible" with `shared` methods.
I don't understand this statement either. Who said they lack
synchronisation? If they need it, they will have it.
There's a good chance they don't need it though, they might not
interact with a thread-unsafe portion of the class.
> > If your shared method is incompatible with other methods, your
> > class is broken, and you violate your promise.
>
> Nope.
So certain...
> class BigCounter {
>
> this() { /* don't even need the mutex if I'm not sharing this
> */ }
>
> this(Mutex m = null) shared {
> this.m = m ? m : new Mutex;
> }
>
> void increment() { value += 1; }
> void increment() shared { synchronized(m)
> *value.assumeUnshared += 1; }
>
> private:
> Mutex m;
> BigInt value;
> }
You've just conflated 2 classes into one. One is a threadlocal
counter, the other is a threadsafe counter. Which is it?
Like I said before: "you can contrive a bad program with literally any
language feature!"
> They're not "compatible" in any shape or form.
Correct, you wrote 2 different things and mashed them together.
> Or would you have
> the unshared ctor also create the mutex and unshared increment
> also take the lock? What's the point of having them then? Better
> disallow mixed implementations altogether (which is actually not
> that bad of an idea).
Right. This is key to my whole suggestion. If you write a shared
thing, you accept that it's shared! You don't just accept it, you jam
the stake in the ground.
There's a relatively small number of things that need to be
threadsafe, you won't see `shared` methods appearing at random. If you
use shared, you promise threadsafety OR the members of the thing are
inaccessible without some sort of lock-&-cast-away treatment.
> > Nobody writes methods of an object such that they don't work
> > with each other... methods are part of a deliberately crafted
> > and packaged
> > entity. If you write a shared object, you do so deliberately,
> > and you buy responsibility of making sure your objects API is
> > thread-safe.
> > If your object is not thread-safe, don't write shared methods.
>
> Ahem... Okay...
>
> import std.concurrency;
> import core.atomic;
>
> void thread(shared int* x) {
> (*x).atomicOp!"+="(1);
> }
>
> shared int c;
>
> void main() {
> int x;
> auto tid = spawn(&thread, &x); // "just" a typo
> }
>
> You're saying that's ok, it should "just" compile. It shouldn't.
> It should produce an error and a mild electric discharge into the
> developer's chair.
Yup. It's a typo. You passed a stack pointer to a scope that outlives
the caller.
That class of issue is not on trial here. There's DIP1000, and all
sorts of things to try and improve safety in terms of lifetimes.
You only managed to contrive this by spawning a thread. If it were
just a normal function, this would be perfectly legitimate, and again,
that's my whole point.
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 5:10 PM Nicholas Wilson via Digitalmars-d wrote: > > On Monday, 15 October 2018 at 23:30:43 UTC, Stanislav Blinov > wrote: > > On Monday, 15 October 2018 at 21:51:43 UTC, Manu wrote: > > > >> I see it this way: > >> If your object has shared methods, then it is distinctly and > >> *deliberately* involved in thread-safety. You have deliberately > >> opted-in to writing a thread-safe object, and you must deliver > >> on your promise. > >> > >> The un-shared API of an object that supports `shared` are not > >> exempt from the thread-safety commitment, they are simply the > >> subset of the API that may not be called from a shared context. > > > > And therefore they lack any synchronization. So I don't see how > > they *can* be "compatible" with `shared` methods. > > > > I think Manu means you have a shared object with some shared > methods and some unshared methods. The shared methods deal with > synchronisation and can therefore be call from anywhere by > anyone, whereas the unshared methods must be called on a locked > object. Yes, except maybe I didn't make it clear that I DO expect the un-shared methods to be aware that a sibling shared method does exist (you wrote it!), and that it may manipulate some state, so *if* the un-shared method does interact with the same data that the shared method may manipulate (in many cases, it won't; it's likely only a small subset of an object's functionality that may have thread-safe access), then the un-shared method does need to acknowledge that functional overlap. So even though a method is un-shared, it still needs to be aware that it may have sibling methods that are shared. If they don't access an overlapping data-set, no special handling is required. If they do overlap, they may need to coordinate appropriately.
Re: shared - i need it to be useful
On Tuesday, 16 October 2018 at 00:36:12 UTC, Manu wrote: *snip* Yes, except maybe I didn't make it clear that I DO expect the un-shared methods to be aware that a sibling shared method does exist (you wrote it!), and that it may manipulate some state, so *if* the un-shared method does interact with the same data that the shared method may manipulate (in many cases, it won't; it's likely only a small subset of an object's functionality that may have thread-safe access), then the un-shared method does need to acknowledge that functional overlap. So even though a method is un-shared, it still needs to be aware that it may have sibling methods that are shared. If they don't access an overlapping data-set, no special handling is required. If they do overlap, they may need to coordinate appropriately. I understand your point but I think the current shared (no implicit conversion) has its uses. It can be quite useful to have one interface for when an object is shared and one for when it is not (one with and without the synchronization cost). Sure, something as trivial as a counter can be re-implemented in both ways but more complex objects would easily result in extreme code duplication. It's also easy to acknowledge that implicit conversion to shared has its uses. Instead of forcing one way or another, how about we leave the decision up to the programmer? Say something like "alias this shared;" enables implicit conversion to shared.
Re: shared - i need it to be useful
On Tuesday, 16 October 2018 at 00:15:54 UTC, Manu wrote:
On Mon, Oct 15, 2018 at 4:35 PM Stanislav Blinov via
Digitalmars-d wrote:
What?!? So... my unshared methods should also perform all
that's necessary for `shared` methods?
Of course! You're writing a threadsafe object... how could you
expect otherwise?
See below.
Just to be clear, what I'm suggesting is a significant
*restriction*
to what shared already does... there will be a whole lot more
safety under my proposal.
I don't see how an *implicit* cast can be a restriction. At all.
The cast gives exactly nothing that attributing a method as
shared
doesn't give you, except that attributing a method shared is so
much
more sanitary and clearly communicates intent at the API level.
It's like we're talking about wholly different things here.
Casting should be done by the caller, i.e. a programmer that uses
some API. If that API expects shared arguments, the caller better
make sure they pass shared values. Implicit conversion destroys
any obligations between the caller and the API.
> You can write bad code with any feature in any number of
> ways.
Yup. For example, passing an int* to a function expecting
shared int*.
I don't understand your example. What's the problem you're
suggesting?
The problem that I'm suggesting is exactly that: an `int*` is
not, and can not, be a `shared int*` at the same time. Substitute
int for any type. But D is not Rust and it can't statically
prevent that, except for disallowing trivial programming
mistakes, which, with implicit conversion introduced, would also
go away.
...And therefore they lack any synchronization. So I don't see
how they *can* be "compatible" with `shared` methods.
I don't understand this statement either. Who said they lack
synchronisation? If they need it, they will have it. There's a
good chance they don't need it though, they might not interact
with a thread-unsafe portion of the class.
Or they might.
> If your shared method is incompatible with other methods,
> your class is broken, and you violate your promise.
Nope.
So certain...
class BigCounter {
this() { /* don't even need the mutex if I'm not sharing
this
*/ }
this(Mutex m = null) shared {
this.m = m ? m : new Mutex;
}
void increment() { value += 1; }
void increment() shared { synchronized(m)
*value.assumeUnshared += 1; }
private:
Mutex m;
BigInt value;
}
You've just conflated 2 classes into one. One is a threadlocal
counter, the other is a threadsafe counter. Which is it?
Like I said before: "you can contrive a bad program with
literally any language feature!"
Because that is exactly the code that a good amount of
"developers" will write. Especially those of the "don't think
about it" variety. Don't be mistaken for a second: if the
language allows it, they'll write it.
They're not "compatible" in any shape or form.
Correct, you wrote 2 different things and mashed them together.
Can you actually provide an example of a mixed shared/unshared
class that even makes sense then? As I said, at this point I'd
rather see such definitions prohibited entirely.
Or would you have
the unshared ctor also create the mutex and unshared increment
also take the lock? What's the point of having them then?
Better disallow mixed implementations altogether (which is
actually not that bad of an idea).
Right. This is key to my whole suggestion. If you write a
shared thing, you accept that it's shared! You don't just
accept it, you jam the stake in the ground.
Then, once more, `shared` should then just be a type qualifier
exclusively, and mixing shared/unshared methods should just not
be allowed.
There's a relatively small number of things that need to be
threadsafe, you won't see `shared` methods appearing at random.
If you use shared, you promise threadsafety OR the members of
the thing are inaccessible without some sort of
lock-&-cast-away treatment.
As above.
import std.concurrency;
import core.atomic;
void thread(shared int* x) {
(*x).atomicOp!"+="(1);
}
shared int c;
void main() {
int x;
auto tid = spawn(&thread, &x); // "just" a typo
}
You're saying that's ok, it should "just" compile. It
shouldn't. It should produce an error and a mild electric
discharge into the developer's chair.
Yup. It's a typo. You passed a stack pointer to a scope that
outlives the caller.
That class of issue is not on trial here. There's DIP1000, and
all sorts of things to try and improve safety in terms of
lifetimes.
I'm sorry, I'm not very good at writing "real" examples for
things that don't exist or don't compile. End of sarcasm.
Let's come back to DIP1000 when it's actually implemented in it's
entirety, ok? Anyway, you're nitpicking while actually missing
the point altogether. The way `shared` is "implemented" today,
the API (`thread` function) *requires* the caller to pass a
`shared int*`. Implicit co
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 7:15 PM Isaac S. via Digitalmars-d wrote: > > On Tuesday, 16 October 2018 at 00:36:12 UTC, Manu wrote: > > *snip* > > > > Yes, except maybe I didn't make it clear that I DO expect the > > un-shared methods to be aware that a sibling shared method does > > exist > > (you wrote it!), and that it may manipulate some state, so *if* > > the > > un-shared method does interact with the same data that the > > shared > > method may manipulate (in many cases, it won't; it's likely > > only a > > small subset of an object's functionality that may have > > thread-safe > > access), then the un-shared method does need to acknowledge that > > functional overlap. > > So even though a method is un-shared, it still needs to be > > aware that > > it may have sibling methods that are shared. If they don't > > access an > > overlapping data-set, no special handling is required. If they > > do > > overlap, they may need to coordinate appropriately. > > I understand your point but I think the current shared (no > implicit conversion) has its uses. It can be quite useful to have > one interface for when an object is shared and one for when it is > not (one with and without the synchronization cost). Sure, > something as trivial as a counter can be re-implemented in both > ways but more complex objects would easily result in extreme code > duplication. If you can give a single 'use', I'm all ears ;) > It's also easy to acknowledge that implicit conversion to shared > has its uses. I actually know of many real uses for this case. > Instead of forcing one way or another, how about we leave the > decision up to the programmer? Say something like "alias this > shared;" enables implicit conversion to shared. That might be fine. Like I said in OP, the first point that I think needs to be agreed on, is that shared can not read or write members. I think that's a pre-requisite for any interesting development.
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 7:25 PM Stanislav Blinov via Digitalmars-d
wrote:
>
> On Tuesday, 16 October 2018 at 00:15:54 UTC, Manu wrote:
> > On Mon, Oct 15, 2018 at 4:35 PM Stanislav Blinov via
> > Digitalmars-d wrote:
>
> >> What?!? So... my unshared methods should also perform all
> >> that's necessary for `shared` methods?
>
> > Of course! You're writing a threadsafe object... how could you
> > expect otherwise?
>
> See below.
>
> > Just to be clear, what I'm suggesting is a significant
> > *restriction*
> > to what shared already does... there will be a whole lot more
> > safety under my proposal.
>
> I don't see how an *implicit* cast can be a restriction. At all.
Because a shared pointer can't access anything.
You can't do anything with a shared instance, so the can be no harm done.
Only if there are shared methods (that promise thread-safety) is it
that shared gets interesting.
Without that, it's just a market for the existing recommended use of
shared; which is lock and cast away.
> > The cast gives exactly nothing that attributing a method as
> > shared
> > doesn't give you, except that attributing a method shared is so
> > much
> > more sanitary and clearly communicates intent at the API level.
>
> It's like we're talking about wholly different things here.
> Casting should be done by the caller, i.e. a programmer that uses
> some API. If that API expects shared arguments, the caller better
> make sure they pass shared values. Implicit conversion destroys
> any obligations between the caller and the API.
Why? What could a function do with shared arguments?
> >> > You can write bad code with any feature in any number of
> >> > ways.
> >>
> >> Yup. For example, passing an int* to a function expecting
> >> shared int*.
> >
> > I don't understand your example. What's the problem you're
> > suggesting?
>
> The problem that I'm suggesting is exactly that: an `int*` is
> not, and can not, be a `shared int*` at the same time. Substitute
> int for any type. But D is not Rust and it can't statically
> prevent that, except for disallowing trivial programming
> mistakes, which, with implicit conversion introduced, would also
> go away.
Why not? The guy who receives the argument receives an argument that
*may be shared*, and as such, he's restricted access to it
appropriately.
Just like if you receive a const thing, you can't write to it, even if
the caller's thing isn't const.
If you receive a shared thing, you can't read or write to it.
> >> ...And therefore they lack any synchronization. So I don't see
> >> how they *can* be "compatible" with `shared` methods.
> >
> > I don't understand this statement either. Who said they lack
> > synchronisation? If they need it, they will have it. There's a
> > good chance they don't need it though, they might not interact
> > with a thread-unsafe portion of the class.
>
> Or they might.
Then you will implement synchronisation, or have violated your
thread-safety promise.
> >> > If your shared method is incompatible with other methods,
> >> > your class is broken, and you violate your promise.
> >>
> >> Nope.
> >
> > So certain...
> >
> >> class BigCounter {
> >>
> >> this() { /* don't even need the mutex if I'm not sharing
> >> this
> >> */ }
> >>
> >> this(Mutex m = null) shared {
> >> this.m = m ? m : new Mutex;
> >> }
> >>
> >> void increment() { value += 1; }
> >> void increment() shared { synchronized(m)
> >> *value.assumeUnshared += 1; }
> >>
> >> private:
> >> Mutex m;
> >> BigInt value;
> >> }
> >
> > You've just conflated 2 classes into one. One is a threadlocal
> > counter, the other is a threadsafe counter. Which is it?
> > Like I said before: "you can contrive a bad program with
> > literally any language feature!"
>
> Because that is exactly the code that a good amount of
> "developers" will write. Especially those of the "don't think
> about it" variety. Don't be mistaken for a second: if the
> language allows it, they'll write it.
This is not even an argument.
Atomic!int must be used with care. Any threading of ANY KIND must be
handled with care.
Saying we shouldn't make shared useful because someone can do
something wrong is like saying we shouldn't have atomic int's and we
shouldn't have spawn(). They're simply too dangerous to give to
users...
> >> They're not "compatible" in any shape or form.
> >
> > Correct, you wrote 2 different things and mashed them together.
>
> Can you actually provide an example of a mixed shared/unshared
> class that even makes sense then? As I said, at this point I'd
> rather see such definitions prohibited entirely.
I think this is a typical sort of construction:
struct ThreadsafeQueue(T)
{
void QueueItem(T*) shared;
T* UnqueueItem() shared;
}
struct SpecialWorkList
{
struct Job { ... }
void MakeJob(int x, float y, string z) shared // <- any thread may
produce a job
{
Job* job = new Job; // <- this is thread-local
PopulateJob(job, x, y, z); // <
Re: shared - i need it to be useful
On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote: I understand your point but I think the current shared (no implicit conversion) has its uses. It can be quite useful to have one interface for when an object is shared and one for when it is not (one with and without the synchronization cost). Sure, something as trivial as a counter can be re-implemented in both ways but more complex objects would easily result in extreme code duplication. If you can give a single 'use', I'm all ears ;) My usages are a custom ref counted template and list types (which are built on top of the former). The ref counted template will initialize naively when not shared but utilize compare-and-set and thread yielding if needed when shared (ensureInitialized can occur any time after declaration). The list types (which are not shared-compatible *yet*) will utilize a read-write mutex only when shared (no cost beyond a little unused memory to non-shared objects). As such, casting any of the non-shared versions of these types to shared would be unsafe. (Technically the types can be safely casted to shared so long as no non-shared reference to them exists and vice versa.) It's also easy to acknowledge that implicit conversion to shared has its uses. I actually know of many real uses for this case. (There was a time when I wanted it as well. I've forgotten what for by now.) That might be fine. Like I said in OP, the first point that I think needs to be agreed on, is that shared can not read or write members. I think that's a pre-requisite for any interesting development. I will agree that *outsiders* should not be able to read/write members of a shared object. I'm not sure whether your design means even within a shared function such members cannot be read from or written to. If it does, the casting required may get a little annoying but isn't unworkable (especially since an unshared template can do the cast in an @trusted manner).
Re: shared - i need it to be useful
On Mon, Oct 15, 2018 at 8:55 PM Isaac S. via Digitalmars-d wrote: > > On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote: > >> I understand your point but I think the current shared (no > >> implicit conversion) has its uses. It can be quite useful to > >> have one interface for when an object is shared and one for > >> when it is not (one with and without the synchronization > >> cost). Sure, something as trivial as a counter can be > >> re-implemented in both ways but more complex objects would > >> easily result in extreme code duplication. > > > > If you can give a single 'use', I'm all ears ;) > > My usages are a custom ref counted template and list types (which > are built on top of the former). The ref counted template will > initialize naively when not shared but utilize compare-and-set > and thread yielding if needed when shared (ensureInitialized can > occur any time after declaration). The list types (which are not > shared-compatible *yet*) will utilize a read-write mutex only > when shared (no cost beyond a little unused memory to non-shared > objects). As such, casting any of the non-shared versions of > these types to shared would be unsafe. > > (Technically the types can be safely casted to shared so long as > no non-shared > reference to them exists and vice versa.) Can you link to code? It doesn't sound incompatible with my proposal. Mutexes are blunt instruments, and easy to model. My whole suggestion has virtually no effect on the mutex protected case, which is what most people seem to talk about. > > That might be fine. > > Like I said in OP, the first point that I think needs to be > > agreed on, > > is that shared can not read or write members. I think that's a > > pre-requisite for any interesting development. > > I will agree that *outsiders* should not be able to read/write > members of a shared object. I'm not sure whether your design > means even within a shared function such members cannot be read > from or written to. Absolutely. Shared references can not access members, that's the rule. It's completely unsafe to read or write to members of a shared thing. Period. Attributing a method shared doesn't change that fact. You can create a thread-local (ie, accessible) object from a shared object by acquiring a lock on it for some duration, and it would be helpful if the mechanism that fetches the lock also cast away shared as part of the lock operation. > If it does, the casting required may get a > little annoying but isn't unworkable (especially since an > unshared template can do the cast in an @trusted manner). I agree users shouldn't write casts explicitly, they should use a lock helper or something that grabs the lock and returns the un-shared reference. The existing behaviour is totally unacceptable though; shared objects can read/write arbitrarily... what's the point of that? It gives the impression that it's okay, but it's completely broken. There is no situation where a shared thing can arbitrarily access its members.
Re: shared - i need it to be useful
On 2018-10-15 20:46, Manu wrote:
1. traditional; assert that the object become thread-local by
acquiring a lock, cast shared away
Instead of having to explicitly cast away shared we could leverage the
synchronized statement. It could be enhanced to allow the following:
shared int a;
Mutex mutex;
synchronized(tlsA; a, mutex)
{
// within this scope "tlsA" is the same as "a" but without
// the "shared" qualifier. "tlsA" is not allowed to escape the block
}
This could also be implemented as a library function, but that would
require casting away shared inside the implementation:
guard(a, mutex, (tlsA){
});
void guard(T)(scope shared T value, Mutex mutex, scope void delegate
(scope T) block)
{
mutex.lock_nothrow();
scope (exit)
mutex.unlock_nothrow();
block(cast(T) value);
}
--
/Jacob Carlborg
Re: shared - i need it to be useful
On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: Current situation where you can arbitrarily access shared members undermines any value it has. The value of shared is existence of thread-local data that's guaranteed to be not shared, so you don't need to worry about thread-local data being shared, and shared protects that value well. Assuming this world... how do you use shared? Unique solution for each case. If you write a lock-free queue for instance, and all the methods are `shared` (ie, threadsafe), then under the current rules, you can't interact with the object when it's not shared, and that's fairly useless. Create it as shared. Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion? All data becomes possibly shared, so you can't assume it's unshared, effectively C-style sharing. BTW D supports the latter already.
Re: shared - i need it to be useful
On 15.10.2018 20:46, Manu wrote: Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion? Unshared becomes useless, and in turn, shared becomes useless. You can't have unshared/shared aliasing. All the risks that I think have been identified previously assume that you can arbitrarily modify the data. That's insanity... assume we fix that... I think the promotion actually becomes safe now...? But useless, because there is no way to ensure thread safety of reads and writes if only one party to the shared state knows about the sharing.
Re: shared - i need it to be useful
On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote: On 15.10.2018 20:46, Manu wrote: Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion? Unshared becomes useless, and in turn, shared becomes useless. why is unshared useless? Unshared means you can read an write to it. If you give it to a function that expect something shared, the function you had given it to can't read or write it, so it can't do any harm. Of course it can handle it threadsave, but as it is local, that is only overhead - reading or changing the value can't do any harm either. I like the idea. But useless, because there is no way to ensure thread safety of reads and writes if only one party to the shared state knows about the sharing. Of course there is. Giving an unshared value to a function that even can handle shared values may create some overhead, but is indeed threadsave.
Re: shared - i need it to be useful
On Tuesday, 16 October 2018 at 03:00:21 UTC, Manu wrote:
I don't see how an *implicit* cast can be a restriction. At
all.
Because a shared pointer can't access anything.
You can't do anything with a shared instance, so the can be no
harm done.
That just doesn't compute. You obviously *can* do "anything" with
a shared instance, per your own requirements all you need is
methods. But at this point, the caller has an unshared reference,
and your API possibliy stole away a shared one without the caller
ever knowing of it happening.
It's like we're talking about wholly different things here.
Casting should be done by the caller, i.e. a programmer that
uses some API. If that API expects shared arguments, the
caller better make sure they pass shared values. Implicit
conversion destroys any obligations between the caller and the
API.
Why? What could a function do with shared arguments?
How, exactly, do you propose getting shared state from one thread
to another? Exclusively through globals?
The problem that I'm suggesting is exactly that: an `int*` is
not, and can not, be a `shared int*` at the same time.
Substitute int for any type. But D is not Rust and it can't
statically prevent that, except for disallowing trivial
programming mistakes, which, with implicit conversion
introduced, would also go away.
Why not? The guy who receives the argument receives an argument
that
*may be shared*, and as such, he's restricted access to it
appropriately.
But the guy that *provides* that argument may have no idea of
this happening. This is unshared aliasing.
module yourapi;
// My code does not know about this function at all
void giveToThread(shared int* ptr) { /* ... */ }
void yourAPI(int* ptr) {
giveToThread(ptr);
}
module mycode;
int x;
void main() {
yourAPI(&x);
}
Just like if you receive a const thing, you can't write to it,
even if the caller's thing isn't const.
You do know why those Rust guys disallowed mutable aliasing? That
is, having both mutable and immutable references at the same
time. That's a long known problem that in C++ and D is only
"solved" by programmer discipline and not much else.
If you receive a shared thing, you can't read or write to it.
You can, per your model, through methods. All the while the
original is being read and written freely.
> a good chance they don't need it though, they might not
> interact with a thread-unsafe portion of the class.
Or they might.
Then you will implement synchronisation, or have violated your
thread-safety promise.
Such a good promise it is when it's simply ignored by an implicit
cast.
Because that is exactly the code that a good amount of
"developers" will write. Especially those of the "don't think
about it" variety. Don't be mistaken for a second: if the
language allows it, they'll write it.
This is not even an argument.
Atomic!int must be used with care. Any threading of ANY KIND
must be
handled with care.
Implicit casts are in another galaxy as far as handling with care
is concerned.
Saying we shouldn't make shared useful because someone can do
something wrong is like saying we shouldn't have atomic int's
and we shouldn't have spawn(). They're simply too dangerous to
give to users...
We should make `shared` useful. We shouldn't allow unshared
aliasing.
Can you actually provide an example of a mixed shared/unshared
class that even makes sense then? As I said, at this point I'd
rather see such definitions prohibited entirely.
I think this is a typical sort of construction:
struct ThreadsafeQueue(T)
{
void QueueItem(T*) shared;
T* UnqueueItem() shared;
}
That's an interface for a multi-producer multi-consumer, yet
you're using it as a multi-producer single-consumer. Those would
have very different implementations.
struct SpecialWorkList
{
struct Job { ... }
void MakeJob(int x, float y, string z) shared // <- any
thread may produce a job
{
Job* job = new Job; // <- this is thread-local
PopulateJob(job, x, y, z); // <- preparation of a job might
be complex, and worthy of the SpecialWorkList implementation
...except you can't call PopulateJob from MakeJob.
The two methods after compiler rewrite are (pseudocode):
void struct_method_SpecialWorkList_MakeJob(shared
SpecialWorkList* this, int x, float y, string z);
void struct_method_SpecialWorkList_PopulateJob(SpecialWorkList*
this, Job* job, ...);
Or are you also suggesting to allow implicitly demoting
shared(T)* to T* ?!? Then you can just throw away `shared`.
jobList.QueueItem(job); // <- QueueItem encapsulates
thread-safety, no need for blunt casts
None were needed here regardless.
void Flush() // <- not shared, thread-local consumer
{
I.e. this can only be called by the thread that owns this
instance, because only that thread can have an unshared reference
to it.
void GetSpecialSystemState() // <- this has NOTHING to do
with the threadsafe part of Sp
Re: shared - i need it to be useful
On 16.10.2018 13:04, Dominikus Dittes Scherkl wrote: On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote: On 15.10.2018 20:46, Manu wrote: Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion? Unshared becomes useless, and in turn, shared becomes useless. why is unshared useless? Unshared means you can read an write to it. If you give it to a function that expect something shared, the function you had given it to can't read or write it, so it can't do any harm. It can do harm to others who hold an unshared alias to the same data and are operating on it concurrently. Of course it can handle it threadsave, but as it is local, that is only overhead - reading or changing the value can't do any harm either. I like the idea. But useless, because there is no way to ensure thread safety of reads and writes if only one party to the shared state knows about the sharing. Of course there is. Please do enlighten me. You have two processors operating (reading/writing) on the same address space on a modern computer architecture with a weak memory model, and you are using an optimizing compiler. How do you ensure sensible results without cooperation from both of them? (Hint: you don't.) Giving an unshared value to a function that even can handle shared values may create some overhead, but is indeed threadsave. Yes, if you give it to one function only, that is the case. However, as you may know, concurrency means that there may be multiple functions operating on the data _at the same time_. If one of them operates on the data as if it was not shared, you will run into trouble. You are arguing as if there was either no concurrency or no mutable aliasing.
Re: shared - i need it to be useful
On 10/15/18 2:46 PM, Manu wrote:
Okay, so I've been thinking on this for a while... I think I have a
pretty good feel for how shared is meant to be.
1. shared should behave exactly like const, except in addition to
inhibiting write access, it also inhibits read access.
I think this is the foundation for a useful definition for shared, and
it's REALLY easy to understand and explain.
Current situation where you can arbitrarily access shared members
undermines any value it has. Shared must assure you don't access
members unsafely, and the only way to do that with respect to data
members, is to inhibit access completely.
I think shared is just const without read access.
Assuming this world... how do you use shared?
1. traditional; assert that the object become thread-local by
acquiring a lock, cast shared away
2. object may have shared methods; such methods CAN be called on
shared instances. such methods may internally implement
synchronisation to perform their function. perhaps methods of a
lock-free queue structure for instance, or operator overloads on
`Atomic!int`, etc.
In practise, there is no functional change in usage from the current
implementation, except we disallow unsafe accesses (which will make
the thing useful).
From there, it opens up another critical opportunity; T* -> shared(T)*
promotion.
Const would be useless without T* -> const(T)* promotion. Shared
suffers a similar problem.
If you write a lock-free queue for instance, and all the methods are
`shared` (ie, threadsafe), then under the current rules, you can't
interact with the object when it's not shared, and that's fairly
useless.
Assuming the rules above: "can't read or write to members", and the
understanding that `shared` methods are expected to have threadsafe
implementations (because that's the whole point), what are the risks
from allowing T* -> shared(T)* conversion?
All the risks that I think have been identified previously assume that
you can arbitrarily modify the data. That's insanity... assume we fix
that... I think the promotion actually becomes safe now...?
Destroy...
This is a step in the right direction. But there is still one problem --
shared is inherently transitive.
So casting away shared is super-dangerous, even if you lock the shared
data, because any of the subreferences will become unshared and
read/writable.
For instance:
struct S
{
int x;
int *y;
}
shared int z;
auto s1 = shared(S)(1, &z);
auto s2 = shared(S)(2, &z);
S* s1locked = s1.lock;
Now I have access to z via s1locked as an unshared int, and I never
locked z. Potentially one could do the same thing via s2, and now there
are 2 mutable references, potentially in 2 threads.
All of this, of course, is manual. So technically we could manually
implement it properly inside S. But this means shared doesn't help us much.
We really need on top of shared, a way to specify something is
tail-shared. That is, all the data in S is unshared, but anything it
points to is still shared. That at least helps the person implementing
the manual locking from doing stupid things himself.
-Steve
Re: shared - i need it to be useful
On 10/16/18 9:25 AM, Steven Schveighoffer wrote: On 10/15/18 2:46 PM, Manu wrote: From there, it opens up another critical opportunity; T* -> shared(T)* promotion. Const would be useless without T* -> const(T)* promotion. Shared suffers a similar problem. If you write a lock-free queue for instance, and all the methods are `shared` (ie, threadsafe), then under the current rules, you can't interact with the object when it's not shared, and that's fairly useless. Oh, I didn't see this part. Completely agree with Timon on this, no implicit conversions should be allowed. If you want to have a lock-free implementation of something, you can abstract the assignments and reads behind the proper mechanisms anyway, and still avoid locking (casting is not locking). -Steve
Re: shared - i need it to be useful
On Tue, Oct 16, 2018 at 2:25 AM Kagamin via Digitalmars-d wrote: > > On Monday, 15 October 2018 at 18:46:45 UTC, Manu wrote: > > Current situation where you can arbitrarily access shared > > members > > undermines any value it has. > > The value of shared is existence of thread-local data that's > guaranteed to be not shared, so you don't need to worry about > thread-local data being shared, and shared protects that value > well. This isn't really an argument in favour of shared being able to arbitrarily access all its members though. I'm not sure how your points addresses my claim? this sounds like that old quote "there can not be peace without war". It's not shared that makes not-shared thread-local by default, it's just that's how D is defined. Shared allows you to break that core assumption. Without shared, it would still be thread-local by default... you just wouldn't have an escape hatch ;) > > Assuming this world... how do you use shared? > > Unique solution for each case. > > > If you write a lock-free queue for instance, and all the > > methods are > > `shared` (ie, threadsafe), then under the current rules, you > > can't > > interact with the object when it's not shared, and that's fairly > > useless. > > Create it as shared. Then you can't use it locally. > > Assuming the rules above: "can't read or write to members", and > > the understanding that `shared` methods are expected to have > > threadsafe implementations (because that's the whole point), > > what are the risks from allowing T* -> shared(T)* conversion? > > All data becomes possibly shared, so you can't assume it's > unshared, effectively C-style sharing. BTW D supports the latter > already. No data becomes 'possibly shared', because it's all inaccessible. Only if your object specifies a threadsafe API may some controlled data become shared... and that's the whole point of writing a threadsafe object. If the threadsafe object doesn't support sharing its own data, then it's not really a threadsafe object.
Re: shared - i need it to be useful
On Tue, Oct 16, 2018 at 3:20 AM Timon Gehr via Digitalmars-d wrote: > > On 15.10.2018 20:46, Manu wrote: > > > > Assuming the rules above: "can't read or write to members", and the > > understanding that `shared` methods are expected to have threadsafe > > implementations (because that's the whole point), what are the risks > > from allowing T* -> shared(T)* conversion? > > > > Unshared becomes useless, and in turn, shared becomes useless. You can't > have unshared/shared aliasing. What aliasing? Please show a reasonable and likely construction of the problem. I've been trying to think of it. > > All the risks that I think have been identified previously assume that > > you can arbitrarily modify the data. That's insanity... assume we fix > > that... I think the promotion actually becomes safe now...? > > But useless, because there is no way to ensure thread safety of reads > and writes if only one party to the shared state knows about the sharing. What? I don't understand this sentence. If a shared method is not threadsafe, then it's an implementation error. A user should expect that a shared method is threadsafe, otherwise it shouldn't be a shared method! Thread-local (ie, normal) methods are for not-threadsafe functionality.
Re: shared - i need it to be useful
On Tue, Oct 16, 2018 at 6:25 AM Timon Gehr via Digitalmars-d wrote: > > On 16.10.2018 13:04, Dominikus Dittes Scherkl wrote: > > On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote: > >> On 15.10.2018 20:46, Manu wrote: > >>> > >>> Assuming the rules above: "can't read or write to members", and the > >>> understanding that `shared` methods are expected to have threadsafe > >>> implementations (because that's the whole point), what are the risks > >>> from allowing T* -> shared(T)* conversion? > >>> > >> > >> Unshared becomes useless, and in turn, shared becomes useless. > > why is unshared useless? > > Unshared means you can read an write to it. > > If you give it to a function that expect something shared, > > the function you had given it to can't read or write it, so it > > can't do any harm. > > It can do harm to others who hold an unshared alias to the same data and > are operating on it concurrently. Nobody else holds an unshared alias. If you pass a value as const, you don't fear that it will become mutable. > > Of course it can handle it threadsave, but as it is local, > > that is only overhead - reading or changing the value can't do > > any harm either. I like the idea. > > > >> But useless, because there is no way to ensure thread safety of reads > >> and writes if only one party to the shared state knows about the sharing. > > Of course there is. > > Please do enlighten me. You have two processors operating > (reading/writing) on the same address space on a modern computer > architecture with a weak memory model, and you are using an optimizing > compiler. How do you ensure sensible results without cooperation from > both of them? (Hint: you don't.) What? This is a weird statement. So, you're saying that nobody has successfully written any threadsafe code, ever... we should stop trying, and we should admit that threadsafe queues and atomics, and mutexes and stuff all don't exist? > without cooperation from both of them? Perhaps this is the key to your statement? Yes. 'cooperation from both of them' in this case means, they are both interacting with a threadsafe api, and they are blocked from accessing members, or any non-threadsafe api. > > Giving an unshared value to a function that > > even can handle shared values may create some overhead, but is > > indeed threadsave. > > > > Yes, if you give it to one function only, that is the case. However, as > you may know, concurrency means that there may be multiple functions > operating on the data _at the same time_. If one of them operates on the > data as if it was not shared, you will run into trouble. Who's doing this, and how? > You are arguing as if there was either no concurrency or no mutable > aliasing. If a class has no shared methods, there's no possibility for mutable aliasing. If the class has shared methods, then the class was carefully designed to be threadsafe.
Re: shared - i need it to be useful
On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d wrote: > > On 10/16/18 9:25 AM, Steven Schveighoffer wrote: > > On 10/15/18 2:46 PM, Manu wrote: > > >>> From there, it opens up another critical opportunity; T* -> shared(T)* > >> promotion. > >> Const would be useless without T* -> const(T)* promotion. Shared > >> suffers a similar problem. > >> If you write a lock-free queue for instance, and all the methods are > >> `shared` (ie, threadsafe), then under the current rules, you can't > >> interact with the object when it's not shared, and that's fairly > >> useless. > >> > > Oh, I didn't see this part. Completely agree with Timon on this, no > implicit conversions should be allowed. Why? > If you want to have a lock-free implementation of something, you can > abstract the assignments and reads behind the proper mechanisms anyway, > and still avoid locking (casting is not locking). Sorry, I don't understand what you're saying. Can you clarify?
Re: shared - i need it to be useful
On 10/16/18 2:10 PM, Manu wrote: On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d wrote: On 10/16/18 9:25 AM, Steven Schveighoffer wrote: On 10/15/18 2:46 PM, Manu wrote: From there, it opens up another critical opportunity; T* -> shared(T)* promotion. Const would be useless without T* -> const(T)* promotion. Shared suffers a similar problem. If you write a lock-free queue for instance, and all the methods are `shared` (ie, threadsafe), then under the current rules, you can't interact with the object when it's not shared, and that's fairly useless. Oh, I didn't see this part. Completely agree with Timon on this, no implicit conversions should be allowed. Why? int x; shared int *p = &x; // allow implicit conversion, currently error passToOtherThread(p); useHeavily(&x); How is this safe? Thread1 is using x without locking, while the other thread has to lock. In order for synchronization to work, both sides have to agree on a synchronization technique and abide by it. If you want to have a lock-free implementation of something, you can abstract the assignments and reads behind the proper mechanisms anyway, and still avoid locking (casting is not locking). Sorry, I don't understand what you're saying. Can you clarify? I'd still mark a lock-free implementation shared, and all its methods shared. shared does not mean you have to lock, just cast away shared. A lock-free container still has to do some special things to make sure it avoids races, and having an "unusable" state aids in enforcing this. -Steve
Re: shared - i need it to be useful
On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via Digitalmars-d wrote: > > On 10/16/18 2:10 PM, Manu wrote: > > On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d > > wrote: > >> > >> On 10/16/18 9:25 AM, Steven Schveighoffer wrote: > >>> On 10/15/18 2:46 PM, Manu wrote: > >> > > From there, it opens up another critical opportunity; T* -> shared(T)* > promotion. > Const would be useless without T* -> const(T)* promotion. Shared > suffers a similar problem. > If you write a lock-free queue for instance, and all the methods are > `shared` (ie, threadsafe), then under the current rules, you can't > interact with the object when it's not shared, and that's fairly > useless. > > >> > >> Oh, I didn't see this part. Completely agree with Timon on this, no > >> implicit conversions should be allowed. > > > > Why? > > int x; > > shared int *p = &x; // allow implicit conversion, currently error > > passToOtherThread(p); > > useHeavily(&x); What does this mean? It can't do anything... that's the whole point here. I think I'm struggling here with people bringing presumptions to the thread. You need to assume the rules I define in the OP for the experiment to work. > How is this safe? Because useHeavily() can't read or write to x. > Thread1 is using x without locking, while the other > thread has to lock. In order for synchronization to work, both sides > have to agree on a synchronization technique and abide by it. Only the owning thread can access x, the shared instance can't access x at all. If some code somewhere decides it wants to cast-away shared, then it needs to determine ownership via some other means. It needs to be confident that the original owner yielded ownership, and any API that leads to this behaviour would need to be designed in such a way to encourage correct behaviour. That's *exactly* how it is now... I haven't changed anything from this perspective. > >> If you want to have a lock-free implementation of something, you can > >> abstract the assignments and reads behind the proper mechanisms anyway, > >> and still avoid locking (casting is not locking). > > > > Sorry, I don't understand what you're saying. Can you clarify? > > > > I'd still mark a lock-free implementation shared, and all its methods > shared. shared does not mean you have to lock, just cast away shared. Shared *should* mean that the function is threadsafe, and you are safe to call it from a shared instance. If a function is not shared, then you MUST cast away shared, and that implies that you need to use external means to create a context where you have thread-local ownership of the instance (usually with a mutex). You shouldn't need to cast-away shared to make a safe function call. By casting away shared, you also gain access to all the non-threadsafe methods and members. Casting shared away to call a shared method makes a safe access into a potentially unsafe access. It's unacceptable to case-away shared. It should be as unacceptable as casting const away. The only situation where it's okay is where you're externally verifying thread-locality by external means, and that's subject to your broader systemic design. > A lock-free container still has to do some special things to make sure it > avoids races, and having an "unusable" state aids in enforcing this. Can you explain how my proposal doesn't model this very neatly?
Re: shared - i need it to be useful
On 10/16/18 4:26 PM, Manu wrote:
On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via
Digitalmars-d wrote:
On 10/16/18 2:10 PM, Manu wrote:
On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d
wrote:
On 10/16/18 9:25 AM, Steven Schveighoffer wrote:
On 10/15/18 2:46 PM, Manu wrote:
From there, it opens up another critical opportunity; T* -> shared(T)*
promotion.
Const would be useless without T* -> const(T)* promotion. Shared
suffers a similar problem.
If you write a lock-free queue for instance, and all the methods are
`shared` (ie, threadsafe), then under the current rules, you can't
interact with the object when it's not shared, and that's fairly
useless.
Oh, I didn't see this part. Completely agree with Timon on this, no
implicit conversions should be allowed.
Why?
int x;
shared int *p = &x; // allow implicit conversion, currently error
passToOtherThread(p);
useHeavily(&x);
What does this mean? It can't do anything... that's the whole point here.
I think I'm struggling here with people bringing presumptions to the
thread. You need to assume the rules I define in the OP for the
experiment to work.
OK, I wrote a whole big response to this, and I went and re-quoted the
above, and now I think I understand what the point of your statement is.
I'll first say that if you don't want to allow implicit casting of
shared to mutable, then you can't allow implicit casting from mutable to
shared. Because it's mutable, races can happen.
There is in fact, no difference between:
int *p;
shared int *p2 = p;
int *p3 = cast(int*)p2;
and this:
int *p;
shared int *p2 = p;
int *p3 = p;
So really, the effort to prevent the reverse cast is defeated by
allowing the implicit cast.
There is a reason we disallow assigning from mutable to immutable
without a cast. Yet, it is done in many cases, because you are sometimes
building an immutable object with mutable pieces, and want to cast the
final result.
In this case, it's ON YOU to make sure it's correct, and the traditional
mechanism for the compiler giving you the responsibility is to require a
cast.
-
OK, so here is where I think I misunderstood your point. When you said a
lock-free queue would be unusable if it wasn't shared, I thought you
meant it would be unusable if we didn't allow the implicit cast. But I
realize now, you meant you should be able to use a lock-free queue
without it being actually shared anywhere.
What I say to this is that it doesn't need to be usable. I don't care to
use a lock-free queue in a thread-local capacity. I'll just use a normal
queue, which is easy to implement, and doesn't have to worry about race
conditions or using atomics. A lock free queue is a special thing, very
difficult to get right, and only really necessary if you are going to
share it. And used for performance reasons!
Why would I want to incur performance penalties when using a lock-free
queue in an unshared mode? I would actually expect 2 separate
implementations of the primitives, one for shared one for unshared.
What about primitives that would be implemented the same? In that case,
the shared method becomes:
auto method() { return (cast(Queue*)&this).method; }
Is this "unusable"? Without a way to say, you can call this on shared or
unshared instances, then we need to do it this way.
But I would trust the queue to handle this properly depending on whether
it was typed shared or not.
-Steve
Re: shared - i need it to be useful
On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote: OK, so here is where I think I misunderstood your point. When you said a lock-free queue would be unusable if it wasn't shared, I thought you meant it would be unusable if we didn't allow the implicit cast. But I realize now, you meant you should be able to use a lock-free queue without it being actually shared anywhere. What I say to this is that it doesn't need to be usable. I don't care to use a lock-free queue in a thread-local capacity. I'll just use a normal queue, which is easy to implement, and doesn't have to worry about race conditions or using atomics. A lock free queue is a special thing, very difficult to get right, and only really necessary if you are going to share it. And used for performance reasons! I think this comes up where the queue was originally shared, you acquired a lock on the thing it is a member of, and you want to continue using it through your exclusive reference.
Re: shared - i need it to be useful
On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote: There is in fact, no difference between: int *p; shared int *p2 = p; int *p3 = cast(int*)p2; and this: int *p; shared int *p2 = p; int *p3 = p; If I understand Manu correctly the first should compile, and the second should error, just like if you replaces shared with const in the above.
Re: shared - i need it to be useful
On Tue, Oct 16, 2018 at 2:20 PM Steven Schveighoffer via Digitalmars-d wrote: > > On 10/16/18 4:26 PM, Manu wrote: > > On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via > > Digitalmars-d wrote: > >> > >> On 10/16/18 2:10 PM, Manu wrote: > >>> On Tue, Oct 16, 2018 at 6:35 AM Steven Schveighoffer via Digitalmars-d > >>> wrote: > > On 10/16/18 9:25 AM, Steven Schveighoffer wrote: > > On 10/15/18 2:46 PM, Manu wrote: > > >>> From there, it opens up another critical opportunity; T* -> > >>> shared(T)* > >> promotion. > >> Const would be useless without T* -> const(T)* promotion. Shared > >> suffers a similar problem. > >> If you write a lock-free queue for instance, and all the methods are > >> `shared` (ie, threadsafe), then under the current rules, you can't > >> interact with the object when it's not shared, and that's fairly > >> useless. > >> > > Oh, I didn't see this part. Completely agree with Timon on this, no > implicit conversions should be allowed. > >>> > >>> Why? > >> > >> int x; > >> > >> shared int *p = &x; // allow implicit conversion, currently error > >> > >> passToOtherThread(p); > >> > >> useHeavily(&x); > > > > What does this mean? It can't do anything... that's the whole point here. > > I think I'm struggling here with people bringing presumptions to the > > thread. You need to assume the rules I define in the OP for the > > experiment to work. > > OK, I wrote a whole big response to this, and I went and re-quoted the > above, and now I think I understand what the point of your statement is. > > I'll first say that if you don't want to allow implicit casting of > shared to mutable, It's critical that this is not allowed. It's totally unreasonable to cast from shared to thread-local without synchronisation. It's as bad as casting away const. > then you can't allow implicit casting from mutable to > shared. Because it's mutable, races can happen. I don't follow... > There is in fact, no difference between: > > int *p; > shared int *p2 = p; > int *p3 = cast(int*)p2; Totally illegal!! You casted away shared. That's as bad as casting away const. > and this: > > int *p; > shared int *p2 = p; > int *p3 = p; There's nothing wrong with this... I don't understand the point? > So really, the effort to prevent the reverse cast is defeated by > allowing the implicit cast. Only the caller has the thread-local instance. You can take a thread-local pointer to a thread-local within the context of a single thread. So, it's perfectly valid for `p` and `p3` to exist in a single scope. `p2` is fine here too... and if that shared pointer were to escape to another thread, it wouldn't be a threat, because it's not readable or writable, and you can't make it back into a thread-local pointer without carefully/deliberately deployed machinery. > There is a reason we disallow assigning from mutable to immutable > without a cast. Yet, it is done in many cases, because you are sometimes > building an immutable object with mutable pieces, and want to cast the > final result. I don't think analogy to immutable has a place in this discussion, or at least, I don't understand the relevance... I think the reasonable analogy is const. > In this case, it's ON YOU to make sure it's correct, and the traditional > mechanism for the compiler giving you the responsibility is to require a > cast. I think what you're talking about are behaviours relating to casting shared *away*, and that's some next-level shit. Handling in that case is no different to the way it exists today. You must guarantee that the pointer you possess becomes thread-local before casting it to a thread-local pointer. In my application framework, I will never cast shared away under my proposed design. We don't have any such global locks. > - > > OK, so here is where I think I misunderstood your point. When you said a > lock-free queue would be unusable if it wasn't shared, I thought you > meant it would be unusable if we didn't allow the implicit cast. But I > realize now, you meant you should be able to use a lock-free queue > without it being actually shared anywhere. Right, a lock-free queue is a threadsafe object, and it's methods work whether the queue is shared or not. The methods are attributed shared because they can be called on shared instances... but they can ALSO be called from a thread-local instance, and under my suggested promotion rules, it's fine for the this-pointer to promote to shared to make the call. > What I say to this is that it doesn't need to be usable. I don't care to > use a lock-free queue in a thread-local capacity. I'll just use a normal > queue, which is easy to implement, and doesn't have to worry about race > conditions or using atomics. A lock free queue is a special thing, very > difficult to get right, and only really necessary if you are going to > share it. And used for performance reasons! I'm more interested in the
Re: shared - i need it to be useful
On Tue, Oct 16, 2018 at 3:25 PM Nicholas Wilson via Digitalmars-d wrote: > > On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer > wrote: > > There is in fact, no difference between: > > > > int *p; > > shared int *p2 = p; > > int *p3 = cast(int*)p2; > > > > and this: > > > > int *p; > > shared int *p2 = p; > > int *p3 = p; > > If I understand Manu correctly the first should compile, and the > second should error, just like if you replaces shared with const > in the above. Why is the second an error? Right, for the sake of the thought experiment, replace shared with const. The first is an abomination (casting away const)... the second is just a no-op.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 00:29:04 UTC, Manu wrote: On Tue, Oct 16, 2018 at 3:25 PM Nicholas Wilson via Digitalmars-d wrote: On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote: > There is in fact, no difference between: > > int *p; > shared int *p2 = p; > int *p3 = cast(int*)p2; > > and this: > > int *p; > shared int *p2 = p; > int *p3 = p; If I understand Manu correctly the first should compile, and the second should error, just like if you replaces shared with const in the above. Why is the second an error? Right, for the sake of the thought experiment, replace shared with const. The first is an abomination (casting away const)... the second is just a no-op. I missed that the third example was *p3 = p; not *p3 = p2;
Re: shared - i need it to be useful
On Tuesday, 16 October 2018 at 06:21:22 UTC, Manu wrote: On Mon, Oct 15, 2018 at 8:55 PM Isaac S. via Digitalmars-d wrote: On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote: >> I understand your point but I think the current shared (no >> implicit conversion) has its uses. >> *snip* > > If you can give a single 'use', I'm all ears ;) My usages are a custom ref counted template and list types (which are built on top of the former). The ref counted template will initialize naively when not shared but utilize compare-and-set and thread yielding if needed when shared (ensureInitialized can occur any time after declaration). The list types (which are not shared-compatible *yet*) will utilize a read-write mutex only when shared (no cost beyond a little unused memory to non-shared objects). As such, casting any of the non-shared versions of these types to shared would be unsafe. (Technically the types can be safely casted to shared so long as no non-shared reference to them exists and vice versa.) Can you link to code? It doesn't sound incompatible with my proposal. Mutexes are blunt instruments, and easy to model. My whole suggestion has virtually no effect on the mutex protected case, which is what most people seem to talk about. On Wednesday, 17 October 2018 at 00:26:58 UTC, Manu wrote: *snip* Overloading for shared and unshared is possible, and may be desirable in many cases. There are also many cases where the code duplication and tech-debt does not carry its weight. It should not be required, because it's not technically required. Overloading for shared and unshared is my reason for not allowing implicit conversion on my types (I have no problems with implicit conversion being optional or disableable). The unshared function performs no synchronization of any kind while the shared function always does. This means that having an unshared and shared reference to the same object is unsafe. As for an actual link to code, I can really only link to my ref counted template as I haven't gotten around to making the containers shared-compatible yet. The main point of interest is at lines 103-131: https://github.com/isaacs-dev/Familiar/blob/66f1a94fc099601465e755d40a2c68bf4200cabd/containers/familiar/containers/safe_ref_counted.d#L103 The unshared ensureInitialized() doesn't worry about threads simultaneously calling it. The shared version of it uses compare-and-set to prevent two threads from initializing it at the same time. An unsafe example would require a shared reference/pointer to an unshared SafeRefCounted (which would be fairly weird to do) and both calling ensureInitialized. While an example using the ref counted template is fairly contrived, a list-type that only uses a read-write mutex when it's shared isn't. As to the larger part of preventing reading/writing of shared objects, I agree with its purpose and have no problems with it.
Re: shared - i need it to be useful
On Tue, Oct 16, 2018 at 8:20 PM Isaac S. via Digitalmars-d wrote: > > On Tuesday, 16 October 2018 at 06:21:22 UTC, Manu wrote: > > On Mon, Oct 15, 2018 at 8:55 PM Isaac S. via Digitalmars-d > > wrote: > >> > >> On Tuesday, 16 October 2018 at 02:26:04 UTC, Manu wrote: > >> >> I understand your point but I think the current shared (no > >> >> implicit conversion) has its uses. > >> >> *snip* > >> > > >> > If you can give a single 'use', I'm all ears ;) > >> > >> My usages are a custom ref counted template and list types > >> (which are built on top of the former). The ref counted > >> template will initialize naively when not shared but utilize > >> compare-and-set and thread yielding if needed when shared > >> (ensureInitialized can occur any time after declaration). The > >> list types (which are not shared-compatible *yet*) will > >> utilize a read-write mutex only when shared (no cost beyond a > >> little unused memory to non-shared objects). As such, casting > >> any of the non-shared versions of these types to shared would > >> be unsafe. > >> > >> (Technically the types can be safely casted to shared so long > >> as no non-shared reference to them exists and vice versa.) > > > > Can you link to code? It doesn't sound incompatible with my > > proposal. Mutexes are blunt instruments, and easy to model. My > > whole suggestion has virtually no effect on the mutex protected > > case, which is what most people seem to talk about. > > On Wednesday, 17 October 2018 at 00:26:58 UTC, Manu wrote: > > *snip* > > > > Overloading for shared and unshared is possible, and may be > > desirable in many cases. > > There are also many cases where the code duplication and > > tech-debt > > does not carry its weight. It should not be required, because > > it's not technically required. > > Overloading for shared and unshared is my reason for not allowing > implicit conversion on my types (I have no problems with implicit > conversion being optional or disableable). The unshared function > performs no synchronization of any kind while the shared function > always does. This means that having an unshared and shared > reference to the same object is unsafe. Okay, so just to be clear, you're objecting to an immensely useful behaviour because you exploit the current design as an optimisation potential? That said, I'm still not taking anything away from you... you can still implement your class that way if you like, no change will affect that decision. You obviously have mechanisms in place to guarantee the thread-local-ness of your object (otherwise it wouldn't work), so, assuming you implement the unshared overload to be unsafe, then situation hasn't changed for you at all...? I don't think a robust middleware would make that trade-off, but an application can do that if it wishes. Trading implicit safety (enforcing it externally via application context) for perf is not unusual at all. > As to the larger part of preventing reading/writing of shared > objects, I agree with its purpose and have no problems with it. Right, the implicit cast thing is secondary to this fundamental part. I think it's worth first focusing on getting that rule right. shared = no read + no write .. I don't think that's objectionable.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 03:50:44 UTC, Manu wrote: On Tue, Oct 16, 2018 at 8:20 PM Isaac S. via Digitalmars-d wrote: *snip* Overloading for shared and unshared is my reason for not allowing implicit conversion on my types (I have no problems with implicit conversion being optional or disableable). The unshared function performs no synchronization of any kind while the shared function always does. This means that having an unshared and shared reference to the same object is unsafe. Okay, so just to be clear, you're objecting to an immensely useful behaviour because you exploit the current design as an optimisation potential? That said, I'm still not taking anything away from you... you can still implement your class that way if you like, no change will affect that decision. You obviously have mechanisms in place to guarantee the thread-local-ness of your object (otherwise it wouldn't work), so, assuming you implement the unshared overload to be unsafe, then situation hasn't changed for you at all...? I don't think a robust middleware would make that trade-off, but an application can do that if it wishes. Trading implicit safety (enforcing it externally via application context) for perf is not unusual at all. I'm not objecting to the behavior (I actually really want it in the language as it allows for some really useful designs) but more-so saying it would be nice if it could be enabled or disabled. If its enabled-by-default, the @disable attribute could be used somehow (maybe "@disable alias this shared;"). Even without the ability to enable/disable shared implicit conversion it won't break my code, it'll just allow a potential bug. Since it does allow for extremely useful designs, I'll support this change (*even if it can't be enabled/disabled*).
Re: shared - i need it to be useful
On 10/15/2018 11:46 AM, Manu wrote: [...] Shared has one incredibly valuable feature - it allows you, the programmer, to identify data that can be accessed by multiple threads. There are so many ways that data can be shared, the only way to comprehend what is going on is to build a wall around shared data. (The exception to this is immutable data. Immutable data does not need synchronization, so there is no need to distinguish between shared and unshared immutable data.) This falls completely apart if shared and unshared data can be aliased. When Andrei and I came up with the rules for: mutable const shared const shared immutable and which can be implicitly converted to what, so far nobody has found a fault in those rules. Timon Gehr has done a good job showing that they still stand unbreached. So, how can mutable data become shared, and vice versa? I've never found a way to do that that is proveably safe. Therefore, it's up to the programmer. If a programmer does the following: T* p; shared(T)* sp; p = cast(T*)sp; (1) sp = cast(shared(T)*) p; (2) those casts will be rejected in @safe code. They'll have to be in @trusted or @system code. It will be up to the programmer to ensure (via mutexes, locks, uniqueness, etc.) that: (1) sp is a unique pointer and that ownership of the data is thus transferred for the lifetime of p (2) p is a unique pointer and that ownership of the data is thus transferred for the lifetime of sp A sensible program should try to minimize the places where there are these "gates" through the wall, as those gates will be where you'll be looking when threading bugs appear. Allowing implicit aliasing between shared and unshared data means the entire program needs to reviewed for threading bugs, rather than just the shared parts. One might as well just not have shared as a language feature at all. D being a systems programming language, one can certainly write code that way, just like one does with C, and with all of the threading bugs one gets doing that. -- Currently, the compiler allows you to read/write shared data without any locks or atomics. I.e. the compiler doesn't attempt to insert any synchronization on your behalf. Given all the ways synchronization can be done, it just seems presumptive and restrictive to impose a particular scheme. It's pretty much left up to the user. Which is why I recommend minimizing the amount of code that actually has to deal with shared data.
Re: shared - i need it to be useful
On Tue, Oct 16, 2018 at 10:45 PM Walter Bright via Digitalmars-d wrote: > > On 10/15/2018 11:46 AM, Manu wrote: > > [...] > > Shared has one incredibly valuable feature - it allows you, the programmer, to > identify data that can be accessed by multiple threads. There are so many ways > that data can be shared, the only way to comprehend what is going on is to > build > a wall around shared data. > > (The exception to this is immutable data. Immutable data does not need > synchronization, so there is no need to distinguish between shared and > unshared > immutable data.) > > This falls completely apart if shared and unshared data can be aliased. What does it mean 'aliased' precisely? I'm trying to prevent that in practical terms. If you can't read/write to a shared object... is it actually aliased? Or do you just have a fancy int that happens to look like a pointer? > When Andrei and I came up with the rules for: > > mutable > const > shared > const shared > immutable > > and which can be implicitly converted to what, so far nobody has found a fault > in those rules. Oh bollocks... everyone has been complaining about this for at least the 10 years I've been here! Shared is uninteresting and mostly useless as spec-ed, everyone knows this. Interaction with shared via barely-controlled blunt casting in @trusted blocks is feeble and boring. It doesn't really give us anything in practice that we don't have in C++. It's an uninspired design. I am working against a design where *everything* may be shared, and it's a very interesting design space. I have no doubt it represents the future of my field. We are limited by what the type system can express on this front. I'm trying to explore opportunities to make shared interesting and useful... and I'm mostly being met with prejudice here. Eject what you think you know about shared from your brain, and try and take a fresh look from the perspective I'm proposing. I think you'll find that the current wisdom and interactions with shared are actually preserved in practice, but new opportunities become available. If not, I want to understand how the design is broken, so I can continue to iterate. If D offered a real advantage here over C++, this would really represent a strong attracting force. > Timon Gehr has done a good job showing that they still stand > unbreached. His last comment was applied to a different proposal. His only comment on this thread wasn't in response to the proposal in this thread. If you nominate Timon as your proxy, then he needs to destroy my proposal, or at least comment on it, rather than make some prejudiced comment generally. > So, how can mutable data become shared, and vice versa? I've never found a way > to do that that is proveably safe. Therefore, it's up to the programmer. I'm proposing a way, and that is: 1. the rule must be applied that shared object can not be read or written 2. attributing a method shared is a statement and a promise that the method is threadsafe The rest just follows naturally. > If a programmer does the following: > > T* p; > shared(T)* sp; > > p = cast(T*)sp; (1) > sp = cast(shared(T)*) p; (2) > > those casts will be rejected in @safe code. They'll have to be in @trusted or > @system code. It will be up to the programmer to ensure (via mutexes, locks, > uniqueness, etc.) that: > > (1) sp is a unique pointer and that ownership of the data is thus > transferred for the lifetime of p > > (2) p is a unique pointer and that ownership of the data is thus > transferred for the lifetime of sp 'Transfer' is a nice idea, but it's not what's happening here. That's not what these operations do. Transferring ownership is a job for move semantics, and that's nowhere in sight. Also, there's no need to 'transfer' a this pointer to shared to call a threadsafe method. You can always just call it safely. > A sensible program should try to minimize the places where there are these > "gates" through the wall, as those gates will be where you'll be looking when > threading bugs appear. I agree, that's my goal here. Shared right now is an unsafe mess; I want to apply aggressive restriction so that you can't do unsafe operations without explicit casts. > Allowing implicit aliasing between shared and unshared data means the entire > program needs to reviewed for threading bugs, rather than just the shared > parts. That's backwards, I'm suggesting conversion TO shared, such that you are able to call threadsafe methods on thread-local things. There's no possible harm in that. I'm trying to eliminate aliasing by removing read/write access. The result is, the only thing you are able to do with a shared reference are explicitly threadsafe things. Any other operation is inaccessible without deliberate blunt casting, and the rules surrounding that are the same as always; those that you listed above. > One might as well just not have shared as a language feature a
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright wrote: When Andrei and I came up with the rules for: mutable const shared const shared immutable and which can be implicitly converted to what, so far nobody has found a fault in those rules... Here's one: shared -> const shared shouldn't be allowed. Mutable aliasing in single-threaded code is bad enough as it is.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright wrote: On 10/15/2018 11:46 AM, Manu wrote: [...] Shared has one incredibly valuable feature - it allows you, the programmer, to identify data that can be accessed by multiple threads. There are so many ways that data can be shared, the only way to comprehend what is going on is to build a wall around shared data. (The exception to this is immutable data. Immutable data does not need synchronization, so there is no need to distinguish between shared and unshared immutable data.) [snip] Isn't that also true for isolated data (data that only allows one alias)?
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 07:20:20 UTC, Manu wrote: [snip] Oh bollocks... everyone has been complaining about this for at least the 10 years I've been here! [snip] As far as I had known from reading the forums, shared was not feature complete. Also, are you familiar with Atila's fearless library for safe sharing? https://github.com/atilaneves/fearless
Re: shared - i need it to be useful
On 16.10.2018 20:07, Manu wrote:
On Tue, Oct 16, 2018 at 6:25 AM Timon Gehr via Digitalmars-d
wrote:
On 16.10.2018 13:04, Dominikus Dittes Scherkl wrote:
On Tuesday, 16 October 2018 at 10:15:51 UTC, Timon Gehr wrote:
On 15.10.2018 20:46, Manu wrote:
Assuming the rules above: "can't read or write to members", and the
understanding that `shared` methods are expected to have threadsafe
implementations (because that's the whole point), what are the risks
from allowing T* -> shared(T)* conversion?
Unshared becomes useless, and in turn, shared becomes useless.
why is unshared useless?
Unshared means you can read an write to it.
If you give it to a function that expect something shared,
the function you had given it to can't read or write it, so it
can't do any harm.
It can do harm to others who hold an unshared alias to the same data and
are operating on it concurrently.
Nobody else holds an unshared alias.
How so? If you allow implicit conversions from unshared to shared, then
you immediately get this situation.
If you pass a value as const, you don't fear that it will become mutable.
...
No, but as I already explained last time, mutable -> const is not at all
like unshared -> shared.
const only takes away capabilities, shared adds new capabilities, such
as sending a reference to another thread. If you have two threads that
share data, you need cooperation from both to properly synchronize accesses.
Of course it can handle it threadsave, but as it is local,
that is only overhead - reading or changing the value can't do
any harm either. I like the idea.
But useless, because there is no way to ensure thread safety of reads
and writes if only one party to the shared state knows about the sharing.
Of course there is.
Please do enlighten me. You have two processors operating
(reading/writing) on the same address space on a modern computer
architecture with a weak memory model, and you are using an optimizing
compiler. How do you ensure sensible results without cooperation from
both of them? (Hint: you don't.)
What? This is a weird statement.
So, you're saying that nobody has successfully written any threadsafe
code, ever... we should stop trying, and we should admit that
threadsafe queues and atomics, and mutexes and stuff all don't exist?
Obviously I am not saying that.
without cooperation from both of them?
Perhaps this is the key to your statement?
Yes.
Yes. 'cooperation from both of them' in this case means, they are both
interacting with a threadsafe api, and they are blocked from accessing
members, or any non-threadsafe api.
...
Yes. Your proposal only enforces this for the shared alias.
Giving an unshared value to a function that
even can handle shared values may create some overhead, but is
indeed threadsave.
Yes, if you give it to one function only, that is the case. However, as
you may know, concurrency means that there may be multiple functions
operating on the data _at the same time_. If one of them operates on the
data as if it was not shared, you will run into trouble.
Who's doing this,
Anyone, it really does not matter. One major point of the type system is
to ensure that _all_ @safe code has defined behavior. You can convert
between shared and unshared, just not in @safe code.
and how?
...
They create a mutable instance of a class, they create a shared alias
using one of your proposed holes, then send the shared alias to another
thread, call some methods on it in both threads and get race conditions.
You are arguing as if there was either no concurrency or no mutable
aliasing.
If a class has no shared methods, there's no possibility for mutable aliasing.
If the class has shared methods, then the class was carefully designed
to be threadsafe.
Not necessarily. Counterexample:
@safe:
class C{
int x;
void foo(){
x+=1; // this can still race with atomicIncrement
}
void bar()shared{
atomicIncrement(x); // presumably you want to allow this
}
}
void main(){
auto c=new C();
shared s=c; // depending on your exact proposed rules, this step
may be more cumbersome
spawn!(()=>s.bar());
s.foo(); // race
}
Now, if a class has only shared members, that is another story. In this
case, all references should implicitly convert to shared. There's a DIP
I meant to write about this. (For all qualifiers, not just shared).
Re: shared - i need it to be useful
On 16.10.2018 19:25, Manu wrote: On Tue, Oct 16, 2018 at 3:20 AM Timon Gehr via Digitalmars-d wrote: On 15.10.2018 20:46, Manu wrote: Assuming the rules above: "can't read or write to members", and the understanding that `shared` methods are expected to have threadsafe implementations (because that's the whole point), what are the risks from allowing T* -> shared(T)* conversion? Unshared becomes useless, and in turn, shared becomes useless. You can't have unshared/shared aliasing. What aliasing? Aliasing means you have two references to the same data. The two references are then said to alias. An implicit conversion from unshared to shared by definition introduces aliasing, where one of the two references is unshared and the other is shared. Please show a reasonable and likely construction of the problem. I've been trying to think of it. ... I have given you an example. I don't care whether it is "reasonable" or "likely". The point of @safe is to have a subset of the language with a sound type system. All the risks that I think have been identified previously assume that you can arbitrarily modify the data. That's insanity... assume we fix that... I think the promotion actually becomes safe now...? But useless, because there is no way to ensure thread safety of reads and writes if only one party to the shared state knows about the sharing. What? I don't understand this sentence. If a shared method is not threadsafe, then it's an implementation error. A user should expect that a shared method is threadsafe, otherwise it shouldn't be a shared method! Thread-local (ie, normal) methods are for not-threadsafe functionality. Your function can be thread safe all you want. If you have a thread unsafe function also operating on the same state, you will still get race conditions. E.g. mutual exclusion is based on cooperation from all threads. If one of them forgets to lock, it does not matter how threadsafe all the others are.
Re: shared - i need it to be useful
On 15.10.2018 23:51, Manu wrote: If a shared method is incompatible with an unshared method, your class is broken. Then what you want is not implicit unshared->shared conversion. What you want is a different way to type shared member access. You want a setup where shared methods are only allowed to access shared members and unshared methods are only allowed to access unshared members. I.e., what you want is that shared is not transitive. You want that if you have a shared(C) c, then it is an error to access c.m iff m is not shared. This way you can have partially shared classes, where part of the class is thread-local, and other parts are shared with other threads. Is this it?
Re: shared - i need it to be useful
On 17.10.2018 14:24, Timon Gehr wrote: On 15.10.2018 23:51, Manu wrote: If a shared method is incompatible with an unshared method, your class is broken. Then what you want is not implicit unshared->shared conversion. What you want is a different way to type shared member access. You want a setup where shared methods are only allowed to access shared members and unshared methods are only allowed to access unshared members. I.e., what you want is that shared is not transitive. You want that if you have a shared(C) c, then it is an error to access c.m iff m is not shared. This way you can have partially shared classes, where part of the class is thread-local, and other parts are shared with other threads. Is this it? (Also, with this new definition of 'shared', unshared -> shared conversion would of course become sound.)
Re: shared - i need it to be useful
On 17.10.2018 09:20, Manu wrote: Timon Gehr has done a good job showing that they still stand unbreached. His last comment was applied to a different proposal. His only comment on this thread wasn't in response to the proposal in this thread. If you nominate Timon as your proxy, then he needs to destroy my proposal, or at least comment on it, rather than make some prejudiced comment generally. There is no "prejudice", just reasoning. Your proposal was "disallow member access on shared aggregates, allow implicit conversion from unshared to shared and keep everything else the same". This is a bad proposal. There may be a good proposal that allows the things you want, but you have not stated what they are, your OP was just: "look at this bad proposal, it might work, no?" I said no, then was met with some hostility. You should focus on finding a good proposal that achieves what you want without breaking the type system instead of attacking me.
Re: shared - i need it to be useful
On 17.10.2018 14:29, Timon Gehr wrote: to access c.m iff m is not shared Unfortunate typo. This should be if, not iff (if and only if).
Re: shared - i need it to be useful
On 10/16/18 6:24 PM, Nicholas Wilson wrote: On Tuesday, 16 October 2018 at 21:19:26 UTC, Steven Schveighoffer wrote: OK, so here is where I think I misunderstood your point. When you said a lock-free queue would be unusable if it wasn't shared, I thought you meant it would be unusable if we didn't allow the implicit cast. But I realize now, you meant you should be able to use a lock-free queue without it being actually shared anywhere. What I say to this is that it doesn't need to be usable. I don't care to use a lock-free queue in a thread-local capacity. I'll just use a normal queue, which is easy to implement, and doesn't have to worry about race conditions or using atomics. A lock free queue is a special thing, very difficult to get right, and only really necessary if you are going to share it. And used for performance reasons! I think this comes up where the queue was originally shared, you acquired a lock on the thing it is a member of, and you want to continue using it through your exclusive reference. Isn't that a locking queue? I thought we were talking lock-free? -Steve
Re: shared - i need it to be useful
On 17.10.2018 14:24, Timon Gehr wrote: and unshared methods are only allowed to access unshared members. This is actually not necessary, let me reformulate: You want: - if you have a C c and a shared(C) s, typeof(s.x) == typeof(c.x). - shared methods are not allowed to access unshared members. - shared is not transitive, and therefore unshared class references implicitly convert to shared class references Applied to pointers, this would mean that you can implicitly convert int* -> shared(int*), but not shared(int*)->int*, int* -> shared(int)* or shared(int)* -> int*. shared(int*) and shared(shared(int)*) would be different types, such that shared(int*) cannot be dereferenced but shared(shared(int)*) can.
Re: shared - i need it to be useful
On 10/16/18 8:26 PM, Manu wrote: On Tue, Oct 16, 2018 at 2:20 PM Steven Schveighoffer via Digitalmars-d wrote: On 10/16/18 4:26 PM, Manu wrote: On Tue, Oct 16, 2018 at 11:30 AM Steven Schveighoffer via Digitalmars-d wrote: int x; shared int *p = &x; // allow implicit conversion, currently error passToOtherThread(p); useHeavily(&x); What does this mean? It can't do anything... that's the whole point here. I think I'm struggling here with people bringing presumptions to the thread. You need to assume the rules I define in the OP for the experiment to work. OK, I wrote a whole big response to this, and I went and re-quoted the above, and now I think I understand what the point of your statement is. I'll first say that if you don't want to allow implicit casting of shared to mutable, It's critical that this is not allowed. It's totally unreasonable to cast from shared to thread-local without synchronisation. OK, so even with synchronization in the second thread when you cast, you still have a thread-local pointer in the originating thread WITHOUT synchronization. It's as bad as casting away const. Of course! But shared has a different problem from const. Const allows the data to change through another reference, shared cannot allow changes without synchronization. Changes without synchronization are *easy* with an unshared reference. Data can't be shared and unshared at the same time. then you can't allow implicit casting from mutable to shared. Because it's mutable, races can happen. I don't follow... You seem to be saying that shared data is unusable. But why the hell have it then? At some point it has to be usable. And the agreed-upon use is totally defeated if you also have some stray non-shared reference to it. There is in fact, no difference between: int *p; shared int *p2 = p; int *p3 = cast(int*)p2; Totally illegal!! You casted away shared. That's as bad as casting away const. But if you can't do anything with shared data, how do you use it? and this: int *p; shared int *p2 = p; int *p3 = p; There's nothing wrong with this... I don't understand the point? It's identical to the top one. You now have a new unshared reference to shared data. This is done WITHOUT any agreed-upon synchronization. So really, the effort to prevent the reverse cast is defeated by allowing the implicit cast. Only the caller has the thread-local instance. You can take a thread-local pointer to a thread-local within the context of a single thread. So, it's perfectly valid for `p` and `p3` to exist in a single scope. `p2` is fine here too... and if that shared pointer were to escape to another thread, it wouldn't be a threat, because it's not readable or writable, and you can't make it back into a thread-local pointer without carefully/deliberately deployed machinery. Huh? If shared data can never be used, why have it? Pretend that p is not a pointer to an int, but a pointer to an UNSHARED type that has shared methods on it and unshared methods (for when you don't need any sync). Now the shared methods will obey the sync, but the unshared ones won't. The result is races. I can't understand how you don't see that. There is a reason we disallow assigning from mutable to immutable without a cast. Yet, it is done in many cases, because you are sometimes building an immutable object with mutable pieces, and want to cast the final result. I don't think analogy to immutable has a place in this discussion, or at least, I don't understand the relevance... I think the reasonable analogy is const. No, immutable is more akin to shared because immutable and mutable are completely different. const can point at mutable or immutable data. shared can't be both shared and unshared. There's no comparison. Data is either shared or not shared, there is no middle ground. There is no equivalent of const to say "this data could be shared, or could be unshared". In this case, it's ON YOU to make sure it's correct, and the traditional mechanism for the compiler giving you the responsibility is to require a cast. I think what you're talking about are behaviours relating to casting shared *away*, and that's some next-level shit. Handling in that case is no different to the way it exists today. You must guarantee that the pointer you possess becomes thread-local before casting it to a thread-local pointer. In my application framework, I will never cast shared away under my proposed design. We don't have any such global locks. OK, so how does shared data actually operate? Somewhere, the magic has to turn into real code. If not casting away shared, what do you suggest? - OK, so here is where I think I misunderstood your point. When you said a lock-free queue would be unusable if it wasn't shared, I thought you meant it would be unusable if we didn't allow the implicit cast. But I realize now, you meant you should be able to use a lock-free queue without it bein
Re: shared - i need it to be useful
Jesus Manu, it's soon 8 pages of dancing around a trivial issue. Implicit casting from mutable to shared is unsafe, case closed. Explicit cast from mutable to unsafe, on the other hand: - is an assertion (on programmer's behalf) that this instance is indeed unique - is self-documenting - is greppable (especially if implemented as assumeShared or other descriptive name). I don't understand why are you so fixated on this. The other parts of your proposal are much more important.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 13:36:53 UTC, Stanislav Blinov wrote: Explicit cast from mutable to unsafe, on the other hand: Blargh, to shared of course.
Re: shared - i need it to be useful
On 10/17/18 8:02 AM, Timon Gehr wrote: Now, if a class has only shared members, that is another story. In this case, all references should implicitly convert to shared. There's a DIP I meant to write about this. (For all qualifiers, not just shared). When you say "shared members", you mean all the data is shared too or just the methods are shared? If not the data, D has a problem with encapsulation. Not only all the methods on the class must be shared, but ALL code in the entire module must be marked as using a shared class instance. Otherwise, other functions could modify the private data without using the proper synch mechanisms. We are better off requiring the cast, or enforcing that one must use a shared object to begin with. I think any sometimes-shared object is in any case going to benefit from parallel implementations for when the thing is unshared. -Steve
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer wrote: On 10/16/18 8:26 PM, Manu wrote: On Tue, Oct 16, 2018 at 2:20 PM Steven Schveighoffer via Digitalmars-d wrote: There is in fact, no difference between: int *p; shared int *p2 = p; int *p3 = cast(int*)p2; Totally illegal!! You casted away shared. That's as bad as casting away const. But if you can't do anything with shared data, how do you use it? and this: int *p; shared int *p2 = p; int *p3 = p; There's nothing wrong with this... I don't understand the point? It's identical to the top one. You now have a new unshared reference to shared data. This is done WITHOUT any agreed-upon synchronization. It isn't, you typo'd it (I originally missed it too). int *p3 = cast(int*)p2; vs int *p3 = p;
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 07:24:13 UTC, Stanislav Blinov wrote: On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright wrote: When Andrei and I came up with the rules for: mutable const shared const shared immutable and which can be implicitly converted to what, so far nobody has found a fault in those rules... Here's one: shared -> const shared shouldn't be allowed. Mutable aliasing in single-threaded code is bad enough as it is. Could you explain that a bit more? I don't understand it, mutable -> const is half the reason const exists. I was thinking that mutable -> shared const as apposed to mutable -> shared would get around the issues that Timon posted.
Re: shared - i need it to be useful
On 17.10.2018 15:40, Steven Schveighoffer wrote:
On 10/17/18 8:02 AM, Timon Gehr wrote:
Now, if a class has only shared members, that is another story. In
this case, all references should implicitly convert to shared. There's
a DIP I meant to write about this. (For all qualifiers, not just shared).
When you say "shared members", you mean all the data is shared too or
just the methods are shared?
If not the data, D has a problem with encapsulation. Not only all the
methods on the class must be shared, but ALL code in the entire module
must be marked as using a shared class instance. Otherwise, other
functions could modify the private data without using the proper synch
mechanisms.
We are better off requiring the cast, or enforcing that one must use a
shared object to begin with.
I think any sometimes-shared object is in any case going to benefit from
parallel implementations for when the thing is unshared.
-Steve
The specific proposal was that, for example, if a class is defined like
this:
shared class C{
// ...
}
then shared(C) and C are implicitly convertible to each other. The
change is not fully backwards-compatible, because right now, this
annotation just makes all members (data and methods) shared, but child
classes may introduce unshared members.
Re: shared - i need it to be useful
On 17.10.2018 16:14, Nicholas Wilson wrote: I was thinking that mutable -> shared const as apposed to mutable -> shared would get around the issues that Timon posted. Unfortunately not. For example, the thread with the mutable reference is not obliged to actually make the changes that are performed on that reference visible to other threads.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 14:26:43 UTC, Timon Gehr wrote: On 17.10.2018 16:14, Nicholas Wilson wrote: I was thinking that mutable -> shared const as apposed to mutable -> shared would get around the issues that Timon posted. Unfortunately not. For example, the thread with the mutable reference is not obliged to actually make the changes that are performed on that reference visible to other threads. Yes, but that is covered by not being able to read non-atomically from a shared reference.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 07:20:20 UTC, Manu wrote: Shared is uninteresting and mostly useless as spec-ed, everyone knows this. Interaction with shared via barely-controlled blunt casting in @trusted blocks is feeble and boring. It doesn't really give us anything in practice that we don't have in C++. It's an uninspired design. +1 I'm pretty sure it's great in theory, in practice it's another invasive type constructor, and not a particular well-understood or easy to deal with one. The fact that this _type constructor_ finds its way into _identifiers_ create some concern: https://github.com/dlang/phobos/blob/656798f2b385437c239246b59e0433148190938c/std/experimental/allocator/package.d#L642
Re: shared - i need it to be useful
On 10/17/18 10:18 AM, Timon Gehr wrote:
On 17.10.2018 15:40, Steven Schveighoffer wrote:
On 10/17/18 8:02 AM, Timon Gehr wrote:
Now, if a class has only shared members, that is another story. In
this case, all references should implicitly convert to shared.
There's a DIP I meant to write about this. (For all qualifiers, not
just shared).
When you say "shared members", you mean all the data is shared too or
just the methods are shared?
If not the data, D has a problem with encapsulation. Not only all the
methods on the class must be shared, but ALL code in the entire module
must be marked as using a shared class instance. Otherwise, other
functions could modify the private data without using the proper synch
mechanisms.
We are better off requiring the cast, or enforcing that one must use a
shared object to begin with.
I think any sometimes-shared object is in any case going to benefit
from parallel implementations for when the thing is unshared.
-Steve
The specific proposal was that, for example, if a class is defined like
this:
shared class C{
   // ...
}
then shared(C) and C are implicitly convertible to each other. The
change is not fully backwards-compatible, because right now, this
annotation just makes all members (data and methods) shared, but child
classes may introduce unshared members.
OK, so the proposal is that all data and function members are shared.
That makes sense.
In one sense, because the class reference is conflated with the type
modifier, having a C that isn't shared, actually have it's class data be
shared, would be useful.
-Steve
Re: shared - i need it to be useful
On 10/17/18 10:33 AM, Nicholas Wilson wrote: On Wednesday, 17 October 2018 at 14:26:43 UTC, Timon Gehr wrote: On 17.10.2018 16:14, Nicholas Wilson wrote: I was thinking that mutable -> shared const as apposed to mutable -> shared would get around the issues that Timon posted. Unfortunately not. For example, the thread with the mutable reference is not obliged to actually make the changes that are performed on that reference visible to other threads. Yes, but that is covered by not being able to read non-atomically from a shared reference. All sides must participate in synchronization for it to make sense. The mutable side has no obligation to use atomics. It can use ++data, and race conditions will happen. -Steve
Re: shared - i need it to be useful
On 10/17/18 9:58 AM, Nicholas Wilson wrote: On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer wrote: It's identical to the top one. You now have a new unshared reference to shared data. This is done WITHOUT any agreed-upon synchronization. It isn't, you typo'd it (I originally missed it too). int *p3 = cast(int*)p2; vs int *p3 = p; It wasn't a typo. It's identical in that both result in a thread-local pointer equivalent to p. Effectively, you can "cast" away shared without having to write a cast. I was trying to demonstrate the ineffectiveness of preventing implicit casting from shared to mutable if you allow unshared data to implicitly cast to shared. It's the same problem with mutable and immutable. It's why we can't allow the implicit casting. Explicit casting is OK as long as you don't later modify the data. In the same vein, explicit casting of local to shared is OK as long as you don't ever treat the data as local again. Which should requires a cast to say "I know what I'm doing, compiler". -Steve
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 15:51:04 UTC, Steven Schveighoffer wrote: On 10/17/18 9:58 AM, Nicholas Wilson wrote: On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer wrote: It's identical to the top one. You now have a new unshared reference to shared data. This is done WITHOUT any agreed-upon synchronization. It isn't, you typo'd it (I originally missed it too). int *p3 = cast(int*)p2; vs int *p3 = p; It wasn't a typo. The first example assigns p2, the second assigns p (which is thread local) _not_ p2 (which is shared), I'm confused.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 14:44:19 UTC, Guillaume Piolat wrote: The fact that this _type constructor_ finds its way into _identifiers_ create some concern: https://github.com/dlang/phobos/blob/656798f2b385437c239246b59e0433148190938c/std/experimental/allocator/package.d#L642 Well, ISharedAllocator is indeed overboard, but at least `allocateShared` would've been a very useful identifier indeed.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 14:14:56 UTC, Nicholas Wilson wrote: On Wednesday, 17 October 2018 at 07:24:13 UTC, Stanislav Blinov wrote: On Wednesday, 17 October 2018 at 05:40:41 UTC, Walter Bright wrote: When Andrei and I came up with the rules for: mutable const shared const shared immutable and which can be implicitly converted to what, so far nobody has found a fault in those rules... Here's one: shared -> const shared shouldn't be allowed. Mutable aliasing in single-threaded code is bad enough as it is. Could you explain that a bit more? I don't understand it, mutable -> const is half the reason const exists. Yes, but `shared` is not `const`. This 'might change' rule is poisonous for shared data: you essentially create extra work for the CPU for very little gain. There's absolutely no reason to share a read-only half-constant anyway. Either give immutable, or just flat out copy. I was thinking that mutable -> shared const as apposed to mutable -> shared would get around the issues that Timon posted. It wouldn't, as Timon already explained. Writes to not-`shared` mutables might just not be propagated beyond the core. Wouldn't happen on amd64, of course, but still. It's not about atomicity of reads, it just depends on architecture. On some systems you have to explicitly synchronize memory.
Re: shared - i need it to be useful
On 10/17/18 12:27 PM, Nicholas Wilson wrote: On Wednesday, 17 October 2018 at 15:51:04 UTC, Steven Schveighoffer wrote: On 10/17/18 9:58 AM, Nicholas Wilson wrote: On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer wrote: It's identical to the top one. You now have a new unshared reference to shared data. This is done WITHOUT any agreed-upon synchronization. It isn't, you typo'd it (I originally missed it too). int *p3 = cast(int*)p2; vs int *p3 = p; It wasn't a typo. The first example assigns p2, the second assigns p (which is thread local) _not_ p2 (which is shared), I'm confused. Here they are again: int *p; shared int *p2 = p; int *p3 = cast(int*)p2; int *p; shared int *p2 = p; int *p3 = p; I'll put some asserts in that show they accomplish the same thing: assert(p3 is p2); assert(p3 is p); assert(p2 is p); What the example demonstrates is that while you are trying to disallow implicit casting of a shared pointer to an unshared pointer, you have inadvertently allowed it by leaving behind an unshared pointer that is the same thing. While we do implicitly allow mutable to cast to const, it's because const is a weak guarantee. It's a guarantee that the data may not change via *this* reference, but could change via other references. Shared doesn't have the same characteristics. In order for a datum to be safely shared, it must be accessed with synchronization or atomics by ALL parties. If you have one party that can simply change it without those, you will get races. That's why shared/unshared is more akin to mutable/immutable than mutable/const. It's true that only one thread will have thread-local access. It's not valid any more than having one mutable alias to immutable data. -Steve
Re: shared - i need it to be useful
On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via
Digitalmars-d wrote:
>
> On 10/17/18 12:27 PM, Nicholas Wilson wrote:
> > On Wednesday, 17 October 2018 at 15:51:04 UTC, Steven Schveighoffer wrote:
> >> On 10/17/18 9:58 AM, Nicholas Wilson wrote:
> >>> On Wednesday, 17 October 2018 at 13:25:28 UTC, Steven Schveighoffer
> >>> wrote:
> It's identical to the top one. You now have a new unshared reference
> to shared data. This is done WITHOUT any agreed-upon synchronization.
> >>>
> >>> It isn't, you typo'd it (I originally missed it too).
> int *p3 = cast(int*)p2;
> >>>
> >>> vs
> >>>
> int *p3 = p;
> >>
> >> It wasn't a typo.
> >
> > The first example assigns p2, the second assigns p (which is thread
> > local) _not_ p2 (which is shared), I'm confused.
> >
>
> Here they are again:
>
> int *p;
> shared int *p2 = p;
> int *p3 = cast(int*)p2;
>
> int *p;
> shared int *p2 = p;
> int *p3 = p;
>
>
> I'll put some asserts in that show they accomplish the same thing:
>
> assert(p3 is p2);
> assert(p3 is p);
> assert(p2 is p);
>
> What the example demonstrates is that while you are trying to disallow
> implicit casting of a shared pointer to an unshared pointer, you have
> inadvertently allowed it by leaving behind an unshared pointer that is
> the same thing.
This doesn't make sense... you're showing a thread-local program.
The thread owning the unshared pointer is entitled to the unshared
pointer. It can make as many copies at it likes. They are all
thread-local.
There's only one owning thread, and you can't violate that without unsafe casts.
> In order for a datum to be
> safely shared, it must be accessed with synchronization or atomics by
> ALL parties.
** Absolutely **
> If you have one party that can simply change it without
> those, you will get races.
*** THIS IS NOT WHAT I'M PROPOSING ***
I've explained it a few times now, but people aren't reading what I
actually write, and just assume based on what shared already does that
they know what I'm suggesting.
You need to eject all presumptions from your mind, take the rules I
offer as verbatim, and do thought experiments from there.
> That's why shared/unshared is more akin to mutable/immutable than
> mutable/const.
Only if you misrepresent my suggestion.
> It's true that only one thread will have thread-local access. It's not
> valid any more than having one mutable alias to immutable data.
And this is why the immutable analogy is invalid. It's like const.
shared offers restricted access (like const), not a different class of
thing.
There is one thread with thread-local access, and many threads with
shared access.
If a shared (threadsafe) method can be defeated by threadlocal access,
then it's **not threadsafe**, and the program is invalid.
struct NotThreadsafe
{
int x;
void local()
{
++x; // <- invalidates the method below, you violate the other
function's `shared` promise
}
void notThreadsafe() shared
{
atomicIncrement(&x);
}
}
struct Atomic(T)
{
void opUnary(string op : "++")() shared { atomicIncrement(&val); }
private T val;
}
struct Threadsafe
{
Atomic!int x;
void local()
{
++x;
}
void threadsafe() shared
{
++x;
}
}
Naturally, local() is redundant, and it's perfectly fine for a
thread-local to call threadsafe() via implicit conversion.
Here's another one, where only a subset of the object is modeled to be
threadsafe (this is particularly interesting to me):
struct Threadsafe
{
int x;
Atomic!int y;
void notThreadsafe()
{
++x;
++y;
}
void threadsafe() shared
{
++y;
}
}
In these examples, the thread-local function *does not* undermine the
threadsafety of threadsafe(), it MUST NOT undermine the threadsafety
of threadsafe(), or else threadsafe() **IS NOT THREADSAFE**.
In the second example, you can see how it's possible and useful to do
thread-local work without invalidating the objects threadsafety
commitments.
I've said this a bunch of times, there are 2 rules:
1. shared inhibits read and write access to members
2. `shared` methods must be threadsafe
>From there, shared becomes interesting and useful.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
I've said this a bunch of times, there are 2 rules:
1. shared inhibits read and write access to members
2. `shared` methods must be threadsafe
From there, shared becomes interesting and useful.
Oh God...
void atomicInc(shared int* i) { /* ... */ }
Now what? There are no "methods" for ints, only UFCS. Those
functions can be as safe as you like, but if you allow implicit
promotion of int* to shared int*, you *allow implicit races*.
Re: shared - i need it to be useful
On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via Digitalmars-d
wrote:
>
> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
>
> > I've said this a bunch of times, there are 2 rules:
> > 1. shared inhibits read and write access to members
> > 2. `shared` methods must be threadsafe
> >
> >>From there, shared becomes interesting and useful.
>
> Oh God...
>
> void atomicInc(shared int* i) { /* ... */ }
>
> Now what? There are no "methods" for ints, only UFCS. Those
> functions can be as safe as you like, but if you allow implicit
> promotion of int* to shared int*, you *allow implicit races*.
This function is effectively an intrinsic. It's unsafe by definition.
It's a tool for implementing threadsafe machinery.
No user can just start doing atomic operations on random ints and say
"it's threadsafe", you must encapsulate the threadsafe functionality
into some sort of object that aggregates all concerns and presents an
intellectually sound api.
Let me try one:
void free(void*) { ... }
Now what? I might have dangling pointers... it's a catastrophe!
It's essentially the same argument.
This isn't a function that professes to do something that people might
misunderstand and try to use in an unsafe way, it's a low-level
implementation device, which is used to build larger *useful*
constructs.
Re: shared - i need it to be useful
On 10/17/18 2:46 PM, Manu wrote:
On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via
What the example demonstrates is that while you are trying to disallow
implicit casting of a shared pointer to an unshared pointer, you have
inadvertently allowed it by leaving behind an unshared pointer that is
the same thing.
This doesn't make sense... you're showing a thread-local program.
The thread owning the unshared pointer is entitled to the unshared
pointer. It can make as many copies at it likes. They are all
thread-local.
It's assumed that shared int pointer can be passed to another thread,
right? Do I have to write a full program to demonstrate?
There's only one owning thread, and you can't violate that without unsafe casts.
The what is the point of shared? Like why would you share data that
NOBODY CAN USE?
At SOME POINT, shared data needs to be readable and writable. Any
correct system is going to dictate how that works. It's a good start to
make shared data unusable unless you cast. But then to make it
implicitly castable from unshared defeats the whole purpose.
In order for a datum to be
safely shared, it must be accessed with synchronization or atomics by
ALL parties.
** Absolutely **
If you have one party that can simply change it without
those, you will get races.
*** THIS IS NOT WHAT I'M PROPOSING ***
I've explained it a few times now, but people aren't reading what I
actually write, and just assume based on what shared already does that
they know what I'm suggesting.
You need to eject all presumptions from your mind, take the rules I
offer as verbatim, and do thought experiments from there.
What seems to be a mystery here is how one is to actually manipulate
shared data. If it's not usable as shared data, how does one use it?
That's why shared/unshared is more akin to mutable/immutable than
mutable/const.
Only if you misrepresent my suggestion.
It's not misrepresentation, I'm trying to fill in the holes with the
only logical possibilities I can think of.
It's true that only one thread will have thread-local access. It's not
valid any more than having one mutable alias to immutable data.
And this is why the immutable analogy is invalid. It's like const.
shared offers restricted access (like const), not a different class of
thing.
No, not at all. Somehow one must manipulate shared data. If shared data
cannot be read or written, there is no reason to share it.
So LOGICALLY, we have to assume, yes there actually IS a way to
manipulate shared data through these very carefully constructed and
guarded things.
There is one thread with thread-local access, and many threads with
shared access.
If a shared (threadsafe) method can be defeated by threadlocal access,
then it's **not threadsafe**, and the program is invalid.
struct NotThreadsafe
{
int x;
void local()
{
++x; // <- invalidates the method below, you violate the other
function's `shared` promise
}
void notThreadsafe() shared
{
atomicIncrement(&x);
}
}
So the above program is invalid. Is it compilable with your added
allowance of implicit casting to shared? If it's not compilable, why
not? If it is compilable, how in the hell does your proposal help
anything? I get the exact behavior today without any changes (except
today, I need to explicitly cast, which puts the onus on me).
struct Atomic(T)
{
void opUnary(string op : "++")() shared { atomicIncrement(&val); }
private T val;
}
struct Threadsafe
{
Atomic!int x;
void local()
{
++x;
}
void threadsafe() shared
{
++x;
}
}
Naturally, local() is redundant, and it's perfectly fine for a
thread-local to call threadsafe() via implicit conversion.
In this case, yes. But that's not because of anything the compiler can
prove.
How does Atomic work? I thought shared data was not usable? I'm being
pedantic because every time I say "well at some point you must be able
to modify things", you explode.
Complete the sentence: "In order to read or write shared data, you have
to ..."
Here's another one, where only a subset of the object is modeled to be
threadsafe (this is particularly interesting to me):
struct Threadsafe
{
int x;
Atomic!int y;
void notThreadsafe()
{
++x;
++y;
}
void threadsafe() shared
{
++y;
}
}
In these examples, the thread-local function *does not* undermine the
threadsafety of threadsafe(), it MUST NOT undermine the threadsafety
of threadsafe(), or else threadsafe() **IS NOT THREADSAFE**.
In the second example, you can see how it's possible and useful to do
thread-local work without invalidating the objects threadsafety
commitments.
I've said this a bunch of times, there are 2 rules:
1. shared inhibits read and write access to members
2. `shared` methods must be threadsafe
From there, shared becomes interesting and useful.
Given rule 1, how does Atomic!int actually work, if it can't read or
writ
Re: shared - i need it to be useful
I don't see any problem with this proposal as long as these points hold: - Shared <-> Unshared is never implicit, either requiring an explicit cast (both ways) or having a language support which allows the conversion gracefully. - Shared methods are called by compiler if the type is shared or if there is no unshared equivalent. - Programmer needs to guarantee that shared -> unshared cast/conversion is thread-safe by hand; such as acquiring a lock, atomic operations... - Programmer needs to guarantee that when unshared -> shared cast/conversion happens, data is not accessed through unshared reference during the lifetime of shared reference(s). Effectively this means a data needs to be treated as shared everywhere at the same time otherwise all things fall apart.
Re: shared - i need it to be useful
On 17.10.18 20:46, Manu wrote:
struct NotThreadsafe
{
int x;
void local()
{
++x; // <- invalidates the method below, you violate the other
function's `shared` promise
}
void notThreadsafe() shared
{
atomicIncrement(&x);
}
}
In the `shared` method you'd get a nice error when attempting `++x;`,
because it's not thread-safe. With your proposal, it's just as unsafe in
`local`, but you don't get any help from the compiler in spotting it.
`local` is effectively `@trusted` without being marked as such.
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via
Digitalmars-d wrote:
On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
> I've said this a bunch of times, there are 2 rules:
> 1. shared inhibits read and write access to members
> 2. `shared` methods must be threadsafe
>
>>From there, shared becomes interesting and useful.
Oh God...
void atomicInc(shared int* i) { /* ... */ }
Now what? There are no "methods" for ints, only UFCS. Those
functions can be as safe as you like, but if you allow
implicit promotion of int* to shared int*, you *allow implicit
races*.
This function is effectively an intrinsic. It's unsafe by
definition.
Only if implicit conversion is allowed. If it isn't, that's
likely @trusted, and this:
void atomicInc(ref shared int);
can even be @safe.
It's a tool for implementing threadsafe machinery.
No user can just start doing atomic operations on random ints
and say
"it's threadsafe", you must encapsulate the threadsafe
functionality
into some sort of object that aggregates all concerns and
presents an
intellectually sound api.
Threadsafety starts and ends with the programmer. By your logic
*all* functions operating on `shared` are unsafe then. As far as
compiler is concerned, there would be no difference between these
two:
struct S {}
void atomicInc(ref shared S);
and
struct S { void atomicInc() shared { /* ... */ } }
The signatures of those two functions are exactly the same. How
is that different from a function taking a shared int pointer or
reference?
Let me try one:
void free(void*) { ... }
Now what? I might have dangling pointers... it's a catastrophe!
One could argue that it should be void free(ref void* p) { /* ...
*/ p = null; }
As a matter of fact, in my own allocators memory blocks allocated
by them are passed by value and are non-copyable, they're not
just void[] as in std.experimental.allocator. One must 'move'
them to pass ownership, and that includes deallocation. But
that's another story altogether.
It's essentially the same argument.
This isn't a function that professes to do something that
people might
misunderstand and try to use in an unsafe way, it's a low-level
implementation device, which is used to build larger *useful*
constructs.
You're missing the point, again. You have an int. You pass a
pointer to it to some API that takes an int*. You continue to use
your int as just an int. The API changes, and now the function
you called previously takes a shared int*. Implicit conversion
works, everything compiles, you have a race. Now, that's of
course an extremely stupid scenario. The point is: the caller of
some API *must* assert that they indeed pass shared data. It's
insufficient for the API alone to "promise" taking shared data.
That's the difference with promotion to `const`.
Re: shared - i need it to be useful
On Wed, Oct 17, 2018 at 12:35 PM Steven Schveighoffer via
Digitalmars-d wrote:
>
> On 10/17/18 2:46 PM, Manu wrote:
> > On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via
>
> >> What the example demonstrates is that while you are trying to disallow
> >> implicit casting of a shared pointer to an unshared pointer, you have
> >> inadvertently allowed it by leaving behind an unshared pointer that is
> >> the same thing.
> >
> > This doesn't make sense... you're showing a thread-local program.
> > The thread owning the unshared pointer is entitled to the unshared
> > pointer. It can make as many copies at it likes. They are all
> > thread-local.
>
> It's assumed that shared int pointer can be passed to another thread,
> right? Do I have to write a full program to demonstrate?
And that shared(int)* provides no access. No other thread with that
pointer can do anything with it.
> > There's only one owning thread, and you can't violate that without unsafe
> > casts.
>
> The what is the point of shared? Like why would you share data that
> NOBODY CAN USE?
You can call shared methods. They promise threadsafety.
That's a small subset of the program, but that's natural; only a very
small subset of the program is safe to be called from a shared
context.
In addition, traditional unsafe interactions which may involve
acquiring locks and doing casts remain exactly the same, and the exact
same design patterns must apply which assure that the object is
handled correctly.
I'm not suggesting any changes that affect that workflow.
> At SOME POINT, shared data needs to be readable and writable. Any
> correct system is going to dictate how that works. It's a good start to
> make shared data unusable unless you cast. But then to make it
> implicitly castable from unshared defeats the whole purpose.
No. No casting! This is antiquated workflow.. I'm not trying to take
it away from you, but it's not an interesting model for the future.
`shared` can model more than just that.
You can call threadsafe methods. Shared methods explicitly dictate how
the system works, and in a very clear and obvious/intuitive way.
The implicit cast makes using threadsafe objects more convenient when
you only have one, which is extremely common.
> >> In order for a datum to be
> >> safely shared, it must be accessed with synchronization or atomics by
> >> ALL parties.
> >
> > ** Absolutely **
> >
> >> If you have one party that can simply change it without
> >> those, you will get races.
> >
> > *** THIS IS NOT WHAT I'M PROPOSING ***
> >
> > I've explained it a few times now, but people aren't reading what I
> > actually write, and just assume based on what shared already does that
> > they know what I'm suggesting.
> > You need to eject all presumptions from your mind, take the rules I
> > offer as verbatim, and do thought experiments from there.
>
> What seems to be a mystery here is how one is to actually manipulate
> shared data. If it's not usable as shared data, how does one use it?
Call shared methods. It's not like I haven't been saying this in every
post since my OP.
The only possible thing that you can safely do with a shared object is
call a method that has been carefully designed for thread-safe
calling. Any other access is invalid under any circumstance.
> >> It's true that only one thread will have thread-local access. It's not
> >> valid any more than having one mutable alias to immutable data.
> >
> > And this is why the immutable analogy is invalid. It's like const.
> > shared offers restricted access (like const), not a different class of
> > thing.
>
> No, not at all. Somehow one must manipulate shared data. If shared data
> cannot be read or written, there is no reason to share it.
Call shared methods.
> So LOGICALLY, we have to assume, yes there actually IS a way to
> manipulate shared data through these very carefully constructed and
> guarded things.
Yes, call the methods, they were carefully constructed to be threadsafe.
Only functions that have made a promise to be threadsafe and
implemented that complexity are valid interactions.
> > There is one thread with thread-local access, and many threads with
> > shared access.
> >
> > If a shared (threadsafe) method can be defeated by threadlocal access,
> > then it's **not threadsafe**, and the program is invalid.
> >
> > struct NotThreadsafe
> > {
> >int x;
> >void local()
> >{
> > ++x; // <- invalidates the method below, you violate the other
> > function's `shared` promise
> >}
> >void notThreadsafe() shared
> >{
> > atomicIncrement(&x);
> >}
> > }
>
> So the above program is invalid. Is it compilable with your added
> allowance of implicit casting to shared? If it's not compilable, why
> not?
All my examples assume my implicit conversion rule. But regardless,
under my proposal, the above program is invalid. This violates the
threadsafe promise.
> If it is compilable, how in the hell does your proposal help
> anything? I g
Re: shared - i need it to be useful
On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via Digitalmars-d
wrote:
>
> On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
> > On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via
> > Digitalmars-d wrote:
> >>
> >> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
> >>
> >> > I've said this a bunch of times, there are 2 rules:
> >> > 1. shared inhibits read and write access to members
> >> > 2. `shared` methods must be threadsafe
> >> >
> >> >>From there, shared becomes interesting and useful.
> >>
> >> Oh God...
> >>
> >> void atomicInc(shared int* i) { /* ... */ }
> >>
> >> Now what? There are no "methods" for ints, only UFCS. Those
> >> functions can be as safe as you like, but if you allow
> >> implicit promotion of int* to shared int*, you *allow implicit
> >> races*.
> >
> > This function is effectively an intrinsic. It's unsafe by
> > definition.
>
> Only if implicit conversion is allowed. If it isn't, that's
> likely @trusted, and this:
>
> void atomicInc(ref shared int);
>
> can even be @safe.
In this case, with respect to the context (a single int) atomicInc()
is ALWAYS safe, even with implicit conversion. You can atomicInc() a
thread-local int perfectly safely.
> > It's a tool for implementing threadsafe machinery.
> > No user can just start doing atomic operations on random ints
> > and say
> > "it's threadsafe", you must encapsulate the threadsafe
> > functionality
> > into some sort of object that aggregates all concerns and
> > presents an
> > intellectually sound api.
>
> Threadsafety starts and ends with the programmer. By your logic
> *all* functions operating on `shared` are unsafe then. As far as
> compiler is concerned, there would be no difference between these
> two:
>
> struct S {}
> void atomicInc(ref shared S);
>
> and
>
> struct S { void atomicInc() shared { /* ... */ } }
>
> The signatures of those two functions are exactly the same. How
> is that different from a function taking a shared int pointer or
> reference?
It's not, atomicInc() of an int is always safe with respect to the int itself.
You can call atomicInc() on an unshared int and it's perfectly fine,
but now you need to consider context, and that's a problem for the
design of the higher-level scope.
To maintain thread-safety, the int in question must be appropriately contained.
The problem is that the same as the example I presented before, which
I'll repeat:
struct InvalidProgram
{
int x;
void fun() { ++x; }
void gun() shared { atomicInc(&x); }
}
The method gun() (and therefore the whole object) is NOT threadsafe by
my definition, because fun() violates the threadsafety of gun().
The situation applies equally here that:
int x;
atomicInc(&x);
++x; // <- by my definition, this 'API' (increment an int) violates
the threadsafety of atomicInc(), and atomicInc() is therefore not
threadsafe.
`int` doesn't present a threadsafe API, so int is by definition, NOT
threadsafe. atomicInc() should be @system, and not @trusted.
If you intend to share an int, use Atomic!int, because it has a threadsafe API.
atomicInc(shared int*) is effectively just an unsafe intrinsic, and
its only use is at ground-level implementation of threadsafe
machinery, like malloc() and free().
> > Let me try one:
> >
> > void free(void*) { ... }
> >
> > Now what? I might have dangling pointers... it's a catastrophe!
>
> One could argue that it should be void free(ref void* p) { /* ...
> */ p = null; }
void *p2 = p;
free(p);
p2.crash();
> As a matter of fact, in my own allocators memory blocks allocated
> by them are passed by value and are non-copyable, they're not
> just void[] as in std.experimental.allocator. One must 'move'
> them to pass ownership, and that includes deallocation. But
> that's another story altogether.
Right, now you're talking about move semantics to implement transfer
of ownership... you might recall I was arguing this exact case to
express transferring ownership of objects between threads earlier.
This talk of blunt casts and "making sure everything is good" is all
just great, but it doesn't mean anything interesting with respect to
`shared`. It should be interesting even without unsafe casts.
> > It's essentially the same argument.
> > This isn't a function that professes to do something that
> > people might
> > misunderstand and try to use in an unsafe way, it's a low-level
> > implementation device, which is used to build larger *useful*
> > constructs.
>
> You're missing the point, again. You have an int. You pass a
> pointer to it to some API that takes an int*. You continue to use
> your int as just an int.
You have written an invalid program. I can think of an infinite number
of ways to write an invalid program.
In this case, don't have an `int`, instead, have an Atomic!int; you
now guarantee appropriate access, problem solved!
If you do have an int, don't pass it to other threads at random when
you don't have any idea what they intend to do with it! That's basic
common sense. Yo
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via
Digitalmars-d wrote:
On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
> On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via
> Digitalmars-d wrote:
>>
>> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
>>
>> > I've said this a bunch of times, there are 2 rules:
>> > 1. shared inhibits read and write access to members
>> > 2. `shared` methods must be threadsafe
>> >
>> >>From there, shared becomes interesting and useful.
>>
>> Oh God...
>>
>> void atomicInc(shared int* i) { /* ... */ }
>>
>> Now what? There are no "methods" for ints, only UFCS. Those
>> functions can be as safe as you like, but if you allow
>> implicit promotion of int* to shared int*, you *allow
>> implicit races*.
>
> This function is effectively an intrinsic. It's unsafe by
> definition.
Only if implicit conversion is allowed. If it isn't, that's
likely @trusted, and this:
void atomicInc(ref shared int);
can even be @safe.
In this case, with respect to the context (a single int)
atomicInc()
is ALWAYS safe, even with implicit conversion. You can
atomicInc() a
thread-local int perfectly safely.
Yes, *you* can. *Another* function can't unless *you* allow for
it to be safe. You can't do that if that function silently
assumes you gave it shared data, when in fact you did not.
The signatures of those two functions are exactly the same.
How is that different from a function taking a shared int
pointer or reference?
It's not, atomicInc() of an int is always safe with respect to
the int itself.
You can call atomicInc() on an unshared int and it's perfectly
fine,
but now you need to consider context, and that's a problem for
the
design of the higher-level scope.
To maintain thread-safety, the int in question must be
appropriately contained.
Exactly. And that means it can't convert to shared without my say
so :)
The problem is that the same as the example I presented before,
which I'll repeat:
struct InvalidProgram
{
int x;
void fun() { ++x; }
void gun() shared { atomicInc(&x); }
}
The method gun() (and therefore the whole object) is NOT
threadsafe by
my definition, because fun() violates the threadsafety of gun().
The situation applies equally here that:
int x;
atomicInc(&x);
++x; // <- by my definition, this 'API' (increment an int)
violates the threadsafety of atomicInc(), and atomicInc() is
therefore not threadsafe.
No. The 'API' is just the atomicInc function. You, the user of
that API, own the int. If the API wants a shared int from you,
you have to be in agreement. You can't have any agreement if the
API is only making promises and assumptions.
`int` doesn't present a threadsafe API, so int is by
definition, NOT threadsafe. atomicInc() should be @system, and
not @trusted.
Exactly. `int` isn't threadsafe and therefore cannot
automatically convert to `shared int`.
If you intend to share an int, use Atomic!int, because it has a
threadsafe API.
No. With current implementation of `shared`, which disallows your
automatic promotion,
your intent is enforced. You cannot share a local `int` unless
*you know* it's safe to do so and therefore can cast that int to
shared.
atomicInc(shared int*) is effectively just an unsafe intrinsic,
and
It is only unsafe if you allow int* to silently convert to shared
int*. If you can't do that, you can't call `atomicInc` on an int*.
One could argue that it should be void free(ref void* p) { /*
... */ p = null; }
void *p2 = p;
free(p);
p2.crash();
That's exactly analogous to what you're proposing: leaking
`shared` references while keeping unshared data.
As a matter of fact, in my own allocators memory blocks
allocated by them are passed by value and are non-copyable,
they're not just void[] as in std.experimental.allocator. One
must 'move' them to pass ownership, and that includes
deallocation. But that's another story altogether.
Right, now you're talking about move semantics to implement
transfer of ownership... you might recall I was arguing this
exact case to express transferring ownership of objects between
threads earlier.
You went off on an irrelevant tangent there, and I feel like you
didn't even see my reply. You don't pass any ownership when you
share. You just share. As an owner, you get to access the
un-`shared` interface freely. Others do not.
This talk of blunt casts and "making sure everything is good"
is all just great, but it doesn't mean anything interesting
with respect to `shared`. It should be interesting even without
unsafe casts.
Again, I feel like you didn't see my reply. It's not talk about
just blunt casts to make sure everything is good. You either have
shared data to begin with, and so can share it freely, or you
*know* that you can share this particular piece of data, even if
it itself isn't marked `shared`, and you *assert* that by
deli
Re: shared - i need it to be useful
I don't have anything to add that hasn't been said yet but it's good to see some thinking on this subject. It feels like progress.
Re: shared - i need it to be useful
On 10/17/18 6:37 PM, Manu wrote: On Wed, Oct 17, 2018 at 12:35 PM Steven Schveighoffer via Digitalmars-d wrote: On 10/17/18 2:46 PM, Manu wrote: On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via What the example demonstrates is that while you are trying to disallow implicit casting of a shared pointer to an unshared pointer, you have inadvertently allowed it by leaving behind an unshared pointer that is the same thing. This doesn't make sense... you're showing a thread-local program. The thread owning the unshared pointer is entitled to the unshared pointer. It can make as many copies at it likes. They are all thread-local. It's assumed that shared int pointer can be passed to another thread, right? Do I have to write a full program to demonstrate? And that shared(int)* provides no access. No other thread with that pointer can do anything with it. So then it's a misnomer -- it's not really shared, because I can't do anything with it. There's only one owning thread, and you can't violate that without unsafe casts. The what is the point of shared? Like why would you share data that NOBODY CAN USE? You can call shared methods. They promise threadsafety. That's a small subset of the program, but that's natural; only a very small subset of the program is safe to be called from a shared context. All I can see is that a shared method promises to be callable on shared or unshared data. In essence, it promises nothing. It's the programmer who must implement the thread safety, and there really is no help at all from the compiler for this. At some level, there will be either casts, or intrinsics, both of which are unsafe without knowing all the context of the object. In any case, it's simply a false guarantee of thread safety, which might as well be a convention of "any function which starts with TS_ is supposed to be thread safe". shared in the current form promises one thing and one thing only -- data marked as shared is actually sharable between threads, and data not marked as shared is actually not shared between threads. This new regime you are proposing does nothing extra or new, except break that guarantee. At SOME POINT, shared data needs to be readable and writable. Any correct system is going to dictate how that works. It's a good start to make shared data unusable unless you cast. But then to make it implicitly castable from unshared defeats the whole purpose. No. No casting! This is antiquated workflow.. I'm not trying to take it away from you, but it's not an interesting model for the future. `shared` can model more than just that. You can call threadsafe methods. Shared methods explicitly dictate how the system works, and in a very clear and obvious/intuitive way. The implicit cast makes using threadsafe objects more convenient when you only have one, which is extremely common. The implicit cast means that you have to look at more than just your method. You have to look at the entire module, and figure out all the interactions, to see if the thread safe method actually is thread safe. That's programming by convention, and fully trusting the programmer. I don't think this thread is going anywhere, so I'll just have to wait and see if someone else can explain it better. I'm a firm no on implicit casting from mutable to shared. -Steve
Re: shared - i need it to be useful
On Wed, Oct 17, 2018 at 5:35 PM Stanislav Blinov via Digitalmars-d
wrote:
>
> On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
> > On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via
> > Digitalmars-d wrote:
> >>
> >> On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
> >> > On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via
> >> > Digitalmars-d wrote:
> >> >>
> >> >> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
> >> >>
> >> >> > I've said this a bunch of times, there are 2 rules:
> >> >> > 1. shared inhibits read and write access to members
> >> >> > 2. `shared` methods must be threadsafe
> >> >> >
> >> >> >>From there, shared becomes interesting and useful.
> >> >>
> >> >> Oh God...
> >> >>
> >> >> void atomicInc(shared int* i) { /* ... */ }
> >> >>
> >> >> Now what? There are no "methods" for ints, only UFCS. Those
> >> >> functions can be as safe as you like, but if you allow
> >> >> implicit promotion of int* to shared int*, you *allow
> >> >> implicit races*.
> >> >
> >> > This function is effectively an intrinsic. It's unsafe by
> >> > definition.
> >>
> >> Only if implicit conversion is allowed. If it isn't, that's
> >> likely @trusted, and this:
> >>
> >> void atomicInc(ref shared int);
> >>
> >> can even be @safe.
> >
> > In this case, with respect to the context (a single int)
> > atomicInc()
> > is ALWAYS safe, even with implicit conversion. You can
> > atomicInc() a
> > thread-local int perfectly safely.
>
> Yes, *you* can. *Another* function can't unless *you* allow for
> it to be safe. You can't do that if that function silently
> assumes you gave it shared data, when in fact you did not.
>
> >> The signatures of those two functions are exactly the same.
> >> How is that different from a function taking a shared int
> >> pointer or reference?
> >
> > It's not, atomicInc() of an int is always safe with respect to
> > the int itself.
> > You can call atomicInc() on an unshared int and it's perfectly
> > fine,
> > but now you need to consider context, and that's a problem for
> > the
> > design of the higher-level scope.
> >
> > To maintain thread-safety, the int in question must be
> > appropriately contained.
>
> Exactly. And that means it can't convert to shared without my say
> so :)
It can though, because `int` doesn't have any shared methods; it's
inaccessible. You can't do anything with a shared int, so it's safe to
distribute regardless.
> > The problem is that the same as the example I presented before,
> > which I'll repeat:
> >
> > struct InvalidProgram
> > {
> > int x;
> > void fun() { ++x; }
> > void gun() shared { atomicInc(&x); }
> > }
> >
> > The method gun() (and therefore the whole object) is NOT
> > threadsafe by
> > my definition, because fun() violates the threadsafety of gun().
> > The situation applies equally here that:
> > int x;
> > atomicInc(&x);
> > ++x; // <- by my definition, this 'API' (increment an int)
> > violates the threadsafety of atomicInc(), and atomicInc() is
> > therefore not threadsafe.
>
> No. The 'API' is just the atomicInc function. You, the user of
> that API, own the int. If the API wants a shared int from you,
> you have to be in agreement. You can't have any agreement if the
> API is only making promises and assumptions.
Okay, here's an idea... atomicInc() should NOT take `shared int*`,
change atomicInt() to receive int*.
We agree that `int` is not a threadsafe type, atomicInc() is not a
threadsafe method, and therefore shouldn't be shared.
If you have a shared(int)*, and you want to do un-threadsafe
atomicInc(), you need to cast to int*.
I think this is actually correct by my definition, because we
recognise int is not a threadsafe type, and atomicInc() is not a
threadsafe method, so hard-cast is required, and programmer must
appropriately validate the context when doing an unsafe operation.
> > `int` doesn't present a threadsafe API, so int is by
> > definition, NOT threadsafe. atomicInc() should be @system, and
> > not @trusted.
>
> Exactly. `int` isn't threadsafe and therefore cannot
> automatically convert to `shared int`.
It can convert to shared int, because it's inaccessible... you can't
do anything with a `shared int` except unsafe cast shared away. Once
you're in unsafe land, you are responsible for assuring correct
conditions.
> > If you intend to share an int, use Atomic!int, because it has a
> > threadsafe API.
>
> No. With current implementation of `shared`, which disallows your
> automatic promotion,
> your intent is enforced. You cannot share a local `int` unless
> *you know* it's safe to do so and therefore can cast that int to
> shared.
Yes, but we're not talking about current definition of shared. Stop
confusing the conversation.
This thread is about the rules I define.
> > atomicInc(shared int*) is effectively just an unsafe intrinsic,
> > and
>
> It is only unsafe if you allow int* to silently convert to shared
> int*. If you can't do that, you can't call `atomicInc` on an
Re: shared - i need it to be useful
On Wednesday, 17 October 2018 at 23:12:48 UTC, Manu wrote:
On Wed, Oct 17, 2018 at 2:15 PM Stanislav Blinov via
Digitalmars-d wrote:
On Wednesday, 17 October 2018 at 19:25:33 UTC, Manu wrote:
> On Wed, Oct 17, 2018 at 12:05 PM Stanislav Blinov via
> Digitalmars-d wrote:
>>
>> On Wednesday, 17 October 2018 at 18:46:18 UTC, Manu wrote:
>>
>> > I've said this a bunch of times, there are 2 rules:
>> > 1. shared inhibits read and write access to members
>> > 2. `shared` methods must be threadsafe
>> >
>> >>From there, shared becomes interesting and useful.
>>
>> Oh God...
>>
>> void atomicInc(shared int* i) { /* ... */ }
>>
>> Now what? There are no "methods" for ints, only UFCS. Those
>> functions can be as safe as you like, but if you allow
>> implicit promotion of int* to shared int*, you *allow
>> implicit races*.
>
> This function is effectively an intrinsic. It's unsafe by
> definition.
Only if implicit conversion is allowed. If it isn't, that's
likely @trusted, and this:
void atomicInc(ref shared int);
can even be @safe.
In this case, with respect to the context (a single int)
atomicInc()
is ALWAYS safe, even with implicit conversion. You can
atomicInc() a
thread-local int perfectly safely.
> It's a tool for implementing threadsafe machinery.
> No user can just start doing atomic operations on random ints
> and say
> "it's threadsafe", you must encapsulate the threadsafe
> functionality
> into some sort of object that aggregates all concerns and
> presents an
> intellectually sound api.
Threadsafety starts and ends with the programmer. By your
logic *all* functions operating on `shared` are unsafe then.
As far as compiler is concerned, there would be no difference
between these two:
struct S {}
void atomicInc(ref shared S);
and
struct S { void atomicInc() shared { /* ... */ } }
The signatures of those two functions are exactly the same.
How is that different from a function taking a shared int
pointer or reference?
It's not, atomicInc() of an int is always safe with respect to
the int itself.
You can call atomicInc() on an unshared int and it's perfectly
fine,
but now you need to consider context, and that's a problem for
the
design of the higher-level scope.
To maintain thread-safety, the int in question must be
appropriately contained.
The problem is that the same as the example I presented before,
which I'll repeat:
struct InvalidProgram
{
int x;
void fun() { ++x; }
void gun() shared { atomicInc(&x); }
}
The method gun() (and therefore the whole object) is NOT
threadsafe by
my definition, because fun() violates the threadsafety of gun().
The situation applies equally here that:
int x;
atomicInc(&x);
++x; // <- by my definition, this 'API' (increment an int)
violates
the threadsafety of atomicInc(), and atomicInc() is therefore
not
threadsafe.
`int` doesn't present a threadsafe API, so int is by
definition, NOT threadsafe. atomicInc() should be @system, and
not @trusted.
If you intend to share an int, use Atomic!int, because it has a
threadsafe API.
atomicInc(shared int*) is effectively just an unsafe intrinsic,
and
its only use is at ground-level implementation of threadsafe
machinery, like malloc() and free().
> Let me try one:
>
> void free(void*) { ... }
>
> Now what? I might have dangling pointers... it's a
> catastrophe!
One could argue that it should be void free(ref void* p) { /*
...
*/ p = null; }
void *p2 = p;
free(p);
p2.crash();
As a matter of fact, in my own allocators memory blocks
allocated by them are passed by value and are non-copyable,
they're not just void[] as in std.experimental.allocator. One
must 'move' them to pass ownership, and that includes
deallocation. But that's another story altogether.
Right, now you're talking about move semantics to implement
transfer of ownership... you might recall I was arguing this
exact case to express transferring ownership of objects between
threads earlier. This talk of blunt casts and "making sure
everything is good" is all just great, but it doesn't mean
anything interesting with respect to `shared`. It should be
interesting even without unsafe casts.
> It's essentially the same argument.
> This isn't a function that professes to do something that
> people might
> misunderstand and try to use in an unsafe way, it's a
> low-level
> implementation device, which is used to build larger *useful*
> constructs.
You're missing the point, again. You have an int. You pass a
pointer to it to some API that takes an int*. You continue to
use your int as just an int.
You have written an invalid program. I can think of an infinite
number
of ways to write an invalid program.
In this case, don't have an `int`, instead, have an Atomic!int;
you
now guarantee appropriate access, problem solved!
If you do have an int, don't pass it to other threads at random
when
you don't have any idea what they intend to do with it! That's
basic
common sense. You don't
Re: shared - i need it to be useful
On Wed, Oct 17, 2018 at 6:50 PM Steven Schveighoffer via Digitalmars-d wrote: > > On 10/17/18 6:37 PM, Manu wrote: > > On Wed, Oct 17, 2018 at 12:35 PM Steven Schveighoffer via > > Digitalmars-d wrote: > >> > >> On 10/17/18 2:46 PM, Manu wrote: > >>> On Wed, Oct 17, 2018 at 10:30 AM Steven Schveighoffer via > >> > What the example demonstrates is that while you are trying to disallow > implicit casting of a shared pointer to an unshared pointer, you have > inadvertently allowed it by leaving behind an unshared pointer that is > the same thing. > >>> > >>> This doesn't make sense... you're showing a thread-local program. > >>> The thread owning the unshared pointer is entitled to the unshared > >>> pointer. It can make as many copies at it likes. They are all > >>> thread-local. > >> > >> It's assumed that shared int pointer can be passed to another thread, > >> right? Do I have to write a full program to demonstrate? > > > > And that shared(int)* provides no access. No other thread with that > > pointer can do anything with it. > > So then it's a misnomer -- it's not really shared, because I can't do > anything with it. **EXACTLY**.. this is the key to the entire design that allows for the implicit promotion. > >>> There's only one owning thread, and you can't violate that without unsafe > >>> casts. > >> > >> The what is the point of shared? Like why would you share data that > >> NOBODY CAN USE? > > > > You can call shared methods. They promise threadsafety. > > That's a small subset of the program, but that's natural; only a very > > small subset of the program is safe to be called from a shared > > context. > > All I can see is that a shared method promises to be callable on shared > or unshared data. In essence, it promises nothing. How is that nothing? It promises threadsafety, such that it can be called on shared data. Normally, you can do NOTHING with shared data short of unsafe operations. It is always safe to call a threadsafe function on an unshared thing, so it should be possible. `shared` opts into a strong commitment to threadsafety that allows for a shared thing to interact with it, otherwise, no interaction is possible. I don't understand how you can claim it promises nothing; this is an extremely strong promise! > It's the programmer who must implement the thread safety, and there > really is no help at all from the compiler for this. I hope we can work on this moving forwards. There are probably some things we can do... but even without compiler support, in reality, there are probably only a small number of core pieces of tooling, but most authors of a shared thing will just aggregate core tools and call through to their API's. Most shared code will use tools written by experts, and implement nothing fancy or magical themselves. > At some level, > there will be either casts, or intrinsics, both of which are unsafe > without knowing all the context of the object. This will tend to be in the core tooling, written by an expert. The whole point of my design is to remove friction from the USERS of those tools, such that they can create aggregate functionality safely without doing anything unsafe. > In any case, it's simply > a false guarantee of thread safety, which might as well be a convention > of "any function which starts with TS_ is supposed to be thread safe". What makes it a false guarantee? It allows you to have confidence in the stack built on top of some core tools. The whole point here is to minimise the number of people writing code like that. I'm trying to make shared generally useful so that people don't have to engage with it at a low-level. If you see an unsafe case in a shared function (beyond the core tooling), then you should immediately be suspicious, and this is a deliberate part of my goal here. The point here is to produce safer and more reliable code by defining access rules that lead to confident threadsafety. What you describe is what we have now. > shared in the current form promises one thing and one thing only -- data > marked as shared is actually sharable between threads, and data not > marked as shared is actually not shared between threads. Ummm. The current form might *say* that, but you still have full unregulated access to all members, and it's completely unsafe, with no attempt to mitigate that. This is a new definition. Take it for what it's worth. The old definition is mostly worthless. I'm trying to make something that's interesting and useful. > This new regime you are proposing does nothing extra or new, except break > that guarantee. There are heaps of advantages: 1. If I have a shared thing, I know I can't ruin its state by manipulating it arbitrarily like I can now 2. I am now able to describe threadsafe objects 3. I relieve users from mental strain of trying to understand how to correctly and safely interact with shared API's, because they can have confidence that a shared thing can only perform threadsafe a
