Re: Thread-safe attribution
struct Bob { threadsafe Atomic!(string[string]) y; } void f(ref threadsafe Bob b) { string[string] aa=b.y; aa["b"]="c"; } Like this?
Re: Thread-safe attribution
On Sunday, 7 October 2018 at 02:59:12 UTC, Manu wrote: On Sat, Oct 6, 2018 at 7:40 PM Nicholas Wilson via Digitalmars-d wrote: [...] One thing that occurred to me is that _objects_ are shared, whereas _functions/methods_ (and their parameters) are thread safe . Theadsafe is kind of like a const (as to mutable/immutable) to threading, a promise to behave correctly in the presence of threading. thread safe references therefore must not escape. Right, that's kinda what I want to model... but the more I think of it, the more I think that experience can fit into `shared`, because it's almost there, and the current incarnation of shared is objectively useless. Consider shared as is today; struct Bob { int x; void f() shared { x = 10; // <- this compiles... WAT?! } } Right now, if you have a shared instance, you can read/write to the members... and that makes *absolutely no sense* no matter how you look at it. There is no reality where you have a shared thing, and accessing members un-controlled can be safe. Conventional wisdom is that when you have a shared thing, and you want to do stuff with it, you must acquire locks (or whatever) and case shared away. That should apply to f() above. struct Bob { int x; void f() shared { auto lock = getLock(); auto unshared = shared_cast(&this); unshared.x = 10; // <- this is now okay. } } If we made a change were `shared` lost the ability to access non-`shared` members, I don't think that would interfere with current or proposed uses of shared in any way whatsoever... and we would make shared useful in the process. I think it should be more like this shared struct Bob { shared int x; void f() shared { Laquire: auto owned_bob = try_aquire(&this, pthread_self()); // type will be the same as Bob* but without shared stripped from variables // and without any functions if (owned_bob is null) { __mmPause(); goto Laquire; } *owned_bob.x = 10; } }
Re: Thread-safe attribution
On Sun, Oct 7, 2018 at 10:00 AM Boris-Barboris via Digitalmars-d wrote: > > On Sunday, 7 October 2018 at 02:01:17 UTC, Manu wrote: > > ... but I'm really struggling > > to express it in terms of the type system... > > I'm pretty sure no simple attribute system is any more useful > than current const\shared idiom. I am yet to see a language with > semantics that actually help with concurrency issues > (serializability, lock domains, lock ordering, deadlock > prevention\detection, consistency, write scew and hundreds of > other problems already known in the domain) on mutable state. SQL > didn't solve it, I still always have to grab pen and paper and > brute-force simulate concurrent access to my data in order to > have any degree of prior knowledge about safety, and > unfortunately this problem does not look like an easy one to > crack for language designers. > > This is why I mainly ignore shared as a feature completely unless > I'm forced to. A simple indication (type system attribute) of the > fact that data is shared is useless for my recent projects just > adds unneeded casts and lines of code without solving any > problems. In our ecosystem within the structure of our scheduling, we would absolutely get loads of use out of shared if it was unable to access non-shared members. That's a major step forward in usefulness compared to now, where you can just read/write to members arbitrarily, which is just plain wrong, and not useful in any way. If shared inhibited access to non-shared members, thereby requiring you interface the instance only via shared attributed methods (or the traditional acquire-lock-and-cast-shared-away method), then it would be infinitely more useful than it is now. If anyone wanted to make an experimental patch with that rule, I could test-drive and see how it goes. Point is, shared is totally useless. Making it useful, even if it's not 100% solving threading issues, is a better place to be, and opens up doors for some new designs, and get more experience.
Re: Thread-safe attribution
On Sunday, 7 October 2018 at 02:01:17 UTC, Manu wrote: ... but I'm really struggling to express it in terms of the type system... I'm pretty sure no simple attribute system is any more useful than current const\shared idiom. I am yet to see a language with semantics that actually help with concurrency issues (serializability, lock domains, lock ordering, deadlock prevention\detection, consistency, write scew and hundreds of other problems already known in the domain) on mutable state. SQL didn't solve it, I still always have to grab pen and paper and brute-force simulate concurrent access to my data in order to have any degree of prior knowledge about safety, and unfortunately this problem does not look like an easy one to crack for language designers. This is why I mainly ignore shared as a feature completely unless I'm forced to. A simple indication (type system attribute) of the fact that data is shared is useless for my recent projects just adds unneeded casts and lines of code without solving any problems.
Re: Thread-safe attribution
On Sunday, 7 October 2018 at 04:16:43 UTC, Manu wrote: We're not trying to 'stuff the nuances' into a keyword... what I'm trying to achieve is a mechanism for attributing that a function has implemented thread-safety *in some way*, and how that works is a detail for the function. What the attribute needs to do, is control the access rights to the object appropriately, so that if you're in custody of a shared object, you should exclusively be limited to performing guaranteed thread-safe operations on the object, OR, you must perform synchronisation and cast shared away (as is current use of shared). Then I maintain that `T*` should *not* implicitly cast to `threadsafe T*`. It should at least be an explicit cast that you could grep for. Consider: struct Bob { int x; int y; void readAndMutate() /* thread-local */ { if (x) y += 1; } void readAndMutate() threadsafe { auto lock = getSomeLockThatPresumablyLocksThisInstance(); auto unshared = cast(Bob*) &this; unshared.readAndMutate(); } } void sendToAnotherThread(threadsafe Bob* bob) { // pass the pointer to some thread... } Bob bob; // not marked `threadsafe` void main() { bob.x = 1; sendToAnotherThread(&bob); bob.x = 0; // <-- that's a bug auto copyOfBob = bob; // <-- that's another bug // do the rest of main... } Basically, *any* non-`threadsafe` access to `bob` should ideally be a compiler error after the cast, but I don't think the language could enforce that. These contrived examples aren't that convincing, I'm sure, but imagine this implicit cast hiding somewhere in a 10Kloc module. In your own words: "I don't think that you should...": then we would need at least some way to succinctly find such problems. There's actually an even more subtle bug in `main` above. Since the `bob` instance is not marked in any way, the compiler (optimizer) would have no idea that e.g. moving reads and writes to bob's fields across that implicit cast must be illegal. I guess a cast should then act as a compiler barrier. You need to read the rest of the thread... I've moved on from this initial post. Yeah, sorry about that. The thread also got split :\
Re: Thread-safe attribution
On Sat, Oct 6, 2018 at 8:10 PM Stanislav Blinov via Digitalmars-d wrote: > > On Sunday, 7 October 2018 at 02:01:17 UTC, Manu wrote: > > >> The thing I'm trying to model is an attribute along the lines > >> of > >> `shared`, but actually useful ;) > >> I'll use the attribute `threadsafe` in place of `shared`, and > >> see > >> where that goes. > >> > >> Consider: > >> struct Bob > >> { > >> int x; > >> threadsafe Atomic!int y; > > Storing shared and local data together? Ew, cache poison. Be that > as it may, given that definition: I don't think mixing would happen in practise, but the rules need to be defined. > 1. Can you copy Bob or assign to it? If so, how? > 2. Can Bob have a destructor? Who calls it if it can? If they're mutable, sure. If Bob was threadsafe, then it would have to have a matching attributed constructor.destructor/copy-ctor, etc. > >> x.m1(); // ERROR, method not threadsafe > >> x.m2(); // fine > >> x.overloaded(); // fine, use the threadsafe overload > > I guess these three should be y., not x.? Yes, sorry. > >> threadsafe Bob* p = &x; // can take threadsafe reference to > >> thread-local object > > I'm not sure what's that supposed to accomplish. It looks like a > silent cast, which is already a red flag. What is the purpose of > this? This is the primitive that allows calling a threadsafe method. You can call a const method because of this same mechanic. > Give that pointer to another thread while the original > continues to treat 'x' as thread-local? I.e. if the arguments > were indeed `ref`, the caller would be blissfully unaware of such > a "transaction" taking place. I'm going to move to the position from the most recent post in the thread (you are responding to the first post). If shared instances were unable to access their members, then it's possible that you *could* pass this to another thread (I don't think you should, but I don't think it's dangerous anymore). Any shared instance to the object is not able to mutate the object... they would need to case shared away, which is unsafe... they would need to do that deliberately by acquiring a lock, or by whatever other mechanism gives them temporary thread-local access to the object. > >> This is loosely what `shared` models, but there's a few > >> differences: > >> 1. thread-local can NOT promote to shared > >> 2. shared `this` applies to members > >> > >> For `shared` to be useful, it should be that a shared > >> reference to something inhibits access to it's thread-local > >> stuff. And in that world, then I believe that thread-local > >> promotion to shared would work like const does. > >> > >> I guess I'm wondering; should `shared` be transitive? Perhaps > >> that's what's wrong with it...? > > IHMO, `shared` should be transitive, but... most uses of `shared` > should just boil down to primitive types (i.e. atomics) and > pointers to shared primitives and structs. With dereferencing > latter requiring to be synchronized, casting away `shared` in the > process, and I don't see how the language as it is can help with > that, as accessing different kinds of data requires different > code. We can't just stuff all the nuances of thread-safety and > multi-core communication into one keyword, whatever that keyword > may be. > What could be the static (compile-time) guarantees of > `threadsafe` methods? We're not trying to 'stuff the nuances' into a keyword... what I'm trying to achieve is a mechanism for attributing that a function has implemented thread-safety *in some way*, and how that works is a detail for the function. What the attribute needs to do, is control the access rights to the object appropriately, so that if you're in custody of a shared object, you should exclusively be limited to performing guaranteed thread-safe operations on the object, OR, you must perform synchronisation and cast shared away (as is current use of shared). You need to read the rest of the thread... I've moved on from this initial post. I agree, shared must be transitive, but I think the effect that it can't access un-shared members may be the innovation we're looking for which will make shared useful.
Re: Thread-safe attribution
On Sunday, 7 October 2018 at 02:01:17 UTC, Manu wrote: The thing I'm trying to model is an attribute along the lines of `shared`, but actually useful ;) I'll use the attribute `threadsafe` in place of `shared`, and see where that goes. Consider: struct Bob { int x; threadsafe Atomic!int y; Storing shared and local data together? Ew, cache poison. Be that as it may, given that definition: 1. Can you copy Bob or assign to it? If so, how? 2. Can Bob have a destructor? Who calls it if it can? [snip] x.m1(); // ERROR, method not threadsafe x.m2(); // fine x.overloaded(); // fine, use the threadsafe overload I guess these three should be y., not x.? threadsafe Bob* p = &x; // can take threadsafe reference to thread-local object I'm not sure what's that supposed to accomplish. It looks like a silent cast, which is already a red flag. What is the purpose of this? Give that pointer to another thread while the original continues to treat 'x' as thread-local? I.e. if the arguments were indeed `ref`, the caller would be blissfully unaware of such a "transaction" taking place. This is loosely what `shared` models, but there's a few differences: 1. thread-local can NOT promote to shared 2. shared `this` applies to members For `shared` to be useful, it should be that a shared reference to something inhibits access to it's thread-local stuff. And in that world, then I believe that thread-local promotion to shared would work like const does. I guess I'm wondering; should `shared` be transitive? Perhaps that's what's wrong with it...? IHMO, `shared` should be transitive, but... most uses of `shared` should just boil down to primitive types (i.e. atomics) and pointers to shared primitives and structs. With dereferencing latter requiring to be synchronized, casting away `shared` in the process, and I don't see how the language as it is can help with that, as accessing different kinds of data requires different code. We can't just stuff all the nuances of thread-safety and multi-core communication into one keyword, whatever that keyword may be. What could be the static (compile-time) guarantees of `threadsafe` methods?
Re: Thread-safe attribution
On Sat, Oct 6, 2018 at 7:01 PM Manu wrote: > > On Sat, Oct 6, 2018 at 6:59 PM Manu wrote: > > > > So I'm working on a SMT infrastructure, and expression of > > thread-safety is a core design mechanic... but I'm really struggling > > to express it in terms of the type system. > > I figure I'll throw some design challenges out here and see if anyone > > can offer some good ideas. > > > > The thing I'm trying to model is an attribute along the lines of > > `shared`, but actually useful ;) > > I'll use the attribute `threadsafe` in place of `shared`, and see > > where that goes. > > > > Consider: > > struct Bob > > { > > int x; > > threadsafe Atomic!int y; > > > > void m1(); > > void m2() threadsafe;; > > > > void overloaded(); > > void overloaded() threadsafe; > > } > > > > void func(Bob x, threadsafe Bob y) > > { > > x.x = 10; // fine > > x.y = 10; // fine > > x.m1(); // fine > > x.m2(); // fine > > x.overloaded(); // fine, use the un-threadsafe overload > > > > y.x = 10; // ERROR, a threadsafe reference can NOT modify an > > un-threadsafe member > > y.y = 10; // fine > > x.m1(); // ERROR, method not threadsafe > > x.m2(); // fine > > x.overloaded(); // fine, use the threadsafe overload > > > > threadsafe Bob* p = &x; // can take threadsafe reference to > > thread-local object > > } > > > > This is loosely what `shared` models, but there's a few differences: > > 1. thread-local can NOT promote to shared > > 2. shared `this` applies to members > > > > For `shared` to be useful, it should be that a shared reference to > > something inhibits access to it's thread-local stuff. And in that > > world, then I believe that thread-local promotion to shared would work > > like const does. > > > > I guess I'm wondering; should `shared` be transitive? Perhaps that's > > what's wrong with it...? > > *** the function arguments should be `ref`! Thinking on this more... perhaps it's the case that shared is transitive, but the effect shared has, is to inhibit read/write to data members. In my example above, there is an atomic data member: struct Bob { int x; Atomic!int y; // not marked `shared` in this example } void fun(ref Bob a, ref shared Bob b) { a.x = 10; // fine a.y = 10; // fine b.x = 10; // error! b is shared, and member 'x' is NOT shared, no access! b.y = 10; // this gets interesting... } So, `b` can't access `y` because it's not shared... but consider this possibility: struct Atomic(T) { T data; void opAssign(T val) shared { // implement assignment using atomic operations } } `b.y.data` is not accessible, but the assignment operator has been attributed shared, which means `b.y = 10` becomes a legal function call. At this point, shared is now useful. So, continue the thought experiment that goes: 1. `shared` instances can NOT access non-shared members (opposite of current behaviour) 2. non-shared can promote to `shared` (like const) I think this restriction in access corrects the issues associated with allowing non-shared -> shared promotion.
Re: Thread-safe attribution
On Sat, Oct 6, 2018 at 7:40 PM Nicholas Wilson via Digitalmars-d wrote: > > [...] > > One thing that occurred to me is that _objects_ are shared, > whereas _functions/methods_ (and their parameters) are thread > safe . > > Theadsafe is kind of like a const (as to mutable/immutable) to > threading, a promise to behave correctly in the presence of > threading. thread safe references therefore must not escape. Right, that's kinda what I want to model... but the more I think of it, the more I think that experience can fit into `shared`, because it's almost there, and the current incarnation of shared is objectively useless. Consider shared as is today; struct Bob { int x; void f() shared { x = 10; // <- this compiles... WAT?! } } Right now, if you have a shared instance, you can read/write to the members... and that makes *absolutely no sense* no matter how you look at it. There is no reality where you have a shared thing, and accessing members un-controlled can be safe. Conventional wisdom is that when you have a shared thing, and you want to do stuff with it, you must acquire locks (or whatever) and case shared away. That should apply to f() above. struct Bob { int x; void f() shared { auto lock = getLock(); auto unshared = shared_cast(&this); unshared.x = 10; // <- this is now okay. } } If we made a change were `shared` lost the ability to access non-`shared` members, I don't think that would interfere with current or proposed uses of shared in any way whatsoever... and we would make shared useful in the process.
Re: Thread-safe attribution
On Sunday, 7 October 2018 at 01:59:21 UTC, Manu wrote: So I'm working on a SMT infrastructure, and expression of thread-safety is a core design mechanic... but I'm really struggling to express it in terms of the type system. I figure I'll throw some design challenges out here and see if anyone can offer some good ideas. The thing I'm trying to model is an attribute along the lines of `shared`, but actually useful ;) I'll use the attribute `threadsafe` in place of `shared`, and see where that goes. Consider: struct Bob { int x; threadsafe Atomic!int y; void m1(); void m2() threadsafe;; void overloaded(); void overloaded() threadsafe; } void func( ref Bob x, ref threadsafe Bob y) { x.x = 10; // fine x.y = 10; // fine x.m1(); // fine x.m2(); // fine x.overloaded(); // fine, use the un-threadsafe overload y.x = 10; // ERROR, a threadsafe reference can NOT modify an un-threadsafe member y.y = 10; // fine x.m1(); // ERROR, method not threadsafe x.m2(); // fine x.overloaded(); // fine, use the threadsafe overload threadsafe Bob* p = &x; // can take threadsafe reference to thread-local object } This is loosely what `shared` models, but there's a few differences: 1. thread-local can NOT promote to shared 2. shared `this` applies to members For `shared` to be useful, it should be that a shared reference to something inhibits access to it's thread-local stuff. And in that world, then I believe that thread-local promotion to shared would work like const does. I guess I'm wondering; should `shared` be transitive? Perhaps that's what's wrong with it...? A delta comparison with shared void func( ref Bob x, ref threadshared /* either shared or threadsafe*/ Bob y) { // threadsafe / shared x.x = 10; // fine / fine x.y = 10; // fine / fine uses atomics x.m1(); // fine / fine x.m2(); // fine / error cannot call shared method on unshared object x.overloaded(); // fine, use the un-threadsafe overload / fine y.x = 10; // ERROR, a threadsafe reference can NOT modify an un-threadsafe member / error y.y = 10; // fine / fine (using atomics) // Assuming these are supposed to be y not x y.m1(); // ERROR, method not threadsafe / error y.m2(); // fine / fine y.overloaded(); // fine, use the threadsafe overload / fine threadsafe Bob* p = &x; // can take threadsafe reference to thread-local object / error } Differences: Can call threadsafe method on thread local / unshared Can take threadsafe reference to thread-local object. One thing that occurred to me is that _objects_ are shared, whereas _functions/methods_ (and their parameters) are thread safe . Theadsafe is kind of like a const (as to mutable/immutable) to threading, a promise to behave correctly in the presence of threading. thread safe references therefore must not escape.
Re: Thread-safe attribution
On Sat, Oct 6, 2018 at 6:59 PM Manu wrote: > > So I'm working on a SMT infrastructure, and expression of > thread-safety is a core design mechanic... but I'm really struggling > to express it in terms of the type system. > I figure I'll throw some design challenges out here and see if anyone > can offer some good ideas. > > The thing I'm trying to model is an attribute along the lines of > `shared`, but actually useful ;) > I'll use the attribute `threadsafe` in place of `shared`, and see > where that goes. > > Consider: > struct Bob > { > int x; > threadsafe Atomic!int y; > > void m1(); > void m2() threadsafe;; > > void overloaded(); > void overloaded() threadsafe; > } > > void func(Bob x, threadsafe Bob y) > { > x.x = 10; // fine > x.y = 10; // fine > x.m1(); // fine > x.m2(); // fine > x.overloaded(); // fine, use the un-threadsafe overload > > y.x = 10; // ERROR, a threadsafe reference can NOT modify an > un-threadsafe member > y.y = 10; // fine > x.m1(); // ERROR, method not threadsafe > x.m2(); // fine > x.overloaded(); // fine, use the threadsafe overload > > threadsafe Bob* p = &x; // can take threadsafe reference to > thread-local object > } > > This is loosely what `shared` models, but there's a few differences: > 1. thread-local can NOT promote to shared > 2. shared `this` applies to members > > For `shared` to be useful, it should be that a shared reference to > something inhibits access to it's thread-local stuff. And in that > world, then I believe that thread-local promotion to shared would work > like const does. > > I guess I'm wondering; should `shared` be transitive? Perhaps that's > what's wrong with it...? *** the function arguments should be `ref`!
Thread-safe attribution
So I'm working on a SMT infrastructure, and expression of thread-safety is a core design mechanic... but I'm really struggling to express it in terms of the type system. I figure I'll throw some design challenges out here and see if anyone can offer some good ideas. The thing I'm trying to model is an attribute along the lines of `shared`, but actually useful ;) I'll use the attribute `threadsafe` in place of `shared`, and see where that goes. Consider: struct Bob { int x; threadsafe Atomic!int y; void m1(); void m2() threadsafe;; void overloaded(); void overloaded() threadsafe; } void func(Bob x, threadsafe Bob y) { x.x = 10; // fine x.y = 10; // fine x.m1(); // fine x.m2(); // fine x.overloaded(); // fine, use the un-threadsafe overload y.x = 10; // ERROR, a threadsafe reference can NOT modify an un-threadsafe member y.y = 10; // fine x.m1(); // ERROR, method not threadsafe x.m2(); // fine x.overloaded(); // fine, use the threadsafe overload threadsafe Bob* p = &x; // can take threadsafe reference to thread-local object } This is loosely what `shared` models, but there's a few differences: 1. thread-local can NOT promote to shared 2. shared `this` applies to members For `shared` to be useful, it should be that a shared reference to something inhibits access to it's thread-local stuff. And in that world, then I believe that thread-local promotion to shared would work like const does. I guess I'm wondering; should `shared` be transitive? Perhaps that's what's wrong with it...?
Re: @safe pointer value modification
On Saturday, September 8, 2018 11:06:20 AM MDT Neia Neutuladh via Digitalmars-d wrote: > On Saturday, 8 September 2018 at 17:01:33 UTC, Jacob Shtokolov > > wrote: > > So, modification of pointer values is prohibited (if I > > understand this sentence correctly). > > @safe code can't manipulate the pointer itself, in order to avoid > memory corruption. > > So this is forbidden: > > void main() @safe > { > int* p = malloc(512); >p++; > } > > But in @safe code, the compiler assumes that all pointers you > receive are valid. And the null pointer is also valid -- > dereferencing it results in a segmentation fault rather than > memory corruption. Also, mutating the data that a pointer points to is not mutating the pointer. So, *foo = 42; is not mutating a pointer, whereas ++foo; would be. So, the first is allowed in @safe code, whereas the second is not. BTW, if you have questions about D, please ask them in D.Learn. This newsgroup / mailing list / forum is intended for general discussion on D, not for answering questions about how the language works. - Jonathan M Davis
Re: @safe pointer value modification
On Saturday, 8 September 2018 at 17:01:33 UTC, Jacob Shtokolov wrote: So, modification of pointer values is prohibited (if I understand this sentence correctly). @safe code can't manipulate the pointer itself, in order to avoid memory corruption. So this is forbidden: void main() @safe { int* p = malloc(512); p++; } But in @safe code, the compiler assumes that all pointers you receive are valid. And the null pointer is also valid -- dereferencing it results in a segmentation fault rather than memory corruption.
@safe pointer value modification
Hi, According to the docs: https://dlang.org/spec/memory-safe-d.html Memory-safe code cannot use certain language features, such as: Casts that break the type system. Modification of pointer values. Taking the address of a local variable or function parameter. So, modification of pointer values is prohibited (if I understand this sentence correctly). However, this code compiles (and will cause a segfault of course): https://run.dlang.io/is/HrUKMy import std.stdio; @safe void main() { int *a; *a = 10; writeln(a); } I'm still learning D so very likely misunderstood something, but isn't that a bug? Thanks!
Re: Is @safe still a work-in-progress?
On Wednesday, 29 August 2018 at 05:01:14 UTC, Walter Bright wrote: On 8/25/2018 5:42 AM, Chris M. wrote: What about my other point then on the syntax? I think something similar to what I suggested would be a much more flexible solution and is worth considering. Much more work would be needed to make that a proposal. Here's some more details that could help with considering it incase you didn't see it: https://forum.dlang.org/post/uxloaftaptfbzhbhq...@forum.dlang.org Seems like it's a super set of return and the current proposal to (from what I can understand at least) make some parameters act like a sink in some situations when a function's return value is of a specific type and the parameters are annotated with some convention? Cheers, - Ali
Re: Is @safe still a work-in-progress?
On Thursday, 30 August 2018 at 16:57:05 UTC, Maksim Fomin wrote: My point is somewhat different from expressed already here by Jonathan. He speaks about whitelisting/blacklisting and that the former model will always contain loopholes. In my view, the reason is that memory safety in (essentially good old C) memory model depends on runtime memory type which is unrelated to static type. Taking random variable - it can be allocated on stack, heap, GC-heap, thread-local and there is no way at CT to determine this in general case. In other words, static type rules is bad approach to determine memory safety. It can be used to detect some obvious bugs, but does not work across compilation boundaries. The better approach in my view is to insert runtime code which performs some tests (at least this how C# works). I think it is understood that any compile time checking must be conservative in the general case to be sound. The same is true for compile time type checking, which must reject type safe code that can only be determined type safe at runtime. Adding runtime checks does sound reasonable, and D already does this for array bounds.
Re: Is @safe still a work-in-progress?
On Wednesday, 29 August 2018 at 13:53:42 UTC, Nicholas Wilson wrote: On Wednesday, 29 August 2018 at 08:35:30 UTC, Walter Bright wrote: On 8/28/2018 10:18 PM, Nicholas Wilson wrote: Bugzilla is not documentation. These are language changes they need to be in release notes and the spec. You asked for a clue: "we have no clue WTF its supposed to do or why the changes are being made" and there it is. There are no barriers to reviewing the idea nor the implementation. I don't know how much more blunt I can be while still being professional about it: _this must be documented properly_. This is not just about us reviewing it, this is also about people using it. I can confirm that, from a user's perspective, (the state of) DIP1000 is not clearly defined. I read the forums so I know Phobos is not completely compatible yet, but I don't know what that means to me. The documentation for -dip1000 [1] simply refers to the DIP text, and after reading that I decide I want it. Then, possibly a while after switching to it, I start getting (mangled) link errors. It is not obvious to me that these relate to -dip1000 (which means wasting some time in trouble shooting) but they disappear after I take away that option. I would have appreciated if [1] would have included or linked to information on the situations in which you can successfully use -dip1000, and what are the symptoms when you move outside that envelope (link errors, presumably). At the very least, [1] could/should stress that -dip switches enable experimental functionality and that certain things are known to not work. I do seem to be able to use -dip25 without issues though. [1] https://dlang.org/dmd-windows.html
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 07:19:25 UTC, Peter Alexander wrote: My question is: what is the status of @safe? I am quite surprised to see such a simple case fail. Is @safe believed to be fully implemented (modulo bugs) and this is just an unfortunate corner case, or is it known work-in-progress? Years ago (2012-2013 https://forum.dlang.org/post/xibbzslaunogifsom...@forum.dlang.org) I was arguing that @safe is crippled: 1) there is high probability that some loopholes are missing (several of them were posted to bugzilla and still not fixed since then) 2) fixing loopholes will make using @safe inconvenient. My point is somewhat different from expressed already here by Jonathan. He speaks about whitelisting/blacklisting and that the former model will always contain loopholes. In my view, the reason is that memory safety in (essentially good old C) memory model depends on runtime memory type which is unrelated to static type. Taking random variable - it can be allocated on stack, heap, GC-heap, thread-local and there is no way at CT to determine this in general case. In other words, static type rules is bad approach to determine memory safety. It can be used to detect some obvious bugs, but does not work across compilation boundaries. The better approach in my view is to insert runtime code which performs some tests (at least this how C# works).
Re: Is @safe still a work-in-progress?
On Wednesday, 29 August 2018 at 08:35:30 UTC, Walter Bright wrote: On 8/28/2018 10:18 PM, Nicholas Wilson wrote: Bugzilla is not documentation. These are language changes they need to be in release notes and the spec. You asked for a clue: "we have no clue WTF its supposed to do or why the changes are being made" and there it is. There are no barriers to reviewing the idea nor the implementation. I don't know how much more blunt I can be while still being professional about it: _this must be documented properly_. This is not just about us reviewing it, this is also about people using it. Think about it this way: I'm a new user and I hear that D is supposed to be memory safe. Am I going to trawl through bugzilla to find about the features of this memory safety? Suppose I do a search for "dlang memory safety" on how I can use this feature. What does a search bring up? First hit: https://dlang.org/spec/memory-safe-d.html Gives me a link to https://dlang.org/articles/safed.html, not very helpful, and to @system/@safe/@trusted, that tells me a bunch of things I'm not allowed to do in safe code. Second hit: https://dlang.org/blog/2016/09/28/how-to-write-trusted-code-in-d/ More (admittedly better) info on @system/@safe/@trusted Third Hit: https://forum.dlang.org/thread/ofkjuq$or6$1...@digitalmars.com welp the forum is down, next! Fourth hit: https://wiki.dlang.org/Memory_Management Fifth hit: https://dconf.org/2017/talks/bright.pdf Some stuff about scope, possibly not up to date(!). Sixth hit is Adam Ruppe's fork of the spec page on memory safety Seventh is https://news.ycombinator.com/item?id=12391370 Number 8: https://www.meetup.com/en-AU/SeaLang/events/246692611/ 9: https://www.reddit.com/r/cpp/comments/6b4xrc/walter_bright_believes_memory_safety_will_kill_c/ 10: is the Wikipdia page on D. Searching for "dlang scope" brings up scope(exit) & friends and some old forum posts including https://forum.dlang.org/thread/obfftm$2m3j$1...@digitalmars.com which feels like déjà vu, followed by a bunch of irrelevant links to various parts of the spec. That was on a not anonymous search, I know precisely what I'm looking for, and the closest thing I found was 4/5ths the way down a 42 slide PDF with no annotations from a year ago. Even someone relatively familiar is going to look at the spec and the changelog, and they're not going to find anything BECAUSE ITS NOT THERE! You know where it is? Bugzilla, because that's where Walter thinks documentation should go.
Re: Is @safe still a work-in-progress?
On 8/28/2018 10:18 PM, Nicholas Wilson wrote: Bugzilla is not documentation. These are language changes they need to be in release notes and the spec. You asked for a clue: "we have no clue WTF its supposed to do or why the changes are being made" and there it is. There are no barriers to reviewing the idea nor the implementation.
Re: Is @safe still a work-in-progress?
On 8/23/2018 6:10 AM, Atila Neves wrote: -- struct S { int x; @safe int* foo() { return &x; } } -- % dmd -o- -dip1000 foo.d % echo $? 0 struct S { int x; @safe int* foo() { return &x; } } int* bar() { S s; return s.foo(); } dmd test -dip1000 test.d(3): Error: returning &this.x escapes a reference to parameter this, perhaps annotate with return Oops: -- int* gPtr; void main() { auto s = S(42); gPtr = s.foo; } -- I get: test.d(3): Error: returning &this.x escapes a reference to parameter this, perhaps annotate with return -- int[] gSlice; void main() { auto s = S([42]); gSlice = s.foo; } struct S { int[] x; @safe int[] foo() return { return x; } } -- T[] is treated like T* as far as scope, etc., are concerned. You should see identical results. And in a struct, `this` is a `ref`, yet `scope` on a member function applies to `this`. Actually, it applies to fields of `this`.
Re: Is @safe still a work-in-progress?
On Wednesday, 29 August 2018 at 05:04:22 UTC, Walter Bright wrote: On 8/25/2018 4:09 AM, Nicholas Wilson wrote: On Saturday, 25 August 2018 at 02:25:41 UTC, Walter Bright wrote: I'm not hostile to debate. I just don't care for "this is uncharted territory, so let's do nothing" which has been going on for probably 4 years now, coincident with "scope is incomplete, D sux". I.e. lead, follow, or get out of the way :-) Document it Already done: https://issues.dlang.org/show_bug.cgi?id=19097 Bugzilla is not documentation. These are language changes they need to be in release notes and the spec.
Re: Is @safe still a work-in-progress?
On 8/28/2018 6:12 AM, Steven Schveighoffer wrote: So this would mean a member function would have to be refactored into a different function with a different calling syntax. i.e: x.foo(target); would have to be refactored to: target.foo(x); or foo(target, x); Maybe it should be anyway. But in terms of put, strictly speaking, any call of some.pipeline.put(x) is wrong. It should be put(some.pipeline, x), to avoid issues with how put was designed. There are some of those in Phobos, too. It's covered by the proposed addition. I don't know how to prove anything with programming languages. I don't mean prove like mathematical proof. I mean try to consider how this affects all cases instead of just the one case that will make phobos compile. "show" is a better verb than "prove". I don't know how to find all cases, either, except by implementing it.
Re: Is @safe still a work-in-progress?
On 8/25/2018 4:09 AM, Nicholas Wilson wrote: On Saturday, 25 August 2018 at 02:25:41 UTC, Walter Bright wrote: I'm not hostile to debate. I just don't care for "this is uncharted territory, so let's do nothing" which has been going on for probably 4 years now, coincident with "scope is incomplete, D sux". I.e. lead, follow, or get out of the way :-) Document it Already done: https://issues.dlang.org/show_bug.cgi?id=19097
Re: Is @safe still a work-in-progress?
On 8/25/2018 5:42 AM, Chris M. wrote: What about my other point then on the syntax? I think something similar to what I suggested would be a much more flexible solution and is worth considering. Much more work would be needed to make that a proposal.
Re: Is @safe still a work-in-progress?
On 8/24/18 10:25 PM, Walter Bright wrote: On 8/23/2018 6:32 AM, Steven Schveighoffer wrote: Furthermore any member function (or UFCS function for that matter) REQUIRES the first parameter to be the aggregate. How do you make a member function that stuffs the return into a different parameter properly typecheck? What I propose is that the function interface be refactored so it does fit into these patterns. Is that an unreasonable requirement? I don't know. But it doesn't seem to be, as I haven't run into it yet. So this would mean a member function would have to be refactored into a different function with a different calling syntax. i.e: x.foo(target); would have to be refactored to: target.foo(x); or foo(target, x); Aside from the adjustment in name that is necessary to make this read correctly, that may cause other problems (sometimes non-member functions aren't available if it's a template instantiation). Phobos doesn't do this by accident. It's how constructors work (see above) and how pipeline programming works. Constructors I agree are reasonable to consider `this` to be the return value. On that point, I would say we should definitely go ahead with making that rule, and I think it will lead to no confusion whatsoever. pipeline programming depends on returning something other than `void`, so I don't see how this applies. grep Phobos for instances of put() and see its signature. It's part of pipeline programming, and it's all over the place. I stand partly corrected! Indeed you can put a void-returning function at the *end* of a pipeline call, I hadn't thought of that. But in terms of put, strictly speaking, any call of some.pipeline.put(x) is wrong. It should be put(some.pipeline, x), to avoid issues with how put was designed. It would restrict your legitimate calls. Maybe that's a good thing. Having multiple simultaneous routes of data out of a function is not good practice (note that it is impossible with functional programming). If you absolutely must have it, the exit routes can be aggregated into a struct, then pass that struct as the first argument. Maybe it's better to designate one sink, and have that be the result. I know that after inout was implemented, there were definitely cases where one wanted to have multiple inout routes (i.e. independent traces between multiple parameters for copying mutability). It may be the same for this, I don't know. I want to stress that it may be a valid solution, but we should strive to prove the solutions are the best possible rather than just use duct-tape methodology. I don't know how to prove anything with programming languages. I don't mean prove like mathematical proof. I mean try to consider how this affects all cases instead of just the one case that will make phobos compile. "show" is a better verb than "prove". It should even be considered that perhaps there are better solutions even than the approach dip1000 has taken. People have hypothesized that for several years, and so far none have been forthcoming beyond a few hand-wavy generalities. I'm just saying if dip1000 cannot fix all the problems, that instead of adding weird exceptions, or the classic "you're just doing it wrong", maybe we should reconsider the approach. Another case which was brought up and pretty much ignored was this one: https://forum.dlang.org/post/qkrdpmdqaxjadgvso...@forum.dlang.org I also want to point out that the attitude of 'we could just fix it, but nobody will pull my request' is unhelpful. We want to make sure we have the best solution possible, don't take criticism as meaningless petty bickering. People are genuinely trying to make sure D is improved. Hostility towards reviews or debate doesn't foster that. I'm not hostile to debate. I just don't care for "this is uncharted territory, so let's do nothing" which has been going on for probably 4 years now, coincident with "scope is incomplete, D sux". I.e. lead, follow, or get out of the way :-) I'm opting for the latter, as the idea of band-aid PRs to get Phobos compiling with dip1000 just to see if dip1000 is going to work seems like the wrong approach to me. The consequence of this is that getting out of the way means your PRs don't get pulled. -Steve
Re: Is @safe still a work-in-progress?
On 8/24/18 10:28 PM, Walter Bright wrote: On 8/23/2018 8:14 AM, Steven Schveighoffer wrote: If I had to design a specific way to allow the common case to be easy, but still provide a mechanism for the uncommon cases, I would say: 1. define a compiler-recognized attribute (e.g. @__sink). 2. If @__sink is applied to any parameter, that is effectively the return value. 3. In the absence of a @__sink designation on non-void-returning functions, it applies to the return value. 4. In the absence of a @__sink designation on void returning functions, it applies to the first parameter. 5. Inference of @__sink happens even on non-templates. 6. If @__sink is attributed on multiple parameters, you assume all return parameters are assigned to all @__sink parameters for the purposes of verifying lifetimes are not exceeded. 'ref' is already @__sink. No, otherwise we wouldn't need the patch you are pushing. -Steve
Re: Is @safe still a work-in-progress?
On Saturday, 25 August 2018 at 02:37:00 UTC, Walter Bright wrote: On 8/23/2018 5:58 PM, Chris M. wrote: Seems to be more of a warning of what issues we may face if DIP25/DIP1000 are finally implemented. It would be good to consider NLLs as well before D is committed. No point in repeating issues that have already been studied. DIP25 waqs "finally implemented" several years ago, and works well. DIP1000 was implemented as well, it works, but it didn't cover the case of returning through a ref parameter. There's no way to "thoroughly vet" them before implementing. It doesn't happen with C++, either, somebody builds an implementation and then people try it out. What about my other point then on the syntax? I think something similar to what I suggested would be a much more flexible solution and is worth considering.
Re: Is @safe still a work-in-progress?
On Saturday, 25 August 2018 at 02:25:41 UTC, Walter Bright wrote: I'm not hostile to debate. I just don't care for "this is uncharted territory, so let's do nothing" which has been going on for probably 4 years now, coincident with "scope is incomplete, D sux". I.e. lead, follow, or get out of the way :-) Document it and we will help you get it reviewed and merged. Until you do that we will stay in your way. Heck, once we understand it we may even lead, but we can't do that if we have no clue WTF its supposed to do or why the changes are being made. At the very least PRs need detailed changelog entries and preferably spec updates and revisions to the DIP1000 proposal document. Heck even a wiki page would be useful. I'm trying to make the spec page on memory safe programming the authoritative source see https://github.com/dlang/dlang.org/pull/2453
Re: Is @safe still a work-in-progress?
On 8/23/2018 5:58 PM, Chris M. wrote: Seems to be more of a warning of what issues we may face if DIP25/DIP1000 are finally implemented. It would be good to consider NLLs as well before D is committed. No point in repeating issues that have already been studied. DIP25 waqs "finally implemented" several years ago, and works well. DIP1000 was implemented as well, it works, but it didn't cover the case of returning through a ref parameter. There's no way to "thoroughly vet" them before implementing. It doesn't happen with C++, either, somebody builds an implementation and then people try it out.
Re: Is @safe still a work-in-progress?
On 8/23/2018 6:32 AM, Steven Schveighoffer wrote: Furthermore any member function (or UFCS function for that matter) REQUIRES the first parameter to be the aggregate. How do you make a member function that stuffs the return into a different parameter properly typecheck? What I propose is that the function interface be refactored so it does fit into these patterns. Is that an unreasonable requirement? I don't know. But it doesn't seem to be, as I haven't run into it yet. Phobos doesn't do this by accident. It's how constructors work (see above) and how pipeline programming works. Constructors I agree are reasonable to consider `this` to be the return value. On that point, I would say we should definitely go ahead with making that rule, and I think it will lead to no confusion whatsoever. pipeline programming depends on returning something other than `void`, so I don't see how this applies. grep Phobos for instances of put() and see its signature. It's part of pipeline programming, and it's all over the place. You would have to consider the shortest liftetime and assume everything goes there. That's right. It would restrict your legitimate calls. Maybe that's a good thing. Having multiple simultaneous routes of data out of a function is not good practice (note that it is impossible with functional programming). If you absolutely must have it, the exit routes can be aggregated into a struct, then pass that struct as the first argument. I want to stress that it may be a valid solution, but we should strive to prove the solutions are the best possible rather than just use duct-tape methodology. I don't know how to prove anything with programming languages. It should even be considered that perhaps there are better solutions even than the approach dip1000 has taken. People have hypothesized that for several years, and so far none have been forthcoming beyond a few hand-wavy generalities. I also want to point out that the attitude of 'we could just fix it, but nobody will pull my request' is unhelpful. We want to make sure we have the best solution possible, don't take criticism as meaningless petty bickering. People are genuinely trying to make sure D is improved. Hostility towards reviews or debate doesn't foster that. I'm not hostile to debate. I just don't care for "this is uncharted territory, so let's do nothing" which has been going on for probably 4 years now, coincident with "scope is incomplete, D sux". I.e. lead, follow, or get out of the way :-)
Re: Is @safe still a work-in-progress?
On 8/23/2018 8:14 AM, Steven Schveighoffer wrote: If I had to design a specific way to allow the common case to be easy, but still provide a mechanism for the uncommon cases, I would say: 1. define a compiler-recognized attribute (e.g. @__sink). 2. If @__sink is applied to any parameter, that is effectively the return value. 3. In the absence of a @__sink designation on non-void-returning functions, it applies to the return value. 4. In the absence of a @__sink designation on void returning functions, it applies to the first parameter. 5. Inference of @__sink happens even on non-templates. 6. If @__sink is attributed on multiple parameters, you assume all return parameters are assigned to all @__sink parameters for the purposes of verifying lifetimes are not exceeded. 'ref' is already @__sink.
Re: Is @safe still a work-in-progress?
On Friday, 24 August 2018 at 00:13:48 UTC, Mike Franklin wrote: On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote: Heck, now that I'm looking at it, DIP25 seems like a more restricted form of Rust's lifetimes. Let me know if I'm just completely wrong about this, but I think DIP 25 is analogous to Problem #3 for Rust's Non-Lexical Lifetimes: http://smallcultfollowing.com/babysteps/blog/2016/04/27/non-lexical-lifetimes-introduction/#problem-case-3-conditional-control-flow-across-functions http://smallcultfollowing.com/babysteps/blog/2016/05/09/non-lexical-lifetimes-adding-the-outlives-relation/#problem-case-3-revisited Seems to be more of a warning of what issues we may face if DIP25/DIP1000 are finally implemented. It would be good to consider NLLs as well before D is committed. No point in repeating issues that have already been studied. would basically be like (pseudosyntax) @safe ref'a int identity(ref'a int x) { return x; // fine } Maybe the more sane thing would be a syntax that visually ties them together as above. Obviously we're looking at possibly breaking changes, but how widespread would they be? void betty(ref'a scope int* r, scope'a int* p); // syntax is not so nice since I just arbitrarily stuck them on different keywords, but that's besides the point here I wish I had been more involved in D when DIP 25 and DIP 1000 were being proposed, as I don't think the designs were thoroughly vetted. It's taken me at least a year to even begin getting a grasp on it. I think DIP 25 and DIP 1000 should have been combined and thought of holistically as simply "annotated lifetimes in D" rather than separate things. I think then it becomes easier to visualize what the problem is and see, potentially many, alternatives. Given the investments that have already been made in DIP 25 and DIP 1000, it's going to take an extremely motivated individual to fight an uphill battle to change direction now, I'm afraid. If working on D was my full-time job, I'd do it, but who in this community has such resources. I think the main problem is that these were created before the DIP process was fully fleshed out, so who knows how much vetting they got. At least they are still compiler switches and probably not widely used, so it's not like we're already too far gone. I do also agree they should be worked on in conjunction, the hard part is finding someone to take ownership and put in the extra effort. Mike
Re: Is @safe still a work-in-progress?
On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote: Heck, now that I'm looking at it, DIP25 seems like a more restricted form of Rust's lifetimes. Let me know if I'm just completely wrong about this, but I think DIP 25 is analogous to Problem #3 for Rust's Non-Lexical Lifetimes: http://smallcultfollowing.com/babysteps/blog/2016/04/27/non-lexical-lifetimes-introduction/#problem-case-3-conditional-control-flow-across-functions http://smallcultfollowing.com/babysteps/blog/2016/05/09/non-lexical-lifetimes-adding-the-outlives-relation/#problem-case-3-revisited would basically be like (pseudosyntax) @safe ref'a int identity(ref'a int x) { return x; // fine } Maybe the more sane thing would be a syntax that visually ties them together as above. Obviously we're looking at possibly breaking changes, but how widespread would they be? void betty(ref'a scope int* r, scope'a int* p); // syntax is not so nice since I just arbitrarily stuck them on different keywords, but that's besides the point here I wish I had been more involved in D when DIP 25 and DIP 1000 were being proposed, as I don't think the designs were thoroughly vetted. It's taken me at least a year to even begin getting a grasp on it. I think DIP 25 and DIP 1000 should have been combined and thought of holistically as simply "annotated lifetimes in D" rather than separate things. I think then it becomes easier to visualize what the problem is and see, potentially many, alternatives. Given the investments that have already been made in DIP 25 and DIP 1000, it's going to take an extremely motivated individual to fight an uphill battle to change direction now, I'm afraid. If working on D was my full-time job, I'd do it, but who in this community has such resources. Mike
Re: Is @safe still a work-in-progress?
On Thursday, 23 August 2018 at 23:58:00 UTC, jmh530 wrote: On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote: Heck, now that I'm looking at it, DIP25 seems like a more restricted form of Rust's lifetimes. Let me know if I'm just completely wrong about this, but [snip] Check out DIP1000 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md I've read through it already, I'm just throwing out an idea for what I think is a more flexible solution to Walter's current issue which seems to be stemming more from DIP25 rather than DIP1000. More effort but I think it'd be worth it in the long run.
Re: Is @safe still a work-in-progress?
On Thursday, 23 August 2018 at 23:36:07 UTC, Chris M. wrote: Heck, now that I'm looking at it, DIP25 seems like a more restricted form of Rust's lifetimes. Let me know if I'm just completely wrong about this, but [snip] Check out DIP1000 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md
Re: Is @safe still a work-in-progress?
On Thursday, 23 August 2018 at 15:48:00 UTC, Chris M. wrote: On Thursday, 23 August 2018 at 15:14:07 UTC, Steven Schveighoffer wrote: On 8/23/18 9:32 AM, Steven Schveighoffer wrote: [...] Actually, thinking about this, the shortest lifetime is dictated by how it is called, so there is no valid way to determine which one makes sense when compiling the function. In order for this to work, you'd have to attribute it somehow. I can see that is likely going to be way more cumbersome than it's worth. If I had to design a specific way to allow the common case to be easy, but still provide a mechanism for the uncommon cases, I would say: 1. define a compiler-recognized attribute (e.g. @__sink). 2. If @__sink is applied to any parameter, that is effectively the return value. 3. In the absence of a @__sink designation on non-void-returning functions, it applies to the return value. 4. In the absence of a @__sink designation on void returning functions, it applies to the first parameter. 5. Inference of @__sink happens even on non-templates. 6. If @__sink is attributed on multiple parameters, you assume all return parameters are assigned to all @__sink parameters for the purposes of verifying lifetimes are not exceeded. Ugly to specify, but might actually be pretty non-intrusive to use. -Steve This is more a general reply to the thread. If I think I'm getting a good grasp on the issue here, it seems like something Rust already solved with lifetime annotations. Could they or something similar be leveraged for D as well? https://doc.rust-lang.org/1.9.0/book/lifetimes.html Current solution just seems too specific and very restrictive. Heck, now that I'm looking at it, DIP25 seems like a more restricted form of Rust's lifetimes. Let me know if I'm just completely wrong about this, but @safe ref int identity(return ref int x) { return x; // fine } would basically be like (pseudosyntax) @safe ref'a int identity(ref'a int x) { return x; // fine } Maybe the more sane thing would be a syntax that visually ties them together as above. Obviously we're looking at possibly breaking changes, but how widespread would they be? void betty(ref'a scope int* r, scope'a int* p); // syntax is not so nice since I just arbitrarily stuck them on different keywords, but that's besides the point here
Re: Is @safe still a work-in-progress?
On Thursday, 23 August 2018 at 15:14:07 UTC, Steven Schveighoffer wrote: On 8/23/18 9:32 AM, Steven Schveighoffer wrote: [...] Actually, thinking about this, the shortest lifetime is dictated by how it is called, so there is no valid way to determine which one makes sense when compiling the function. In order for this to work, you'd have to attribute it somehow. I can see that is likely going to be way more cumbersome than it's worth. If I had to design a specific way to allow the common case to be easy, but still provide a mechanism for the uncommon cases, I would say: 1. define a compiler-recognized attribute (e.g. @__sink). 2. If @__sink is applied to any parameter, that is effectively the return value. 3. In the absence of a @__sink designation on non-void-returning functions, it applies to the return value. 4. In the absence of a @__sink designation on void returning functions, it applies to the first parameter. 5. Inference of @__sink happens even on non-templates. 6. If @__sink is attributed on multiple parameters, you assume all return parameters are assigned to all @__sink parameters for the purposes of verifying lifetimes are not exceeded. Ugly to specify, but might actually be pretty non-intrusive to use. -Steve This is more a general reply to the thread. If I think I'm getting a good grasp on the issue here, it seems like something Rust already solved with lifetime annotations. Could they or something similar be leveraged for D as well? https://doc.rust-lang.org/1.9.0/book/lifetimes.html Current solution just seems too specific and very restrictive.
Re: Is @safe still a work-in-progress?
On 8/23/18 9:32 AM, Steven Schveighoffer wrote: On 8/23/18 4:58 AM, Walter Bright wrote: On 8/22/2018 6:50 AM, Steven Schveighoffer wrote: As for things being made "more flexible in the future" this basically translates to code breakage. For example, if you are depending on only the first parameter being considered the "return" value, and all of a sudden it changes to encompass all your parameters, your existing code may fail to compile, even if it's correctly safe and properly annotated. It's a good point. But I don't see an obvious use case for considering all the ref parameters as being returns. You would have to consider the shortest liftetime and assume everything goes there. It would restrict your legitimate calls. Only mitigating factor may be if you take the ones you aren't going to modify as const or inout. Actually, thinking about this, the shortest lifetime is dictated by how it is called, so there is no valid way to determine which one makes sense when compiling the function. In order for this to work, you'd have to attribute it somehow. I can see that is likely going to be way more cumbersome than it's worth. If I had to design a specific way to allow the common case to be easy, but still provide a mechanism for the uncommon cases, I would say: 1. define a compiler-recognized attribute (e.g. @__sink). 2. If @__sink is applied to any parameter, that is effectively the return value. 3. In the absence of a @__sink designation on non-void-returning functions, it applies to the return value. 4. In the absence of a @__sink designation on void returning functions, it applies to the first parameter. 5. Inference of @__sink happens even on non-templates. 6. If @__sink is attributed on multiple parameters, you assume all return parameters are assigned to all @__sink parameters for the purposes of verifying lifetimes are not exceeded. Ugly to specify, but might actually be pretty non-intrusive to use. -Steve
Re: Is @safe still a work-in-progress?
On 8/23/18 4:58 AM, Walter Bright wrote: On 8/22/2018 6:50 AM, Steven Schveighoffer wrote: What about: size_t put(sink, parameters...) Does this qualify as the sink being the "return" type? Obviously the real return can't contain any references, so it trivially can be ruled out as the destination of any escaping parameters. Your reasoning is correct, but currently it only applies with 'void' return types. Or how about a member function that takes a ref parameter? Is `this` the "return" or is the ref parameter the "return"? `this` is the ref parameter. In particular, consider a constructor: struct S { int* p; this(return scope int* p) { this.p = p; } } int i; S s = S(&i); This code appears in Phobos, and it is very reasonable to expect it to check as safe. What I mean to say is, we have a semantic today -- the return value is hooked to any `return` parameters, end of story. This is clear, concise, and easy to understand. You are saying that in some cases, the return value is actually deposited in the `this` parameter. In cases where the actual return type is void, OK, I see that we can tack on that rule without issues. Furthermore any member function (or UFCS function for that matter) REQUIRES the first parameter to be the aggregate. How do you make a member function that stuffs the return into a different parameter properly typecheck? What rule do we tack on then? It's going to be confusing to anyone who writes their API thinking about how it's call syntax reads, not how the compiler wants to do flow analysis. Not to mention, the keyword is `return`, not `returnorfirstparam`. It's still going to be confusing no matter how you look at it. My problem with the idea is that it is going to seem flaky -- we are using convention to dictate what is actually the return parameter, vs. what semantically happens inside the function. It's going to confuse anyone trying to do it a different way. I've experienced this in the past with things like toHash, where if you didn't define it with the exact signature, it wouldn't actually be used. I realize obviously, that `put` is already specified. But as I said in the bug report, we should think twice about defining rules based solely on how Phobos does things, and calling that the solution. Phobos doesn't do this by accident. It's how constructors work (see above) and how pipeline programming works. Constructors I agree are reasonable to consider `this` to be the return value. On that point, I would say we should definitely go ahead with making that rule, and I think it will lead to no confusion whatsoever. pipeline programming depends on returning something other than `void`, so I don't see how this applies. It's more when you are setting members via properties where this comes into play. We need it -- we need this ability to tell the compiler "this parameter connects to this other parameter". I just don't know if the proposed rules are a) good enough for the general case, and b) don't cause more confusion than they are worth. As for things being made "more flexible in the future" this basically translates to code breakage. For example, if you are depending on only the first parameter being considered the "return" value, and all of a sudden it changes to encompass all your parameters, your existing code may fail to compile, even if it's correctly safe and properly annotated. It's a good point. But I don't see an obvious use case for considering all the ref parameters as being returns. You would have to consider the shortest liftetime and assume everything goes there. It would restrict your legitimate calls. Only mitigating factor may be if you take the ones you aren't going to modify as const or inout. > I want to ensure Atila is successful with this. But that means Phobos has to compile with dip1000. So I need to make it work. I think it's a very worthy goal to make Phobos work, and a great proof of concept for dip1000's veracity. However, one-off rules just to make it work with existing code go against that goal IMO. Rules that stand on their own I think will fare better than ones that are loopholes to allow existing code to compile. I couldn't come up with a better idea than this, and this one works. I want to stress that it may be a valid solution, but we should strive to prove the solutions are the best possible rather than just use duct-tape methodology. It should even be considered that perhaps there are better solutions even than the approach dip1000 has taken. I also want to point out that the attitude of 'we could just fix it, but nobody will pull my request' is unhelpful. We want to make sure we have the best solution possible, don't take criticism as meaningless petty bickering. People are genuinely trying to make sure D is improved. Hostility towards reviews or debate doesn't foster that. -Steve
Re: Is @safe still a work-in-progress?
On Thursday, 23 August 2018 at 08:48:15 UTC, Walter Bright wrote: On 8/22/2018 3:52 AM, Atila Neves wrote: On Wednesday, 22 August 2018 at 09:05:13 UTC, Walter Bright wrote: On 8/21/2018 8:58 PM, Nicholas Wilson wrote: On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote: The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. Kagamin just told me I needed to use `return` instead of `scope` to get things to work and I'm still not sure why. The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`. Another way to think about it is this: S s; return &s; We all know that is an error. The idea is to have a way to express that for: S s; return s.foo(); and: S s; return foo(&s); so that the compiler knows that the return value of foo() is attached to the lifetime of s. Pretty much everything flows from that. Would the guideline below be correct? "Add scope to every non-template member function that isn't meant to escape this and add return to every non-template member function that returns all or part of `this` by pointer or ref if you want the compiler to check that nothing gets escaped in @safe code." Being a template doesn't make any difference, except that it will helpfully infer these things. The reason I wrote "non-template" is precisely because attributes get inferred for templates and therefore it would be at best redundant to annotate. What about the guideline being correct or not? Also, since 'this' is passed by 'ref' to struct member functions, it cannot escape anyway with dip1000: struct S { int x; @safe ref int foo() { return x; } } dmd test -dip1000 test.d(4): Error: returning this.x escapes a reference to parameter this, perhaps annotate with return Returning by ref is good, since it's a lot harder to escape it. Since variables can't be declared ref, I can't pass it to a ref global. However: -- struct S { int x; @safe int* foo() { return &x; } } -- % dmd -o- -dip1000 foo.d % echo $? 0 Oops: -- int* gPtr; void main() { auto s = S(42); gPtr = s.foo; } -- "Don't use pointers then!". Ok: -- int[] gSlice; void main() { auto s = S([42]); gSlice = s.foo; } struct S { int[] x; @safe int[] foo() return { return x; } } -- % dmd -o- -dip1000 bar.d % echo $? 0 Oops. I can't say `int[] x;` since it doesn't apply to fields. `scope` is for pointers, `ref` does not need further annotation. I always forget this, it's confusing. It doesn't help that slices have pointers, so I guess `scope` is for them too? And in a struct, `this` is a `ref`, yet `scope` on a member function applies to `this`. Atila
Re: Is @safe still a work-in-progress?
On Thursday, 23 August 2018 at 08:48:15 UTC, Walter Bright wrote: Being a template doesn't make any difference, except that it will helpfully infer these things. Including the `return` attribute. Also, since 'this' is passed by 'ref' to struct member functions, it cannot escape anyway with dip1000: struct S { int x; @safe ref int foo() { return x; } } dmd test -dip1000 test.d(4): Error: returning this.x escapes a reference to parameter this, perhaps annotate with return If we add the `return` attribute, we can escape a pointer to `this`. And since attributes are often inferred, it might come as a surprise to the programmer. However, as far as I understand DIP 1000, it should make sure that the returned pointer won't outlive the struct instance. And I guess that's why it's ok to infer the `return` attribute? That depends on DMD not over-estimating the lifetimes of things, though. Unfortunately, it does just that. When in doubt, DMD goes with infinite (e.g., `malloc`). I think that's a problem. But I've already demonstrated my ignorance regarding DIP 1000 in a long, not very fruitful discussion with Atila on Bugzilla [1], so maybe I just don't get it. [1] https://issues.dlang.org/show_bug.cgi?id=19183
Re: Is @safe still a work-in-progress?
On 8/22/2018 6:50 AM, Steven Schveighoffer wrote: What about: size_t put(sink, parameters...) Does this qualify as the sink being the "return" type? Obviously the real return can't contain any references, so it trivially can be ruled out as the destination of any escaping parameters. Your reasoning is correct, but currently it only applies with 'void' return types. Or how about a member function that takes a ref parameter? Is `this` the "return" or is the ref parameter the "return"? `this` is the ref parameter. In particular, consider a constructor: struct S { int* p; this(return scope int* p) { this.p = p; } } int i; S s = S(&i); This code appears in Phobos, and it is very reasonable to expect it to check as safe. My problem with the idea is that it is going to seem flaky -- we are using convention to dictate what is actually the return parameter, vs. what semantically happens inside the function. It's going to confuse anyone trying to do it a different way. I've experienced this in the past with things like toHash, where if you didn't define it with the exact signature, it wouldn't actually be used. I realize obviously, that `put` is already specified. But as I said in the bug report, we should think twice about defining rules based solely on how Phobos does things, and calling that the solution. Phobos doesn't do this by accident. It's how constructors work (see above) and how pipeline programming works. As for things being made "more flexible in the future" this basically translates to code breakage. For example, if you are depending on only the first parameter being considered the "return" value, and all of a sudden it changes to encompass all your parameters, your existing code may fail to compile, even if it's correctly safe and properly annotated. It's a good point. But I don't see an obvious use case for considering all the ref parameters as being returns. > I want to ensure Atila is successful with this. But that means Phobos has to compile with dip1000. So I need to make it work. I think it's a very worthy goal to make Phobos work, and a great proof of concept for dip1000's veracity. However, one-off rules just to make it work with existing code go against that goal IMO. Rules that stand on their own I think will fare better than ones that are loopholes to allow existing code to compile. I couldn't come up with a better idea than this, and this one works.
Re: Is @safe still a work-in-progress?
On 8/22/2018 3:52 AM, Atila Neves wrote: On Wednesday, 22 August 2018 at 09:05:13 UTC, Walter Bright wrote: On 8/21/2018 8:58 PM, Nicholas Wilson wrote: On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote: The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. Kagamin just told me I needed to use `return` instead of `scope` to get things to work and I'm still not sure why. The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`. Another way to think about it is this: S s; return &s; We all know that is an error. The idea is to have a way to express that for: S s; return s.foo(); and: S s; return foo(&s); so that the compiler knows that the return value of foo() is attached to the lifetime of s. Pretty much everything flows from that. Would the guideline below be correct? "Add scope to every non-template member function that isn't meant to escape this and add return to every non-template member function that returns all or part of `this` by pointer or ref if you want the compiler to check that nothing gets escaped in @safe code." Being a template doesn't make any difference, except that it will helpfully infer these things. Also, since 'this' is passed by 'ref' to struct member functions, it cannot escape anyway with dip1000: struct S { int x; @safe ref int foo() { return x; } } dmd test -dip1000 test.d(4): Error: returning this.x escapes a reference to parameter this, perhaps annotate with return `scope` is for pointers, `ref` does not need further annotation.
Re: Is @safe still a work-in-progress?
On 8/22/18 5:23 AM, Walter Bright wrote: On 8/21/2018 6:07 PM, Mike Franklin wrote: The proposed idea wants to make the first parameter, if it's `ref`, special. This is because Phobos is written with functions of the form: void put(sink, parameters...) which corresponds to: sink.put(parameters...) The two forms are fairly interchangeable, made more so by the Uniform Function Call Syntax. > Why not the first `ref` parameter regardless of whether it's the absolute first in the list. Why not the last `ref` parameter? Why not all `ref` parameters? Good question. If this fairly restricted form solves the problems, then there is no need for the more flexible form. Things can always be made more flexible in the future, but tightening things can be pretty disruptive. Hence, unless there is an obvious and fairly strong case case for the flexibility, then it should be avoided for now. What about: size_t put(sink, parameters...) Does this qualify as the sink being the "return" type? Obviously the real return can't contain any references, so it trivially can be ruled out as the destination of any escaping parameters. Or how about a member function that takes a ref parameter? Is `this` the "return" or is the ref parameter the "return"? My problem with the idea is that it is going to seem flaky -- we are using convention to dictate what is actually the return parameter, vs. what semantically happens inside the function. It's going to confuse anyone trying to do it a different way. I've experienced this in the past with things like toHash, where if you didn't define it with the exact signature, it wouldn't actually be used. I realize obviously, that `put` is already specified. But as I said in the bug report, we should think twice about defining rules based solely on how Phobos does things, and calling that the solution. As for things being made "more flexible in the future" this basically translates to code breakage. For example, if you are depending on only the first parameter being considered the "return" value, and all of a sudden it changes to encompass all your parameters, your existing code may fail to compile, even if it's correctly safe and properly annotated. I want to ensure Atila is successful with this. But that means Phobos has to compile with dip1000. So I need to make it work. I think it's a very worthy goal to make Phobos work, and a great proof of concept for dip1000's veracity. However, one-off rules just to make it work with existing code go against that goal IMO. Rules that stand on their own I think will fare better than ones that are loopholes to allow existing code to compile. -Steve
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 11:02:00 UTC, Seb wrote: No, it's behind a flag, so you can't really say that we're shipping it as "production ready release". The changes to Phobos are not behind a flag. We're making changes to Phobos in the release branch to accommodate a draft/experimental/choose-your-adjective feature. Mike
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 02:18:15 UTC, Mike Franklin wrote: On Wednesday, 22 August 2018 at 01:07:28 UTC, Mike Franklin wrote: But what bothers me the most... Something else that rubs me the wrong way is that DIP 1000 is currently in a status of `DRAFT`: https://github.com/dlang/DIPs/blob/master/DIPs/README.md What the heck is going on here? We're adding features to the compiler and modifying Phobos in production releases based on a `DRAFT` proposal? No, it's behind a flag, so you can't really say that we're shipping it as "production ready release". In fact I think we should have a hell of a lot more of such experimental flags. This would allow us to be able to merge things quickly, and gain real-world feedback and testing on complicated matters instead of PRs stalling to death in the queue. For reference, Rust has currently 148 opt-in experimental language features: https://doc.rust-lang.org/unstable-book/index.html
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 09:05:13 UTC, Walter Bright wrote: On 8/21/2018 8:58 PM, Nicholas Wilson wrote: On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote: The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. Kagamin just told me I needed to use `return` instead of `scope` to get things to work and I'm still not sure why. The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`. Another way to think about it is this: S s; return &s; We all know that is an error. The idea is to have a way to express that for: S s; return s.foo(); and: S s; return foo(&s); so that the compiler knows that the return value of foo() is attached to the lifetime of s. Pretty much everything flows from that. Would the guideline below be correct? "Add scope to every non-template member function that isn't meant to escape this and add return to every non-template member function that returns all or part of `this` by pointer or ref if you want the compiler to check that nothing gets escaped in @safe code."
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 09:23:26 UTC, Walter Bright wrote: dip1000 has been around for two years, and its predecessor dip25 several years now. Plenty of time for anyone to comment and/or propose something better. Part of the problem is that the implementation keeps changing without keeping the documentation in sync. For example you're implementing all of these inference rules without documenting them: https://github.com/dlang/dmd/pull/8346 https://github.com/dlang/dmd/pull/8408 I asked you about, instead of inferring the attributes, allowing users to add such logic themselves: void foo(T)(T x) if (__traits(isPointer, T)) { T = scope T; } { } But, then the PR got rubber-stamped, and now here we are. Mike
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 09:23:26 UTC, Walter Bright wrote: I want to ensure Atila is successful with this. But that means Phobos has to compile with dip1000. So I need to make it work. There's a good chance you'll get your PR merged when it's documented and vetted, then you'll be able to make it work for Atila. The only think holding it up is you. I'm not asking for much. Sufficiently document the idea, open it up for comment and Q & A, transfer the documentation to the spec to accompany the DMD implementation PR. Assuming there's no major flaws in the design, it should get merged. Mike P.S. Actually, I've been trying to document it myself, since you don't seem willing to, but it's going to take me a lot longer to figure out what's in your head than it would take you.
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 09:23:26 UTC, Walter Bright wrote: The proposed idea wants to make the first parameter, if it's `ref`, special. This is because Phobos is written with functions of the form: void put(sink, parameters...) which corresponds to: sink.put(parameters...) The two forms are fairly interchangeable, made more so by the Uniform Function Call Syntax. Makes perfect sense. This is the kind of stuff I'd like you to put in a formal document and present to us as an RFC along with your PR. Then transfer that information to the spec to accompany the DMD PR, after you've received feedback. Mike
Re: Is @safe still a work-in-progress?
On 8/21/2018 6:07 PM, Mike Franklin wrote: The proposed idea wants to make the first parameter, if it's `ref`, special. This is because Phobos is written with functions of the form: void put(sink, parameters...) which corresponds to: sink.put(parameters...) The two forms are fairly interchangeable, made more so by the Uniform Function Call Syntax. > Why not the first `ref` parameter regardless of whether it's the absolute first in the list. Why not the last `ref` parameter? Why not all `ref` parameters? Good question. If this fairly restricted form solves the problems, then there is no need for the more flexible form. Things can always be made more flexible in the future, but tightening things can be pretty disruptive. Hence, unless there is an obvious and fairly strong case case for the flexibility, then it should be avoided for now. But what bothers me the most is I think it's missing the bigger picture: D needs a way to annotate lifetimes. Maybe `scope` and `return` with weird conditions based on the order of parameters and their attributes are the way to go. Maybe there's another way that hasn't yet been considered. Put together a thorough description of the proposal, justify it, ask the larger community for comment, vet it, and document it. At least that's what it's going to take to get me to take action on the PR. dip1000 has been around for two years, and its predecessor dip25 several years now. Plenty of time for anyone to comment and/or propose something better. --- I want to ensure Atila is successful with this. But that means Phobos has to compile with dip1000. So I need to make it work.
Re: Is @safe still a work-in-progress?
On 8/21/2018 8:58 PM, Nicholas Wilson wrote: On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote: The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. Kagamin just told me I needed to use `return` instead of `scope` to get things to work and I'm still not sure why. The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`. Another way to think about it is this: S s; return &s; We all know that is an error. The idea is to have a way to express that for: S s; return s.foo(); and: S s; return foo(&s); so that the compiler knows that the return value of foo() is attached to the lifetime of s. Pretty much everything flows from that.
Re: Is @safe still a work-in-progress?
On 8/21/2018 2:17 PM, Atila Neves wrote: Well, no. The syntax isn't the same for member functions. The examples from the actual DIP don't compile. There it says: --- scope can be applied to function return values (even though it is not a type qualifier). It must be applied to the left of the declaration, in the same way ref is: scope int* foo(); // applies to return value Except: --- struct MyStruct { scope int* foo() scope; } foo.d(1): Error: redundant attribute scope --- Meaning the first `scope` actually applies to `this`. Writing this out as a non-member function won't help me declare member functions! I still don't know how to return a ref/pointer that's scoped. And I thought I'd written code that did that. Maybe I did. I'm very confused. Here's how you make it work: --- @safe: struct MyStruct { ref int foo() return; int* bar() return; } ref int sun() { MyStruct s; return s.foo(); // returning s.foo() escapes a reference to local variable s } int* moon() { MyStruct s; return s.bar(); // returning s.bar() escapes a reference to local variable s } --- In effect, the 'return' on 'foo()' says: The return value of foo() contains the address of 'this', and if the return escapes the scope of what 'this' is a ref to, then it's an error.
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 03:58:42 UTC, Nicholas Wilson wrote: On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote: The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. Kagamin just told me I needed to use `return` instead of `scope` to get things to work and I'm still not sure why. The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`. I guess my problem is that DIP1000 talks about returning scope values and they don't seem to actually exist in the implementation.
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 04:49:15 UTC, Mike Franklin wrote: On Wednesday, 22 August 2018 at 04:23:52 UTC, Jonathan M Davis wrote: The reality of the matter is that the DIP system is a formal way to propose language changes in order to convince Walter and Andrei that those changes should be implemented, whereas if Walter or Andrei writes the DIP, they're already convinced. This isn't a democracy. Walter is the BDFL, and it's his call. So, I really don't think that it's hypocritical Walter and Andrei need to have their ideas vetted by the community, not in an effort to convince anyone, but for quality assurance, to ensure they're not overlooking something. It is hypocritical an arrogant to believe that only our ideas have flaws and require scrutiny. The formal DIP process was put in place after DIP1000. I would even daresay that the process was put in place because of the issue with DIP1000 (the rigorously checked DIP's are all >1000 for that reason).
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 02:18:15 UTC, Mike Franklin wrote: Something else that rubs me the wrong way is that DIP 1000 is currently in a status of `DRAFT`: https://github.com/dlang/DIPs/blob/master/DIPs/README.md What the heck is going on here? We're adding features to the compiler and modifying Phobos in production releases based on a `DRAFT` proposal? Furthermore, I find it hypocritical that some of us are put through a disproportionately burdensome DIP process requiring thorough documentation, multiple peer reviews, excessive delays, and judgment that defaults to "no" for some of the most minute changes to the language, but a game-changing feature like DIP 1000 can just be amended on a whim. +1, For valuable contributors like you, it's far from friendly. Friendly atmosphere increases creativity. Although, for an industrial-level language, it should be careful to add many features (A core engine that can simulate a lot of easy to use sugars is important, for those potential hackers. )
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 05:39:05 UTC, Mike Franklin wrote: I understand that Walter's DIPs have been put through the process just like the others, but with regard to the specific issue in this thread (https://issues.dlang.org/show_bug.cgi?id=19097), the accompanying PR (https://github.com/dlang/dmd/pull/8504), and similarly undocumented PR (https://github.com/dlang/dmd/pull/8408), they are amending DIP 1000 and DIP 25 "under the table". There is no accompanying PR to the specification, no formal rational, no RFC from the community, etc... A 3rd example (https://github.com/dlang/dmd/pull/8346) to throw a little more salt on the wound. The DIP25/1000 rabbit hole is deepening behind the curtain. Mike
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 05:04:25 UTC, Mike Parker wrote: Whatever the status of DIP 1000, I would point out that that one of Walter's DIPs is in Community Review right now after sitting in the PR queue in Draft Review for a while. Once this review stage is done, it will go back into the queue to await Final Review like any other DIP. The only difference between this and other DIPs is that there will be no Formal Assessment for it. And this is the second DIP from Walter that has gone through the process since I've come on board, with DIP 1008 being put on pause at his request. I understand that Walter's DIPs have been put through the process just like the others, but with regard to the specific issue in this thread (https://issues.dlang.org/show_bug.cgi?id=19097), the accompanying PR (https://github.com/dlang/dmd/pull/8504), and similarly undocumented PR (https://github.com/dlang/dmd/pull/8408), they are amending DIP 1000 and DIP 25 "under the table". There is no accompanying PR to the specification, no formal rational, no RFC from the community, etc... Yet I and others have to go through the DIP process for much less significant changes to the language, and rightly so: https://github.com/dlang/dmd/pull/7310 https://github.com/dlang/dmd/pull/7079 All I want to see from Walter is: 1) a sufficiently documented proposal for his idea 2) an RFC from the community 3) a PR to the spec documenting the final design I don't think that's too much to ask. Mike
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 04:49:15 UTC, Mike Franklin wrote: It is hypocritical an arrogant to believe that only our ideas have flaws and require scrutiny. Sorry, that was poorly stated and conveyed the wrong intent. It should read: It is hypocritical an arrogant to believe that only our ideas should require thorough documentation and scrutiny. Mike
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 02:18:15 UTC, Mike Franklin wrote: Furthermore, I find it hypocritical that some of us are put through a disproportionately burdensome DIP process requiring thorough documentation, multiple peer reviews, excessive delays, and judgement that defaults to "no" for some of the most minute changes to the language, but a game-changing feature like DIP 1000 can just be amended on a whim. DIP 1000 is in a bit of limbo at the moment. When I took over the process from Mihails, he told me it was stalled and that the proposal did not match the implementation. So I haven't touched it, which is why it's still marked as Draft. At some point, Walter will revise it to match the implementation and then we'll discuss how to handle it. Whatever the status of DIP 1000, I would point out that that one of Walter's DIPs is in Community Review right now after sitting in the PR queue in Draft Review for a while. Once this review stage is done, it will go back into the queue to await Final Review like any other DIP. The only difference between this and other DIPs is that there will be no Formal Assessment for it. And this is the second DIP from Walter that has gone through the process since I've come on board, with DIP 1008 being put on pause at his request.
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 04:23:52 UTC, Jonathan M Davis wrote: The reality of the matter is that the DIP system is a formal way to propose language changes in order to convince Walter and Andrei that those changes should be implemented, whereas if Walter or Andrei writes the DIP, they're already convinced. This isn't a democracy. Walter is the BDFL, and it's his call. So, I really don't think that it's hypocritical Walter and Andrei need to have their ideas vetted by the community, not in an effort to convince anyone, but for quality assurance, to ensure they're not overlooking something. It is hypocritical an arrogant to believe that only our ideas have flaws and require scrutiny. Mike
Re: Is @safe still a work-in-progress?
On Tuesday, August 21, 2018 8:18:15 PM MDT Mike Franklin via Digitalmars-d wrote: > On Wednesday, 22 August 2018 at 01:07:28 UTC, Mike Franklin wrote: > > But what bothers me the most... > > Something else that rubs me the wrong way is that DIP 1000 is > currently in a status of `DRAFT`: > https://github.com/dlang/DIPs/blob/master/DIPs/README.md > > What the heck is going on here? We're adding features to the > compiler and modifying Phobos in production releases based on a > `DRAFT` proposal? > > Furthermore, I find it hypocritical that some of us are put > through a disproportionately burdensome DIP process requiring > thorough documentation, multiple peer reviews, excessive delays, > and judgement that defaults to "no" for some of the most minute > changes to the language, but a game-changing feature like DIP > 1000 can just be amended on a whim. The reality of the matter is that the DIP system is a formal way to propose language changes in order to convince Walter and Andrei that those changes should be implemented, whereas if Walter or Andrei writes the DIP, they're already convinced. This isn't a democracy. Walter is the BDFL, and it's his call. So, I really don't think that it's hypocritical, but I also do think that DIP 1000 probably should have gone through more peer review. From what I can tell, outside of the simple cases, pretty much everyone has a really hard time understanding it. The situation will likely improve once Phobos properly supports it, and you can more reasonably use it, but the whole thing defnitely seems to be overly complicated given what it's supposed to be doing and what benefits you get from it. Personally, I think that it seems pretty reasonable as long as user-defined types don't get involved, but once they do, it's a mess. - Jonathan M Davis
Re: Is @safe still a work-in-progress?
On Tuesday, 21 August 2018 at 14:31:02 UTC, Atila Neves wrote: The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. Kagamin just told me I needed to use `return` instead of `scope` to get things to work and I'm still not sure why. The way I think about it is if you have a function that takes a pointer, any pointer, and either returns it or a pointer derived from it (dereferencing or indexing) that argument must be marked `return`. In your case it was a pointer derived from `this` so `return` must be applied to `this`.
Re: Is @safe still a work-in-progress?
On Wednesday, 22 August 2018 at 01:07:28 UTC, Mike Franklin wrote: But what bothers me the most... Something else that rubs me the wrong way is that DIP 1000 is currently in a status of `DRAFT`: https://github.com/dlang/DIPs/blob/master/DIPs/README.md What the heck is going on here? We're adding features to the compiler and modifying Phobos in production releases based on a `DRAFT` proposal? Furthermore, I find it hypocritical that some of us are put through a disproportionately burdensome DIP process requiring thorough documentation, multiple peer reviews, excessive delays, and judgement that defaults to "no" for some of the most minute changes to the language, but a game-changing feature like DIP 1000 can just be amended on a whim. Mike
Re: Is @safe still a work-in-progress?
On Tuesday, 21 August 2018 at 21:17:25 UTC, Atila Neves wrote: I don't have merge rights. I took a look anyway and it mostly looks ok, but I'm not familiar enough with that part of the codebase. It's not the implementation that's preventing it from being merged. It's the idea itself, weak rationale, lack of peer review, lack of consideration for alternatives, and lack of documentation supporting it. It reeks of "designed on a whim to quickly patch some oversight in the design DIP1000 while trying to get Phobos to compile with -dip1000". With the proposed idea, order of parameters matters. We'll need to establish a convention that return parameters must be declared first, which is opposite of https://dlang.org/phobos/std_algorithm_mutation.html#copy. Is that a good idea? Maybe it is. We haven't vetted the design yet, so I'm not sure. Why haven't we vetted the design? Because there currently isn't one; there's just an informal back-of-the-napkin memo uploaded to a bugzilla entry. The proposed idea wants to make the first parameter, if it's `ref`, special. Why not the first `ref` parameter regardless of whether it's the absolute first in the list. Why not the last `ref` parameter? Why not all `ref` parameters? But what bothers me the most is I think it's missing the bigger picture: D needs a way to annotate lifetimes. Maybe `scope` and `return` with weird conditions based on the order of parameters and their attributes are the way to go. Maybe there's another way that hasn't yet been considered. Put together a thorough description of the proposal, justify it, ask the larger community for comment, vet it, and document it. At least that's what it's going to take to get me to take action on the PR. Or maybe someone else is willing to just rubber stamp it in the interest of expediency. Mike
Re: Is @safe still a work-in-progress?
On Tuesday, 21 August 2018 at 19:36:39 UTC, Walter Bright wrote: On 8/21/2018 7:31 AM, Atila Neves wrote: The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. That's what I was saying :-) The way to deal with this is make a copy of the code, then rewrite it so it does the exact same thing, but with pointers and refs only. No member functions, no delegates, no dynamic arrays. Then it is MUCH MUCH easier to see where the annotations go. Well, no. The syntax isn't the same for member functions. The examples from the actual DIP don't compile. There it says: --- scope can be applied to function return values (even though it is not a type qualifier). It must be applied to the left of the declaration, in the same way ref is: scope int* foo(); // applies to return value Except: --- struct MyStruct { scope int* foo() scope; } foo.d(1): Error: redundant attribute scope --- Meaning the first `scope` actually applies to `this`. Writing this out as a non-member function won't help me declare member functions! I still don't know how to return a ref/pointer that's scoped. And I thought I'd written code that did that. Maybe I did. I'm very confused. BTW, the annotations do not break things. The worst that will happen is the compiler will complain in @safe code that they are incorrect, and you'll need to fix it or make it @trusted. The compiler is also pretty good about inferring the correct annotations, at least for templates and lambdas, which helps enormously. In my opinion, the worst that can happen is I successfully compile my @safe code with -dip1000 and the resulting binary isn't memory-safe, which is what's been happening to me. However, dip1000 not working with Phobos is a huge impediment to success, and so pulling 8504 is critical. I don't have merge rights. I took a look anyway and it mostly looks ok, but I'm not familiar enough with that part of the codebase.
Re: Is @safe still a work-in-progress?
On 8/21/2018 7:31 AM, Atila Neves wrote: The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. That's what I was saying :-) The way to deal with this is make a copy of the code, then rewrite it so it does the exact same thing, but with pointers and refs only. No member functions, no delegates, no dynamic arrays. Then it is MUCH MUCH easier to see where the annotations go. Kagamin just told me I needed to use `return` instead of `scope` to get things to work and I'm still not sure why. Also: destructors? Always `scope`? Sometimes? I just add `scope` when the compiler complains at this point. I think it's interesting that when I played with Rust I didn't have problems fighting the borrow checker at all. DIP1000 is supposed to have the same safety without the Rust complications but currently Rust is much easier to understand for me. It doesn't help that the current implementation of -dip1000 doesn't match the document it's supposedly based on. All good points. But I cannot make any progress when nobody is willing to pull my PRs that improve the situation. https://github.com/dlang/dmd/pull/8504 BTW, the annotations do not break things. The worst that will happen is the compiler will complain in @safe code that they are incorrect, and you'll need to fix it or make it @trusted. The compiler is also pretty good about inferring the correct annotations, at least for templates and lambdas, which helps enormously. However, dip1000 not working with Phobos is a huge impediment to success, and so pulling 8504 is critical.
Re: Is @safe still a work-in-progress?
On Tuesday, 21 August 2018 at 00:09:03 UTC, Walter Bright wrote: On 8/20/2018 6:46 AM, Steven Schveighoffer wrote: I would, but I have no idea how dip1000 is supposed to work. I think only you understand it. Even looking at the PR that you have been citing over and over, I can't make heads or tails of what it does or what it allows. The way to understand dip1000 is to forget about `this`, arrays, structs, classes, delegates, etc. Just think about pointers - functions that take pointers as arguments, and return pointers. That simplifies things enormously. Once that makes sense, then deconstruct the higher level constructs into pointers, and then you'll see how they work. This is why my presentation does it all in terms of pointers: http://dconf.org/2017/talks/bright.html If you're having trouble understanding a particular example, rewrite it in terms of pointers. If it still is inscrutable, then ask about the pointer version here. (When someone gives me some complex thing and says "I don't understand scope here", I always first rewrite it in terms of pointers. It's the same thing I do with bug reports that use templates - I manually instantiate the templates first.) The problem is that the code we write doesn't deal directly with pointers - see the recent confusion in this forum over where `scope` on the left applies to the `this` pointer or the one returned by the member function. Kagamin just told me I needed to use `return` instead of `scope` to get things to work and I'm still not sure why. Also: destructors? Always `scope`? Sometimes? I just add `scope` when the compiler complains at this point. I think it's interesting that when I played with Rust I didn't have problems fighting the borrow checker at all. DIP1000 is supposed to have the same safety without the Rust complications but currently Rust is much easier to understand for me. It doesn't help that the current implementation of -dip1000 doesn't match the document it's supposedly based on.
Re: Is @safe still a work-in-progress?
On 8/20/2018 6:46 AM, Steven Schveighoffer wrote: I would, but I have no idea how dip1000 is supposed to work. I think only you understand it. Even looking at the PR that you have been citing over and over, I can't make heads or tails of what it does or what it allows. The way to understand dip1000 is to forget about `this`, arrays, structs, classes, delegates, etc. Just think about pointers - functions that take pointers as arguments, and return pointers. That simplifies things enormously. Once that makes sense, then deconstruct the higher level constructs into pointers, and then you'll see how they work. This is why my presentation does it all in terms of pointers: http://dconf.org/2017/talks/bright.html If you're having trouble understanding a particular example, rewrite it in terms of pointers. If it still is inscrutable, then ask about the pointer version here. (When someone gives me some complex thing and says "I don't understand scope here", I always first rewrite it in terms of pointers. It's the same thing I do with bug reports that use templates - I manually instantiate the templates first.)
Re: Is @safe still a work-in-progress?
On 8/20/2018 6:08 AM, Atila Neves wrote: As mentioned in my other thread, I went to file one and it was declared not a bug: https://issues.dlang.org/show_bug.cgi?id=17927 I'll look at it again.
Re: Is @safe still a work-in-progress?
On Monday, August 20, 2018 7:46:48 AM MDT Steven Schveighoffer via Digitalmars-d wrote: > I would, but I have no idea how dip1000 is supposed to work. I think > only you understand it. Even looking at the PR that you have been citing > over and over, I can't make heads or tails of what it does or what it > allows. The basic idea of DIP 1000 seems simple enough, but once you get into the details, it gets hard to understand fast. And from what work I've done to make std.datetime work with it better, I'm very much inclined to think that it's just too intrusive for too little gain. I mean, it's great that you can know that a pointer isn't going to escape, but once you have to start marking up member functions all over the place just so that the type can work with scope? It just seems like it's too viral. I honestly do not understand why much of anything in std.datetime.systime needs scope, but I was forced to mark up SysTime all over the place in order for scope variables of type SysTime to work just because it contains a class reference - one which cannot posssibly ever be on the stack, since you can't pass a scope argument to a member function to have it then stored in a member variable. So, why it has any effect on why scope is required on member functions, I have no idea. User-defined types can't contain anything that's scope (since that would mean that they escaped), and yet, they potentially have to be marked up with scope all over the place in case someone qualifies a variable of that type with scope somewhere. And of course, because of how stuff like ranges work, you're quickly in trouble if they're marked with scope, since then they can't be wrapped by other ranges, making them useless for pretty much anything other than foreach. scope seems great for toy examples or stuff that's extern(C), but once user-defined types get involved, it's a mess. I've got partially completed changes to std.datetime.interval towards making it work with scope better that I need to get back to, but to be honest, the more I've tried to do anything with -dip1000, the less I've like it, and it's really hard to be motivated to go to the effort of making anything compatible with it. - Jonathan M Davis
Re: Is @safe still a work-in-progress?
On 8/17/18 11:04 PM, Walter Bright wrote: On 8/17/2018 11:17 AM, bachmeier wrote: This is a good example of D needing to evolve or fizzle out. I don't see evidence that the community has yet figured out how to evolve the language. If it had, these problems would not be around for so many years. We deprecate features of D all the time. (Remember the D1 => D2 wrenching change?) Hm... if you are going for "all the time", the example of D1 to D2 transition is pretty dated. I'd say more like the addition of UDAs was a big evolution. Or maybe UFCS. The reason @safe cannot be default at the moment it because -dip1000 needs work, and nobody is willing to pitch in and review/pull my PRs on it. I would, but I have no idea how dip1000 is supposed to work. I think only you understand it. Even looking at the PR that you have been citing over and over, I can't make heads or tails of what it does or what it allows. -Steve
Re: Is @safe still a work-in-progress?
On Saturday, 18 August 2018 at 02:17:01 UTC, Walter Bright wrote: On 8/17/2018 4:33 AM, Atila Neves wrote: I've been using -dip1000 a lot lately. Great news! :) I hit two bugs yesterday. In D? or in your code? In dmd. Bugs don't count if they're not in bugzilla! As mentioned in my other thread, I went to file one and it was declared not a bug: https://issues.dlang.org/show_bug.cgi?id=17927 I don't know how that's possible given that supposedly @safe code is doing pretty bad things. That disheartened me enough to not continue although I did file this one: https://issues.dlang.org/show_bug.cgi?id=19173
Re: Is @safe still a work-in-progress?
On 8/17/2018 5:30 AM, Atila Neves wrote: It's not easy though. You have to either be building your own phobos Not really. You only have to be able to import phobos files. You don't have to compile something that will link. I.e. just copy the phobos source tree, add @safe to it, and point the compiler at that one. or sudo editing files in /usr/include. You can, but it's a pain. Then there's finding out exactly where in a chain of 10 template functions that it became @system... I recently fixed the compiler to say which template overload was used. It's a huge time saver. But you're currently doomed on this anyway, as an awful lot of Phobos winds up calling put(), which cannot be inferred as @safe without: https://github.com/dlang/dmd/pull/8504
Re: Is @safe still a work-in-progress?
On 8/17/2018 11:17 AM, bachmeier wrote: This is a good example of D needing to evolve or fizzle out. I don't see evidence that the community has yet figured out how to evolve the language. If it had, these problems would not be around for so many years. We deprecate features of D all the time. (Remember the D1 => D2 wrenching change?) The reason @safe cannot be default at the moment it because -dip1000 needs work, and nobody is willing to pitch in and review/pull my PRs on it.
Re: Is @safe still a work-in-progress?
On 8/17/2018 4:33 AM, Atila Neves wrote: I've been using -dip1000 a lot lately. Great news! I hit two bugs yesterday. In D? or in your code? When it works, it's great, _except_: . @safe isn't default . -dip1000 isn't default . Good luck figuring out why your template functions aren't @safe and coaxing the compiler to tell you why it's inferring the attributes it is. I empathize with that, having run into the same problem. A couple months ago I improved the resulting error messages, which make it a lot easier to track down why it is inferring what it does. Bonus points if it's a Phobos function so you can't slap `@safe` on its definition. Sometimes I think it is better to just slap them with @safe and to hell with unsafe parameters. :-) Did I mention the bugs? Bugs don't count if they're not in bugzilla!
Re: Is @safe still a work-in-progress?
On 8/17/2018 3:01 AM, Jonathan M Davis wrote: we still don't have a real plan as to how we're going to make -dip1000 the default. We do have a plan. Get Phobos to compile with -dip1000. As to why that hasn't happened yet, see: https://github.com/dlang/dmd/pull/8504
Re: Is @safe still a work-in-progress?
On 8/17/2018 12:50 AM, Jonathan M Davis wrote: > That particular bug is a duplicate of > https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed > based on the fact that -dip1000 fixes the problem by treating marking the > slice of a static array with scope. It's still quite broken without -dip1000 > though. D will never be @safe until -dip1000 is the default. However, I cannot get any traction with improving this: https://github.com/dlang/dmd/pull/8504 blocks progress on getting Phobos to compile with -dip1000, and nobody will help me with it. > Honestly, the reality of the matter is that @safe is probably always going > to be somewhat broken, because it's implemented via blacklisting rather than > whitelisting. Instead of @safe only allowing stuff that's been proven to be > @safe, it disallows stuff that a programmer decided was @system. The bug you > ran into is a pretty glaring one that arguably should have been fixed ages > ago, It was fixed with -dip1000 ages ago. > but given how hard it is to prove what is and isn't @safe, there are > bound to be corner cases which have been missed. As we find them, they'll be > fixed, but who knows how many are left or whether we'll ever actually get > them all. The whitelisting idea has come up before. I see it, though, as a way to avoid dealing with the issues as nobody is willing to do any work at all on this approach. Furthermore, I see no evidence that whitelisting will produce superior results. On the other hand, I see plenty of evidence that @safe bugs that are found and posted to bugzilla get fixed. If people want to see @safe be watertight, 1. post any problems to bugzilla, and tag them with the 'safe' keyword. 2. investigate solutions to any issues tagged with 'safe' keyword. 3. review/approve PR 8504.
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 16:00:26 UTC, 12345swordy wrote: On Friday, 17 August 2018 at 15:27:22 UTC, Mike Franklin wrote: I actually started writing a DIP for this about a year ago, but I need to pick my battles. Mike Is it on github? Alex No,but here are some notes I found in my files. @safe by default Before Transition Write a program that parses D code and explicitly adds @system if no function is not decorated with any safety attribute Add a transitional -explicit-safety compiler switch that emits a warning if a safety attribute is not explicitly specified. Update all dlang repositories and maybe others with explicit attributes using aforementioned program. Turn on -explicit-safety for all CIs so any new PRs so explicit safety is added to all new functions. Add transitional @system-by-default compiler switch for backward compatibility This will be especially useful to keep our CIs running with legacy code This will also be useful for programs with dub dependencies that need to continue running Add -@system-main compiler switch for backward compatibility Add -@safe-main and -@safe-by-default compiler switch for forward-looking D projects Update all of D's tutorials, examples, etc… with explicit safety attributes. Transition Add changelog entry warning that @safe-by-default is coming and how they can use the aforementioned tools eliminate disruption Run this change log for 1 year (4 releases)? After Transition Remove @safe attributes in D's example code. Remove @safe attributes from phobos/druntime/dmd etc.. Turn off -explicit-safety for those libraries that have updated their code. Deprecate @safe-main and @safe-by-default compiler switch through the normal deprecation process. -explicit-safety, @system-main and @system-by-default remain for an indeterminate amount of time. They should eventually be deprecated, but only after they have outlived their usefulness. Difficulties The difficulty will be with printed material (books, etc…) that will become out-of-date. Recommendation is to add information to their published errata, or create a new edition. Mike
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 17:59:45 UTC, H. S. Teoh wrote: As opposed to today's situation, where somebody tries something illegal and the compiler fails to detect it, and nobody even know there's a problem until it becomes an exploitable bug that causes a security problem. Given today's security climate, this would be utterly disastrous for D's reputation. This is a good example of D needing to evolve or fizzle out. I don't see evidence that the community has yet figured out how to evolve the language. If it had, these problems would not be around for so many years.
Re: Is @safe still a work-in-progress?
On Fri, Aug 17, 2018 at 05:26:48PM +, jmh530 via Digitalmars-d wrote: > On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote: > > [...] > > And that is exactly why the whole implementation of @safe is > > currently rather laughable. By blacklisting rather than > > whitelisting, we basically open the door wide open to loopholes -- > > anything that we haven't thought of yet could potentially be a > > @safe-breaking combination, and we wouldn't know until somebody > > discovers and reports it. [...] > Fundamentally, I see it as a good idea. Walter has talked about how > important memory safety is for D. People thinking their @safe code is > safe is a big problem when that turns out to not be the case. Imagine > the black eye D would have if a company was hacked because of > something like this? Indeed. > IMO, the problem is that you can't just replace @safe as it is now. > You could introduce something like @whitelist or @safewhitelist and > begin implementing it, but it would probably be some time before it > could replace @safe. Like when @whitelist is only breaking unsafe > code. The way I envision it is that the implementation would begin as a separate topic branch in the dmd repo, and gradually brought up to the point where it begins passing the testsuite. The good thing is that nowadays with our CI system put in place, any breakages caused in major D projects like vibe.d would be detected by the autotester, so we can immediately fill in any obvious missing whitelist items. Then we'd merge it into master but controlled by a command-line switch, say -dip if we put it under a DIP, and put out for 1 or 2 releases. People who depend on @safe can then opt in to try it out, and report any bugs (missing whitelist items) that they find. We fix those bugs until no more bugs are reported. Then we make -dip the default, perhaps with a reverse switch for reverting to the old @safe implementation just in case some projects out there depend on it but haven't gotten around to reporting bugs in the new implementation or whatever. After another release or so, we finally remove the switch and delete the old implementation. Then going forward, we will have eliminated @safe loopholes pretty much completely (the only exception being codegen bugs), and the only likely bugs are missing items in the whitelist. Which should be much more manageable and much less disastrous -- the worst that can happen is that somebody tried to do something legal and got rejected by a big fat compiler error. As opposed to today's situation, where somebody tries something illegal and the compiler fails to detect it, and nobody even know there's a problem until it becomes an exploitable bug that causes a security problem. Given today's security climate, this would be utterly disastrous for D's reputation. T -- It is of the new things that men tire --- of fashions and proposals and improvements and change. It is the old things that startle and intoxicate. It is the old things that are young. -- G.K. Chesterton
Re: Is @safe still a work-in-progress?
On 8/17/18 1:26 PM, jmh530 wrote: On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote: [...] And that is exactly why the whole implementation of @safe is currently rather laughable. By blacklisting rather than whitelisting, we basically open the door wide open to loopholes -- anything that we haven't thought of yet could potentially be a @safe-breaking combination, and we wouldn't know until somebody discovers and reports it. Sadly, it seems there is little interest in reimplementing @safe to use whitelisting instead of blacklisting. T Fundamentally, I see it as a good idea. Walter has talked about how important memory safety is for D. People thinking their @safe code is safe is a big problem when that turns out to not be the case. Imagine the black eye D would have if a company was hacked because of something like this? This will always be a possibility thanks to @trusted. IMO, the problem is that you can't just replace @safe as it is now. You could introduce something like @whitelist or @safewhitelist and begin implementing it, but it would probably be some time before it could replace @safe. Like when @whitelist is only breaking unsafe code. I have to say, I don't see how this all helps. In theory, black-listing and white-listing will get you to the same position. Mechanisms to get or use pointers aren't really being added to the language any more, so the set of "things" to "list" either black or white is finite. In this thread, we are talking about something that should have been black-listed LONG ago, but was not because it was "too useful" (i.e. would break too much code). If @safe code was white-listed, nobody would use it until it was finished, so it would be theoretical anyway. Nobody wants a feature that is @safe, but not useful. However, a bigger problem is that we have a bug that is "fixed" (slicing static arrays) but only if you use a feature that doesn't work correctly (dip1000). Why? I think the bug should be reopened until dip1000 is the default, or it simply gets fixed (i.e. without requiring dip1000). -Steve
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote: [...] And that is exactly why the whole implementation of @safe is currently rather laughable. By blacklisting rather than whitelisting, we basically open the door wide open to loopholes -- anything that we haven't thought of yet could potentially be a @safe-breaking combination, and we wouldn't know until somebody discovers and reports it. Sadly, it seems there is little interest in reimplementing @safe to use whitelisting instead of blacklisting. T Fundamentally, I see it as a good idea. Walter has talked about how important memory safety is for D. People thinking their @safe code is safe is a big problem when that turns out to not be the case. Imagine the black eye D would have if a company was hacked because of something like this? IMO, the problem is that you can't just replace @safe as it is now. You could introduce something like @whitelist or @safewhitelist and begin implementing it, but it would probably be some time before it could replace @safe. Like when @whitelist is only breaking unsafe code.
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 15:27:22 UTC, Mike Franklin wrote: I actually started writing a DIP for this about a year ago, but I need to pick my battles. Mike Is it on github? Alex
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 14:26:07 UTC, H. S. Teoh wrote: On Fri, Aug 17, 2018 at 01:50:32AM -0600, Jonathan M Davis via Digitalmars-d wrote: [...] Honestly, the reality of the matter is that @safe is probably always going to be somewhat broken, because it's implemented via blacklisting rather than whitelisting. Instead of @safe only allowing stuff that's been proven to be @safe, it disallows stuff that a programmer decided was @system. Sigh: https://issues.dlang.org/show_bug.cgi?id=12941 This was reported 4 years ago, but was unfortunately closed as invalid. It will continue to be a problem as long as @safe is implemented via blacklisting, because every single time there's a new language feature, there's a chance that a loophole is introduced into @safe. And that's not counting the combinatorial explosion of existing language features that might lead to @safe loopholes, that we simply haven't thought of yet. It's like allowing anyone to enter your house freely except those few people whom you've explicitly named. You can't possibly expect *not* to get robbed that way. I knew there was something fundamentally wrong with @safe, but I could never put my finger on it. Now that you and Jonathan mention this, it becomes clear. This makes me exceptionally sad. D is great in so many ways, but then this taints the pool. I asked if D was ever going to be @safe by default at DConf (https://youtu.be/HvqsUO77FGI?t=13242), but it didn't elicit a very positive answer. It seems D is backtracking in some ways (@nogc, -betterC), trying to evolve it into something it wasn't originally envisioned to be, and now we have another one to add to the list. The bug you ran into is a pretty glaring one that arguably should have been fixed ages ago, but given how hard it is to prove what is and isn't @safe, there are bound to be corner cases which have been missed. As we find them, they'll be fixed, but who knows how many are left or whether we'll ever actually get them all. [...] And that is exactly why the whole implementation of @safe is currently rather laughable. By blacklisting rather than whitelisting, we basically open the door wide open to loopholes -- anything that we haven't thought of yet could potentially be a @safe-breaking combination, and we wouldn't know until somebody discovers and reports it. Sadly, it seems there is little interest in reimplementing @safe to use whitelisting instead of blacklisting. I think there is probably some interest, though maybe not from the ones with the position or ability to make it happen. A DIP might be the way forward, but it seems like quite a difficult task to turn it right-side-up at this point. I actually started writing a DIP for this about a year ago, but I need to pick my battles. Mike
Re: Is @safe still a work-in-progress?
On 8/17/18 7:26 AM, H. S. Teoh wrote: It will continue to be a problem as long as @safe is implemented via blacklisting, because every single time there's a new language feature, there's a chance that a loophole is introduced into @safe. And that's not counting the combinatorial explosion of existing language features that might lead to @safe loopholes, that we simply haven't thought of yet. It's like allowing anyone to enter your house freely except those few people whom you've explicitly named. You can't possibly expect *not* to get robbed that way. We should tag bugs like these as #safecracking
Re: Is @safe still a work-in-progress?
On Fri, Aug 17, 2018 at 01:50:32AM -0600, Jonathan M Davis via Digitalmars-d wrote: [...] > Honestly, the reality of the matter is that @safe is probably always > going to be somewhat broken, because it's implemented via blacklisting > rather than whitelisting. Instead of @safe only allowing stuff that's > been proven to be @safe, it disallows stuff that a programmer decided > was @system. Sigh: https://issues.dlang.org/show_bug.cgi?id=12941 This was reported 4 years ago, but was unfortunately closed as invalid. It will continue to be a problem as long as @safe is implemented via blacklisting, because every single time there's a new language feature, there's a chance that a loophole is introduced into @safe. And that's not counting the combinatorial explosion of existing language features that might lead to @safe loopholes, that we simply haven't thought of yet. It's like allowing anyone to enter your house freely except those few people whom you've explicitly named. You can't possibly expect *not* to get robbed that way. > The bug you ran into is a pretty glaring one that arguably should have > been fixed ages ago, but given how hard it is to prove what is and > isn't @safe, there are bound to be corner cases which have been > missed. As we find them, they'll be fixed, but who knows how many are > left or whether we'll ever actually get them all. [...] And that is exactly why the whole implementation of @safe is currently rather laughable. By blacklisting rather than whitelisting, we basically open the door wide open to loopholes -- anything that we haven't thought of yet could potentially be a @safe-breaking combination, and we wouldn't know until somebody discovers and reports it. Sadly, it seems there is little interest in reimplementing @safe to use whitelisting instead of blacklisting. T -- A program should be written to model the concepts of the task it performs rather than the physical world or a process because this maximizes the potential for it to be applied to tasks that are conceptually similar and, more important, to tasks that have not yet been conceived. -- Michael B. Allen
Re: Is @safe still a work-in-progress?
On 18/08/2018 1:29 AM, vit wrote: On Friday, 17 August 2018 at 13:02:20 UTC, Jonathan M Davis wrote: On Friday, August 17, 2018 6:30:26 AM MDT Atila Neves via Digitalmars-d wrote: On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote: > On 17/08/2018 11:33 PM, Atila Neves wrote: >> . Good luck figuring out why your template functions aren't >> @safe and coaxing the compiler to tell you why it's >> inferring the attributes it is. Bonus points if it's a >> Phobos function so you can't slap `@safe` on its definition. > > Sure you can. It's a template so it'll be initialized as its > required. > > I've done that to fix a bug in druntime before and in my own > libraries. It's not easy though. You have to either be building your own phobos or sudo editing files in /usr/include. You can, but it's a pain. Then there's finding out exactly where in a chain of 10 template functions that it became @system... The reality of the matter is that you tend to have to be very motivated to make your code @safe if you're doing much with templates (the same with other attributes such as pure). And sadly, it can quickly get to the point that it just feels like it's not worth it in spite of the fact that there is definitely value to be had in using them. I wonder what it would take to make it so that the compiler could actually tell you where in the function call chain it first becomes @system. - Jonathan M Davis Does -dip1000 affect alias template parameters and closures? example: import std.experimental.all; void main()@safe @nogc{ int i = 2; const x = iota(0, 10) .filter!((x)scope => x == i) .map!((x)scope => x * x) .front; assert(x == 4); } onlineapp.d(4): Error: function `D main` is @nogc yet allocates closures with the GC onlineapp.d(8): onlineapp.main.__lambda1 closes over variable i at onlineapp.d(5) No it shouldn't. You used i there which automatically requires a heap allocation for a delegate. Hence it uses GC.
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 13:02:20 UTC, Jonathan M Davis wrote: On Friday, August 17, 2018 6:30:26 AM MDT Atila Neves via Digitalmars-d wrote: On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote: > On 17/08/2018 11:33 PM, Atila Neves wrote: >> . Good luck figuring out why your template functions aren't >> @safe and coaxing the compiler to tell you why it's >> inferring the attributes it is. Bonus points if it's a >> Phobos function so you can't slap `@safe` on its definition. > > Sure you can. It's a template so it'll be initialized as its > required. > > I've done that to fix a bug in druntime before and in my own > libraries. It's not easy though. You have to either be building your own phobos or sudo editing files in /usr/include. You can, but it's a pain. Then there's finding out exactly where in a chain of 10 template functions that it became @system... The reality of the matter is that you tend to have to be very motivated to make your code @safe if you're doing much with templates (the same with other attributes such as pure). And sadly, it can quickly get to the point that it just feels like it's not worth it in spite of the fact that there is definitely value to be had in using them. I wonder what it would take to make it so that the compiler could actually tell you where in the function call chain it first becomes @system. - Jonathan M Davis Does -dip1000 affect alias template parameters and closures? example: import std.experimental.all; void main()@safe @nogc{ int i = 2; const x = iota(0, 10) .filter!((x)scope => x == i) .map!((x)scope => x * x) .front; assert(x == 4); } onlineapp.d(4): Error: function `D main` is @nogc yet allocates closures with the GC onlineapp.d(8):onlineapp.main.__lambda1 closes over variable i at onlineapp.d(5)
Re: Is @safe still a work-in-progress?
On Friday, August 17, 2018 6:30:26 AM MDT Atila Neves via Digitalmars-d wrote: > On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote: > > On 17/08/2018 11:33 PM, Atila Neves wrote: > >> . Good luck figuring out why your template functions aren't > >> @safe and coaxing the compiler to tell you why it's inferring > >> the attributes it is. Bonus points if it's a Phobos function > >> so you can't slap `@safe` on its definition. > > > > Sure you can. It's a template so it'll be initialized as its > > required. > > > > I've done that to fix a bug in druntime before and in my own > > libraries. > > It's not easy though. You have to either be building your own > phobos or sudo editing files in /usr/include. You can, but it's a > pain. Then there's finding out exactly where in a chain of 10 > template functions that it became @system... The reality of the matter is that you tend to have to be very motivated to make your code @safe if you're doing much with templates (the same with other attributes such as pure). And sadly, it can quickly get to the point that it just feels like it's not worth it in spite of the fact that there is definitely value to be had in using them. I wonder what it would take to make it so that the compiler could actually tell you where in the function call chain it first becomes @system. - Jonathan M Davis
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 11:37:54 UTC, rikki cattermole wrote: On 17/08/2018 11:33 PM, Atila Neves wrote: . Good luck figuring out why your template functions aren't @safe and coaxing the compiler to tell you why it's inferring the attributes it is. Bonus points if it's a Phobos function so you can't slap `@safe` on its definition. Sure you can. It's a template so it'll be initialized as its required. I've done that to fix a bug in druntime before and in my own libraries. It's not easy though. You have to either be building your own phobos or sudo editing files in /usr/include. You can, but it's a pain. Then there's finding out exactly where in a chain of 10 template functions that it became @system...
Re: Is @safe still a work-in-progress?
On 17/08/2018 11:33 PM, Atila Neves wrote: . Good luck figuring out why your template functions aren't @safe and coaxing the compiler to tell you why it's inferring the attributes it is. Bonus points if it's a Phobos function so you can't slap `@safe` on its definition. Sure you can. It's a template so it'll be initialized as its required. I've done that to fix a bug in druntime before and in my own libraries.
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 08:23:20 UTC, vit wrote: On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote: On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via Digitalmars-d wrote: [...] That particular bug is a duplicate of https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed based on the fact that -dip1000 fixes the problem by treating marking the slice of a static array with scope. It's still quite broken without -dip1000 though. [...] What's the state of -dip1000? I've been using -dip1000 a lot lately. I hit two bugs yesterday. When it works, it's great, _except_: . @safe isn't default . -dip1000 isn't default . Good luck figuring out why your template functions aren't @safe and coaxing the compiler to tell you why it's inferring the attributes it is. Bonus points if it's a Phobos function so you can't slap `@safe` on its definition. Take excel-d for instance. The other day somebody filed a bug because Excel was crashing when they wrapped some D code that uses std.regex. After investigating I found out that regex memoizes the string with the regular expression in it, effectively escaping the string passed from Excel to the D function. Unfortunately for everyone involved, that string came from a region allocator, and after bumping the pointer back, boom. Telling people to not escape reference parameters doesn't really work, because nobody will read that part of the README, and, as in the example above, the user didn't even know they were doing it! Ok, so as a good library author, I tried to make it a compile-time error to shoot yourself in the foot that way, and I did. Great, right? Well, no. I can't force the user to write @safe functions, much less to compile with -dip1000. I changed the example XLL to use -dip1000 because more likely than not users will copy the example and edit it. But I can't force them to write @safe functions, and if they don't (the likeliest scenario given it's not the default), they'll be able to escape whatever it is they want, Excel will crash, and the compiler will be ok with it. And if they _do_ write @safe functions and compile with -dip1000, chances are at some point they won't understand the compiler error message, make their function @trusted and... crash. I don't know that because I have a crystal ball, I know that because I did it myself this week. Twice. Did I mention the bugs? I wrote about my interesting experiences from this week with -dip1000 here: https://forum.dlang.org/post/lxavagebcmaxjdzis...@forum.dlang.org
Re: Is @safe still a work-in-progress?
On Friday, August 17, 2018 2:23:20 AM MDT vit via Digitalmars-d wrote: > On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote: > > On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via > > > > Digitalmars-d wrote: > >> [...] > > > > That particular bug is a duplicate of > > https://issues.dlang.org/show_bug.cgi?id=8838, which was closed > > as fixed based on the fact that -dip1000 fixes the problem by > > treating marking the slice of a static array with scope. It's > > still quite broken without -dip1000 though. > > > > [...] > > What's the state of -dip1000? Well, I _think_ that aside from bugs (which may or may not be a small number), the compiler implementation is complete. However, there's still a fair bit of work to do in Phobos to make it fully work with -dip1000. And AFAIK, we still don't have a real plan as to how we're going to make -dip1000 the default. But given how much -dip1000 seems to break, I don't know how we're going to switch to it being the default without being highly disruptive in the process. - Jonathan M Davis
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote: On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via Digitalmars-d wrote: [...] That particular bug is a duplicate of https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed based on the fact that -dip1000 fixes the problem by treating marking the slice of a static array with scope. It's still quite broken without -dip1000 though. [...] What's the state of -dip1000?
Re: Is @safe still a work-in-progress?
On Friday, 17 August 2018 at 07:50:32 UTC, Jonathan M Davis wrote: That particular bug is a duplicate of https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed based on the fact that -dip1000 fixes the problem by treating marking the slice of a static array with scope. It's still quite broken without -dip1000 though. It still appears to be broke even with -dip1000: https://run.dlang.io/is/gJi2Fa
Re: Is @safe still a work-in-progress?
On Friday, August 17, 2018 1:19:25 AM MDT Peter Alexander via Digitalmars-d wrote: > > import std.algorithm, std.stdio; > > @safe: > > auto foo() { >int[6] xs = [0, 1, 2, 3, 4, 5]; >return xs[].map!(x => x); > } > > void main() { >writeln(foo()); > } > > > https://run.dlang.io/is/qC7HUR > > For me this gives: > > > [0, 0, -2132056872, 22008, 0, 0] > > > Which looks like its just reading arbitrary memory. > > I've filed https://issues.dlang.org/show_bug.cgi?id=19175 > > My question is: what is the status of @safe? I am quite surprised > to see such a simple case fail. Is @safe believed to be fully > implemented (modulo bugs) and this is just an unfortunate corner > case, or is it known work-in-progress? That particular bug is a duplicate of https://issues.dlang.org/show_bug.cgi?id=8838, which was closed as fixed based on the fact that -dip1000 fixes the problem by treating marking the slice of a static array with scope. It's still quite broken without -dip1000 though. Honestly, the reality of the matter is that @safe is probably always going to be somewhat broken, because it's implemented via blacklisting rather than whitelisting. Instead of @safe only allowing stuff that's been proven to be @safe, it disallows stuff that a programmer decided was @system. The bug you ran into is a pretty glaring one that arguably should have been fixed ages ago, but given how hard it is to prove what is and isn't @safe, there are bound to be corner cases which have been missed. As we find them, they'll be fixed, but who knows how many are left or whether we'll ever actually get them all. - Jonathan M Davis
Is @safe still a work-in-progress?
import std.algorithm, std.stdio; @safe: auto foo() { int[6] xs = [0, 1, 2, 3, 4, 5]; return xs[].map!(x => x); } void main() { writeln(foo()); } https://run.dlang.io/is/qC7HUR For me this gives: [0, 0, -2132056872, 22008, 0, 0] Which looks like its just reading arbitrary memory. I've filed https://issues.dlang.org/show_bug.cgi?id=19175 My question is: what is the status of @safe? I am quite surprised to see such a simple case fail. Is @safe believed to be fully implemented (modulo bugs) and this is just an unfortunate corner case, or is it known work-in-progress?