Re: DIP69 - Implement scope for escape proof references
On Monday, 22 December 2014 at 20:51:46 UTC, Walter Bright wrote:
On 12/22/2014 12:04 AM, Dicebot wrote:
Point of transitive scope is to make easy to expose complex
custom data
structures without breaking memory safety.
I do understand that. Making it work with the type system is
another matter entirely - it's far more complex than just
adding a qualifier. 'inout' looks simple but still has ongoing
issues.
And the thing is, wrappers can be used instead of qualifiers,
in the same places in the same way. It's much simpler.
Ok, lets consider some examples of code I'd want to work with
wrapper approach.
Basic tree structure:
struct Node
{
Node* left, right;
Payload payload;
}
struct Tree
{
Node* root;
}
Restricting public access to nodes to prevent storing persistent
node pointers.
Your proposed approach:
struct Wrapper(T)
{
scope T wrapped;
alias wrapped this;
}
struct Node
{
private:
Node* _left, _right;
public:
auto left() @property { return Wrapper!(Node*)(_left); }
auto right() @property { return Wrapper!(Node*)(_right); }
Payload payload;
}
struct Tree
{
private:
Node* _root;
public:
auto root() @property { return Wrapper!(Node*)(_root); }
}
Am I correct?
If yes, this is exactly the breakage issue I was speaking about :
any code that looked like `is(typeof(Tree.root) == Node*)` will
stop working.
With my latest proposal (pseudo-transitive storage class which is
not part of a type) getters would look simply like `scope Node*
root() @property { return _root; }` and `is(typeof(Tree.root) ==
Node*)` still remained true. Only impact on existing would be
restriction of operations to take address or assign a pointer.
I am pretty sure I am missing many complicated parts here but
this seems like a big deal in retro-fitting existing libraries to
use scope after it was added.
Re: DIP69 - Implement scope for escape proof references
On Monday, 22 December 2014 at 03:07:53 UTC, Walter Bright wrote: On 12/21/2014 2:06 AM, Dicebot wrote: No, it is exactly the other way around. The very point of what I am saying is that you DOESN'T CARE about ownership as long as worst case scenario is assumed. I have zero idea why you identify it is conflating as ownership when it is explicitly designed to be distinct. The point of transitive scoping would be if the root owned the data reachable through the root. Quoting myself: For me scopeness is a property of view, not object itself - this also makes ownership method of actual data irrelevant. Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available. It doesn't matter of root owns the data. We _assume_ that as worst case scenario and allowed actions form a strict subset of allowed actions for any other ownership situation. Such `scope` for stack/GC is same as `const` for mutable/immutable - common denominator. Point of transitive scope is to make easy to expose complex custom data structures without breaking memory safety.
Re: DIP69 - Implement scope for escape proof references
On 12/22/2014 12:04 AM, Dicebot wrote: Point of transitive scope is to make easy to expose complex custom data structures without breaking memory safety. I do understand that. Making it work with the type system is another matter entirely - it's far more complex than just adding a qualifier. 'inout' looks simple but still has ongoing issues. And the thing is, wrappers can be used instead of qualifiers, in the same places in the same way. It's much simpler.
Re: DIP69 - Implement scope for escape proof references
On Saturday, 20 December 2014 at 21:39:44 UTC, Walter Bright wrote: On 12/19/2014 9:44 PM, Dicebot wrote: Such notion of view requires at least some elements of transitivity to be practical in my opinion. I have no idea how some elements of transitivity can even work. It's either transitive or its not. Please don't think of scope in terms of ownership, ownership is an orthogonal issue. .. and here I was about you to do exactly the same :P What in example I show makes you think of ownership? When I was speaking about some elements of transitivity I was thinking in a way of keeping scope storage class but transitively applying same restrictions to all data accessible through it AS IF it had scope storage class on its own - while still making illegal to use scope as a separate type qualifier. Also with my definition in mind your example of tree that stores scope nodes makes absolutely no sense unless whole tree itself is scoped (and nodes are thus scoped transitively). Such view is always assumes worst case about ownership and shouldn't persist in any form (as that would require some serious ownership tracking). This is definitely conflating scope and ownership. No, it is exactly the other way around. The very point of what I am saying is that you DOESN'T CARE about ownership as long as worst case scenario is assumed. I have zero idea why you identify it is conflating as ownership when it is explicitly designed to be distinct.
Re: DIP69 - Implement scope for escape proof references
On 12/21/2014 2:06 AM, Dicebot wrote: No, it is exactly the other way around. The very point of what I am saying is that you DOESN'T CARE about ownership as long as worst case scenario is assumed. I have zero idea why you identify it is conflating as ownership when it is explicitly designed to be distinct. The point of transitive scoping would be if the root owned the data reachable through the root.
Re: DIP69 - Implement scope for escape proof references
On 12/19/2014 9:44 PM, Dicebot wrote: Such notion of view requires at least some elements of transitivity to be practical in my opinion. I have no idea how some elements of transitivity can even work. It's either transitive or its not. Please don't think of scope in terms of ownership, ownership is an orthogonal issue. Also with my definition in mind your example of tree that stores scope nodes makes absolutely no sense unless whole tree itself is scoped (and nodes are thus scoped transitively). Such view is always assumes worst case about ownership and shouldn't persist in any form (as that would require some serious ownership tracking). This is definitely conflating scope and ownership.
Re: DIP69 - Implement scope for escape proof references
On 12/6/14 4:49 PM, Manu via Digitalmars-d wrote: I need, at least, forceinline to complete it, but that one*is* controversial - we've talked about this for years. I'm still 883 messages behind so here's a drive-by comment - it's time to revisit this, I think the need has become a lot clearer. -- Andrei
Re: DIP69 - Implement scope for escape proof references
On 12/6/14 4:49 PM, Manu via Digitalmars-d wrote:
In the situation where templates are involved, it would be nice to be
able to make that explicit statement that some type is ref or not at
the point of template instantiation, and the resolution should work
according to the well-defined rules that we are all familiar with.
Another drive-by comment: I understand the motivation for this and the
difficulties involved. There needs to be a clear understanding that
adding new type qualifiers is extremely intrusive and expensive. Because
of that, I think we should best address binding generation via a set of
tactical tools i.e. standard library artifacts that do all that mixin
business in an encapsulated and reusable manner.
(As an aside forcing a template instantiation to decide ref vs. no ref
should be easy but currently can't be done, i.e. this code should work
but currently doesn't:
T fun(T)(ref T x) { return x + 1; }
void main(string[] group)
{
int function(int) f1 = fun!int;
int function(ref int) f2 = fun!int;
}
)
Andrei
Re: DIP69 - Implement scope for escape proof references
On Tuesday, 16 December 2014 at 01:00:47 UTC, Walter Bright wrote: On 12/15/2014 5:38 AM, deadalnix wrote: On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote: For me scopeness is a property of view, not object itself - this also makes ownership method of actual data irrelevant. Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available. Ha finally something start to make sense here. Well, the DIP does defined scope in terms of a 'view' in just this manner. I am obviously terrible at explaining things. Such notion of view requires at least some elements of transitivity to be practical in my opinion. Also with my definition in mind your example of tree that stores scope nodes makes absolutely no sense unless whole tree itself is scoped (and nodes are thus scoped transitively). Such view is always assumes worst case about ownership and shouldn't persist in any form (as that would require some serious ownership tracking). I don't think you have explained your case that bad - we simply have a very different use cases in mind as primary use case.
Re: DIP69 - Implement scope for escape proof references
On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile wrote: Walter Bright: I'm afraid I don't understand at all what you wrote. Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_systemredirect=no#Linear_type_systems But note that we want a relaxed linear type system. In general, we are fine with multiple aliases, although there are some applications where strict uniqueness is useful. Deadalnix' proposal goes in this direction.
Re: DIP69 - Implement scope for escape proof references
On Wednesday, 17 December 2014 at 11:13:02 UTC, Marc Schütz wrote: On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile wrote: Walter Bright: I'm afraid I don't understand at all what you wrote. Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_systemredirect=no#Linear_type_systems But note that we want a relaxed linear type system. In general, we are fine with multiple aliases, although there are Pointer aliasing is an optimization killer, which is why C99 added the «restrict» keyword. I think in most cases the use of stack allocated objects tend to be alias-free, so it is important that this knowledge is retained so that the compiler can make use of it. http://en.wikipedia.org/wiki/Pointer_aliasing
Re: DIP69 - Implement scope for escape proof references
On Wednesday, 17 December 2014 at 05:28:35 UTC, Walter Bright wrote: On 12/16/2014 5:37 AM, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: The current proposal is either too limiting or not limiting enough. I'm afraid I don't understand at all what you wrote. Well, look at it this way: D1 took 1990s style C++ application programming model and made it more convenient with a GC. Having a GC-memory model with the possibility of stack/heap allocations as optimizations can keep the language simple. D2 is more of a library author's language, but without GC you need to deal with different types of ownership. If you toss out the GC, and want the same level of safety and generality you either: 1. need to track ownership add dynamic runtime checks 2. provide a full blown proof system (not realistic) 3. provide means for the programmer to help the semantic analysis What you want is a combination of 1 and 3, but you don't need to keep them separate. Since refcounting is expensive, you don't want that everywhere. For stack allocated objects you have an implicit refcounting in the nesting of scopes, you only need to track scope-depth. If the compiler knows that a returned object is stack allocated, then it can check that by comparing against the stack frame address since the stack grows downwards in an ordered fashion. And in most cases it should be able to elide the check. If you also maintain information about order (stack depth) among the stack allocated objects a function receives through reference parameters (for every function call), then you can safely combine the stack allocated objects you receive through parameters in aggregates, like inserting an stack allocated element into a container. Since D is increasingly becoming a library author's language, you probably also would benefit more from strengthening the template support by having a templated ref-type rather than scope ref. You could define protocols (e.g. predefined UDAs) that tells the compiler what kind of key properties the library type provides, such as ownership. Then use that for a more generic approach to optimization and escape analysis and let the ownership-reference-types be library types. If semantic analysis gets rid of the dynamic aspects of a referencs (such as reference counting) then the compiler should be able to use the same function body, and in other cases just add a inferred wrapper to a generic function body. So the code bloat can be limited. If all references to external resources (like the heap) in the stack allocated object is owned by the object, then you can treat those as a whole. But you need the compiler to understand what ownership is if it is to guarantee memory safety in the kind of situations where memory safety is known to be difficult to achieve. The alternative is to state that @safe is not covering non-GC memory and that one should stick to vetted library code when writing @safe code that does not use the GC. I actually find that acceptable for D2. In D2 you would be better off spending the effort at finding ways to turn automatically turn GC allocations into stack allocations as optimizations. Then you can find a more general and powerful ownership-oriented approach for non-GC applications for a new major version of D. It is not realistic that people will annotate with scope ref, and compiler inference could make updates to libraries break application code if you get false negatives. Function signatures are contracts, they ought to be explicit for libraries. So having library functions act as scope ref without it being explicit in the signature is not necessarily a good idea.
Re: DIP69 - Implement scope for escape proof references
On Wednesday, 17 December 2014 at 12:29:21 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 17 December 2014 at 11:13:02 UTC, Marc Schütz wrote: On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile wrote: Walter Bright: I'm afraid I don't understand at all what you wrote. Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_systemredirect=no#Linear_type_systems But note that we want a relaxed linear type system. In general, we are fine with multiple aliases, although there are Pointer aliasing is an optimization killer, which is why C99 added the «restrict» keyword. I think in most cases the use of stack allocated objects tend to be alias-free, so it is important that this knowledge is retained so that the compiler can make use of it. http://en.wikipedia.org/wiki/Pointer_aliasing I know, this is one of those applications. The one I was thinking of is const-borrowing (useful to prevent iterator invalidation and a bunch of other things). I think that enforcing alias-freeness by default (the way a strict linear type system requires) doesn't fit too well into the existing language. Some way of signalling this to the compiler will therefore be required. It might be as simple as C's `restrict` keyword (but it needs to be enforced!), or better yet, an `unrestrict` keyword. Parameters would be alias-free by default unless explicitly specified otherwise. Local aliases (inside one function) are however ok, as long as not more than one is passed to another function at the same time.
Re: DIP69 - Implement scope for escape proof references
On Wednesday, 17 December 2014 at 12:29:21 UTC, Ola Fosheim Grøstad wrote: On Wednesday, 17 December 2014 at 11:13:02 UTC, Marc Schütz wrote: On Wednesday, 17 December 2014 at 07:48:52 UTC, bearophile wrote: Walter Bright: I'm afraid I don't understand at all what you wrote. Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_systemredirect=no#Linear_type_systems But note that we want a relaxed linear type system. In general, we are fine with multiple aliases, although there are Pointer aliasing is an optimization killer, which is why C99 added the «restrict» keyword. I think in most cases the use of stack allocated objects tend to be alias-free, so it is important that this knowledge is retained so that the compiler can make use of it. http://en.wikipedia.org/wiki/Pointer_aliasing If you have owner ship, you have free. If you have a pair alloc/free then you can promote on stack. This is a much more powerful way to handle things, as this take advantage of inlining.
Re: DIP69 - Implement scope for escape proof references
On Wednesday, 17 December 2014 at 13:48:24 UTC, deadalnix wrote: If you have owner ship, you have free. If you have a pair alloc/free then you can promote on stack. This is a much more powerful way to handle things, as this take advantage of inlining. More powerful than what? Rather than thinking in terms of alloc/free it might be helpful to think about it more generally as resources, use, reuse and side-effects. On the stack you get to reuse memory when the variable is not longer live, meaning you junk writes if the results cannot be read and reuse the memory prematurely. http://en.wikipedia.org/wiki/Live_variable_analysis If you are saying that turning GC allocations automatically into stack allocations is a good direction for D2, then I agree.
Re: DIP69 - Implement scope for escape proof references
On Wednesday, 17 December 2014 at 13:35:53 UTC, Marc Schütz wrote: I think that enforcing alias-freeness by default (the way a strict linear type system requires) doesn't fit too well into the existing language. Some way of signalling this to the compiler will therefore be required. It might be as simple as C's `restrict` keyword (but it needs to be enforced!), or better yet, an `unrestrict` keyword. Parameters would be alias-free by default unless explicitly specified otherwise. Local aliases (inside one function) are however ok, as long as not more than one is passed to another function at the same time. Hmm, I kind of feel that D is having too many parameter keywords/types/qualifiers. Parameter specification is primarily useful where it provides information about constraints that the programmer need to know about when using a library. restricted scope ref could be a useful default for non-templated functions, then use templated ownership reference typing or inference to go beyond it. I think that if one improved the templating system so that it was capable of analysing similarities between instantiations and use heuristics to analyse when a new instance is profitable then you could make functions templated by default and propagate knowledge from the call site to the template-instantiation mechanism of the compiler. So if the reference is known to be aliasfree then the compiler can use heuristics to determine if it should instantiate a restricted version or use a slower general version of the templated function. Similarily, if refcounters always are on offset 0 and destructors are virtual, then you only need a single container method template instance for refcouting references, even if you have many types using the same method. If refcounters are big enough, you can have a unique_ptr that just set it to a large value and use the same template instance for a unique_ptr and refcounted_ptr behind the scene when referencing objects that provide a refcount field. There are also many other things that could be done to enhance the template system, like querying properties of the block in loops that iterate over a range, so that you can instantiate different optimized ranges depending on whether the loop contains a break statement or not. Or like Jonathan Blow's language, where you can issue a remove element and continue like statement within a foreach-loop, which might require a different range iteration implementation.
Re: DIP69 - Implement scope for escape proof references
On Tuesday, 16 December 2014 at 01:00:47 UTC, Walter Bright wrote: On 12/15/2014 5:38 AM, deadalnix wrote: On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote: For me scopeness is a property of view, not object itself - this also makes ownership method of actual data irrelevant. Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available. Ha finally something start to make sense here. Well, the DIP does defined scope in terms of a 'view' in just this manner. I am obviously terrible at explaining things. No, I actually think the concept of view in the proposal is a good one. However, the proposal itself is too limiting.
Re: DIP69 - Implement scope for escape proof references
On Tuesday, 16 December 2014 at 01:00:47 UTC, Walter Bright wrote: On 12/15/2014 5:38 AM, deadalnix wrote: On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote: For me scopeness is a property of view, not object itself - this also makes ownership method of actual data irrelevant. Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available. Ha finally something start to make sense here. Well, the DIP does defined scope in terms of a 'view' in just this manner. I am obviously terrible at explaining things. Don't you think there is a contradiction between this notion of view and being so focused on lifetimes? Does it make sense to explicitly throw away lifetime information and then spend a lot of effort on preventing lifetime-violating assignments to take place? The current proposal is either too limiting or not limiting enough. Too limiting, because programmers expect to gain something for making parameter types explicit. A naive make no assumptions view is more suitable as a default, then let the programmer specify the specifics where possible/needed, but you need to figure out the specifics first and let the make no assumptions view default be the leftovers… The proposal is not limiting enough, because retaining knowledge about allocation order and aliasing brings important benefits: An ordered linear parameter type would have properties such as: - being aliasfree - being on the stack, and cheaply checkable against the stack pointer Sure, ref counting is fairly general for dynamic tracking, but it is also quite expensive. A check against the stack-pointer is much cheaper. I think the uncanny valley metaphor that was brought up was quite good. If you are going to put in constraints on references, then do it in a manner that is easy to deal with and which brings benefits such as aliasfree references. If constraints become arbitrary then usability will suffer and it will seem pointless to put in extra work to use them. You have already gone more or less wholesale for templates, so you might as well use it here as well, to bring substantial benefits to stack-allocation, because the current proposal is a very weak in terms of benefits.
Re: DIP69 - Implement scope for escape proof references
On 12/16/2014 5:37 AM, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: The current proposal is either too limiting or not limiting enough. I'm afraid I don't understand at all what you wrote.
Re: DIP69 - Implement scope for escape proof references
Walter Bright: I'm afraid I don't understand at all what you wrote. Perhaps reading about linear type systems could help: http://en.wikipedia.org/w/index.php?title=Substructural_type_systemredirect=no#Linear_type_systems Bye, bearophile
Re: DIP69 - Implement scope for escape proof references
On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote: C++ seems to do fine without it for const. It's a convenience feature. C++ const does not really restrict or affect anything for real, it is non-existent feature. `scope` as proposed would result in inability to store result of predicates if those are ever to accept scope data - unless you defined two versions for each function that may possibly accept scope data with absolutely identical body. 2. This is what scope inference is all about. Which only works with templates and lack of scope on arguments does not affect function body - templates are not necessary, same as inout. It also works with all the lambdas, since source for them is always available. I also wanted to make it (i.e. inference) work with auto functions, but Don Clugston was the primary objector :-) This is not really answering my objections but side-stepping. I do support attribute inference. I would actually support full attribute inference in the language (and help convincing Don about it ;)) if it was introduced as foundation of the language and not arbitrary hack (== rethink the way .di interfaces and static libraries are defined) I do view inference as something we need to extend to more functions. That would help with many issues but it need careful design to be well-accepted. How about teaming up to do a DIP about it at some point where this discussion is over? :P 2) at least optional transitivity to be able to express to protect with scope data referenced by slice or owned linked list referenced from root node. 1. that won't work unless scope is a type constructor I know and this is why I am leaning toward it being qualifier despite all related issues. That would be a truly massive change to D, and I'm not at all sure it would be worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and idea that had seemed straightforward. Yes, I know and this why I am leaning towards that path but willing to accept any storage-class based solution - as long as it fits my basic criteria. I wonder if some hybrid approach is possible - for example, keep it storage class but pretend anything transitively accessible though it as if it was of scope storage class on its own. But that quickly gets into exceptions for special cases land :( 2. it can be achieved by using wrappers that only allow by-scope references to their data (RC is an example of such a wrapper) It is viral approach and backwards incompatible one - we can't change signatures of Phobos functions to return Wrapped!(char[]) instead of char[] for example. While theoretically possible I can't call this approach practical - not until see some convincing application example. But you can change signatures to add annotations? Such annotation would only prevent from compiling code which was already undefined by documentation. Changing return type (which wasn't Voldemort type before) may break any program that has lines akin to `char[] data = foo()` - perfectly legal working code. This sounds totally against my understanding of scope. I want scope exactly to prohibit such actions. However it is possible in slightly modified way - where you don't directly insert pointer to AST but use public method of AST control structure that checks if supplied scope pointer belongs to list of nodes it owns and casts away scope after that. Wrappers do that, too. I know that 'wrapper' sounds bad to you, but one of the goals of D's UDT's is to make it easy to adapt the behavior of a type by wrapping it in a struct, rather than build new behavior into the language. Problem is that our UDT are simply not that good enough yet. Wrappers are fine in general but I want to wrap for less common cases and support more common ones natively. I suppose this is your intention too but we disagree on what is actually the common case and what is the special case.
Re: DIP69 - Implement scope for escape proof references
On Sunday, 14 December 2014 at 23:27:15 UTC, Walter Bright wrote:
I think I said that clearly: What do you get when you take a
pointer
of a function with an auto-ref
arg? ...an error, because it's not actually a function!
What's baffling to me is why even declare a function pointer
parameter as auto-ref? I don't use floating point to index
strings, either, that doesn't mean floating point is a failed
concept.
That's not what he's talking about. My understanding is that Manu
is talking about something like this:
void foo(auto ref someConcreteType) { ... }
auto fooP = foo; //error, foo is not a function
Re: DIP69 - Implement scope for escape proof references
On Monday, 15 December 2014 at 01:44:27 UTC, Dicebot wrote: On Saturday, 13 December 2014 at 00:10:07 UTC, Walter Bright wrote: The proposal provides escape proof passing of arguments. This is necessary in order to make rc safe, for example. What are you looking for? I am looking for a tool to prevent escaping references to user-defined entities / resources from specific scope. RC case is not interesting at all to me though I recognize the potential benefits of tweaking the scope system to help with it. But there is a big difference between tweak for one specific use case or designing feature around it (so it becomes almost useless in other cases). I have already provided you an example of code I want to be enforced with scope and two major issues with existing proposal that make it lacking. This example is akin to litmus test for any scope proposal - if it doesn't allow to express such design, that proposal is simply not good enough. In case you have forgotten, I am reminding about two critical points that are necessary to make it fly: 1) inout analog for scope to be able to deduce borrowship of return values based on borrowship of input arguments. Current system is conservative and results in template bloat (complicated by the fact that it is storage class thus not actually part of a type) Is it be possible to make `inout` a more general tool for type erasure? I don't think yet another ad-hoc mechanism should be introduced. 2) at least optional transitivity to be able to express to protect with scope data referenced by slice or owned linked list referenced from root node. -- In your tree example I would have never wanted scope protection of one specific node of such tree - but a transitive scope protection of whole tree view available through on of node pointers/references. It doesn't matter who and how owns the data for borrowship implementation - only thing that matters that _it is not me_. I think there are two cases that are relevant. 1) the tree nodes don't own their children, instead they are managed by - let's say - a region allocator, and 2) the tree nodes do own their children. In both cases, they can declare the children as scope members (assuming a proposal that allows that). The only case I can think of where the children should not be scope members is when they are GC managed. But in this case, you don't need to worry about references to them escaping. Therefore, I don't see a use case for transitivity of scope, from a conceptional point of view. If a particular design makes transitivity necessary, this points to a flaw in the design, IMO.
Re: DIP69 - Implement scope for escape proof references
On Monday, 15 December 2014 at 11:23:28 UTC, Marc Schütz wrote: I think there are two cases that are relevant. 1) the tree nodes don't own their children, instead they are managed by - let's say - a region allocator, and 2) the tree nodes do own their children. In both cases, they can declare the children as scope members (assuming a proposal that allows that). The only case I can think of where the children should not be scope members is when they are GC managed. But in this case, you don't need to worry about references to them escaping. Therefore, I don't see a use case for transitivity of scope, from a conceptional point of view. If a particular design makes transitivity necessary, this points to a flaw in the design, IMO. Actually I think in neither case children should be declared as scope members. Instead those should be declared as private members but methods that return pointers/references to children would be declared to return (transitive) scope pointers/references. For me scopeness is a property of view, not object itself - this also makes ownership method of actual data irrelevant. Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available.
Re: DIP69 - Implement scope for escape proof references
On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote: On 12/14/2014 7:41 PM, Dicebot wrote: Being forced to duplicate every single function in two flavors to actually make scope system usable? This is as much critical as it can be. C++ seems to do fine without it for const. It's a convenience feature. It's more than that. It has the potential (well, could have) to be used for type erasure, which is a useful and worthwhile concept by itself. I know and this is why I am leaning toward it being qualifier despite all related issues. That would be a truly massive change to D, and I'm not at all sure it would be worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and idea that had seemed straightforward. To be honest, I think one reason for those bugs is the awkward implementation. I had a look at mtype.c to see how difficult it would be to add another type modifier. It's no fun - the current implementation requires a dozen groups of helper functions to be written for every possible combination of type modifiers. Therefore there are helpers like `makeConst`, `makeImmutable`, `makeShared`, `makeSharedConst`, `makeWild` (this is `inout`), `makeWildConst`, `makeSharedWild`, `makeSharedWildConst`, and `makeMutable`. There are also some monster switch statements in the same fashion. No wonder this is error prone. I suspect that with a cleaner implementation most of these problems will go away.
Re: DIP69 - Implement scope for escape proof references
On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote: On Monday, 15 December 2014 at 11:23:28 UTC, Marc Schütz wrote: I think there are two cases that are relevant. 1) the tree nodes don't own their children, instead they are managed by - let's say - a region allocator, and 2) the tree nodes do own their children. In both cases, they can declare the children as scope members (assuming a proposal that allows that). The only case I can think of where the children should not be scope members is when they are GC managed. But in this case, you don't need to worry about references to them escaping. Therefore, I don't see a use case for transitivity of scope, from a conceptional point of view. If a particular design makes transitivity necessary, this points to a flaw in the design, IMO. Actually I think in neither case children should be declared as scope members. Instead those should be declared as private members but methods that return pointers/references to children would be declared to return (transitive) scope pointers/references. I see this as the same thing, conceptually (module the transitive in in your parentheses). It's just a different mechanism. Scope members are just a short-cut instead of writing an accessor method that returns scope. For me scopeness is a property of view, not object itself - this also makes ownership method of actual data irrelevant. Yes, and this would be true for scope members just the same. (If the aggregate owns it's scope members, it will however get in trouble when it tries to free() them, therefore your mechanism is more appropriate in this case.) Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available. Agreed.
Re: DIP69 - Implement scope for escape proof references
On Saturday, 13 December 2014 at 20:45:53 UTC, Walter Bright
wrote:
On 12/13/2014 2:10 AM, bearophile wrote:
Walter Bright:
struct Tree {
RefCount!(Tree*) left;
RefCount!(Tree*) right;
...
}
... I don't think I'd ever have a use for this code.
You have no use for tree structures?
Giving a reference counter to every pointer in a binary tree
sounds a bit too much.
DMD, for example, uses ASTs where the nodes have multiple
parents.
It is a DAG then.
Re: DIP69 - Implement scope for escape proof references
On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote: For me scopeness is a property of view, not object itself - this also makes ownership method of actual data irrelevant. Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available. Ha finally something start to make sense here.
Re: DIP69 - Implement scope for escape proof references
On 12/15/2014 3:34 AM, Marc Schütz schue...@gmx.net wrote: No wonder this is error prone. I suspect that with a cleaner implementation most of these problems will go away. Sometimes, it isn't possible to see a cleaner way unless the dirty version is written first. Sometimes, there isn't a cleaner way :-( But there are more issues with inout than are in mtype.c. The template implementation is full of it.
Re: DIP69 - Implement scope for escape proof references
On 12/15/2014 1:28 AM, Dicebot wrote: On Monday, 15 December 2014 at 06:12:05 UTC, Walter Bright wrote: C++ seems to do fine without it for const. It's a convenience feature. C++ const does not really restrict or affect anything for real, it is non-existent feature. C++ programmers constantly copy/pasta functions for the sole purpose of having one with const and one without. `scope` as proposed would result in inability to store result of predicates if those are ever to accept scope data - unless you defined two versions for each function that may possibly accept scope data with absolutely identical body. If that is necessary, it is no worse than C++ is with const. It isn't a critical issue, or C++ would not be usable. I'd like to wait and see how this plays out. 2. This is what scope inference is all about. Which only works with templates and lack of scope on arguments does not affect function body - templates are not necessary, same as inout. It also works with all the lambdas, since source for them is always available. I also wanted to make it (i.e. inference) work with auto functions, but Don Clugston was the primary objector :-) This is not really answering my objections but side-stepping. Inference addresses the issue. That would help with many issues but it need careful design to be well-accepted. How about teaming up to do a DIP about it at some point where this discussion is over? :P Ok, sure.
Re: DIP69 - Implement scope for escape proof references
On 12/15/2014 5:38 AM, deadalnix wrote: On Monday, 15 December 2014 at 11:32:11 UTC, Dicebot wrote: For me scopeness is a property of view, not object itself - this also makes ownership method of actual data irrelevant. Only difference between GC owned data and stack allocated one is that former can have scoped view optionally but for the latter compiler must force it as the only available. Ha finally something start to make sense here. Well, the DIP does defined scope in terms of a 'view' in just this manner. I am obviously terrible at explaining things.
Re: DIP69 - Implement scope for escape proof references
On Sunday, 14 December 2014 at 00:45:06 UTC, Manu via Digitalmars-d wrote: On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d A function template is a function that takes both compile-time args and run-time args. C++ tried to make them completely different animals, when they are not. Don't think template, think compile-time argument to a function. I convinced Andrei to never mention template in his D book, as it brings forth all kinds of connotations and expectations that impede understanding what D templates actually are. I think the result was successful. You can spin it however you like, but it's exactly the same thing. They are completely different animals. A function template is not a function at all until it's instantiated. When it's instantiated, it becomes a function, or even, one of a suite of functions. And it's not known to the author where the function is. (Note: I define 'function' in this context to mean 'some code that is emitted') Philippe Sigaud Excellent D Templates: A Tutorial... the MANTRA... XXX templates are not XXXs, they are templates. With XXX being any of (function, struct, class, interface, union). --- /Paolo
Re: DIP69 - Implement scope for escape proof references
On Sun, 14 Dec 2014 09:04:58 + Paolo Invernizzi via Digitalmars-d digitalmars-d@puremagic.com wrote: Philippe Sigaud Excellent D Templates: A Tutorial... the MANTRA... XXX templates are not XXXs, they are templates. With XXX being any of (function, struct, class, interface, union). and that still doesn't work. ;-) as a somehow related example i recall my first expirience with C. my first serious language was Pascal, and it has that nice range-checking feature for arrays. it was very hard for me to understand that C has no arrays at all, not to say range checking. why there is no arrays if i have this nice indexing and all that?! what do you mean by that buffer overflow nonsence? if it looks like an array it must be an array! signature.asc Description: PGP signature
Re: DIP69 - Implement scope for escape proof references
On Sunday, 14 December 2014 at 00:45:06 UTC, Manu via
Digitalmars-d wrote:
Perhaps scope and RC are different things?
Solve scope purely for RC, and you've broken scope for other
purposes
that it's useful; that is, inhibiting escaping of data.
Exactly. It shouldn't matter which type this data has - both
references and value types are valid. I would define `scope` like
this:
A value designated with `scope` may not be assigned to a
variable (or parameter, ...) with a lifetime longer than its
owner's lifetime.
Maybe there's a compromise. If we say scope isn't
'transitive', but it
is transitive when it comes to aggregates?
One thing I've tried very hard to make work in D is have basic
types /
aggregages / arrays be interchangeable.
It should only be applicable to whatever is explicitly marked as
`scope`. For references, that's the reference itself, but not
what it points to. For aggregates, it's the entire aggregate with
all it's members. That way, all types will be treated uniformly.
Of course, this can only work with a type modifier, not with a
storage class.
I'm looking for ways to make scope fit your proposed mould and
be useful to me.
If I can't do this then my most common use case is unsatisfied:
struct Wrap
{
OpaqueThing *ptr;
this() {}
~this() {} // - constructors that twiddle RC effectively.
this(this) {}
this() scope {}
~this() scope {} // - Overloadable for 'scope', in which
case,
don't twiddle the RC!
this(this) scope {}
// lots of operations on OpaqueThing wrapped up here...
}
void f(scope Wrap x) -- RC twiddling is effectively elided
here due
to constructor overloads
{
}
If you can propose an alternative solution?
This is representative of my 99% case. Sometimes the struct is
more
than a single pointer though.
Unfortunately Walter rejected this when I had proposed it.
In the current proposal, we cannot overload on scope. The
proposed solution is to not borrow the wrapper, but only its
payload.
But this is restrictive; even in the case of RC, there are
use-cases that cannot be served with this technique, namely when
the callee can only decide at runtime whether it wants to make a
copy of an RC value it received. The correct solution would be to
pass the RC wrapper itself as scope, as you show in your example,
thereby delegating the decision of whether to adjust the refcount
from the caller to the callee.
Re: DIP69 - Implement scope for escape proof references
On 12/8/2014 7:23 AM, Steven Schveighoffer wrote: What I wanted to know was, when I see scope ref, why is it there? When should I use it? It's there to indicate a parameter is NOT returned by ref. When should I use scope? What nasty things are prevented if I use it? Examples would be preferable. http://wiki.dlang.org/DIP69#Ref has examples of errors that are prevented.
Re: DIP69 - Implement scope for escape proof references
On 12/13/2014 4:44 PM, Manu via Digitalmars-d wrote: On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d digitalmars-d@puremagic.com wrote: On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote: I did just give some examples, I'll repeat; auto ref fails when the function is extern. Don't make it extern, then. Do you think I just interact with other languages for fun or something? I'm not trying to insult you - but why need 'auto ref' for extern functions? The extern functions are ref or not-ref, and declare the interface to match. There's no point to auto ref there. If there's source to the function, it'll often be inlined which will remove the indirection. Not if it's extern (applicable to ref, not auto-ref), or wrapped, or if I ever want to capture a function pointer. It's also semantically different; I can change the caller's value. Again, if it is extern it has a fixed definition of ref or not. The D side doesn't decide if it's ref or not, it just follows whatever the extern declaration is. This is why I do not understand why you are running into this issue everywhere. What do you get when you take a pointer of a function with an auto-ref arg? ...an error, because it's not actually a function! So in a context where I'm dealing with functions and function pointers (very, very frequent), I suddenly have something that's not a function find it's way into my meta and I have to special-case the hell out of it. Why are function pointers and ints going to the same argument of a function? I thought you weren't using templates? I'm confused. I'm talking about capturing function pointers. Not about function arguments. Capturing pointers of functions with auto-ref args (aka, template functions) is a serious nuisance, and impossible outside the site of instantiation. WHY are you doing this? I have no idea what coding pattern would need to pass functions by reference. I wonder what is the need for the code that you are writing. General function adaptation. There are lots of reasons to wrap functions. Adaptation to existing or external API's is the most common in my experience. Are those API's all templated? My guess is no. If they're not, you don't need templates or auto-ref to interface to them. Functions are what programs are! I really struggle to understand why you have such trouble sympathising with my view here. Because you don't provide any examples of why you need this stuff. You say I need this feature because I'm doing X, but I have no idea what X is or why you need X. I.e. you present and advocate for particular solutions, but not the problems. Languages generate code, which is packaged into blocks we call 'functions'... and they require to adhere to strict ABI requirements. That is programming in a nutshell. Surely it's reasonable to want to retain complete control over this most primitive and basic of tasks. One important aspect of that control is where the code is generated; is it 'here', or 'there'? Why is that important? And that's a critical distinction between functions and templates. You can control that with function templates, too. The compiler won't instantiate a template locally if the import instantiates it. Look at LuaD. Lots of awkward cases have emerged there. There are many more to come too; the known bug list are mostly nasty issues of this nature that I've been putting off. I can't offer any insight into my commercial code sadly, and I no longer have access to it :/ Situations like this appear frequently: https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R178 And something like this tends to appear as one aspect of the solution: https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R68 'struct Ref(T)' leads to its own problems though, in that it's a localised concept. No external code anywhere understands it as a 'ref', so if 3rd party code has any special treatment for ref, that code now fails. Then more magic like this: https://github.com/JakobOvrum/LuaD/pull/76/files#diff-ec8c532aeca798240de4d70ee639fc16R90 Since we need to recognise 'ref' and substitute it for our magic 'struct Ref(T)'. I've probably spent more hours wrangling D meta of this sort than most people. This sort of thing always happens. If Ref!T is the tool that resolves the situation, then it's clear demonstration that ref should be part of the type. I'll take a look at your references in a moment. When I'm writing a function that's not a template, I intend, and expect, to write a function that's _not a template_. Templates and functions are different things. I think it's a massive mistake to have created a way to write a template that looks nothing like a template. A function template is a function that takes both compile-time args and run-time args. C++ tried to make them completely different animals, when they are not. Don't think template, think compile-time
Re: DIP69 - Implement scope for escape proof references
On Saturday, 13 December 2014 at 00:10:07 UTC, Walter Bright wrote: The proposal provides escape proof passing of arguments. This is necessary in order to make rc safe, for example. What are you looking for? I am looking for a tool to prevent escaping references to user-defined entities / resources from specific scope. RC case is not interesting at all to me though I recognize the potential benefits of tweaking the scope system to help with it. But there is a big difference between tweak for one specific use case or designing feature around it (so it becomes almost useless in other cases). I have already provided you an example of code I want to be enforced with scope and two major issues with existing proposal that make it lacking. This example is akin to litmus test for any scope proposal - if it doesn't allow to express such design, that proposal is simply not good enough. In case you have forgotten, I am reminding about two critical points that are necessary to make it fly: 1) inout analog for scope to be able to deduce borrowship of return values based on borrowship of input arguments. Current system is conservative and results in template bloat (complicated by the fact that it is storage class thus not actually part of a type) 2) at least optional transitivity to be able to express to protect with scope data referenced by slice or owned linked list referenced from root node. -- In your tree example I would have never wanted scope protection of one specific node of such tree - but a transitive scope protection of whole tree view available through on of node pointers/references. It doesn't matter who and how owns the data for borrowship implementation - only thing that matters that _it is not me_.
Re: DIP69 - Implement scope for escape proof references
On 12/14/2014 5:44 PM, Dicebot wrote: I am looking for a tool to prevent escaping references to user-defined entities / resources from specific scope. RC case is not interesting at all to me though I recognize the potential benefits of tweaking the scope system to help with it. But there is a big difference between tweak for one specific use case or designing feature around it (so it becomes almost useless in other cases). I have already provided you an example of code I want to be enforced with scope and two major issues with existing proposal that make it lacking. This example is akin to litmus test for any scope proposal - if it doesn't allow to express such design, that proposal is simply not good enough. In case you have forgotten, I am reminding about two critical points that are necessary to make it fly: 1) inout analog for scope to be able to deduce borrowship of return values based on borrowship of input arguments. Current system is conservative and results in template bloat (complicated by the fact that it is storage class thus not actually part of a type) 1. All inout actually does is reduce code copy/pasta, it is not critical. 2. This is what scope inference is all about. 2) at least optional transitivity to be able to express to protect with scope data referenced by slice or owned linked list referenced from root node. 1. that won't work unless scope is a type constructor 2. it can be achieved by using wrappers that only allow by-scope references to their data (RC is an example of such a wrapper) In your tree example I would have never wanted scope protection of one specific node of such tree - but a transitive scope protection of whole tree view available through on of node pointers/references. It doesn't matter who and how owns the data for borrowship implementation - only thing that matters that _it is not me_. As I explained to Manu, transitive scope makes things like doing a symbol table lookup, then putting a pointer to the found symbol in an AST, not workable.
Re: DIP69 - Implement scope for escape proof references
On Monday, 15 December 2014 at 02:45:04 UTC, Walter Bright wrote: 1. All inout actually does is reduce code copy/pasta, it is not critical. Being forced to duplicate every single function in two flavors to actually make scope system usable? This is as much critical as it can be. 2. This is what scope inference is all about. Which only works with templates and lack of scope on arguments does not affect function body - templates are not necessary, same as inout. 2) at least optional transitivity to be able to express to protect with scope data referenced by slice or owned linked list referenced from root node. 1. that won't work unless scope is a type constructor I know and this is why I am leaning toward it being qualifier despite all related issues. 2. it can be achieved by using wrappers that only allow by-scope references to their data (RC is an example of such a wrapper) It is viral approach and backwards incompatible one - we can't change signatures of Phobos functions to return Wrapped!(char[]) instead of char[] for example. While theoretically possible I can't call this approach practical - not until see some convincing application example. In your tree example I would have never wanted scope protection of one specific node of such tree - but a transitive scope protection of whole tree view available through on of node pointers/references. It doesn't matter who and how owns the data for borrowship implementation - only thing that matters that _it is not me_. As I explained to Manu, transitive scope makes things like doing a symbol table lookup, then putting a pointer to the found symbol in an AST, not workable. This sounds totally against my understanding of scope. I want scope exactly to prohibit such actions. However it is possible in slightly modified way - where you don't directly insert pointer to AST but use public method of AST control structure that checks if supplied scope pointer belongs to list of nodes it owns and casts away scope after that.
Re: DIP69 - Implement scope for escape proof references
On 12/14/2014 7:41 PM, Dicebot wrote: On Monday, 15 December 2014 at 02:45:04 UTC, Walter Bright wrote: 1. All inout actually does is reduce code copy/pasta, it is not critical. Being forced to duplicate every single function in two flavors to actually make scope system usable? This is as much critical as it can be. C++ seems to do fine without it for const. It's a convenience feature. 2. This is what scope inference is all about. Which only works with templates and lack of scope on arguments does not affect function body - templates are not necessary, same as inout. It also works with all the lambdas, since source for them is always available. I also wanted to make it (i.e. inference) work with auto functions, but Don Clugston was the primary objector :-) I do view inference as something we need to extend to more functions. 2) at least optional transitivity to be able to express to protect with scope data referenced by slice or owned linked list referenced from root node. 1. that won't work unless scope is a type constructor I know and this is why I am leaning toward it being qualifier despite all related issues. That would be a truly massive change to D, and I'm not at all sure it would be worth it. We've (i.e. Kenji) have been fixing bugs with inout for years, and idea that had seemed straightforward. 2. it can be achieved by using wrappers that only allow by-scope references to their data (RC is an example of such a wrapper) It is viral approach and backwards incompatible one - we can't change signatures of Phobos functions to return Wrapped!(char[]) instead of char[] for example. While theoretically possible I can't call this approach practical - not until see some convincing application example. But you can change signatures to add annotations? You're going to get one or the other. This sounds totally against my understanding of scope. I want scope exactly to prohibit such actions. However it is possible in slightly modified way - where you don't directly insert pointer to AST but use public method of AST control structure that checks if supplied scope pointer belongs to list of nodes it owns and casts away scope after that. Wrappers do that, too. I know that 'wrapper' sounds bad to you, but one of the goals of D's UDT's is to make it easy to adapt the behavior of a type by wrapping it in a struct, rather than build new behavior into the language.
Re: DIP69 - Implement scope for escape proof references
Walter Bright:
struct Tree {
RefCount!(Tree*) left;
RefCount!(Tree*) right;
...
}
... I don't think I'd ever have a use for this code.
You have no use for tree structures?
Giving a reference counter to every pointer in a binary tree
sounds a bit too much.
Bye,
bearophile
Re: DIP69 - Implement scope for escape proof references
On Saturday, 13 December 2014 at 02:55:50 UTC, Manu via Digitalmars-d wrote: Templates and functions are different things. I think it's a massive mistake to have created a way to write a template that looks nothing like a template. That is a misconception spread by C++. Templates are pure functions applied at compile time. Function templates are functions partially applied at compile time.
Re: DIP69 - Implement scope for escape proof references
On 2014-12-13 01:09, Walter Bright wrote: The proposal provides escape proof passing of arguments. This is necessary in order to make rc safe, for example. What are you looking for? Passing references to stack allocated data to functions safely is one idea that comes to mind. -- /Jacob Carlborg
Re: DIP69 - Implement scope for escape proof references
On 2014-12-13 06:11, Walter Bright wrote: Don't make it extern, then. He's said a couple of time he's interfacing with C and/or C++ code. Why are function pointers and ints going to the same argument of a function? I thought you weren't using templates? I think he meant taking the address of a function with auto ref. -- /Jacob Carlborg
Re: DIP69 - Implement scope for escape proof references
On 12/13/2014 8:01 AM, Jacob Carlborg wrote: On 2014-12-13 01:09, Walter Bright wrote: The proposal provides escape proof passing of arguments. This is necessary in order to make rc safe, for example. What are you looking for? Passing references to stack allocated data to functions safely is one idea that comes to mind. 'scope ref' does that.
Re: DIP69 - Implement scope for escape proof references
On 12/13/2014 2:10 AM, bearophile wrote:
Walter Bright:
struct Tree {
RefCount!(Tree*) left;
RefCount!(Tree*) right;
...
}
... I don't think I'd ever have a use for this code.
You have no use for tree structures?
Giving a reference counter to every pointer in a binary tree sounds a bit too
much.
DMD, for example, uses ASTs where the nodes have multiple parents.
Re: DIP69 - Implement scope for escape proof references
On 12/13/2014 8:19 AM, Jacob Carlborg wrote: On 2014-12-13 06:11, Walter Bright wrote: Don't make it extern, then. He's said a couple of time he's interfacing with C and/or C++ code. Yes, and C++ code is either ref or not ref, and it is not a problem to insert 'ref' or not, manually. Why are function pointers and ints going to the same argument of a function? I thought you weren't using templates? I think he meant taking the address of a function with auto ref. Right, but why is that even being attempted?
Re: DIP69 - Implement scope for escape proof references
On 12/13/2014 6:10 AM, Max Samukha wrote: On Saturday, 13 December 2014 at 02:55:50 UTC, Manu via Digitalmars-d wrote: Templates and functions are different things. I think it's a massive mistake to have created a way to write a template that looks nothing like a template. That is a misconception spread by C++. Templates are pure functions applied at compile time. Function templates are functions partially applied at compile time. It took me an embarrassingly long time to get past the C++ notion that templates are something orthogonal from 'normal' functions and data declarations. Even the C++ notion of a 'primary' template is wholly unnecessary, but is deeply embedded in how it handles the semantics of templates. C++ did get right the partial ordering of templates, and D applied partial ordering to 'normal' functions as well.
Re: DIP69 - Implement scope for escape proof references
On 13 December 2014 at 15:11, Walter Bright via Digitalmars-d digitalmars-d@puremagic.com wrote: On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote: I did just give some examples, I'll repeat; auto ref fails when the function is extern. Don't make it extern, then. Do you think I just interact with other languages for fun or something? It is also wrong that when I pass an int or float, it is passed by ref instead of by value... that is never what I want! If there's source to the function, it'll often be inlined which will remove the indirection. Not if it's extern (applicable to ref, not auto-ref), or wrapped, or if I ever want to capture a function pointer. It's also semantically different; I can change the caller's value. What do you get when you take a pointer of a function with an auto-ref arg? ...an error, because it's not actually a function! So in a context where I'm dealing with functions and function pointers (very, very frequent), I suddenly have something that's not a function find it's way into my meta and I have to special-case the hell out of it. Why are function pointers and ints going to the same argument of a function? I thought you weren't using templates? I'm confused. I'm talking about capturing function pointers. Not about function arguments. Capturing pointers of functions with auto-ref args (aka, template functions) is a serious nuisance, and impossible outside the site of instantiation. You need information that's only available at the point of instantiation. That changes the landscape quite a bit. The solution in this case is to wrap it in a non-auto-ref shim with the ref-ness explicitly stated in the way I expect... which is in itself another problem, because 'ref' is not part of the type! So the annoying de-auto-ref-ing shim must be a text mixin with some very complex, practically unreadable, definitely unmaintainable logic surrounding it. It's insanity on top of insanity! I also run the invisible risk of generating 2^num_args instances of the code for functions containing auto-ref args. I wonder what is the need for the code that you are writing. General function adaptation. There are lots of reasons to wrap functions. Adaptation to existing or external API's is the most common in my experience. Functions are what programs are! I really struggle to understand why you have such trouble sympathising with my view here. Languages generate code, which is packaged into blocks we call 'functions'... and they require to adhere to strict ABI requirements. That is programming in a nutshell. Surely it's reasonable to want to retain complete control over this most primitive and basic of tasks. One important aspect of that control is where the code is generated; is it 'here', or 'there'? And that's a critical distinction between functions and templates. Look at LuaD. Lots of awkward cases have emerged there. There are many more to come too; the known bug list are mostly nasty issues of this nature that I've been putting off. I can't offer any insight into my commercial code sadly, and I no longer have access to it :/ Situations like this appear frequently: https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R178 And something like this tends to appear as one aspect of the solution: https://github.com/JakobOvrum/LuaD/pull/76/files#diff-bcb370a5bc6fe75a9d5c04f2e1c17eb0R68 'struct Ref(T)' leads to its own problems though, in that it's a localised concept. No external code anywhere understands it as a 'ref', so if 3rd party code has any special treatment for ref, that code now fails. Then more magic like this: https://github.com/JakobOvrum/LuaD/pull/76/files#diff-ec8c532aeca798240de4d70ee639fc16R90 Since we need to recognise 'ref' and substitute it for our magic 'struct Ref(T)'. I've probably spent more hours wrangling D meta of this sort than most people. This sort of thing always happens. If Ref!T is the tool that resolves the situation, then it's clear demonstration that ref should be part of the type. When I'm writing a function that's not a template, I intend, and expect, to write a function that's _not a template_. Templates and functions are different things. I think it's a massive mistake to have created a way to write a template that looks nothing like a template. A function template is a function that takes both compile-time args and run-time args. C++ tried to make them completely different animals, when they are not. Don't think template, think compile-time argument to a function. I convinced Andrei to never mention template in his D book, as it brings forth all kinds of connotations and expectations that impede understanding what D templates actually are. I think the result was successful. You can spin it however you like, but it's exactly the same thing. They are completely different animals. A function template is not a function at all until it's instantiated. When it's
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 06:57:54 UTC, deadalnix wrote: On Friday, 12 December 2014 at 06:06:40 UTC, Dicebot wrote: On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright wrote: Consider a ref counted type, RC!T. If scope were transitive, then you could not have, say, a tree where the edges were RC!T. I.e., the payload of an RC type should not be forced to be scope. I don't see how this is related. It would be perfectly ok to declare root of such tree scope if it was transitive (as long as it only controls access and does not attempt early destruction). That is my proposal. However, it is not as simply as you think it is without making it a type qualifier (which is not desirable). In effect, that mean that you can see something that has infinite lifetime through a scope reference, so you need to track lifetime rvalue and lvalue differently. I'm still for it. The current proposal is not powerful enough to pull its weight. I don't hope it will be simple. It is all about making scope as simple as possible to keep it useful for idiomatic D code - but not simpler.
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote: Are you suggesting two kinds of scope - transitive and non-transitive? Non transitive scope can be added without any language extension. There is no point doing much with the scope keyword if indirection aren't in the loop. The more I think about it, the more ref counting is the definitive case, as just about everything else can be explained in terms of how ref counting works. ref counting is just one form of ownership.
Re: DIP69 - Implement scope for escape proof references
On 11 December 2014 at 23:22, ixid via Digitalmars-d digitalmars-d@puremagic.com wrote: Manu have you shown Walter some of the code where ref is problematic? He seems to have asked to see examples repeatedly as he's not convinced there is a problem. I have discussed issues here in the past, but possibly not as concretely as would be useful to this discussion. I want to take some time to contribute value to this issue, but I have virtually no time at all lately, and I can't even keep up with this posts thread, let alone sit and consider+craft a bunch of demonstration cases. This thread is moving faster than I am able to, so I'm not sure what I can do. There's no TL;DR conversation, and almost all discussion happens outside of my timezone, which wasn't such a problem when I was nocturnal, but I have had to change lifestyle recently :/ I have about 30 posts to try and catch up on and understand right now in 5 minutes before I have to leave _
Re: DIP69 - Implement scope for escape proof references
On 11 December 2014 at 23:55, via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via Digitalmars-d wrote:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough. For
example,
not
being able to represent idiom of `scope ref int foo(scope ref int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet, and which
wasn't obvious for me either, at the beginning:
* A `ref` parameter means that it cannot escape the function, _except_ by
return.
* A `scope ref` parameter means that it cannot escape the function _ever_,
not even by return.
* A `scope ref` return means that it cannot leave the current statement.
Therefore, a `scope ref` return value can be passed on to the next function
as a `ref` argument. If that function again returns a reference (even if not
explicitly designated as `scope`), the compiler will treat it as if it were
`scope ref`.
Ummm. I reckon there's a good reason that people don't seem to
understand this... because it would have difficulty being any more
unintuitive!
It's contrary to the behaviour of basically everything else in D!
const, pure, nothrow, etc... they all work a reliable and consistent way.
Also, I still don't think I get it... a scope-ref return, which has a
restriction as you say, can be subverted by being piped through a
function that receives and returns it as ref?
What is the point?
I'm also quite uneasy with the fact that scope would not be transitive
as a storage class. What happens when it's applied to a value type,
like a struct, that contains some pointers? An adaptation wrapper for
a single pointer is super common; ie, a tiny struct passed by value
with scope needs to have it's contained pointer receive the scope-ness
of the argument.
I agree, this is important. In my proposal, this works without transitivity.
The wrapper stores the pointer as a `scope` member, then by copying the
wrapper, the pointer gets copied implicitly, to which the normal scope
restrictions apply (`scope` on members means will not outlive the
aggregate). If it stored it as normal non-scope pointer, it couldn't get
assigned in the first place. (Additionally, some higher level tricks are
possible if we allow scope for value types and overloading on scope.)
How can a member be marked scope? Apparently it's a storage class, not
a type constructor... we can only attribute storage classes to
function args/results(?).
Re: DIP69 - Implement scope for escape proof references
On 12/11/2014 11:55 PM, Dicebot wrote: On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote: I don't see how this is related. It would be perfectly ok to declare root of such tree scope if it was transitive (as long as it only controls access and does not attempt early destruction). Are you suggesting two kinds of scope - transitive and non-transitive? The more I think about it, the more ref counting is the definitive case, as just about everything else can be explained in terms of how ref counting works. I don't see applicability of non-transitive scope because I don't understand the problem with the tree or reference counting you have mentioned - that is why I suggested to amend DIP to put explanation there. Consider every pointer in a tree to be a ref counted pointer. There is no purpose to making scope transitive - traveling from one node to the next goes through the usual ref counting mechanism, and the ref counting wrapper can control that. Also, consider a data structure: A - B C - B If scope is transitive, how is C going to simultaneously access B?
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 12:16 AM, deadalnix wrote: On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote: Are you suggesting two kinds of scope - transitive and non-transitive? Non transitive scope can be added without any language extension. In order for it to work, the holes in the language that enabled escapes had to be plugged. That's what this proposal does - plug the holes. ref counting is just one form of ownership. My point is if that works with this proposal, then the other forms can work, too.
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 08:44:56 UTC, Walter Bright wrote: On 12/12/2014 12:16 AM, deadalnix wrote: On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote: Are you suggesting two kinds of scope - transitive and non-transitive? Non transitive scope can be added without any language extension. In order for it to work, the holes in the language that enabled escapes had to be plugged. That's what this proposal does - plug the holes. The holes covered by a non transitive scope are already undefined behavior. ref counting is just one form of ownership. My point is if that works with this proposal, then the other forms can work, too. No, unless we wrap every single indirection (pointer, slice, classes, delegates) into a wrapper. That is not going to fly very far.
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 00:13:10 UTC, deadalnix wrote:
On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz
wrote:
This is a point that most people don't seem to understand yet,
and which wasn't obvious for me either, at the beginning:
* A `ref` parameter means that it cannot escape the function,
_except_ by return.
* A `scope ref` parameter means that it cannot escape the
function _ever_, not even by return.
* A `scope ref` return means that it cannot leave the current
statement.
Therefore, a `scope ref` return value can be passed on to the
next function as a `ref` argument. If that function again
returns a reference (even if not explicitly designated as
`scope`), the compiler will treat it as if it were `scope ref`.
No, it understood.
Steven hadn't, evidently.
It is simply not useful.
I'm not convinced either.
I agree, this is important. In my proposal, this works without
transitivity. The wrapper stores the pointer as a `scope`
member, then by copying the wrapper, the pointer gets copied
implicitly, to which the normal scope restrictions apply
(`scope` on members means will not outlive the aggregate).
If it stored it as normal non-scope pointer, it couldn't get
assigned in the first place.
Wut ? You cante store anything with grear lifetime, including
non scope things (as they'll have infinite lifetime). Meaning
the only thing you know, is that thing are possibly scoped.
Meaning you have to assume infinite lifetime with every
indirection, which make this proposal useless.
My comments above don't refer to this proposal, but my original
one:
struct Wrapper(T) {
scope T payload;
// used to be:
// scope!this T payload;
}
scope int a;
auto w = Wrapper!(int*)(a); // ok
scope int b;
w.payload = b; // error, w (and therefore
w.payload)
// lives longer than b
Wrapper!(int*) w2 = w; // ok, lifetime(w2) lifetime(w)
w = w2; // error, equivalent to
// w.payload = w2.payload
Therefore, wrapping is safe without transitivity of scope.
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 08:42:22 UTC, Manu via Digitalmars-d wrote: On 11 December 2014 at 23:55, via Digitalmars-d digitalmars-d@puremagic.com wrote: This is a point that most people don't seem to understand yet, and which wasn't obvious for me either, at the beginning: * A `ref` parameter means that it cannot escape the function, _except_ by return. * A `scope ref` parameter means that it cannot escape the function _ever_, not even by return. * A `scope ref` return means that it cannot leave the current statement. Therefore, a `scope ref` return value can be passed on to the next function as a `ref` argument. If that function again returns a reference (even if not explicitly designated as `scope`), the compiler will treat it as if it were `scope ref`. Ummm. I reckon there's a good reason that people don't seem to understand this... because it would have difficulty being any more unintuitive! It's contrary to the behaviour of basically everything else in D! const, pure, nothrow, etc... they all work a reliable and consistent way. Unfortunately I have to agree. Also, I still don't think I get it... a scope-ref return, which has a restriction as you say, can be subverted by being piped through a function that receives and returns it as ref? What is the point? It cannot be subverted. As soon as you pass something into a function G() as `ref` that's been returned from a function F() via `scope ref`, the compiler needs to treat what is returned from G() as `scope ref`, even if it's only declared as `ref`. As you say, it's unintuitive and non-obvious. I'm also quite uneasy with the fact that scope would not be transitive as a storage class. What happens when it's applied to a value type, like a struct, that contains some pointers? An adaptation wrapper for a single pointer is super common; ie, a tiny struct passed by value with scope needs to have it's contained pointer receive the scope-ness of the argument. I agree, this is important. In my proposal, this works without transitivity. The wrapper stores the pointer as a `scope` member, then by copying the wrapper, the pointer gets copied implicitly, to which the normal scope restrictions apply (`scope` on members means will not outlive the aggregate). If it stored it as normal non-scope pointer, it couldn't get assigned in the first place. (Additionally, some higher level tricks are possible if we allow scope for value types and overloading on scope.) How can a member be marked scope? Apparently it's a storage class, not a type constructor... we can only attribute storage classes to function args/results(?). I was talking about my proposal here, where it's a type modifier, not a storage class. deadalnix also brought up the problem of transitivity. This, too, would simply go away if it's a type modifier.
Re: DIP69 - Implement scope for escape proof references
On 12 December 2014 at 07:41, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough. For
example,
not
being able to represent idiom of `scope ref int foo(scope ref int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
Yes, it can be.
I don't have the perfect proposal, but I feel very strongly about 2
things:
1. It must not be a storage class; the concept was a disaster with
ref, and I struggle with this more frequently than any other 'feature'
in D.
I simply do not understand why distinguishing beteen ref and not-ref is a
cornerstone of everything you do.
I've said so many times, it's the single greatest regular frustration
I encounter, by far.
That is of course a relative measure. It's not the 'cornerstone of
everything I do'; I don't start discussions about things that are not
broken and otherwise painless.
It is the thing that is *the most broken*, and leads to the most edge
cases, general bloat, and text mixins.
It comes up the most frequently, and by that metric alone, I consider
it highest priority on my list.
I've also said many times before, it possibly stems from the fact that
one of the key cornerstones of almost everything I do in D is interact
with other languages.
This is a practical reality, I can't write all my code in D, and have
mountains of existing code to interact with.
Boilerplate is the result. D has powerful systems to automate
boilerplate, like a carrot dangling right in front of my face, but
it's almost always thwarted by ref, and almost exclusively so.
My recent work updating LuaD to support all the features I required
wasted about 80% of my time dealing with ref issues.
In my past C++ bindings solutions, much code, which was otherwise
simple, readable, and elegant, turned into a mess of text mixins,
almost exclusively because ref is broken.
2. I feel it's a big mistake to separate it from the type system,
which I think most agree, is D's greatest asset by far. Manipulating
types is simple and convenient using the type system, and I think
manipulating scope will be just as important as any other attribute as
soon as even slightly complex use cases begin to arise.
Consider a ref counted type, RC!T. If scope were transitive, then you could
not have, say, a tree where the edges were RC!T. I.e., the payload of an RC
type should not be forced to be scope.
I'm not sure I quite visualise this correctly...
It sounds like you're talking about the same problem where people
complain with respect to 'const' and you happily tell them to stop
whingeing and deal with it?
(FWIW, I think you made the right call on const. It's never caused me
any hair loss.)
So you're saying that a pointer itself shouldn't be able to escape a
call tree, but the thing it points to should be able to escape just
fine?
It feels like that kinda defeats the purpose... or at least makes the
concept about as reliable as 'const' is in C++.
I guess you're seeing a situation where 'scope' is almost exclusively
useful as a mechanism to tell the RC that it doesn't need to worry
about ref-fiddling, and the thing we're passing isn't interested in
scope restrictions at any level other than that RC optimisation?
I guess I can see your angle... but my reaction is if that's all you
are concerned about, maybe scope is the wrong tool for eliding ref
fiddling in that case :/
I can see how you arrived at the opinion from your other comment
(wherever it was) that you are kinda likening this whole problem to
the RC case, if that's how you're looking at it.
It sounds like what you want is something like head-scope, in the same
way that people often want head-const?
Re: DIP69 - Implement scope for escape proof references
On 12 December 2014 at 08:53, John Colvin via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright wrote:
On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough. For
example,
not
being able to represent idiom of `scope ref int foo(scope ref int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
Yes, it can be.
I don't have the perfect proposal, but I feel very strongly about 2
things:
1. It must not be a storage class; the concept was a disaster with
ref, and I struggle with this more frequently than any other 'feature'
in D.
I simply do not understand why distinguishing beteen ref and not-ref is a
cornerstone of everything you do.
Because he requires control over function ABIs for both inter-language
communication and performance.
In binding D to IDL (Interactive Data, not Interface Description) I found
ref often required special casing.
Thank you. I feel like I'm just repeating myself over and over again.
Re: DIP69 - Implement scope for escape proof references
On 12 December 2014 at 22:18, via Digitalmars-d digitalmars-d@puremagic.com wrote: On Friday, 12 December 2014 at 08:42:22 UTC, Manu via Digitalmars-d wrote: How can a member be marked scope? Apparently it's a storage class, not a type constructor... we can only attribute storage classes to function args/results(?). I was talking about my proposal here, where it's a type modifier, not a storage class. Oh okay. Is that still on the table? deadalnix also brought up the problem of transitivity. This, too, would simply go away if it's a type modifier. Just for clarity, does type modifier mean the same thing as type constructor?
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 12:34:40 UTC, Manu via Digitalmars-d wrote: On 12 December 2014 at 22:18, via Digitalmars-d digitalmars-d@puremagic.com wrote: On Friday, 12 December 2014 at 08:42:22 UTC, Manu via Digitalmars-d wrote: How can a member be marked scope? Apparently it's a storage class, not a type constructor... we can only attribute storage classes to function args/results(?). I was talking about my proposal here, where it's a type modifier, not a storage class. Oh okay. Is that still on the table? Dunno, maybe? DIP69 doesn't look very well-received at the moment... deadalnix also brought up the problem of transitivity. This, too, would simply go away if it's a type modifier. Just for clarity, does type modifier mean the same thing as type constructor? Yes. Type modifier = const, shared, immutable, etc.; type constructor = the syntactical construct by which a type is constructed/modified. Andrei suggested this differentiation some time ago (sorry, don't have a reference at hand).
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 12:34:40 UTC, Manu via Digitalmars-d wrote: deadalnix also brought up the problem of transitivity. This, too, would simply go away if it's a type modifier. Just for clarity, does type modifier mean the same thing as type constructor? It appears to me that scopeness is a completely different beast than type qualifier, and type qualifier won't work. I think type modifier was just made up here. And why not ? It is more like some metadata (lifetime) being associated with value and symbols, like a type would, but that isn't really per se a type as D understands it right now.
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 13:38:10 UTC, Marc Schütz wrote: Yes. Type modifier = const, shared, immutable, etc.; type constructor = the syntactical construct by which a type is constructed/modified. Andrei suggested this differentiation some time ago (sorry, don't have a reference at hand). That won't fly. scope turtle left, type qualifier turtle right.
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 22:05:21 UTC, Walter Bright wrote: forms can work, too. No, unless we wrap every single indirection (pointer, slice, classes, delegates) into a wrapper. That is not going to fly very far. The beauty of GC is that you don't have to do any of this. To have safe other methods of allocation, there has to be annotation everywhere or use a wrapped type in the same places. The whole point of scope is to reduce that cost, both in term of runtime (as copy can be elided) but also on the user.
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 2:28 AM, deadalnix wrote: On Friday, 12 December 2014 at 08:44:56 UTC, Walter Bright wrote: On 12/12/2014 12:16 AM, deadalnix wrote: On Friday, 12 December 2014 at 07:48:21 UTC, Walter Bright wrote: Are you suggesting two kinds of scope - transitive and non-transitive? Non transitive scope can be added without any language extension. In order for it to work, the holes in the language that enabled escapes had to be plugged. That's what this proposal does - plug the holes. The holes covered by a non transitive scope are already undefined behavior. Right, but since the current compiler cannot detect them, it is a hole. ref counting is just one form of ownership. My point is if that works with this proposal, then the other forms can work, too. No, unless we wrap every single indirection (pointer, slice, classes, delegates) into a wrapper. That is not going to fly very far. The beauty of GC is that you don't have to do any of this. To have safe other methods of allocation, there has to be annotation everywhere or use a wrapped type in the same places.
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 4:19 AM, Manu via Digitalmars-d wrote:
I simply do not understand why distinguishing beteen ref and not-ref is a
cornerstone of everything you do.
I've said so many times, it's the single greatest regular frustration
I encounter, by far.
That is of course a relative measure. It's not the 'cornerstone of
everything I do'; I don't start discussions about things that are not
broken and otherwise painless.
It is the thing that is *the most broken*, and leads to the most edge
cases, general bloat, and text mixins.
It comes up the most frequently, and by that metric alone, I consider
it highest priority on my list.
I've also said many times before, it possibly stems from the fact that
one of the key cornerstones of almost everything I do in D is interact
with other languages.
This is a practical reality, I can't write all my code in D, and have
mountains of existing code to interact with.
Boilerplate is the result. D has powerful systems to automate
boilerplate, like a carrot dangling right in front of my face, but
it's almost always thwarted by ref, and almost exclusively so.
My recent work updating LuaD to support all the features I required
wasted about 80% of my time dealing with ref issues.
In my past C++ bindings solutions, much code, which was otherwise
simple, readable, and elegant, turned into a mess of text mixins,
almost exclusively because ref is broken.
You've said this before, many times, but what is lacking is an explanation of
WHY. What is the pattern, and why do you need that pattern? You say you don't
like auto ref because it doesn't give you exact control, but what are the cases
where auto ref is wrong?
Consider a ref counted type, RC!T. If scope were transitive, then you could
not have, say, a tree where the edges were RC!T. I.e., the payload of an RC
type should not be forced to be scope.
I'm not sure I quite visualise this correctly...
struct Tree {
RefCount!(Tree*) left;
RefCount!(Tree*) right;
...
}
So you're saying that a pointer itself shouldn't be able to escape a
call tree, but the thing it points to should be able to escape just
fine?
Yes.
It feels like that kinda defeats the purpose...
You're arguing that a data structure with only one access point is the only kind
of data structure in use. With transitive scope, such a data structure would be
the only one possible!
I guess you're seeing a situation where 'scope' is almost exclusively
useful as a mechanism to tell the RC that it doesn't need to worry
about ref-fiddling, and the thing we're passing isn't interested in
scope restrictions at any level other than that RC optimisation?
I guess I can see your angle... but my reaction is if that's all you
are concerned about, maybe scope is the wrong tool for eliding ref
fiddling in that case :/
'scope' is a way to say that this use of a pointer does not escape this scope.
That is incredibly useful.
Recall my statements that pervasive refcounting is a terrible performance
problem because of all the inc/dec? Knowing that references cannot escape means
an inc/dec pair can be elided.
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 4:20 AM, Manu via Digitalmars-d wrote: On 12 December 2014 at 08:53, John Colvin via Digitalmars-d Because he requires control over function ABIs for both inter-language communication and performance. In binding D to IDL (Interactive Data, not Interface Description) I found ref often required special casing. Thank you. I feel like I'm just repeating myself over and over again. Example, please.
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 2:09 PM, deadalnix wrote: On Friday, 12 December 2014 at 22:05:21 UTC, Walter Bright wrote: forms can work, too. No, unless we wrap every single indirection (pointer, slice, classes, delegates) into a wrapper. That is not going to fly very far. The beauty of GC is that you don't have to do any of this. To have safe other methods of allocation, there has to be annotation everywhere or use a wrapped type in the same places. The whole point of scope is to reduce that cost, both in term of runtime (as copy can be elided) but also on the user. Yes, I do understand that :-) but currently doing a safe refcounted type is impossible in D. This proposal fixes that.
Re: DIP69 - Implement scope for escape proof references
On Friday, 12 December 2014 at 22:34:27 UTC, Walter Bright wrote: On 12/12/2014 2:09 PM, deadalnix wrote: Yes, I do understand that :-) but currently doing a safe refcounted type is impossible in D. This proposal fixes that. Actually, no it doesn't. It allow to ensure that this is safe, module the RC system is the only owner. This proposal allow for easy to use and fast RC, but certainly do not make it safe.
Re: DIP69 - Implement scope for escape proof references
On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote: How not? scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes !
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 2:48 PM, deadalnix wrote: On Friday, 12 December 2014 at 22:34:27 UTC, Walter Bright wrote: On 12/12/2014 2:09 PM, deadalnix wrote: Yes, I do understand that :-) but currently doing a safe refcounted type is impossible in D. This proposal fixes that. Actually, no it doesn't. It allow to ensure that this is safe, module the RC system is the only owner. This proposal allow for easy to use and fast RC, but certainly do not make it safe. How not?
Re: DIP69 - Implement scope for escape proof references
On 12/11/2014 10:10 PM, Dicebot wrote: So far I tend to agree, unfortunately. Considering all provided answers and explanations suggested semantics are simply not powerful enough to be useful even for relatively simple idiomatic D code and with no clear way of backward compatible improvements this DIP does not pull own weight. Those are not scopes you are looking for The proposal provides escape proof passing of arguments. This is necessary in order to make rc safe, for example. What are you looking for?
Re: DIP69 - Implement scope for escape proof references
On 12/11/2014 4:13 PM, deadalnix wrote:
On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz wrote:
On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via Digitalmars-d wrote:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough. For example,
not
being able to represent idiom of `scope ref int foo(scope ref int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet, and which
wasn't obvious for me either, at the beginning:
* A `ref` parameter means that it cannot escape the function, _except_ by
return.
* A `scope ref` parameter means that it cannot escape the function _ever_,
not even by return.
* A `scope ref` return means that it cannot leave the current statement.
Therefore, a `scope ref` return value can be passed on to the next function as
a `ref` argument. If that function again returns a reference (even if not
explicitly designated as `scope`), the compiler will treat it as if it were
`scope ref`.
No, it understood. It is simply not useful.
The use case is described in the DIP under the 'Scope Ref' heading.
I agree, this is important. In my proposal, this works without transitivity.
The wrapper stores the pointer as a `scope` member, then by copying the
wrapper, the pointer gets copied implicitly, to which the normal scope
restrictions apply (`scope` on members means will not outlive the
aggregate). If it stored it as normal non-scope pointer, it couldn't get
assigned in the first place.
Wut ? You cante store anything with grear lifetime, including non scope things
(as they'll have infinite lifetime). Meaning the only thing you know, is that
thing are possibly scoped.
Meaning you have to assume infinite lifetime with every indirection, which make
this proposal useless.
It means you'll have to wrap the next level of indirection in a ref counting (or
equivalent) manner.
The great thing about GC is you can leave everything to the GC. But when
explicitly managing memory, you have to decide for EVERY pointer who owns it,
when it can be released, etc. Having a transitive scope is NOT going to resolve
that for you, it isn't even going to help.
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 4:07 PM, deadalnix wrote: On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote: How not? scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes ! RC needs to take a type as a parameter, not a symbol.
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 4:07 PM, deadalnix wrote: On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote: How not? scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes ! What's the type signature of RC ?
Re: DIP69 - Implement scope for escape proof references
On Saturday, 13 December 2014 at 00:52:46 UTC, Walter Bright wrote: On 12/12/2014 4:07 PM, deadalnix wrote: On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote: How not? scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes ! What's the type signature of RC ? RCWrapper!T RC(T)(scope T t) or somethign along these lines.
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 4:59 PM, deadalnix wrote: On Saturday, 13 December 2014 at 00:52:46 UTC, Walter Bright wrote: On 12/12/2014 4:07 PM, deadalnix wrote: On Saturday, 13 December 2014 at 00:06:18 UTC, Walter Bright wrote: How not? scope a = new Stuff(); scope b = a; auto rc1 = RC(a); auto rc2 = RC(b); // Enjoy the free when it comes ! What's the type signature of RC ? RCWrapper!T RC(T)(scope T t) or somethign along these lines. And how does RC assign it to the wrapper? (It will fail to compile, and so won't be a safety hole.)
Re: DIP69 - Implement scope for escape proof references
On 13 December 2014 at 08:14, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/12/2014 4:19 AM, Manu via Digitalmars-d wrote:
I simply do not understand why distinguishing beteen ref and not-ref is a
cornerstone of everything you do.
I've said so many times, it's the single greatest regular frustration
I encounter, by far.
That is of course a relative measure. It's not the 'cornerstone of
everything I do'; I don't start discussions about things that are not
broken and otherwise painless.
It is the thing that is *the most broken*, and leads to the most edge
cases, general bloat, and text mixins.
It comes up the most frequently, and by that metric alone, I consider
it highest priority on my list.
I've also said many times before, it possibly stems from the fact that
one of the key cornerstones of almost everything I do in D is interact
with other languages.
This is a practical reality, I can't write all my code in D, and have
mountains of existing code to interact with.
Boilerplate is the result. D has powerful systems to automate
boilerplate, like a carrot dangling right in front of my face, but
it's almost always thwarted by ref, and almost exclusively so.
My recent work updating LuaD to support all the features I required
wasted about 80% of my time dealing with ref issues.
In my past C++ bindings solutions, much code, which was otherwise
simple, readable, and elegant, turned into a mess of text mixins,
almost exclusively because ref is broken.
You've said this before, many times, but what is lacking is an explanation
of WHY. What is the pattern, and why do you need that pattern? You say you
don't like auto ref because it doesn't give you exact control, but what are
the cases where auto ref is wrong?
I did just give some examples, I'll repeat; auto ref fails when the
function is extern.
It is also wrong that when I pass an int or float, it is passed by ref
instead of by value... that is never what I want!
What do you get when you take a pointer of a function with an auto-ref
arg? ...an error, because it's not actually a function!
So in a context where I'm dealing with functions and function pointers
(very, very frequent), I suddenly have something that's not a function
find it's way into my meta and I have to special-case the hell out of
it.
The solution in this case is to wrap it in a non-auto-ref shim with
the ref-ness explicitly stated in the way I expect... which is in
itself another problem, because 'ref' is not part of the type! So the
annoying de-auto-ref-ing shim must be a text mixin with some very
complex, practically unreadable, definitely unmaintainable logic
surrounding it. It's insanity on top of insanity!
I also run the invisible risk of generating 2^num_args instances of
the code for functions containing auto-ref args.
When I'm writing a function that's not a template, I intend, and
expect, to write a function that's _not a template_.
Templates and functions are different things. I think it's a massive
mistake to have created a way to write a template that looks nothing
like a template.
auto-ref is not, and never has been a tool I have wanted. I don't have
any use for auto-ref, and it has only proven to make an already severe
problem worse. I've tried to use it before in some instances, but it
made ref's out of int's and floats, so even in the rare potentially
useful cases, I had to reject it.
At the time, you introduced auto-ref as a 'solution' to an issue that
I was the primary complainant (although the solution was mainly pushed
by Andrei iirc?). I absolutely rejected it then, and said it would
have the disastrous effect of setting ref in stone, which I was
already very critical about. I reject it even more now that I have had
some experience with it.
Who was the customer that you designed it for? (It wasn't me!)
What issue was it designed to address?
Consider a ref counted type, RC!T. If scope were transitive, then you
could
not have, say, a tree where the edges were RC!T. I.e., the payload of an
RC
type should not be forced to be scope.
I'm not sure I quite visualise this correctly...
struct Tree {
RefCount!(Tree*) left;
RefCount!(Tree*) right;
...
}
... I don't think I'd ever have a use for this code.
I've been using trees for a long time, and I can't imagine a situation
where each node in a tree would want to be ref counted.
It sounds like a disaster for performance, and I can't imagine any
advantage it would offer?
Perhaps in some complex graph structure... but I expect that sort of
thing would be far more specialised.
I can see common cases where nodes may contain a refcounted object as
node data, but that's different than the situation you demonstrate
here.
I'll need to think about how that situation would be affected in this case.
So you're saying that a pointer itself shouldn't be able to escape a
call tree, but the thing it points to should be able to escape just
fine?
Yes.
Re: DIP69 - Implement scope for escape proof references
On 12/12/2014 6:55 PM, Manu via Digitalmars-d wrote:
I did just give some examples, I'll repeat; auto ref fails when the
function is extern.
Don't make it extern, then.
It is also wrong that when I pass an int or float, it is passed by ref
instead of by value... that is never what I want!
If there's source to the function, it'll often be inlined which will remove the
indirection.
What do you get when you take a pointer of a function with an auto-ref
arg? ...an error, because it's not actually a function!
So in a context where I'm dealing with functions and function pointers
(very, very frequent), I suddenly have something that's not a function
find it's way into my meta and I have to special-case the hell out of
it.
Why are function pointers and ints going to the same argument of a function? I
thought you weren't using templates?
The solution in this case is to wrap it in a non-auto-ref shim with
the ref-ness explicitly stated in the way I expect... which is in
itself another problem, because 'ref' is not part of the type! So the
annoying de-auto-ref-ing shim must be a text mixin with some very
complex, practically unreadable, definitely unmaintainable logic
surrounding it. It's insanity on top of insanity!
I also run the invisible risk of generating 2^num_args instances of
the code for functions containing auto-ref args.
I wonder what is the need for the code that you are writing.
When I'm writing a function that's not a template, I intend, and
expect, to write a function that's _not a template_.
Templates and functions are different things. I think it's a massive
mistake to have created a way to write a template that looks nothing
like a template.
A function template is a function that takes both compile-time args and run-time
args. C++ tried to make them completely different animals, when they are not.
Don't think template, think compile-time argument to a function. I convinced
Andrei to never mention template in his D book, as it brings forth all kinds
of connotations and expectations that impede understanding what D templates
actually are. I think the result was successful.
auto-ref is not, and never has been a tool I have wanted. I don't have
any use for auto-ref, and it has only proven to make an already severe
problem worse. I've tried to use it before in some instances, but it
made ref's out of int's and floats, so even in the rare potentially
useful cases, I had to reject it.
If it's a rare useful case, why is it a pervasive problem?
At the time, you introduced auto-ref as a 'solution' to an issue that
I was the primary complainant (although the solution was mainly pushed
by Andrei iirc?). I absolutely rejected it then, and said it would
have the disastrous effect of setting ref in stone, which I was
already very critical about. I reject it even more now that I have had
some experience with it.
Who was the customer that you designed it for? (It wasn't me!)
What issue was it designed to address?
I still have no idea what code you are developing that you need to send ints and
function pointers to the same argument of a function template, yet you don't use
function templates. Nor do I understand what pattern you need that simply must
mix up ref and value parameters, and why that pattern appears pervasively in
your code.
struct Tree {
RefCount!(Tree*) left;
RefCount!(Tree*) right;
...
}
... I don't think I'd ever have a use for this code.
You have no use for tree structures?
I've been using trees for a long time, and I can't imagine a situation
where each node in a tree would want to be ref counted.
You have more than one parent of a node. You never write data structures like
that? dmd uses such pervasively (Type and Expression).
It sounds like a disaster for performance, and I can't imagine any
advantage it would offer?
How do you propose to manage the memory explicitly otherwise? Use a GC? :-)
I can see common cases where nodes may contain a refcounted object as
node data, but that's different than the situation you demonstrate
here.
That would fail if scope were transitive.
But there are more things than pointers which need to be protected
against escaping their scope, in particular, things that contain
pointers...
Solve it in the general case (this proposal) and RC now works.
Maybe there's a compromise. If we say scope isn't 'transitive', but it
is transitive when it comes to aggregates?
One thing I've tried very hard to make work in D is have basic types /
aggregages / arrays be interchangeable.
Ie, you can apply scope to a by-val struct, and all aggregate members
have scope applied.
It is not transitive in the sense that it does not follow through
pointer members, but any such pointer members may not escape as if it
were a pointer argument alone.
This is unlike any other behaviour in D, but without that, I think
this proposal fails to suit my most common use cases.
I have no idea what your
Re: DIP69 - Implement scope for escape proof references
On Thursday, 11 December 2014 at 07:30:03 UTC, Walter Bright wrote: On 12/10/2014 8:56 PM, deadalnix wrote: On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright wrote: I disagree. It's critical for chaining one function to the next. I one can't return, one can't chain. I guess I'm not seeing the problem. a.foo().bar() is the same as bar(foo(a)), that is, using the returned value as first parameter. If you can't return, you can't chain.
Re: DIP69 - Implement scope for escape proof references
On Wed, 10 Dec 2014 23:06:13 -0800 Shammah Chancellor via Digitalmars-d digitalmars-d@puremagic.com wrote: Was afraid that would break too much code. No, this is super important. Break it all! alas, not in this life. each break our code just making resistance harder. ;-) signature.asc Description: PGP signature
Re: DIP69 - Implement scope for escape proof references
On 12/10/2014 11:55 PM, deadalnix wrote: On Thursday, 11 December 2014 at 07:30:03 UTC, Walter Bright wrote: On 12/10/2014 8:56 PM, deadalnix wrote: On Thursday, 11 December 2014 at 03:27:08 UTC, Walter Bright wrote: I disagree. It's critical for chaining one function to the next. I one can't return, one can't chain. I guess I'm not seeing the problem. a.foo().bar() is the same as bar(foo(a)), that is, using the returned value as first parameter. If you can't return, you can't chain. Right, and you can chain when the declarations are: scope ref S foo(ref T a); U bar(ref S s); a.foo().bar(); // works
Re: DIP69 - Implement scope for escape proof references
On 12/10/2014 10:43 PM, Dicebot wrote: On Thursday, 11 December 2014 at 03:30:07 UTC, Walter Bright wrote: If you want data to 'escape' from r.front, then it shouldn't be marked as scope. Definitely, using scope successfully will require some rethinking of how code is written. Allowing or prohibiting escaping of r.front in that example depends on definition of mapped range. There is nothing to rethink about it - case is almost identical to const+inout. One reason why with templates, they'll do scope inference.
Re: DIP69 - Implement scope for escape proof references
On Thursday, 11 December 2014 at 00:35:46 UTC, deadalnix wrote: It is always safe to consider scopeness of the retrun value (if marked scope) as being the intersection of the lifetime of parameters. That should cover most bases, and we can still extends later if this is too limited (I suspect it is ok for most cases). Linear typing is already extremely limiting, by limiting it even further you end up with something annoying. You basically get a version of memory safety that does not solve any typical memory unsafe situations. By having pointers that do scope-depth-tracking you do at least get a generic solution that can be optimized away when possible. The D authors have to accept that you need to embed ownership in pointers if you want memory safety and convenience, or that you have to provide means to guide the semantic analysis. You need one or the other, or both, but you cannot pretend that you can do without. Arbitrary constraints are annoying, not convenient. If I as a programmer know that something is safe, then the compiler should accept it, and the language should allow me express it.
Re: DIP69 - Implement scope for escape proof references
On 11.12.14 08:25, Shammah Chancellor wrote: This is exactly why this feature should be default behavior with compiler warnings generated when things escape scope. It can be compiler switch escape=I(gnore)|W(arning)|E(rror).
Re: DIP69 - Implement scope for escape proof references
On Thursday, 4 December 2014 at 10:00:37 UTC, Walter Bright wrote: On 12/4/2014 1:51 AM, eles wrote: Making it implicit and requiring an explicit escape for un-scoped variables? Was afraid that would break too much code. Compiler switch: escape=I(gnore)|W(arning)|E(rror). For transition default Ignore.
Re: DIP69 - Implement scope for escape proof references
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough. For example,
not
being able to represent idiom of `scope ref int foo(scope ref int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
I also don't consider `ref` design as a storage class any kind of success
at all
and generally agree with Manu on this topic. At the same time alternative
proposals that make it a qualifier (like Marc did) do impact existing
language
much more and this no small concern.
My experience with C++ ref as type qualifier is very, very bad. It's a
special case EVERYWHERE. Doing type deduction with it is an exercise in a
completely baffling set of rules and a different rule for every occasion -
Scott Meyers has a great piece on this.
There are probably only a handful of people on the planet who actually
understand C++ ref. I wished very hard to avoid that with D ref.
I do completely sympathise with you on this point. I understand where
you were coming from on 'ref', and I do applaud the effort. Nobody
said that ref should be just the same as C++, but I am saying that the
D solution is not a success.
I'm all about a ref that improves on C++, but I don't think that's
what we have. We have something that's just complex and unwieldy in a
different (and for my money, even more frustrating) way.
I'm just saying, don't repeat that mistake with scope! There must be
another way...
I'm also quite uneasy with the fact that scope would not be transitive
as a storage class. What happens when it's applied to a value type,
like a struct, that contains some pointers? An adaptation wrapper for
a single pointer is super common; ie, a tiny struct passed by value
with scope needs to have it's contained pointer receive the scope-ness
of the argument.
I don't have the perfect proposal, but I feel very strongly about 2 things:
1. It must not be a storage class; the concept was a disaster with
ref, and I struggle with this more frequently than any other 'feature'
in D.
2. I feel it's a big mistake to separate it from the type system,
which I think most agree, is D's greatest asset by far. Manipulating
types is simple and convenient using the type system, and I think
manipulating scope will be just as important as any other attribute as
soon as even slightly complex use cases begin to arise.
Re: DIP69 - Implement scope for escape proof references
Manu have you shown Walter some of the code where ref is problematic? He seems to have asked to see examples repeatedly as he's not convinced there is a problem.
Re: DIP69 - Implement scope for escape proof references
On 11/12/2014 09:07, Ola Fosheim Grøstad ola.fosheim.grostad+dl...@gmail.com wrote: If I as a programmer know that something is safe, then the compiler should accept it, and the language should allow me express it. It will, use @system. Maybe this proposal will be supplemented later, but there are always costs to making the language more complicated. We need to be certain they are worth it before adding them. A cautious approach can be extended later, if the situation demands it. Sometimes innovative workarounds are developed that are difficult to foresee in advance - e.g. in Rust their type system can be restrictive, but they rely on trusted library functions/types to make more things possible in a safe way.
Re: DIP69 - Implement scope for escape proof references
On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via
Digitalmars-d wrote:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough.
For example,
not
being able to represent idiom of `scope ref int foo(scope ref
int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet,
and which wasn't obvious for me either, at the beginning:
* A `ref` parameter means that it cannot escape the function,
_except_ by return.
* A `scope ref` parameter means that it cannot escape the
function _ever_, not even by return.
* A `scope ref` return means that it cannot leave the current
statement.
Therefore, a `scope ref` return value can be passed on to the
next function as a `ref` argument. If that function again returns
a reference (even if not explicitly designated as `scope`), the
compiler will treat it as if it were `scope ref`.
I'm also quite uneasy with the fact that scope would not be
transitive
as a storage class. What happens when it's applied to a value
type,
like a struct, that contains some pointers? An adaptation
wrapper for
a single pointer is super common; ie, a tiny struct passed by
value
with scope needs to have it's contained pointer receive the
scope-ness
of the argument.
I agree, this is important. In my proposal, this works without
transitivity. The wrapper stores the pointer as a `scope` member,
then by copying the wrapper, the pointer gets copied implicitly,
to which the normal scope restrictions apply (`scope` on members
means will not outlive the aggregate). If it stored it as
normal non-scope pointer, it couldn't get assigned in the first
place. (Additionally, some higher level tricks are possible if we
allow scope for value types and overloading on scope.)
Re: DIP69 - Implement scope for escape proof references
On 09/12/2014 16:25, Steven Schveighoffer wrote: Will ref just automatically bind to any scoped reference? A ref parameter essentially is a scope ref parameter, but it can also be returned: http://forum.dlang.org/post/m64v3g$2bga$1...@digitalmars.com
Re: DIP69 - Implement scope for escape proof references
On Thursday, 11 December 2014 at 13:55:55 UTC, Marc Schütz wrote:
On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via
Digitalmars-d wrote:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough.
For example,
not
being able to represent idiom of `scope ref int foo(scope
ref int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet,
and which wasn't obvious for me either, at the beginning:
* A `ref` parameter means that it cannot escape the function,
_except_ by return.
* A `scope ref` parameter means that it cannot escape the
function _ever_, not even by return.
* A `scope ref` return means that it cannot leave the current
statement.
Therefore, a `scope ref` return value can be passed on to the
next function as a `ref` argument. If that function again
returns a reference (even if not explicitly designated as
`scope`), the compiler will treat it as if it were `scope ref`.
OH! I had totally misunderstood that. Cheers for the explanation.
Re: DIP69 - Implement scope for escape proof references
On Thursday, 11 December 2014 at 13:43:07 UTC, Nick Treleaven wrote: It will, use @system. Yes… Maybe this proposal will be supplemented later, but there are always costs to making the language more complicated. We need to be certain they are worth it before adding them. A cautious approach can be extended later, if the situation demands it. Can it be extended later? Without breaking stuff? You need to get it right from the start, otherwise you end up with N different, but incompatible ways of doing the same thing. That means you need to get ownership right first.
Re: DIP69 - Implement scope for escape proof references
Nick Treleaven: Sometimes innovative workarounds are developed that are difficult to foresee in advance - e.g. in Rust their type system can be restrictive, but they rely on trusted library functions/types to make more things possible in a safe way. Ideally a type system should be flexible enough, in practice giving it a high flexibility has significant costs (in compilation times, amount of code that implements the system, bugs in such implementation, costs for the final programmer in inventing a way to express the semantics, etc), so most languages avoid a too much complex type system (even Haskell does this) and accept reasonable workarounds... Bye, bearophile
Re: DIP69 - Implement scope for escape proof references
On 12/11/14 8:55 AM, Marc =?UTF-8?B?U2Now7x0eiI=?= schue...@gmx.net
wrote:
On Thursday, 11 December 2014 at 12:48:05 UTC, Manu via Digitalmars-d
wrote:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough. For
example,
not
being able to represent idiom of `scope ref int foo(scope ref int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
This is a point that most people don't seem to understand yet, and which
wasn't obvious for me either, at the beginning:
* A `ref` parameter means that it cannot escape the function, _except_
by return.
* A `scope ref` parameter means that it cannot escape the function
_ever_, not even by return.
* A `scope ref` return means that it cannot leave the current statement.
Therefore, a `scope ref` return value can be passed on to the next
function as a `ref` argument. If that function again returns a reference
(even if not explicitly designated as `scope`), the compiler will treat
it as if it were `scope ref`.
Please, put this in the DIP! This is exactly the thing I had been asking
for here: http://forum.dlang.org/post/m5sitd$1ki0$1...@digitalmars.com
Thanks, I'll re-read the proposal with this in mind.
-Steve
Re: DIP69 - Implement scope for escape proof references
On 12/11/2014 8:26 AM, Steven Schveighoffer wrote: Please, put this in the DIP! This is exactly the thing I had been asking for here: http://forum.dlang.org/post/m5sitd$1ki0$1...@digitalmars.com Thanks, I'll re-read the proposal with this in mind. Marc is clearly better at explaining things than I am, as I've explained that point in this thread about 5 times now.
Re: DIP69 - Implement scope for escape proof references
On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough. For example,
not
being able to represent idiom of `scope ref int foo(scope ref int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
Yes, it can be.
I don't have the perfect proposal, but I feel very strongly about 2 things:
1. It must not be a storage class; the concept was a disaster with
ref, and I struggle with this more frequently than any other 'feature'
in D.
I simply do not understand why distinguishing beteen ref and not-ref is a
cornerstone of everything you do.
2. I feel it's a big mistake to separate it from the type system,
which I think most agree, is D's greatest asset by far. Manipulating
types is simple and convenient using the type system, and I think
manipulating scope will be just as important as any other attribute as
soon as even slightly complex use cases begin to arise.
Consider a ref counted type, RC!T. If scope were transitive, then you could not
have, say, a tree where the edges were RC!T. I.e., the payload of an RC type
should not be forced to be scope.
Re: DIP69 - Implement scope for escape proof references
12-Dec-2014 00:41, Walter Bright пишет: On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote: On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d digitalmars-d@puremagic.com wrote: I don't have the perfect proposal, but I feel very strongly about 2 things: 1. It must not be a storage class; the concept was a disaster with ref, and I struggle with this more frequently than any other 'feature' in D. I simply do not understand why distinguishing beteen ref and not-ref is a cornerstone of everything you do. I would hazard a guess that const T is something lots of C++ got addicted to as the most common way of parameter passing. There is nothing exactly like that in D, auto ref was quite close but only for templates and it instantiates 2 distinct bodies for ref and non-ref at the moment. It allows easy pass by ref for both r-values and l-values with logical const and is likely what Manu is referring(!) to. Truth be told it's not always the fastest way, in fact Boost C++ has template for optimal parameter passing - call_traitsT::param_type but it must be too cumbersome for actual use as I never seen it used in the wild. http://www.boost.org/doc/libs/1_57_0/libs/utility/call_traits.htm -- Dmitry Olshansky
Re: DIP69 - Implement scope for escape proof references
On Thursday, 11 December 2014 at 21:41:11 UTC, Walter Bright
wrote:
On 12/11/2014 4:47 AM, Manu via Digitalmars-d wrote:
On 8 December 2014 at 07:29, Walter Bright via Digitalmars-d
digitalmars-d@puremagic.com wrote:
On 12/7/2014 6:12 AM, Dicebot wrote:
But from existing cases it doesn't seem working good enough.
For example,
not
being able to represent idiom of `scope ref int foo(scope
ref int x) {
return x;
}` seems very limiting.
scope ref int foo(ref int x);
will do it.
Will it? It looks like foo can't be called with scope data?
Yes, it can be.
I don't have the perfect proposal, but I feel very strongly
about 2 things:
1. It must not be a storage class; the concept was a disaster
with
ref, and I struggle with this more frequently than any other
'feature'
in D.
I simply do not understand why distinguishing beteen ref and
not-ref is a cornerstone of everything you do.
Because he requires control over function ABIs for both
inter-language communication and performance.
In binding D to IDL (Interactive Data, not Interface Description)
I found ref often required special casing.
