Re: aliasing expressions and identifiers
On Monday, 30 May 2016 at 14:18:38 UTC, Nick Treleaven wrote: On Monday, 30 May 2016 at 10:55:57 UTC, Marc Schütz wrote: auto tmp = stdin.byLine; auto lines = tmp.array; Here, `lines` contains references to the buffer owned by `tmp`, but doesn't escape (assuming `array` takes its argument by `scope` or however the final solution will look like). tmp and stdin.byLine are of type ByLine, whose front could be scope/return to prevent escaping. Above array() does escape ByLine.front so can't mark its argument with scope - the compiler would error. Sorry, I meant using array(scope R range) would not affect the compiler error when array attempts to escape range.front into its slice.
Re: aliasing expressions and identifiers
On Monday, 30 May 2016 at 10:55:57 UTC, Marc Schütz wrote: On Sunday, 29 May 2016 at 14:27:51 UTC, Nick Treleaven wrote: What about if the RCArray (of ref count 1) is assigned to a different one after the local ref is initialised? That is what we're discussing -it's your example above(!) Exactly, and then one of the two suggested approaches will have to be used to prevent the use-after-free. But that's something that needs to happen on assignment, not when the reference is created. Well my solution does work, disallowing the problematic local refs. But it wouldn't be great for generic code or consistency with the @rc DIP. For the latter, we can add a temporary RC object to keep the referenced memory alive, but we only need to do this when the local ref is initialized from a function both (1) returning ref and (2) with a parameter marked return. I think we should just prevent front from escaping. It doesn't necessarily need to escape for the problem to occur. Well, it does in this example, but it can be trivially rewritten: auto tmp = stdin.byLine; auto lines = tmp.array; Here, `lines` contains references to the buffer owned by `tmp`, but doesn't escape (assuming `array` takes its argument by `scope` or however the final solution will look like). tmp and stdin.byLine are of type ByLine, whose front could be scope/return to prevent escaping. Above array() does escape ByLine.front so can't mark its argument with scope - the compiler would error.
Re: aliasing expressions and identifiers
On Sunday, 29 May 2016 at 14:27:51 UTC, Nick Treleaven wrote: What about if the RCArray (of ref count 1) is assigned to a different one after the local ref is initialised? That is what we're discussing -it's your example above(!) Exactly, and then one of the two suggested approaches will have to be used to prevent the use-after-free. But that's something that needs to happen on assignment, not when the reference is created. It can be solved in one of two ways: Either by making the owner (`arr`) non-mutable during the existence of the references, thereby forbidding the call to `bar()` (I would prefer this one, as it's cleaner and can be used for many more things, e.g. the byLine problem) I don't see directly how this affects byLine.front, that does not return a reference. It returns a reference in the wider sense, namely a slice to a private buffer that gets overwritten by each call to `byLine`. Currently, DIP25 only applies to `ref`s in the narrow sense, but I'm assuming it will be generalized to include pointer, slices, class references, AAs and hidden context pointers. Making the ByLine range constant as long as there's a reference to its buffer would prevent surprises like this: auto lines = stdin.byLine.array; // => all elements of `lines` are the same, should have used `byLineCopy` So your solution would statically prevent popFront because front has escaped. Yes. I think we should just prevent front from escaping. It doesn't necessarily need to escape for the problem to occur. Well, it does in this example, but it can be trivially rewritten: auto tmp = stdin.byLine; auto lines = tmp.array; Here, `lines` contains references to the buffer owned by `tmp`, but doesn't escape (assuming `array` takes its argument by `scope` or however the final solution will look like).
Re: aliasing expressions and identifiers
On Friday, 27 May 2016 at 11:50:42 UTC, Marc Schütz wrote:
On Friday, 27 May 2016 at 10:04:14 UTC, Nick Treleaven wrote:
On Thursday, 26 May 2016 at 08:29:41 UTC, Marc Schütz wrote:
RCArray!int arr = [7];
ref r = arr[0];
arr = [9];// this releases the old array
r++; // use after free
...
statically prevent the above from compiling using:
RCArray(T) {
...
ref opIndex(size_t) return;
Local refs cannot be assigned from a function returning ref if
that function has any parameters marked with the return
attribute. If there is no attribute, local refs + function
returning ref is OK.
Huh? `return` means that the returned reference is owned by the
RCArray struct and must therefore not outlive it. If the
RCArray is a local variable (or parameter), the local ref is
always declared after it (because it must be initialized
immediately), and will have a shorter scope than the RCArray.
Therefore, such an assignment is always accepted.
What about if the RCArray (of ref count 1) is assigned to a
different one after the local ref is initialised? That is what
we're discussing -it's your example above(!)
It can be solved in one of two ways: Either by making the
owner (`arr`) non-mutable during the existence of the
references, thereby forbidding the call to `bar()` (I would
prefer this one, as it's cleaner and can be used for many
more things, e.g. the byLine problem)
I don't see directly how this affects byLine.front, that does
not return a reference.
It returns a reference in the wider sense, namely a slice to a
private buffer that gets overwritten by each call to `byLine`.
Currently, DIP25 only applies to `ref`s in the narrow sense,
but I'm assuming it will be generalized to include pointer,
slices, class references, AAs and hidden context pointers.
Making the ByLine range constant as long as there's a reference
to its buffer would prevent surprises like this:
auto lines = stdin.byLine.array;
// => all elements of `lines` are the same, should have
used `byLineCopy`
So your solution would statically prevent popFront because front
has escaped. I think we should just prevent front from escaping.
Re: aliasing expressions and identifiers
On Friday, 27 May 2016 at 10:04:14 UTC, Nick Treleaven wrote:
On Thursday, 26 May 2016 at 08:29:41 UTC, Marc Schütz wrote:
To elaborate: neither `scope` nor reference counting can ever
protect you against explicit premature destruction of a
still-referenced owner. But there is a slightly different
problematic scenario:
RCArray!int arr = [7];
ref r = arr[0];
arr = [9];// this releases the old array
r++; // use after free
This is the same situation. There's nothing special about
destroy, it just assigns arr = arr.init. destroy is @safe so in
fact it must work with safe RC.
If this is what destroy does, then yes, that's ok. But it is a
misleading name then, IMO.
You seem to have ignored my suggestion (which maybe wasn't
clear enough) to statically prevent the above from compiling
using:
RCArray(T) {
...
ref opIndex(size_t) return;
Local refs cannot be assigned from a function returning ref if
that function has any parameters marked with the return
attribute. If there is no attribute, local refs + function
returning ref is OK.
Huh? `return` means that the returned reference is owned by the
RCArray struct and must therefore not outlive it. If the RCArray
is a local variable (or parameter), the local ref is always
declared after it (because it must be initialized immediately),
and will have a shorter scope than the RCArray. Therefore, such
an assignment is always accepted.
But this issue exists even without locale `ref`s:
void foo() {
RCArray!int arr = [7];
bar(arr, arr[0]);
}
void bar(ref RCArray!int arr, ref int r) {
arr = [9];// this releases the old array
r++; // use after free
}
This is the reason for the @rc DIP.
It can be solved in one of two ways: Either by making the
owner (`arr`) non-mutable during the existence of the
references, thereby forbidding the call to `bar()` (I would
prefer this one, as it's cleaner and can be used for many more
things, e.g. the byLine problem)
I don't see directly how this affects byLine.front, that does
not return a reference.
It returns a reference in the wider sense, namely a slice to a
private buffer that gets overwritten by each call to `byLine`.
Currently, DIP25 only applies to `ref`s in the narrow sense, but
I'm assuming it will be generalized to include pointer, slices,
class references, AAs and hidden context pointers.
Making the ByLine range constant as long as there's a reference
to its buffer would prevent surprises like this:
auto lines = stdin.byLine.array;
// => all elements of `lines` are the same, should have used
`byLineCopy`
By the way, there's a theoretical third possibility:
Force-invalidating the references after the owner has
(potentially) been mutated. But this would require very advanced
data flow tracking and is probably impossible to implement in the
general case.
Re: aliasing expressions and identifiers
On Thursday, 26 May 2016 at 08:29:41 UTC, Marc Schütz wrote:
On Wednesday, 25 May 2016 at 19:47:06 UTC, Nick Treleaven wrote:
On 24/05/2016 14:48, Nick Treleaven wrote:
@safe unittest
{
RCArray!int arr;
+ arr.length = 1;
ref r = arr[0];
arr.destroy; // refcount drops to zero, arr.impl memory
freed
r++; // writes to unallocated memory
}
Here I think local refs must be prevented from initialization
by return ref (-dip25). The @rc DIP and RCArray would use
return ref. It would be OK to initialize a local ref with a
ref function result that is not return ref.
...
To elaborate: neither `scope` nor reference counting can ever
protect you against explicit premature destruction of a
still-referenced owner. But there is a slightly different
problematic scenario:
RCArray!int arr = [7];
ref r = arr[0];
arr = [9];// this releases the old array
r++; // use after free
This is the same situation. There's nothing special about
destroy, it just assigns arr = arr.init. destroy is @safe so in
fact it must work with safe RC. You seem to have ignored my
suggestion (which maybe wasn't clear enough) to statically
prevent the above from compiling using:
RCArray(T) {
...
ref opIndex(size_t) return;
Local refs cannot be assigned from a function returning ref if
that function has any parameters marked with the return
attribute. If there is no attribute, local refs + function
returning ref is OK.
But this issue exists even without locale `ref`s:
void foo() {
RCArray!int arr = [7];
bar(arr, arr[0]);
}
void bar(ref RCArray!int arr, ref int r) {
arr = [9];// this releases the old array
r++; // use after free
}
This is the reason for the @rc DIP.
It can be solved in one of two ways: Either by making the owner
(`arr`) non-mutable during the existence of the references,
thereby forbidding the call to `bar()` (I would prefer this
one, as it's cleaner and can be used for many more things, e.g.
the byLine problem)
I don't see directly how this affects byLine.front, that does not
return a reference.
Re: aliasing expressions and identifiers
On Wednesday, 25 May 2016 at 19:47:06 UTC, Nick Treleaven wrote:
On 24/05/2016 14:48, Nick Treleaven wrote:
What about:
@safe unittest
{
RCArray!int arr;
+ arr.length = 1;
ref r = arr[0];
arr.destroy; // refcount drops to zero, arr.impl memory
freed
r++; // writes to unallocated memory
}
Here I think local refs must be prevented from initialization
by return ref (-dip25). The @rc DIP and RCArray would use
return ref. It would be OK to initialize a local ref with a ref
function result that is not return ref. Naturally, this would
be @safe:
auto slice = [7];
ref r = slice[0];
slice.destroy;
r++; // slice memory still allocated
To elaborate: neither `scope` nor reference counting can ever
protect you against explicit premature destruction of a
still-referenced owner. But there is a slightly different
problematic scenario:
RCArray!int arr = [7];
ref r = arr[0];
arr = [9];// this releases the old array
r++; // use after free
But this issue exists even without locale `ref`s:
void foo() {
RCArray!int arr = [7];
bar(arr, arr[0]);
}
void bar(ref RCArray!int arr, ref int r) {
arr = [9];// this releases the old array
r++; // use after free
}
It can be solved in one of two ways: Either by making the owner
(`arr`) non-mutable during the existence of the references,
thereby forbidding the call to `bar()` (I would prefer this one,
as it's cleaner and can be used for many more things, e.g. the
byLine problem), or by making the owner live longer by inserting
the appropriate AddRef/Release pairs whenever such a situation
arises.
Re: aliasing expressions and identifiers
On Tuesday, 24 May 2016 at 13:48:41 UTC, Nick Treleaven wrote:
What about:
@safe unittest
{
RCArray!int arr;
ref r = arr[0];
arr.destroy; // refcount drops to zero, arr.impl memory freed
r++; // writes to unallocated memory
}
You're calling `destroy` explicitly, what else would you expect
to happen?
Re: aliasing expressions and identifiers
On 24/05/2016 14:48, Nick Treleaven wrote:
On Monday, 23 May 2016 at 17:03:32 UTC, Marc Schütz wrote:
On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:
I think the reason D doesn't support local refs is because it would
make it harder to design @safe, particularly with the planned @rc
ref-counting.
They wouldn't pose a problem for lifetime tracking,
because they can never be assigned to, only initialized once.
What about:
@safe unittest
{
RCArray!int arr;
+ arr.length = 1;
ref r = arr[0];
arr.destroy; // refcount drops to zero, arr.impl memory freed
r++; // writes to unallocated memory
}
Here I think local refs must be prevented from initialization by return
ref (-dip25). The @rc DIP and RCArray would use return ref. It would be
OK to initialize a local ref with a ref function result that is not
return ref. Naturally, this would be @safe:
auto slice = [7];
ref r = slice[0];
slice.destroy;
r++; // slice memory still allocated
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
Re: aliasing expressions and identifiers
On Monday, 23 May 2016 at 17:43:49 UTC, deed wrote: On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote: Recomputation or not, @safe or @system, should be no different from the expanded code. So within a scope, after aliasing 'm', 'm' should be replaced by 'matrix.rawArr'. Everything stays the same as is, even error messages. I think it's more useful to support local refs than alias expressions. In fact apart from the rvalue support, I would rather use a local ref than the existing with statement (assuming the ref was memory safe). The only exception is when using `with (EnumType)`. Also, compile-time expressions use enum, not alias. It could be changed, but I think it's useful when reading meta-programming code to clearly see the difference. Currently aliased symbols are expanded by the compiler in error messages, but that's probably not ideal.
Re: aliasing expressions and identifiers
On Monday, 23 May 2016 at 17:03:32 UTC, Marc Schütz wrote:
On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:
If we had local refs, we could use this instead:
ref m = matrix.rawArr;
Note that this wouldn't work with rvalues, which `with`
supports.
OK. I suppose supporting local refs is a good reason not to allow
rvalues to be passed as const ref arguments, if the function also
returns by ref.
I think the reason D doesn't support local refs is because it
would make it harder to design @safe, particularly with the
planned @rc ref-counting. Because M() above is only a return
reference, it can't live longer than the data it references.
My ref m could persist longer than matrix.rawArr using (naive)
reference counting.
At some point during the `scope` discussion, Walter wanted to
allow local `ref`s, so I guess he's not opposed to the idea. As
Great :-)
far as I understand, the compiler already supports them
internally, as they can result from lowering certain
constructs, there's just no syntax for them. They wouldn't pose
a problem for lifetime tracking, because they can never be
assigned to, only initialized once.
What about:
@safe unittest
{
RCArray!int arr;
ref r = arr[0];
arr.destroy; // refcount drops to zero, arr.impl memory freed
r++; // writes to unallocated memory
}
Re: aliasing expressions and identifiers
On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:
If we had local refs, we could use this instead:
ref m = matrix.rawArr;
The difference is m is already computed, it is not recomputed
each time m is read, unlike M(). I think the reason D doesn't
support local refs is because it would make it harder to design
@safe, particularly with the planned @rc ref-counting. Because
M() above is only a return reference, it can't live longer than
the data it references. My ref m could persist longer than
matrix.rawArr using (naive) reference counting.
BTW local refs are possible in @system code:
http://forum.dlang.org/post/lmuokynffgljzvrpv...@forum.dlang.org
Just to be clear: I'm not looking for anything touching the
semantics, just a simple replacement mechanism/macro expansion.
Recomputation or not, @safe or @system, should be no different
from the expanded code. So within a scope, after aliasing 'm',
'm' should be replaced by 'matrix.rawArr'. Everything stays the
same as is, even error messages.
{
// Some scope
alias i = instance.targetIdx;
alias m = matrix.rawArr;
m[i] = m[j] + m[k];
// detected and conceptually replaced in one of the earlier
compiler passes by
// matrix.rawArr[instance.targetIdx] = matrix.rawArr[j] +
matrix.rawArr[k]
}
Re: aliasing expressions and identifiers
On Monday, 23 May 2016 at 15:18:51 UTC, Nick Treleaven wrote:
On Monday, 23 May 2016 at 14:05:43 UTC, deed wrote:
Some thoughts about extending the with-statement were brought
up here earlier:
http://forum.dlang.org/post/txpifmwpmmhsvcpbc...@forum.dlang.org
I don't care much whether it would be with, alias or possibly
something clever already existing, but a solution should be
easy to use, easy to read and shouldn't introduce any possible
overhead at runtime.
From the linked thread:
ref M() { return matrix.rawArr; }
ref Ex1() { return e1.someProperties.someModulusX; }
If we had local refs, we could use this instead:
ref m = matrix.rawArr;
Note that this wouldn't work with rvalues, which `with` supports.
The difference is m is already computed, it is not recomputed
each time m is read, unlike M(). I think the reason D doesn't
support local refs is because it would make it harder to design
@safe, particularly with the planned @rc ref-counting. Because
M() above is only a return reference, it can't live longer than
the data it references. My ref m could persist longer than
matrix.rawArr using (naive) reference counting.
At some point during the `scope` discussion, Walter wanted to
allow local `ref`s, so I guess he's not opposed to the idea. As
far as I understand, the compiler already supports them
internally, as they can result from lowering certain constructs,
there's just no syntax for them. They wouldn't pose a problem for
lifetime tracking, because they can never be assigned to, only
initialized once.
Re: aliasing expressions and identifiers
On Monday, 23 May 2016 at 14:05:43 UTC, deed wrote:
Some thoughts about extending the with-statement were brought
up here earlier:
http://forum.dlang.org/post/txpifmwpmmhsvcpbc...@forum.dlang.org
I don't care much whether it would be with, alias or possibly
something clever already existing, but a solution should be
easy to use, easy to read and shouldn't introduce any possible
overhead at runtime.
From the linked thread:
ref M() { return matrix.rawArr; }
ref Ex1() { return e1.someProperties.someModulusX; }
If we had local refs, we could use this instead:
ref m = matrix.rawArr;
The difference is m is already computed, it is not recomputed
each time m is read, unlike M(). I think the reason D doesn't
support local refs is because it would make it harder to design
@safe, particularly with the planned @rc ref-counting. Because
M() above is only a return reference, it can't live longer than
the data it references. My ref m could persist longer than
matrix.rawArr using (naive) reference counting.
BTW local refs are possible in @system code:
http://forum.dlang.org/post/lmuokynffgljzvrpv...@forum.dlang.org
aliasing expressions and identifiers
As I've been finding myself constantly desiring to alias expressions and identifiers for many years, I'd like to ask again whether it would be possible to extend either the alias-statement or the with-statement. It would be great to relieve the constant tension between descriptiveness, (often outside your control), and brevity in a straightforward manner. There is something missing when your fields and parameters start out with proper names but end up as cryptic abbreviations when you start using them. Sometimes you want to look at an identifier and understand its purpose, other times you want to look at a bunch of expressions and statements and understand their pattern. I think an expressive language should support both needs without tension. It would also isolate names and reduce editing when names change. Some thoughts about extending the with-statement were brought up here earlier: http://forum.dlang.org/post/txpifmwpmmhsvcpbc...@forum.dlang.org I don't care much whether it would be with, alias or possibly something clever already existing, but a solution should be easy to use, easy to read and shouldn't introduce any possible overhead at runtime. 1) Are there any technical blockers for extending the alias- or with-statement? 2) Does it carry its own weight?
