Re: delegate confusion

[Timon Gehr via Digitalmars-d]

On 04.08.2017 18:57, bitwise wrote:
I'm confused about how D's lambda capture actually works, and can't find 
any clear specification on the issue. I've read the comments on the bug 
about what's described below, but I'm still confused. The conversation 
there dropped off in 2016, and the issue hasn't been fixed, despite high 
bug priority and plenty of votes.

Consider this code:

void foo() {
     void delegate()[] funs;

     foreach(i; 0..5)
         funs ~= (){ writeln(i); };

     foreach(fun; funs)
         fun();
}

void bar() {
     void delegate()[] funs;

     foreach(i; 0..5)
     {
         int j = i;
         funs ~= (){ writeln(j); };
     }
     foreach(fun; funs)
         fun();
}


void delegate() baz() {
     int i = 1234;
     return (){ writeln(i); };
}

void overwrite() {
     int i = 5;
     writeln(i);
}

int main(string[] argv)
{
     foo();
     bar();

     auto fn = baz();
     overwrite();
     fn();

     return 0;
}

First, I run `foo`. The output is "4 4 4 4 4".
So I guess `i` is captured by reference, and the second loop in `foo` 
works because the stack hasn't unwound, and `i` hasn't been overwritten, 
and `i` contains the last value that was assigned to it.

Next I run `bar`. I get the same output of "4 4 4 4 4". While this hack 
works in C#,

It's very important to understand that the C# is different, even though 
it looks similar. In D, the foreach loop variable is a distinct 
declaration for each loop iteration, while in C#, the same loop variable 
is repeatedly reassigned. In C#, the issue is bad language design, while 
in D, the issue is a buggy compiler implementation leading to memory 
corruption.

I suppose it's reasonable to assume the D compiler would 
just reuse stack space for `j

It's reasonable to assume that the D compiler uses the same memory 
location for all of the distinct variables. This is a dangling pointer 
bug, if you wish. Both of your examples should print "0 1 2 3 4".

and that the C# compiler has some 
special logic built in to handle this.
...

The C# compiler just uses the correct rules for creating closures. (It 
is hard for the compiler to screw this up, because the underlying 
platform aims to prevents memory corruption.)

Now, I test my conclusions above, and run `baz`, `overwrite` and `fn`. 
The result? total confusion.
The output is "5" then "1234". So if the lambdas are referencing the 
stack, why wasn't 1234 overwritten?
...

The lambdas are referencing the heap, but all of them reference 
identical heap locations. This should not happen. Distinct variables 
shouldn't share the same memory.

Take a simple C++ program for example:

int* foo() {
     int i = 1234;
     return &i;
}

void overwrite() {
     int i = 5;
     printf("%d\n", i);
}

int main()
{
     auto a = foo();
     overwrite();
     printf("%d\n", *a);
     return 0;
}

This outputs "5" and "5" which is exactly what I expect, because I'm 
overwriting the stack space where the first `i` was stored with "5".
 > So now, I'm thinking.... D must be storing these captures on the heap
then..right? So why would I get "4 4 4 4 4" instead of "0 1 2 3 4" for 
`foo` and `bar`?

This makes absolutely no sense at all.

It seems like there are two straight forward approaches available here:

1) capture everything by reference, in which case the `overwrite` 
example would work just like the C++ version. Then, it would be up to 
the programmer to heap allocate anything living beyond the current scope.
...

Capturing by reference is not the same as creating stack references. The 
language semantics don't even need to be implemented using a stack.

2) heap allocate a chunk of space for each lambda's captures, and copy 
everything captured into that space when the lambda is constructed. This 
of course, would mean that `foo` and `bar` would both output "0 1 2 3 4".
...

3) heap allocate a chunk of space for each captured scope (as in lisp 
and C#).

The way to go is 3). 1) is bad, because it completely prevents closures 
from being escaped, 2) is bad because it does not allow sharing of 
closure memory.

When I look at the output I get from the code above though, it seems 
like neither of these things were done, and that someone has gone way 
out of their way to implement some very strange behavior.
...

Absolutely not. The current behavior was quite straightforward to 
implement, but it is wrong. Bugs often lead to strange behavior. This 
does not imply that such bugs are intentional.

What I would prefer, would be a mixture of reference and value capture 
like C++, where I could explicitly state whether I wanted (1) or (2). I 
would settle for (2) though.
...

"Like C++" does not work: in C++, each lambda has its own unique type.

While I'm sure there is _some_ reason that things currently work the way 
they do, the current behavior is very unintuitive, and gives no control 
over how things are captured.


You can work around the bug like this:

foreach(i;0..5)(){
    int j=i;
    funs~=(){ writeln(j); };
}()