Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 16:52:06 UTC, Paul Backus wrote:
On Monday, 13 November 2023 at 16:23:05 UTC, Tim wrote:
The visitor can already be `extern(D)`. Only member functions
overriding those in the base class need to be `extern(C++)`.
Other member functions can then use paratemers, which would be
incompatible with `extern(C++)`.
Wow, it really was that simple all along. Thanks for the tip!
That's not a complete solution. If you template the visitor and
have some member functions that override the extern(C++) base
class methods, then you will still end up with errors if you use
any types that use non-mapable C++ types because of the mangling.
For example:
```d
extern(C++) class DMDVisitor
{
void visit(Expression e) {}
}
class MyVisitor(T) : DMDVisitor
{
T field;
extern(C++) override void visit(Expression e) {} //
troubled line
}
```
Instantiating MyVisitor with a string type will issue an error at
the troubled line because the mangler needs to take into account
the template instance. Since C++ has no dynamic array
correspondent you essentially cannot call MyVisitor.visit from
C++ and therefore the compiler issues an error.
However, if you don't use templated visitors you are fine, you
can just mark your introducing functions/fields as extern(D) and
even if the class is extern(C++) it will work.
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 16:09:51 UTC, Guillaume Piolat wrote: On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 Thanks for the detailed summary, I'm reading them all! Seconded, though I didn't follow all of it...
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 16:23:05 UTC, Tim wrote: The visitor can already be `extern(D)`. Only member functions overriding those in the base class need to be `extern(C++)`. Other member functions can then use paratemers, which would be incompatible with `extern(C++)`. Wow, it really was that simple all along. Thanks for the tip!
Re: DLF September 2023 Monthly Meeting Summary
On Sunday, 12 November 2023 at 21:55:31 UTC, Paul Backus wrote: Personally, my number-one complaint with dmd-as-a-library is that I am forced to use `extern(C++)` when creating my own `Visitor` classes. In [`dmdtags`][1], this requirement has made it necessary for me to implement my own C++-compatible [`Span`][2] and [`Appender`][3] types, just to avoid the C++ mangling errors caused by D's built-in `T[]` slices. I have no use for overriding AST nodes, but the ability to use `extern(D)` visitors with dmd-as-a-library would be a welcome improvement. The visitor can already be `extern(D)`. Only member functions overriding those in the base class need to be `extern(C++)`. Other member functions can then use paratemers, which would be incompatible with `extern(C++)`.
Re: DLF September 2023 Monthly Meeting Summary
On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 Thanks for the detailed summary, I'm reading them all!
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 10:01:23 UTC, RazvanN wrote: On Sunday, 12 November 2023 at 21:55:31 UTC, Paul Backus wrote: I have no use for overriding AST nodes, but the ability to use `extern(D)` visitors with dmd-as-a-library would be a welcome improvement. I have already brought that up to one of our work group meetings regarding dmd as a library as I have stumbled upon this also. We have a solution for this, I'm going to try to implement it this week. That's great to hear! Thanks for letting me know.
Re: DLF September 2023 Monthly Meeting Summary
Agreed, getting good locking support is critical for safe memory handling at the process level. Shared as we know it today, is just a distraction from this goal.
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 00:18:35 UTC, Richard (Rikki)
Andrew Cattermole wrote:
Part of the problem with shared is that it is completely
inverse of what it should be.
It fundamentally does not annotate memory with anything extra
that is useful.
At the CPU level there are no guarantees that memory is mapped
only to one thread, nor at the language level.
Therefore all memory is shared between threads.
As long as people have this inverted mindset in place and
instead of wanting to prove its thread owned shared is going to
be a problem for us.
Remove shared, add atomic storage class. Figure out thread
owned/shared memory some other day.
Just to once again add my own view, since I don't see it
represented a lot:
Remove shared or don't, at least *stop making `synchronized`
imply `shared`.* I don't care about shared, I don't care about
atomics, I want to use the mutex style of protecting access,
where I manually think about who exclusively owns what memory.
But I can't use `synchronized`, because then I have to cast
*every access to `this`,* which is clearly insane, and as a
result invariant-using class code is *straight up incorrect,*
because I have literally no choice but to do this:
```
class Foo
{
private int[] array;
invariant { synchronized (this) { assert(this.array.all!(i => i
> 0)); } }
void foo(int i)
in (i > 0)
{
synchronized (this)
{
// Note that the invariant on `array` is not actually
guaranteed here!
// Some other method in another thread might currently have
set up a violation
// for it, but switched out before its out invariant could
be locked and tested.
// So we can have a *wrong invariant in this method,* which
has done absolutely
// nothing wrong. Very fun!!
array ~= i;
}
}
}
```
The only way to prevent this is `synchronized class`, but even if
all missing features are implemented, `array` would be `shared`.
Which is just wrong, as `array` is *privately owned storage of
the class,* a fact I cannot prove to the compiler but is
nonetheless true. `synchronized class` brings in the one feature
I don't want to use and makes it an unavoidable condition of
*literally writing correct code at all*. So yeah `shared`, I
didn't *want* to be your enemy but apparently you wanted to be
mine. `shared` delenda est.
Re: DLF September 2023 Monthly Meeting Summary
On Sunday, 12 November 2023 at 21:55:31 UTC, Paul Backus wrote: On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 There was a side discussion about how the `extern(C++)` interface affects dmd-as-a-library. Personally, my number-one complaint with dmd-as-a-library is that I am forced to use `extern(C++)` when creating my own `Visitor` classes. In [`dmdtags`][1], this requirement has made it necessary for me to implement my own C++-compatible [`Span`][2] and [`Appender`][3] types, just to avoid the C++ mangling errors caused by D's built-in `T[]` slices. I have no use for overriding AST nodes, but the ability to use `extern(D)` visitors with dmd-as-a-library would be a welcome improvement. [1]: https://code.dlang.org/packages/dmdtags [2]: https://github.com/pbackus/dmdtags/blob/v1.1.1/src/dmdtags/span.d [3]: https://github.com/pbackus/dmdtags/blob/v1.1.1/src/dmdtags/appender.d I have already brought that up to one of our work group meetings regarding dmd as a library as I have stumbled upon this also. We have a solution for this, I'm going to try to implement it this week. RazvanN
Re: DLF September 2023 Monthly Meeting Summary
On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote: Well. For the first time in all my years of using these forums, I've managed to post something that exceeds the byte limit. You'll find the September 2023 Monthly Meeting Summary at the following link: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 Thank you for posting. As always, it's interesting read. And it's great for the community to get the feeling for what's happening. Nice to see that people such as Adam and Timon joined these discussions recently. Topic-wise, I would be interested in how the "shared" is going to be shaped. Especially, I remember Manu advocating for a quite simple in principle approach to it, but he never got to much positive vibes from the community. I noticed his suggestions mentioned in the summary notes of the DLF meeting, and wondered whether some of those ideas will be part of the discussion.
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 04:47:42 UTC, matheus wrote: That's it! Matheus. Thank you, Matheus.Have a nice day.
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 02:23:22 UTC, Imperatorn wrote: https://paste.myst.rs/u074ali8 Thank you,Have a nice day!
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 03:07:07 UTC, Mike Parker wrote: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 I can't access it,please post it here. I can't. It's too big. That's why I posted it there. I can't access `https://gist.github.com`, Anyway, thank you for `your efforts`.
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 04:46:44 UTC, matheus wrote: ... Part 3: Átila Átila said he'd asked Roy after DConf if he wanted to write a spec for the new shared semantics but hadn't yet received a reply. He thought we could go ahead with it anyway, as it's probably going to be obvious in all the cases what we should do. He'd also been thinking about Robert's suggestions regarding safe code, and the more he'd done so, the more they made sense. He didn't think we needed to wait for editions, because then we could solve all the problems we've had with DIP 1000, possibly in one fell swoop. The whole issue was we were making people have to add scope everywhere because they use @safe, even if the functions weren't doing anything "unscopey". They wouldn't have to anymore because it would only apply to @trusted. He thought we should go ahead with that, too. Another thing: he'd been trying to play around with Timon's idea of pointers that have a compile-time enum saying if they're isolated or not. He wanted to do a proof of concept with a vector type and some allocators to see what that looks like. That might inform some decisions on how to add isolated to the language. He'd also been thinking about how we could do editions. I brought up an email Átila had sent recently about what features should go in the next edition. I suggested that's not what we need to be thinking about right now. We should first define exactly how editions were going to be implemented. For example, Walter had suggested the approach of attaching editions to module declarations. There are other possibilities. So we need to define what that looks like first. We should also be thinking about which features we need to have stabilized in the current language, which is going to be the default edition. What's broken now that we're not going to fix now, but want to defer until the first new edition? Átila agreed. He said he'd wanted to know what we could maybe do with the new edition so that it could inform what potential issues could arise rather than dealing with it in the abstract. It's more because of that. Not even, "that's what we're going to do", but "what would we do if we could do anything" so that we could think of what could go wrong. I thought this should be a high priority for us. I proposed that we put it on the agenda of our next planning session. Everyone agreed. (SPOILER: I'll post the September planning update a day or so after this summary, but in that session, we agreed that Átila would write up the proposal for editions by November 1st. We've since extended that to December.) Before yielding his time, Átila said he'd like to write a program that fetches all of the dub packages and checks which ones still built so he could get a list of stuff we need to test the kinds of modifications like the shared and safe stuff, along with as many future dub packages as possible, instead of the hand-selected, curated list we have now. Dennis thought even Phobos would break. He said that DIP 1000 was not a breaking change. The breaking change was fixing the accepts invalid bug that, so far, slicing a local static array has been allowed. Robert's proposal was strictly more breakage than DIP 1000. Átila said he understood that, but that code was obviously wrong anyway. The code was broken. It's just that the compiler didn't tell people. And the issue with DIP 1000 was adoption. The issue with that was having to add scope everywhere. I said that's the kind of thing I was talking about earlier. If fixing shared or scope or anything was going to break a lot of code, shouldn't we defer fixing them to a new edition and not do it in the default language? Walter said DIP 1000 clearly had to be in an edition. Átila said, "Okay." But the shared thing is a no-brainer. The code is completely wrong and we're not going to break anything by lowering it to atomics. Walter agreed. Timon asked if the idea is to do operations on shared variables atomically. Walter said yes, for the ones where the CPU supports an atomic operation. If the CPU doesn't support an atomic operation on a variable, then it will not be allowed. It varies by target. Timon said yes, but an atomic operation is not the only way to synchronize something. Átila said that's true, but the thing is, there's always an opt-out by casting away shared yourself. If you're going to do something to shared directly, you should either pass it to a function that takes shared, or cast it away and lock a mutex, or whatever you want. Timon brought up Manu's push to just reject all of those operations. Átila said that doesn't work. He'd been trying to fix it this year and it's been one bug after another. Timon asked what the issue with that was. Átila said there were many, but gave one example: synchronized(mutex) doesn't compile with -preview=nosharedaccess because you're accessing the mutex. The runtime
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 04:46:07 UTC, matheus wrote:
...
Part 2:
AST nodes in dmd-as-a-library
Since DConf, Razvan had been considering how dmd-as-a-library
could offer the possibility to override any kind of AST.
He gave us this example of the expression class hierarchy used by
the AST implementation:
module expression;
import std.stdio;
class Expression {}
class BinExp : Expression
{
void fun()
{
writeln("expression.fun");
BinExp p = new BinExp();
Expression e = new Expression();
}
}
class BinAssignExp : BinExp
{}
class AddAssignExp : BinAssignExp
{
override void fun()
{
writeln("AddAssignExp");
}
}
class MulAssignExp : BinAssignExp
{}
Then in ast_family.d, the default ASTCodegen looks like this:
struct ASTCodegen
{
public import expression;
}
To create a custom AST and override the default behavior of e.g.,
BinExp, you need to do something like this:
struct MyAST
{
import expression;
import std.stdio;
alias Expression = expression.Expression;
class MyBinExp : expression.BinExp
{
override void fun()
{
writeln("MyBinExp.fun");
}
}
alias BinExp = MyBinExp;
alias BinAssignExp = ?
}
The problem with this is that you now have to declare all of the
AST nodes that inherit from BinExp so that they use your custom
implementation. This is not a workable solution. We need the
ability to specify not only that we're overriding a particular
node, but that other nodes need to use it.
First, he thought about templating the AST nodes and inheriting
from the templated version, but that means heavily modifying the
compiler. Then he came up with a solution using mixins. You just
mixin the code of the AST node that you want.
With this approach, the AST nodes are now mixin templates:
module expression_mixins;
import std.stdio;
mixin template Epression_code()
{
class Expression {}
}
mixin template BinExp_code()
{
class BinExp : Expression
{
void fun()
{
writeln("expression.fun");
BinExp p = new BinExp();
Expression e = new Expression();
}
}
}
mixin template BinAssignExp_code()
{
class BinAssignExp : BinExp
{}
}
mixin template AddAssignExp_code()
{
class AddAssignExp : BinAssignExp
{
override void fun()
{
writeln("AddAssignExp");
}
}
}
mixin template MulAssignExp_code()
{
class MulAssignExp : BinAssignExp
{}
}
And then the expression module becomes:
module expression;
import expression_mixins;
import std.stdio;
mixin Expression_code();
mixin BinExp_code();
mixin BinAssignExp_code();
mixin AddAssignExp_code();
mixin MulAssignExp_code();
In ast_family, ASTCodegen remains the same, but now you can do
this for your custom AST:
struct MyAst
{
import expression_mixins;
import std.stdio;
mixin Expression_code();
mixin BinExp_code() t;
class MyBinExp : t.BinExp
{
override void fun()
{
writeln("MyBinExp.fun");
}
}
alias BinExp = MyBinExp;
mixin BinAssignExp_code();
mixin AddAssignExp_code();
mixin MulAssignExp_code();
}
We could have something in the frontend library to generate the
boilerplate automatically. But the main thing is that now you can
replace any default node in the hierarchy with your custom
implementation without needing to redeclare everything. In this
example, everything that inherits from BinExp is now going to
inherit from MyBinExp instead. This works. He showed a runnable
example.
He doesn't think this is that ugly. And for what it gives us,
basically a pluggable AST, any perceived ugliness is worth it. He
said it would be great if we could reach a consensus on how to go
forward.
Átila said he liked it.
Razvan noted that a problem is that the semantic routine visitors
aren't going to work anymore. But the cool thing is you can also
put those in mixins. You mix those in with your custom AST, you
inherit from and override whatever visiting nodes you want, and
you get all of the functionality you need.
Timon said his main concern with this kind of scheme is that he
has tried them in the past, and usually dmd dies when it tries to
build itself. He thinks the current version of dmd will choke on
this at some point. It always appears to work at the start, but
if you scale it up enough, random stuff starts to break, like an
"undefined identifier string", a general forward reference error,
or an ICE, etc.
Razvan said he had encountered the undefined identifier thing,
but you can work around that by inserting an alias in the problem
spot. Regardless, he argued that any such error is a compiler bug
that needs to be fixed.
Timon agreed, but his question was how do you navigate that when
the compiler can't build itself because of a compiler bug? Razvan
said he'd fix the bug.
Dennis noted
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 03:07:07 UTC, Mike Parker wrote:
On Monday, 13 November 2023 at 00:55:37 UTC, zjh wrote:
On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote:
https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581
I can't access it,please post it here.
I can't. It's too big. That's why I posted it there.
Well maybe splitting in 2 parts? - Let's try:
Part 1:
DLF September 2023 Monthly Meeting Summary
The D Language Foundation's monthly meeting for September 2023
took place on Friday the 15th at 15:00 UTC. After we'd had one of
our shortest meetings ever the previous month, this one was a bit
of a long one, lasting a little over two hours.
Note that this was Timon Gehr's first time joining us for a
monthly. I'd spoken with him at DConf, and he'd expressed
interest in joining both our monthlies and our planning sessions
when he has the time for them. I'll be inviting him as a
permanent member as long as he's willing to be one.
The Attendees
The following people attended the meeting:
Walter Bright
Timon Gehr
Martin Kinkelin
Dennis Korpel
Mathias Lang
Átila Neves
Razvan Nitu
Mike Parker
Adam D. Ruppe
Robert Schadek
Steven Schveighoffer
Robert
Robert got us started by letting us know he had done some
preliminary JSON 5 work at DConf. He also gave an update on his
script for the Bugzilla to GitHub migration. He had changed it to
use a "hidden" API that someone from GitHub revealed to him when
he reached out for assistance. Though there are still rate limits
to deal with, his script was now much faster. The previous script
would have taken days to migrate the dmd issues, but for a test
run, he was able to do it in one sitting at DConf. He was ready
to show me how to use it so I could test it and provide him with
feedback.
Other than that, he'd done a few small things on DScanner and was
waiting on Jan Jurzitza (Webfreak) to merge them. He noted that
Walter had asked him to write an article for the blog related to
his DConf talk. Robert had an idea for an article related to the
DScanner updates to supplement the talk.
(UPDATE: During a subsequent planning session, Robert reminded me
that the only reason I had volunteered to do the migration was
that he didn't have admin access to our repositories. That was
easily rectified. He will now be doing the migration. At our most
recent planning session, we talked about a migration plan. Before
taking any steps, he's going to chat with Vladimir Panteleev.
Vladimir raised several concerns with me a while back about the
Dlang bot and other things the migration might affect. Robert
wants to get up to speed with all of that before moving forward.)
Me
I told everyone I'd just gotten home from my DConf/vacation trip
two days before the meeting and had spent a chunk of that time
decompressing. I did manage to get a little bit of post-DConf
admin out of the way the night before by going through all the
receipts from everyone eligible for reimbursement to let them
know how much was due to them. I went into some details on how I
was going to make those payments. The big news there was that we
managed to get enough in revenue from registrations and
sponsorships that we came in under budget, i.e., the amount
Symmetry needed to send us to make up the difference was less
than the total they'd allocated for reimbursements. (Thanks to
Ahrefs, Ucora, Decard, Funkwerk, and Weka for helping out with
that!)
I then reported that I'd started editing Saeed's video. The venue
had provided me access to all of their footage this year. Last
year, they only gave me footage from one camera and wanted us to
pay extra for more. This year, I have footage from three cameras
('main', 'wide', and 'left') as well as the video feed of the
slides.
Next, I noted that John Colvin and I had participated in an
after-action meeting with Sarah and Eden from Brightspace, our
event planners (if you were at DConf, Eden was the young woman
sitting out front all four days, and Sarah was with her on the
first day). We all agreed that, aside from the unfortunate laptop
theft and COVID outbreak, the mechanics of the conference went
well this year. We went through some feedback we'd received to
discuss how to improve things next year (more info on badges, an
actual registration form to get that info, etc.), and tossed
around some ideas on how to prevent future thefts and mitigate
the risk of COVID outbreaks. One thing we agreed on there is to
have an extra person at the door whose main job is to check
badges. There will surely be other steps to take once we consult
with the venue. They're evaluating what measures they can take to
avoid a repeat at any event they host.
I also let everyone know what Sarah said about our community. Due
to disruptions at Heathrow at the start of the conference,
several attendees found themselves with cancel
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 00:55:37 UTC, zjh wrote: On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 I can't access it,please post it here. I can't. It's too big. That's why I posted it there.
Re: DLF September 2023 Monthly Meeting Summary
On Monday, 13 November 2023 at 00:55:37 UTC, zjh wrote: On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 I can't access it,please post it here. https://paste.myst.rs/u074ali8
Re: DLF September 2023 Monthly Meeting Summary
On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 I can't access it,please post it here.
Re: DLF September 2023 Monthly Meeting Summary
Part of the problem with shared is that it is completely inverse of what it should be. It fundamentally does not annotate memory with anything extra that is useful. At the CPU level there are no guarantees that memory is mapped only to one thread, nor at the language level. Therefore all memory is shared between threads. As long as people have this inverted mindset in place and instead of wanting to prove its thread owned shared is going to be a problem for us. Remove shared, add atomic storage class. Figure out thread owned/shared memory some other day.
Re: DLF September 2023 Monthly Meeting Summary
On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 There was a side discussion about how the `extern(C++)` interface affects dmd-as-a-library. Personally, my number-one complaint with dmd-as-a-library is that I am forced to use `extern(C++)` when creating my own `Visitor` classes. In [`dmdtags`][1], this requirement has made it necessary for me to implement my own C++-compatible [`Span`][2] and [`Appender`][3] types, just to avoid the C++ mangling errors caused by D's built-in `T[]` slices. I have no use for overriding AST nodes, but the ability to use `extern(D)` visitors with dmd-as-a-library would be a welcome improvement. [1]: https://code.dlang.org/packages/dmdtags [2]: https://github.com/pbackus/dmdtags/blob/v1.1.1/src/dmdtags/span.d [3]: https://github.com/pbackus/dmdtags/blob/v1.1.1/src/dmdtags/appender.d
Re: DLF September 2023 Monthly Meeting Summary
On Sunday, 12 November 2023 at 19:50:02 UTC, Mike Parker wrote: Well. For the first time in all my years of using these forums, I've managed to post something that exceeds the byte limit. You'll find the September 2023 Monthly Meeting Summary at the following link: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581 Two notes: - Making the build bit of build.d simpler would be nice although reading the source code will reveal that it doesn't just build the compiler. - I think Martin is wrong about not upstreaming stuff from LDC into DMD. Nothing good will come of them diverging (which they already have). The UX of having multiple compilers is already terrible — conservatively let's estimate that "oh you need to download LDC if you want a compiler with an optimizer that works to a modern standard" halves the number of people bothering to try D. Every little detail that you have to think about other than actually writing code hurts D both in terms of adoption and massive fragmentation of the projects people have to contribute to. Standardising proper hooks such that LDC doesn't have to use a fork of the frontend (it's not really dmd-as-a-library in any sense that would be tolerated for a smaller project) is also important dogfooding for the frontend as a codebase.
DLF September 2023 Monthly Meeting Summary
Well. For the first time in all my years of using these forums, I've managed to post something that exceeds the byte limit. You'll find the September 2023 Monthly Meeting Summary at the following link: https://gist.github.com/mdparker/f28c9ae64f096cd06db6b987318cc581
