Re: Debugging mixins - we need to do something
On Saturday, 8 September 2018 at 22:01:23 UTC, Manu wrote: [snip] Is anyone interested in this issue? https://github.com/dlang/dmd/pull/8677
Re: Debugging mixins - we need to do something
On Sunday, 9 September 2018 at 03:02:41 UTC, tide wrote:
Oh another one from 2008, 10 years ago.
https://issues.dlang.org/show_bug.cgi?id=1870
Oh hey, a wild me appears.
So, yeah, the tricky bit with this is where to emit the source
code.
If you just want a best-effort aid, you could specify a file
manually to record string mixins in, and then error messages
within string mixins would refer to locations within that file.
(Which should just be a file number offset. There's already
special handling, so that shouldn't be a huge amount of
refactoring.) No attempt to append to an existing file; it's
going to blow away the existing contents each time.
This is a partial solution that gets a fair amount of value, and
it's where I'd probably start if I were trying to solve this.
But if you want to support this well enough for debug information
to point at it, that's harder.
Consider:
---
module sourcelib;
template Mixin(string toMixIn)
{
mixin(toMixIn);
}
---
module foo;
import sourcelib;
Mixin!`static assert(1 == 1);`;
---
module bar;
import sourcelib;
Mixin!`static assert(2 == 2);`;
---
Compile this with:
dmd -c foo.d
dmd -c bar.d
Now sourcelib.d-mixin.d contains:
static assert(1 == 1);
static assert(2 == 2);
You recompile foo.d and now it's got:
static assert(1 == 1);
static assert(2 == 2);
static assert(1 == 1);
Keep doing incremental builds and the length of this file is only
limited by your disk space. Plus your debug info has to point at
lines in this file, and the only way to determine that is by
reading the entire file, so that's going to slow down your build.
Okay, let's include the name of the file you're compiling. Now we
have two mixin logs:
foo.d-sourcelib.d-mixin.d:
static assert(1 == 1);
bar.d-sourcelib.d-mixin.d:
static assert(2 == 2);
Good, yes? Except now we change it so foo imports bar. What do we
see now?
foo.d-sourcelib.d-mixin.d:
static assert(2 == 2);
static assert(1 == 1);
bar.d-sourcelib.d-mixin.d:
static assert(2 == 2);
So we end up with very large mixin logs whenever we compile
nontrivial projects. Cue the command-line options for ignoring
mixins based on the name of the module doing the mixing in.
What if there are more layers of indirection? You need to walk up
the chain of templates to find the ultimate source to see if it's
listed on the command line. Or, in a more forward-looking way of
doing things, you can record that in whatever context object you
might have during template instantiation. Faster but potentially
more buggy.
It's also not terribly readable. We could add template
stacktraces to it:
foo.d-sourcelib.d-mixin.d:
// bar.d:3
// sourcelib.d:3
static assert(2 == 2);
// foo.d:3
// sourcelib.d:3
static assert(1 == 1);
bar.d-sourcelib.d-mixin.d:
// bar.d:3
// sourcelib.d:3
static assert(2 == 2);
I think this would work (not certain), but it's not simple.
That's why I'd prefer the best-effort version.
Re: Debugging mixins - we need to do something
On Sunday, 9 September 2018 at 00:58:15 UTC, Nicholas Wilson wrote: Obligatory "Bugzilla issue?". Obligatory "it already exists and has exited for X amount of years" (4 in this case). https://issues.dlang.org/show_bug.cgi?id=12790 As with most issues in bugzilla, no one has so much as made a comment on it or any other kind of activity. Oh another one from 2008, 10 years ago. https://issues.dlang.org/show_bug.cgi?id=1870 At least this one has someone adding to the issue, is Bill Baxter still around? Then 6 years later, someone says they want the feature and 2 years after that the guy that made the other issue adds a comment. Followed by 4 years of nothing. This just about describes Dlang's bug tracking as accurately as it gets.
Re: Debugging mixins - we need to do something
On Saturday, 8 September 2018 at 22:01:23 UTC, Manu wrote: TL;DR, debugging is critical and neglected. Mixin is worst offender. So, string mixin breaks the debugger. All of a sudden there is code that's executing that doesn't exist anywhere. We need a solution to this problem... As I and others have suggested before; mixin expansion should emit a `[sourcefile].d.mixin` file to the object directory, this file should accumulate mixin instantiations, and perhaps it would be ideal to also emit surrounding code for context. The debuginfo should point into that file when compiling mixin code, so that it's possible to step and inspect within the mixin. How is this done in C/C++ w.r.t macros? Point to within the macro? I initially imagined an 'expanded' copy of the file would be cool, but that doesn't work when mixins are driven by template args, since the expansion may be different for every instantiation of the surrounding code. There has been lots of chatter in the past, can we prioritise it? It would help me a lot. I think this is a high-importance ticket. I have absolutely no idea where to start with this, and I don't have anywhere near enough time to put towards my current initiatives (extern(C++) stuff). Is anyone interested in this issue? Obligatory "Bugzilla issue?". I don't think is should be too difficult to come up with a really dumb solution, just dump CompileStatement's string to a file, set the Lexer's loc to that file and let regular debug info generation do its thing. I'm busy for the next few days, but I'll take a crack at it if no-one beats me to it.
Re: Debugging mixins - we need to do something
On Saturday, 8 September 2018 at 22:01:23 UTC, Manu wrote: As I and others have suggested before; mixin expansion should emit a `[sourcefile].d.mixin` file to the object directory, this file should accumulate mixin instantiations, and perhaps it would be ideal to also emit surrounding code for context. This implies also emitting all template instantiation instances, right? I agree this is an important problem.
Debugging mixins - we need to do something
TL;DR, debugging is critical and neglected. Mixin is worst offender. So, string mixin breaks the debugger. All of a sudden there is code that's executing that doesn't exist anywhere. We need a solution to this problem... As I and others have suggested before; mixin expansion should emit a `[sourcefile].d.mixin` file to the object directory, this file should accumulate mixin instantiations, and perhaps it would be ideal to also emit surrounding code for context. The debuginfo should point into that file when compiling mixin code, so that it's possible to step and inspect within the mixin. I initially imagined an 'expanded' copy of the file would be cool, but that doesn't work when mixins are driven by template args, since the expansion may be different for every instantiation of the surrounding code. There has been lots of chatter in the past, can we prioritise it? It would help me a lot. Some background on this issue: (TL;DR, don't read) D's biggest draw for me is it's powerful meta, and ability to obliterate a huge quantity of typical C++ boilerplate. As C++ continues to close the gap, this becomes more and more the defining feature, and probably the primary basis for an argument that you might make to switch a C++ workflow to D. (closely followed by modules and build-times) Most uninterested and/or unopinionated people I've interacted with recently have agreed that D feels 'pretty good', and they'd be willing to explore it further if there was a compelling argument. The focus of resistance in my recent experience has been related 90% to tooling. In particular, IDE integration, and debug experience. (the other 10% being implementation issues in extern(C++), which I've been working to improve) >From my own experience over the last decade, there is a critical tension between this core value proposition; which is meta and boilerplate-reduction vs debugging experience, which is also critical (and neglected). I think the state of this tension is the most likely point where a case for D will fail with my particular set of colleagues. While I might balance that boilerplate reduction and code simplicity is worth the trade, they will argue (and I think they will win) that the fact you practically lose the ability to debug your code when there is meta nearby is very bad indeed. I think this is a high-importance ticket. I have absolutely no idea where to start with this, and I don't have anywhere near enough time to put towards my current initiatives (extern(C++) stuff). Is anyone interested in this issue?
Re: debugging mixins
On Wednesday, 5 October 2016 at 02:45:53 UTC, Stefan Koch wrote:
On Tuesday, 4 October 2016 at 01:59:11 UTC, Stefan Koch wrote:
On Tuesday, 4 October 2016 at 01:20:01 UTC, Manu wrote:
On 4 October 2016 at 04:21, Stefan Koch via Digitalmars-d
wrote:
On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler
wrote:
Yes, having the mixins expanded without the surrounding
code would make it difficult to debug in some cases. Maybe
generating the entire source with the expanded mixins is
another option?
mycode.d
obj/mycode_processed.d
That was my intention.
Maybe this idea could also be expanded to template
instantiation?
Oh yes. it is not that more much work :)
A small update on this.
The POC works rather well ...
Except for cases of massive template recursion. (binderoo and
most of std.traits)
In such cases a stack overflow occurs inside the prettyPrinter.
I am trying to find a work-around.
The simplest workaround is to run recursion heavy code in a fiber
with large stack size (e.g. 64MB).
// before
auto newExpr = expr.ctfeInterpret(); // some recursion heavy code
// after
Expression newExpr;
import core.thread : Fiber;
new Fiber(
{
newExpr = expr.ctfeInterpret(); // some recursion heavy code
}, 64 * 1024 * 1024).call();
Re: debugging mixins
On Tuesday, 4 October 2016 at 01:59:11 UTC, Stefan Koch wrote: On Tuesday, 4 October 2016 at 01:20:01 UTC, Manu wrote: On 4 October 2016 at 04:21, Stefan Koch via Digitalmars-d wrote: On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler wrote: Yes, having the mixins expanded without the surrounding code would make it difficult to debug in some cases. Maybe generating the entire source with the expanded mixins is another option? mycode.d obj/mycode_processed.d That was my intention. Maybe this idea could also be expanded to template instantiation? Oh yes. it is not that more much work :) A small update on this. The POC works rather well ... Except for cases of massive template recursion. (binderoo and most of std.traits) In such cases a stack overflow occurs inside the prettyPrinter. I am trying to find a work-around.
Re: debugging mixins
On Tuesday, 4 October 2016 at 01:20:01 UTC, Manu wrote: On 4 October 2016 at 04:21, Stefan Koch via Digitalmars-d wrote: On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler wrote: Yes, having the mixins expanded without the surrounding code would make it difficult to debug in some cases. Maybe generating the entire source with the expanded mixins is another option? mycode.d obj/mycode_processed.d That was my intention. Maybe this idea could also be expanded to template instantiation? Oh yes. it is not that more much work :) What case of template instantiation where there are no mixins involved would this make significantly simpler to debug? (I don't know this is a critical debugability problem as it is...) Do you mean just substituting 'T' with actual types? resolving static if's? Hard to know what it should do... Actually, one case that often bites me is static-foreach unrolling. That's borderline impossible to debug. foreach(m; __traits(allMembers,T)) is the classic impossible to debug case. static ifs are resolved when the compiler sees the template-instance in semantic3. And that makes a huge difference in some cases where a template is generated by a string-mixin for example. is not that big of a deal to print out unrolled static foreach. (as in I can implement in the compiler within 2 days)
Re: debugging mixins
On 4 October 2016 at 04:21, Stefan Koch via Digitalmars-d wrote: > On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler wrote: > >> >> Yes, having the mixins expanded without the surrounding code would make it >> difficult to debug in some cases. Maybe generating the entire source with >> the expanded mixins is another option? >> >> mycode.d >> obj/mycode_processed.d > > That was my intention. >> >> Maybe this idea could also be expanded to template instantiation? > > Oh yes. it is not that more much work :) What case of template instantiation where there are no mixins involved would this make significantly simpler to debug? (I don't know this is a critical debugability problem as it is...) Do you mean just substituting 'T' with actual types? resolving static if's? Hard to know what it should do... Actually, one case that often bites me is static-foreach unrolling. That's borderline impossible to debug. foreach(m; __traits(allMembers,T)) is the classic impossible to debug case.
Re: debugging mixins
On Monday, 3 October 2016 at 15:23:40 UTC, Jonathan Marler wrote: Yes, having the mixins expanded without the surrounding code would make it difficult to debug in some cases. Maybe generating the entire source with the expanded mixins is another option? mycode.d obj/mycode_processed.d That was my intention. Maybe this idea could also be expanded to template instantiation? Oh yes. it is not that more much work :)
Re: debugging mixins
On Sunday, 2 October 2016 at 03:36:31 UTC, Manu wrote: This comes up a lot. As far as I know, it's not solved. What shall we do? I feel like a simple solution would be to have the compiler emit a _mixin.d file populated with all the mixin expansions beside the .obj files, and have the debuginfo refer to that fabricated source file? It might look a little bit weird jumping into code where the surrounding scope is not visible (unless that were copied over too?), but it's better than what we have now. Are there any other commonly proposed solutions? Yes, having the mixins expanded without the surrounding code would make it difficult to debug in some cases. Maybe generating the entire source with the expanded mixins is another option? mycode.d obj/mycode_processed.d Maybe this idea could also be expanded to template instantiation?
Re: debugging mixins
On 10/03/2016 07:42 AM, Stefan Koch wrote: On Sunday, 2 October 2016 at 12:27:23 UTC, Andrei Alexandrescu wrote: Yes, Stefan it would be terrific if you could keep an eye on it while working on the engine. A file with properly handed dependencies could serve as instantiation cache and save a ton of re-instantiation waste. Thanks! -- Andrei instantiation ? Do you mean mixin-expansion ? or are you talking about mixins created in templates ? Mixin expansions. Although Caching mixin expansions hits on the same problem as chaching template-instances, it is more difficult since a mixin can appear anywhere. and one has to worry about a much wider scope. I'll see what I can come up with, for now the debugging is priority. @Ethan can you share code illustrating your usecase. (My head is inside compiler internals and building test-code is a rather unpleasant context switch) Understood, then keep this on the back burner and forge ahead with the warm cache. Thanks! -- Andrei
Re: debugging mixins
On Monday, 3 October 2016 at 11:42:25 UTC, Stefan Koch wrote: @Ethan can you share code illustrating your usecase. (My head is inside compiler internals and building test-code is a rather unpleasant context switch) You should be able to do everything with the example code you have. My intention there is to compile everything in the acorelibrary module in to a library and statically link against that. As such, you'll want to do the .di generation on those files. The behaviour right now is that all the Binderoo mixins will not expand.
Re: debugging mixins
On Sunday, 2 October 2016 at 12:27:23 UTC, Andrei Alexandrescu wrote: Yes, Stefan it would be terrific if you could keep an eye on it while working on the engine. A file with properly handed dependencies could serve as instantiation cache and save a ton of re-instantiation waste. Thanks! -- Andrei instantiation ? Do you mean mixin-expansion ? or are you talking about mixins created in templates ? Although Caching mixin expansions hits on the same problem as chaching template-instances, it is more difficult since a mixin can appear anywhere. and one has to worry about a much wider scope. I'll see what I can come up with, for now the debugging is priority. @Ethan can you share code illustrating your usecase. (My head is inside compiler internals and building test-code is a rather unpleasant context switch)
Re: debugging mixins
On 10/02/2016 10:44 PM, Manu via Digitalmars-d wrote: On 3 October 2016 at 00:08, Ethan Watson via Digitalmars-d wrote: On Sunday, 2 October 2016 at 05:00:07 UTC, Stefan Koch wrote: We should create a file where the string-mixins are expanded, I agree. Further to this. I tried generating .di files the other day for code based on Binderoo. None of the mixins were expanded, which resulted in the compiler needing to do all that work again anyway and negating the effect of a precompiled library. If you get to doing that work, it would be fab if you could apply it to .di generation. If not as the default, then at least as a switch I can provide on the command line. That's genius! :) Yah, very very interesting idea. -- Andrei
Re: debugging mixins
On 3 October 2016 at 00:08, Ethan Watson via Digitalmars-d wrote: > On Sunday, 2 October 2016 at 05:00:07 UTC, Stefan Koch wrote: >> >> We should create a file where the string-mixins are expanded, I agree. > > > Further to this. I tried generating .di files the other day for code based > on Binderoo. None of the mixins were expanded, which resulted in the > compiler needing to do all that work again anyway and negating the effect of > a precompiled library. > > If you get to doing that work, it would be fab if you could apply it to .di > generation. If not as the default, then at least as a switch I can provide > on the command line. That's genius! :)
Re: debugging mixins
On Sunday, 2 October 2016 at 05:00:07 UTC, Stefan Koch wrote: We should create a file where the string-mixins are expanded, I agree. Further to this. I tried generating .di files the other day for code based on Binderoo. None of the mixins were expanded, which resulted in the compiler needing to do all that work again anyway and negating the effect of a precompiled library. If you get to doing that work, it would be fab if you could apply it to .di generation. If not as the default, then at least as a switch I can provide on the command line.
Re: debugging mixins
On 10/02/2016 01:00 AM, Stefan Koch wrote: We should create a file where the string-mixins are expanded, I agree. However I am not sure if this is a quick-fix or a more tricky thing. Yes, Stefan it would be terrific if you could keep an eye on it while working on the engine. A file with properly handed dependencies could serve as instantiation cache and save a ton of re-instantiation waste. Thanks! -- Andrei
Re: debugging mixins
On Sunday, 2 October 2016 at 03:36:31 UTC, Manu wrote: This comes up a lot. As far as I know, it's not solved. What shall we do? I feel like a simple solution would be to have the compiler emit a _mixin.d file populated with all the mixin expansions beside the .obj files, and have the debuginfo refer to that fabricated source file? It might look a little bit weird jumping into code where the surrounding scope is not visible (unless that were copied over too?), but it's better than what we have now. Are there any other commonly proposed solutions? We should create a file where the string-mixins are expanded, I agree. However I am not sure if this is a quick-fix or a more tricky thing. In theory the PrettyPrinter we have should give as a way to produce such a file. I have been meaning to work on this issue for a while now, alas there is always something else to do :)
debugging mixins
This comes up a lot. As far as I know, it's not solved. What shall we do? I feel like a simple solution would be to have the compiler emit a _mixin.d file populated with all the mixin expansions beside the .obj files, and have the debuginfo refer to that fabricated source file? It might look a little bit weird jumping into code where the surrounding scope is not visible (unless that were copied over too?), but it's better than what we have now. Are there any other commonly proposed solutions?
