Re: Our template emission strategy is broken

[Jonathan M Davis via Digitalmars-d]

On Monday, 16 November 2015 at 09:05:16 UTC, Marco Leise wrote:

Am Wed, 11 Nov 2015 14:23:15 +
schrieb Martin Nowak :

Think of that for a moment, no package manager allows you to 
have cycles in your dependencies.


There are package managers that allow packages to mutually
depend on each other. Order is established by qualifying the
dependency similar to how you declare an owning and a weak
reference in a parent/child pointer relation.


LOL. Yeah. I expect that 99+% of Linux boxes contain mutually 
dependent packages. Circular dependencies make setting up Gentoo 
that much more entertaining (e.g. IIRC, OpenSSL and Cyrus SASL 
depend on each other normally). At least some dependencies can be 
eliminated with binary packages, because they're only build 
dependencies, but not all of them are that way, unfortunately.


- Jonathan M Davis

Re: Our template emission strategy is broken

[Marco Leise via Digitalmars-d]

Am Wed, 11 Nov 2015 14:23:15 +
schrieb Martin Nowak :

> Think of that for a moment, no package manager allows you to
> have cycles in your dependencies.

There are package managers that allow packages to mutually
depend on each other. Order is established by qualifying the
dependency similar to how you declare an owning and a weak
reference in a parent/child pointer relation.
E.g. You must install DMD before druntime/Phobos, but DMD is
only usable after druntime/Phobos has also been installed.

-- 
Marco

Re: Our template emission strategy is broken

[Tofu Ninja via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

...


Sorry I haven't read this whole thread so maybe this has been 
mentioned before or not. I am not even sure if its really related 
to this topic, but it seemed related to me so I figured I would 
just post it here rather than make a new thread.


ModuleInfo.localClasses doesn't list template class instances. I 
just ran into this(like 5 min ago) and thought it was odd. I 
expected it too list template instances as well so I thought it 
might be a bug, not really sure. Thankfully I don't really need 
it to list template instances of classes so it not affecting my 
project, but I could see it being a problem for others.

Re: Our template emission strategy is broken

[deadalnix via Digitalmars-d]

On Saturday, 14 November 2015 at 03:43:38 UTC, Walter Bright 
wrote:

On 11/13/2015 9:23 AM, Johannes Pfau wrote:

Ellery filed a bug report and I've posted a reduced test case:

https://issues.dlang.org/show_bug.cgi?id=15324


Thank you!


https://issues.dlang.org/show_bug.cgi?id=15333

Also, thanks to whoever was able to reduce it more than I.

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/13/2015 07:51 AM, Stefan wrote:

On Thursday, 12 November 2015 at 23:08:57 UTC, Jakob Ovrum wrote:


> [...dependency check...]



It can be implemented as an external tool with the -deps compiler switch.


there is depend [1].

It warns about cycles and unintended dependencies if you specify the
target dependencies.
It helps us to maintain our commercial code base. src as well as unittest.


[1]
https://github.com/funkwerk/depend


Hi Stefan, it was great to meet in Stuttgart!

This is a simple but very useful tool, it ought to be made part of the 
standard distribution. I just created 
https://issues.dlang.org/show_bug.cgi?id=15337, in summary:


1. Relicensing the work under Boost

2. Make the tool easier to use, e.g. have it run dmd automatically if 
needed etc.


3. Integrate the tool within the tools/ repository and make it part of 
the standard dmd distribution.


Good tooling has been a huge asset for Go - we could take a page from 
their book.


Stefan, Dragos - do you guys think you could take this to completion?


Thanks,

Andrei

Re: Our template emission strategy is broken

[Dicebot via Digitalmars-d]

I know what the problem is. Hence you need to import a in b and 
b in a. There is not really anyway around this because this is 
intrinsically how mixin works.


It could be all much easier if resolving aliases was controllable 
in generic code, most importantly in regards of .stringof

Re: Our template emission strategy is broken

[Robert burner Schadek via Digitalmars-d]

On Friday, 13 November 2015 at 02:50:07 UTC, Daniel Murphy wrote:
You also need to modify root/rmem.d to actually use the GC as 
the allocator.


I should have known that it couldn't be that simple. Anyway, 
after doing so. Building druntime and phobos die with a segfault, 
but all dmd tests pass, except runnable/arrayop.d


Furthermore, I had to create a function called:

extern (C) void* allocmemory(size_t m_size) {
return GC.malloc(m_size);
}

Re: Our template emission strategy is broken

[Johannes Pfau via Digitalmars-d]

Am Thu, 12 Nov 2015 15:15:24 -0800
schrieb Walter Bright :

> On 11/11/2015 5:47 AM, Johannes Pfau wrote:
> > Ellery Newcomer recently reported a template emission bug where
> > templates are emitted twice:
> > http://forum.dlang.org/thread/n1omke$1bh5$1...@digitalmars.com  
> 
> Please file bug reports in bugzilla. Filing them in the n.g. means
> they will most likely get overlooked.
> 

Ellery filed a bug report and I've posted a reduced test case:

https://issues.dlang.org/show_bug.cgi?id=15324

Re: Our template emission strategy is broken

[Daniel Murphy via Digitalmars-d]

On 13/11/2015 8:26 PM, Robert burner Schadek wrote:

On Friday, 13 November 2015 at 02:50:07 UTC, Daniel Murphy wrote:

You also need to modify root/rmem.d to actually use the GC as the
allocator.


I should have known that it couldn't be that simple. Anyway, after doing
so. Building druntime and phobos die with a segfault, but all dmd tests
pass, except runnable/arrayop.d

Furthermore, I had to create a function called:

 extern (C) void* allocmemory(size_t m_size) {
 return GC.malloc(m_size);
 }




Yep, that sounds about right.  It's probably just some malloced memory 
being used to store GC pointers, somewhere in there.

Re: Our template emission strategy is broken

[Walter Bright via Digitalmars-d]

On 11/13/2015 9:23 AM, Johannes Pfau wrote:

Ellery filed a bug report and I've posted a reduced test case:

https://issues.dlang.org/show_bug.cgi?id=15324


Thank you!

Re: Our template emission strategy is broken

[Atila Neves via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

Hi all,

Kenji and Walter have been working on improving the template 
emission strategy during the last couple of releases, i.e. 
whether a template instance is emitted to a given object file 
or not. Nevertheless, I've been continually hearing complaints 
from various people with large D code bases (our commercial 
users and some of the more complex open source projects) that 
they have problems compiling their code doing anything else 
than an all-at-once build.


[...]


I've had problems compiling part of my code as a static library 
and linking to a file that used a template mixin that in turn did 
compile-time reflection. The mangling was different between the 
two and I had linker errors. It was weird and frustrating, and I 
had to go back to compiling all at once, which is slower.


Atila

Re: Our template emission strategy is broken

[Stefan via Digitalmars-d]

On Thursday, 12 November 2015 at 23:08:57 UTC, Jakob Ovrum wrote:


> [...dependency check...]


It can be implemented as an external tool with the -deps 
compiler switch.


there is depend [1].

It warns about cycles and unintended dependencies if you specify 
the target dependencies.
It helps us to maintain our commercial code base. src as well as 
unittest.



[1]
https://github.com/funkwerk/depend

Re: Our template emission strategy is broken

[Atila Neves via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:
On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:
I think we should really fix these issues, working separate 
compilation is very important.


No, it's not. Separate compilation is C++-ishly slow by design 
(b/c all dependencies get imported over and over again, just 
like headers).
We should endorse compiling small (rather independent) packages 
to static/shared libraries.


But compiling by package isn't slow by design. Also, it depends 
on the modules you're compiling. If, like me, you have all your 
unit tests in separate files from the implementation, it's faster 
when only changing the test to recompile that one file and relink 
than compiling the whole program again.


So... it depends.

Atila

Re: Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

On Thursday, 12 November 2015 at 02:13:04 UTC, Martin Nowak wrote:
There shouldn't be anything quadratic left for template 
instantiation in the frontend.


Maybe not for a single instantiation, but constructing the module 
member list is overall still quadratic in the number of members 
(which might be quite a lot if you heavily use templates, because 
all the instances get added first, whether they make it to 
codegen or not). See: 
https://issues.dlang.org/show_bug.cgi?id=15323.


 — David

Re: Our template emission strategy is broken

[Walter Bright via Digitalmars-d]

On 11/11/2015 11:45 PM, Iain Buclaw via Digitalmars-d wrote:

 > Nor do I believe we need to catch C++ objects with catch(...). Catching C++
objects will need an explicit catch(std.exception*).
 >

Well, as demonstrated, that is the one thing that can be supported now for
free.  Either by allowing C++-style catch-all or using language specific
catch-all class matching.


I don't agree with implementing things just because we can. There has to be a 
good case for it. I cannot see a case for catching ints, strings, etc. My 
understanding is that this is more or less acknowledged by the C++ community, 
and std::exceptions are used.




Any type could be caught, again I demonstrated this, but I would have more
confidence if g++ generated the typeinfo bindings for us using pragma(mangle) to
some D-friendly symbol.  Rather than us pushing in more frail mangling support
from our end.


Having dealt with mangling issues for decades, I'm not overly concerned about 
it.

Re: Our template emission strategy is broken

[deadalnix via Digitalmars-d]

On Friday, 13 November 2015 at 00:58:57 UTC, Walter Bright wrote:

On 11/12/2015 2:58 PM, deadalnix wrote:

You can have a template that
do mixin without creating loop because then you pass it down 
identifiers that

won't resolve.


My brain seg faulted trying to parse that.


module a;

struct S(E) {
import std.bitfield;
mixin(bitfield!(
bool, "flag", 1,
T, "myT", SizeOf!T,
uint, "", 32 - SizeOf!T - 1,
));
}

module b;

enum MyEnum { Val1, Val2, Val3 }

import a;
alias UsefulStruct = S!MyEnum; // Boom, MyEnum is undefined

Now you end up having to tie stupid node into your code, like 
adding all user of library X as import in library X. Which sucks, 
but is forced by the language design.

Re: Our template emission strategy is broken

[Walter Bright via Digitalmars-d]

On 11/12/2015 5:38 AM, Andrei Alexandrescu wrote:

I notice you use a star - did you back off from the idea of defining C++'s
std::exception as an extern(C++) class (as opposed to an extern(C++) struct)? --



I just wanted to emphasize receiving it by reference rather than by value.

Re: Our template emission strategy is broken

[Adam D. Ruppe via Digitalmars-d]

On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:
I hadn't anticipated that code would inevitably be constructed 
so that every module imports every other module.


One day, I was cleaning my house and found all kinds of the cat's 
stuff under the couch. I wondered: why does the cat love shoving 
her stuff down there?


Then I realized the answer: she randomly bats it around the whole 
floor. If it lands out in the open, she knocks it somewhere else. 
When it randomly ends up under the couch though, she can no 
longer reach it, so that is where it stays. Eventually, 
everything will accumulate there.



I guess this is just a long winded way of saying that if 
something is even a little useful and possible, it will end up 
being used until it reaches some point where we can't change it 
anymore...

Re: Our template emission strategy is broken

[deadalnix via Digitalmars-d]

On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:

On 11/11/2015 9:19 AM, David Nadlinger wrote:
Of course, many of the problems could have probably been 
avoided if there was an
iron-clad rule that the module dependency graph must remain 
acyclic from the
beginning of development (at least at the level of units of 
compilation).


In hindsight, I would have made D enforce that (Go does).



That's impossible because of the way we do mixin. You can have a 
template that do mixin without creating loop because then you 
pass it down identifiers that won't resolve.


That has been a giant pain in various situations in project I 
work on.

Re: Our template emission strategy is broken

[Walter Bright via Digitalmars-d]

On 11/12/2015 2:58 PM, deadalnix wrote:

You can have a template that
do mixin without creating loop because then you pass it down identifiers that
won't resolve.


My brain seg faulted trying to parse that.

Re: Our template emission strategy is broken

[Daniel Murphy via Digitalmars-d]

On 12/11/2015 10:44 AM, Robert burner Schadek wrote:


I just run make -f posix.mak unittest -j10 on phobos after removing
GC.disable() from ddmd. It worked just fine. Everything build everything
passed the tests.

(the time program told 1:41.25 for nogc and 1:49.80 with gc. This is
with running all the tests.)

Shouldn't we at least add a command line switch to dmd to activate the GC.


You also need to modify root/rmem.d to actually use the GC as the allocator.

Re: Our template emission strategy is broken

[deadalnix via Digitalmars-d]

On Friday, 13 November 2015 at 04:03:23 UTC, Walter Bright wrote:

On 11/12/2015 5:14 PM, deadalnix wrote:

module a;

struct S(E) {
 import std.bitfield;
 mixin(bitfield!(
 bool, "flag", 1,
 T, "myT", SizeOf!T,
 uint, "", 32 - SizeOf!T - 1,
 ));
}

module b;

enum MyEnum { Val1, Val2, Val3 }

import a;
alias UsefulStruct = S!MyEnum; // Boom, MyEnum is undefined

Now you end up having to tie stupid node into your code, like 
adding all user of
library X as import in library X. Which sucks, but is forced 
by the language

design.



The trouble there is that E is turned into MyEnum in the mixin 
string, and MyEnum would be unknown to module a.


You can file an enhancement request on this in bugzilla if you 
like.


I know what the problem is. Hence you need to import a in b and b 
in a. There is not really anyway around this because this is 
intrinsically how mixin works.

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/12/2015 01:15 AM, Walter Bright wrote:


What D does need to do is support catching an std::exception*, or a
class derived from std::exception. I believe we are on solid ground in
not supporting any other C++ types being thrown.

Nor do I believe we need to catch C++ objects with catch(...). Catching
C++ objects will need an explicit catch(std.exception*).


I notice you use a star - did you back off from the idea of defining 
C++'s std::exception as an extern(C++) class (as opposed to an 
extern(C++) struct)? -- Andrei

Re: Our template emission strategy is broken

[Walter Bright via Digitalmars-d]

On 11/12/2015 5:14 PM, deadalnix wrote:

module a;

struct S(E) {
 import std.bitfield;
 mixin(bitfield!(
 bool, "flag", 1,
 T, "myT", SizeOf!T,
 uint, "", 32 - SizeOf!T - 1,
 ));
}

module b;

enum MyEnum { Val1, Val2, Val3 }

import a;
alias UsefulStruct = S!MyEnum; // Boom, MyEnum is undefined

Now you end up having to tie stupid node into your code, like adding all user of
library X as import in library X. Which sucks, but is forced by the language
design.



The trouble there is that E is turned into MyEnum in the mixin string, and 
MyEnum would be unknown to module a.


You can file an enhancement request on this in bugzilla if you like.

Re: Our template emission strategy is broken

[Walter Bright via Digitalmars-d]

On 11/11/2015 9:19 AM, David Nadlinger wrote:

Of course, many of the problems could have probably been avoided if there was an
iron-clad rule that the module dependency graph must remain acyclic from the
beginning of development (at least at the level of units of compilation).


In hindsight, I would have made D enforce that (Go does).


But how could they have known how bad it would get otherwise?


I hadn't anticipated that code would inevitably be constructed so that every 
module imports every other module.

Re: Our template emission strategy is broken

[Jakob Ovrum via Digitalmars-d]

On Thursday, 12 November 2015 at 23:06:01 UTC, rsw0x wrote:
On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:

On 11/11/2015 9:19 AM, David Nadlinger wrote:
Of course, many of the problems could have probably been 
avoided if there was an
iron-clad rule that the module dependency graph must remain 
acyclic from the
beginning of development (at least at the level of units of 
compilation).


In hindsight, I would have made D enforce that (Go does).


Would there be any gain from adding an optional compiler 
warning for it?


It can be implemented as an external tool with the -deps compiler 
switch.

Re: Our template emission strategy is broken

[rsw0x via Digitalmars-d]

On Thursday, 12 November 2015 at 22:50:08 UTC, Walter Bright 
wrote:

On 11/11/2015 9:19 AM, David Nadlinger wrote:
Of course, many of the problems could have probably been 
avoided if there was an
iron-clad rule that the module dependency graph must remain 
acyclic from the
beginning of development (at least at the level of units of 
compilation).


In hindsight, I would have made D enforce that (Go does).


Would there be any gain from adding an optional compiler warning 
for it?

Re: Our template emission strategy is broken

[Walter Bright via Digitalmars-d]

On 11/11/2015 5:47 AM, Johannes Pfau wrote:

Ellery Newcomer recently reported a template emission bug where
templates are emitted twice:
http://forum.dlang.org/thread/n1omke$1bh5$1...@digitalmars.com


Please file bug reports in bugzilla. Filing them in the n.g. means they will 
most likely get overlooked.

Re: Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

On Wednesday, 11 November 2015 at 17:19:51 UTC, Iain Buclaw wrote:
Please no.  I've done my fair share of investigating this in 
libunwind, and it just isn't worth it.


It isn't worth it, or you couldn't figure out how to? ;P

In other words, what I'm asking is: Yes, all the lifetime 
management depends crucially on the C++ standard library 
internals. But given that we do this anyway (name mangling, 
etc.), how would you decide that the effort is not worth it?


 — David

Re: Our template emission strategy is broken

[rsw0x via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:
On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:
I think we should really fix these issues, working separate 
compilation is very important.


No, it's not. Separate compilation is C++-ishly slow by design 
(b/c all dependencies get imported over and over again, just 
like headers).
We should endorse compiling small (rather independent) packages 
to static/shared libraries.


This would be great if cross-module inlining actually worked. 
pragma(inline,true) doesn't even work across modules afaik

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/11/2015 12:19 PM, Iain Buclaw via Digitalmars-d wrote:

On 11 November 2015 at 17:25, Andrei Alexandrescu via Digitalmars-d
> wrote:

On 11/11/2015 08:08 AM, David Nadlinger wrote:
[snip]

Regarding the top-level issue. Walter and I agree it's an important
problem, and also that plugging holes as they start leaking (which
is what we've been doing so far) is not going to work well in the
long haul.

A redesign of template instantiation is necessary, and Walter needs
to be fully involved in it. However, please give it time. Walter is
currently working full time on catching C++ exceptions from D code,
and as we all know the best way of getting many things done is to do
one thing at a given time and do it fully. It should take him at
least two weeks' time to get there.


Please no.  I've done my fair share of investigating this in libunwind,
and it just isn't worth it.


Could you please provide Walter more detail? What's not worth it - 
catching C++ exceptions from D code? -- Andrei

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/11/2015 12:13 PM, David Nadlinger wrote:

On Wednesday, 11 November 2015 at 15:07:15 UTC, Andrei Alexandrescu wrote:

Phobos takes about 600MB max to build.


When was the last time you did "dmd -unittest -main $(find std -name
'*.d')"? On current master, dmd takes just below 9 GiB to complete the
build.


"Then don't do that". Seriously, I agree reducing memory consumed is 
necessary but there are always extreme cases to construct.



Also, consider that many of the features in Phobos – apart from the
underlying molasses consisting of std.traits and friends – are not
really widely used internally. This is not surprising, considering that
large parts of std.* are mostly a grab-bag of orthogonal stuff, and the
unit tests are, as they should, fairly isolated. For example, how many
Phobos modules actually use std.regex internally, let alone its
compile-time variants (which we are quick to advertise)? What about
range "pipelines" longer than the one or two stages you'd typically
write in a unit test?


My perception is Phobos uses its own dog food intensively, which also 
leads to the criticisms that everything in Phobos depends on everything 
else. That said, clearly the library + unittests themselves are not a 
large project using the library.



Apologies for maybe being a bit too confrontational about this,


There is agreement that a problem exists. No need to get overly carried 
over this; I recall last time you got really upset, you said something 
that was so incredibly offensive, Walter and I had to hold a pow-wow and 
decide to simply ignore it and just give you time to calm down. There's 
no need or time for all that. We're on the same boat.



but I
don't think anybody is helped by dismissing entire classes of problems
just because they tend not to be so apparent from a library writer's
perspective. If you were to, say, visit the folks at Weka, you might be
surprised to see how easy it is to do the latter without even realizing so.


Agreed. Glad you're helping them out!


Andrei

Re: Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

On Wednesday, 11 November 2015 at 15:07:15 UTC, Andrei 
Alexandrescu wrote:

Phobos takes about 600MB max to build.


When was the last time you did "dmd -unittest -main $(find std 
-name '*.d')"? On current master, dmd takes just below 9 GiB to 
complete the build.


Also, consider that many of the features in Phobos – apart from 
the underlying molasses consisting of std.traits and friends – 
are not really widely used internally. This is not surprising, 
considering that large parts of std.* are mostly a grab-bag of 
orthogonal stuff, and the unit tests are, as they should, fairly 
isolated. For example, how many Phobos modules actually use 
std.regex internally, let alone its compile-time variants (which 
we are quick to advertise)? What about range "pipelines" longer 
than the one or two stages you'd typically write in a unit test?


Apologies for maybe being a bit too confrontational about this, 
but I don't think anybody is helped by dismissing entire classes 
of problems just because they tend not to be so apparent from a 
library writer's perspective. If you were to, say, visit the 
folks at Weka, you might be surprised to see how easy it is to do 
the latter without even realizing so.


 — David

Re: Our template emission strategy is broken

[Iain Buclaw via Digitalmars-d]

On 11 November 2015 at 17:25, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d@puremagic.com> wrote:

> On 11/11/2015 08:08 AM, David Nadlinger wrote:
> [snip]
>
> Regarding the top-level issue. Walter and I agree it's an important
> problem, and also that plugging holes as they start leaking (which is what
> we've been doing so far) is not going to work well in the long haul.
>
> A redesign of template instantiation is necessary, and Walter needs to be
> fully involved in it. However, please give it time. Walter is currently
> working full time on catching C++ exceptions from D code, and as we all
> know the best way of getting many things done is to do one thing at a given
> time and do it fully. It should take him at least two weeks' time to get
> there.
>
>
Please no.  I've done my fair share of investigating this in libunwind, and
it just isn't worth it.

Re: Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

On Wednesday, 11 November 2015 at 15:04:01 UTC, Andrei 
Alexandrescu wrote:
Yes, compiling package-at-a-time should be the mode endorsed by 
the core community. Liran Zvibel and I discussed this privately 
following his DConf 2015 talk, and I recall he mentioned the 
package-at-a-time solved essentially all of their build 
problems.


If you have a look at the bug report I linked in the original 
post, you'll find that this case occurs precisely with 
per-package compilation. Yes, there is a single cyclic 
inter-package dependency hidden somewhere deep inside the 
dependency graph, but getting rid of it after the fact would have 
either required re-structuring large parts of a ˜10^5 LOC code 
base, or would have again involved creating logical packages so 
large that the memory inefficiencies (CTFE is the biggest problem 
here) and run-time scaling issues (e.g. O(n^2) behavior in the 
overall number of templates) in the front-end become a problem 
again.


Of course, many of the problems could have probably been avoided 
if there was an iron-clad rule that the module dependency graph 
must remain acyclic from the beginning of development (at least 
at the level of units of compilation). But how could they have 
known how bad it would get otherwise? I don't think this is 
reflected in our documentation anywhere, at least not in a 
prominent place.


 — David

Re: Our template emission strategy is broken

[Adam D. Ruppe via Digitalmars-d]

On Wednesday, 11 November 2015 at 15:04:01 UTC, Andrei 
Alexandrescu wrote:
Yes, compiling package-at-a-time should be the mode endorsed by 
the core community. Liran Zvibel and I discussed this privately 
following his DConf 2015 talk, and I recall he mentioned the 
package-at-a-time solved essentially all of their build 
problems.


They must have encountered more since then since their build has 
had a lot of difficulty. I tried to hack around the problems they 
had using the various tricks I've accumulated (btw you can hit 
several of these problems without necessarily having a huge 
codebase, though reproducing them in a reduced test is extremely 
difficult) and failed. They tried throwing more hardware at it 
and that didn't work either with newer compiler versions. David 
has been working on fixing the fundamental problems and surely, 
if he succeeded completely already, he wouldn't have made these 
posts.


We, as a community, need to start believing people when they say 
they are having troubles instead of blaming the customer for 
doing it wrong.

Re: Our template emission strategy is broken

[Paolo Invernizzi via Digitalmars-d]

On Wednesday, 11 November 2015 at 14:31:54 UTC, Martin Nowak 
wrote:
On Wednesday, 11 November 2015 at 14:15:54 UTC, Johannes Pfau 
wrote:


Separate compilation works fine (and predictable) but is slow 
(due to reparsing), you do need to link all objects from all 
your sources though.




That's not the experience we have in my company: we are not able 
to compile and link project right now, using separate compilation.


From dmd 2.066 up to 2.069.
---
Paolo

Re: Our template emission strategy is broken

[Sönke Ludwig via Digitalmars-d]

Am 11.11.2015 um 15:15 schrieb Johannes Pfau:

I use docker hub to build docker images. One Dockerfile compiles one
small D tool which depends on vibe.d for JSON parsing. Without
separate compilation the build fails as docker hub kills the build
process, out of memory...



Sorry for OT: the latest 0.7.27-alpha.1 of vibe.d is now separated into 
sub packages ("vibe-d:data" in particular contains only JSON/BSON and 
the serialization mechanics).

Re: Our template emission strategy is broken

[Johannes Pfau via Digitalmars-d]

Am Wed, 11 Nov 2015 14:31:53 +
schrieb Martin Nowak :

> On Wednesday, 11 November 2015 at 14:15:54 UTC, Johannes Pfau 
> wrote:
> > And where do you draw the line? Do we expect ARM users to have 
> > 4GB of RAM to compile phobos? Is 512MB enough? 64 MB? Even huge 
> > C++ codebases such as GCC compile with 128MB memory.  
> 
> OK let's clarify the terminology first.
> 
> separate compilation: -c single_source.d
> incremental compilation: -c source_a.d source_b.d (as well as -c 
> -ofsingle.o source_a.d source_b.d)
> library compilation: -lib all_pkg_sources
> 
> Separate compilation works fine (and predictable) but is slow 
> (due to reparsing), you do need to link all objects from all your 
> sources though.
> Choosing this strategy makes sense to mitigate OOM issues.
> 

OK, that makes sense. I thought there were some issues with separate
compilation as well, but I'm not sure.

Anyway, I meant working _separate_ compilation is important, incremental
compilations is much less important for me. GDC doesn't even support
multiobj style compilation.

BTW: shouldn't incremental compilation with one single output object
have exactly the same result as compiling into one static library? So I
guess we already avoid most of these problems in gdc by simply not
offering a multiobj mode ;-)

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/11/15 8:56 AM, Martin Nowak wrote:

On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau wrote:

I think we should really fix these issues, working separate
compilation is very important.


No, it's not. Separate compilation is C++-ishly slow by design (b/c all
dependencies get imported over and over again, just like headers).
We should endorse compiling small (rather independent) packages to
static/shared libraries.


Yes, compiling package-at-a-time should be the mode endorsed by the core 
community. Liran Zvibel and I discussed this privately following his 
DConf 2015 talk, and I recall he mentioned the package-at-a-time solved 
essentially all of their build problems.


It would be fantastic if somebody could write a wiki/dlang.org article 
about this with details on how to do it and measurements. One prime 
example is Phobos itself, for which conversion to package-at-a-time 
build (http://forum.dlang.org/thread/mkvpgn$2fg3$2...@digitalmars.com) has 
led to dramatic improvements in memory consumed and build times 
(https://github.com/D-Programming-Language/phobos/pull/3379).



Andrei

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/11/15 9:14 AM, Daniel Kozak via Digitalmars-d wrote:

V Wed, 11 Nov 2015 13:56:50 +
Martin Nowak via Digitalmars-d  napsáno:


On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau
wrote:

I think we should really fix these issues, working separate
compilation is very important.


No, it's not. Separate compilation is C++-ishly slow by design
(b/c all dependencies get imported over and over again, just like
headers).
We should endorse compiling small (rather independent) packages
to static/shared libraries.



This isn't a solution it's a workaround.
It seems to me like "It's not a bug it's a feature"


Rust, Haskell, and Go have similar approaches if that helps. -- Andrei

Re: Our template emission strategy is broken

[Martin Nowak via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:


Let's please make sure that we come out of this discussion with 
some concrete actions to work on. Ideally you'd fill our backlog 
with some stories, even if they are just research stories.

Re: Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

On Wednesday, 11 November 2015 at 14:50:27 UTC, Johannes Pfau 
wrote:
Anyway, I meant working _separate_ compilation is important, 
incremental compilations is much less important for me. GDC 
doesn't even support multiobj style compilation.


The particular issue I mentioned in the original post (that will 
hit GDC as well, by the way, if/as soon as you are using 
TemplateInstance::needsCodegen in your glue layer) has nothing to 
do with "multiobj" compilation.


BTW: shouldn't incremental compilation with one single output 
object have exactly the same result as compiling into one 
static library?


Yes, and both are broken.

 — David

Re: Our template emission strategy is broken

[Paolo Invernizzi via Digitalmars-d]

On Wednesday, 11 November 2015 at 15:12:16 UTC, Adam D. Ruppe 
wrote:


We, as a community, need to start believing people when they 
say they are having troubles instead of blaming the customer 
for doing it wrong.


+10_000

---
Paolo

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/11/15 10:12 AM, Adam D. Ruppe wrote:

We, as a community, need to start believing people when they say they
are having troubles instead of blaming the customer for doing it wrong.


Total agreement with that! -- Andrei

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/11/2015 08:08 AM, David Nadlinger wrote:
[snip]

Regarding the top-level issue. Walter and I agree it's an important 
problem, and also that plugging holes as they start leaking (which is 
what we've been doing so far) is not going to work well in the long haul.


A redesign of template instantiation is necessary, and Walter needs to 
be fully involved in it. However, please give it time. Walter is 
currently working full time on catching C++ exceptions from D code, and 
as we all know the best way of getting many things done is to do one 
thing at a given time and do it fully. It should take him at least two 
weeks' time to get there.


This is a good time to collect cases known to be troublesome and to 
discuss high-level approaches.



Thanks,

Andrei

Re: Our template emission strategy is broken

[Martin Nowak via Digitalmars-d]

On Wednesday, 11 November 2015 at 14:06:43 UTC, Dicebot wrote:
Note that many of template emission issues that affect "plain" 
separate compilation are likely to affect package-based static 
libraties as well, thus in context of this thread distinction 
is not very important.


Yes, you can get similar issues with package libraries, but only 
if the packages are mutually dependent on each other. Think of 
that for a moment, no package manager allows you to have cycles 
in your dependencies.


The case where you do compile packages with circular dependencies 
seems to mostly come from artificially splitting packages to deal 
with OOM, and we better address that at it's root.


In general you save many troubles with a library b/c you always 
link with all objects that were compiled (so it's only important 
that a template was emitted but not into which object).

Re: Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

On Wednesday, 11 November 2015 at 14:31:54 UTC, Martin Nowak 
wrote:

OK let's clarify the terminology first.

separate compilation: -c single_source.d
incremental compilation: -c source_a.d source_b.d (as well as 
-c -ofsingle.o source_a.d source_b.d)

library compilation: -lib all_pkg_sources

Separate compilation works fine (and predictable) but is slow 
(due to reparsing), you do need to link all objects from all 
your sources though.

Choosing this strategy makes sense to mitigate OOM issues.

Incremental compilation doesn't work b/c templates are easily 
emitted into different objects, causing all sorts of linker 
errors. It works somewhat when you delete all objects and 
recompile all of them.


Library compilation works and recompiles a whole library 
whenever something changes.

It's fast b/c modules/semantic can be reused by the compiler.
Choose this strategy unless your running into OOM issues.


That terminology seems a bit misleading for discussing the 
problem at hand. There is nothing incremental about compiling 
per-package with a single .o file, and it is in fact rather 
similar to what you call "library compilation". As noted on 
Bugzilla, the particular template instantiation bug that was 
plaguing Weka occurs with "-lib" just the same as it does with 
"-c -osingleobj.o".


 — David

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/11/15 9:15 AM, Johannes Pfau wrote:

Am Wed, 11 Nov 2015 13:56:50 +
schrieb Martin Nowak :


On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau
wrote:

I think we should really fix these issues, working separate
compilation is very important.


No, it's not. Separate compilation is C++-ishly slow by design
(b/c all dependencies get imported over and over again, just like
headers).
We should endorse compiling small (rather independent) packages
to static/shared libraries.



And where do you draw the line? Do we expect ARM users to have 4GB of
RAM to compile phobos? Is 512MB enough? 64 MB? Even huge C++ codebases
such as GCC compile with 128MB memory.


Large codebases are best organized to have more packages, not larger 
packages.


Phobos takes about 600MB max to build.


Andrei

Re: Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:
No, it's not. Separate compilation is C++-ishly slow by design 
(b/c all dependencies get imported over and over again, just 
like headers).


To a certain extent this is not true. As long as the amount 
repeated work is not too big, the ability to parallelize the 
actual semantic3/glue/codegen parts in a separate compilation 
setting still leads to a considerable speedup. All-at-once 
compilation forces you to throw away 31 of the 32 cores in your 
nice heavy duty development box.


(Yes, in theory the different parts of the compilation process 
could be parallelized even for an all-at-once build, but by the 
time we reach the level of separation/well-definedness of the 
internal state required for this, all the by-package compilation 
problems would have probably disappeared rather automatically.)


 — David

Re: Our template emission strategy is broken

[Johannes Pfau via Digitalmars-d]

Am Wed, 11 Nov 2015 14:44:39 +0100
schrieb Johannes Pfau :

> Am Wed, 11 Nov 2015 13:08:06 +
> schrieb David Nadlinger :
> 
> > Hi all,
> > 
> > Kenji and Walter have been working on improving the template 
> > emission strategy during the last couple of releases, i.e. 
> > whether a template instance is emitted to a given object file or 
> > not. Nevertheless, I've been continually hearing complaints from 
> > various people with large D code bases (our commercial users and 
> > some of the more complex open source projects) that they have 
> > problems compiling their code doing anything else than an 
> > all-at-once build.
> > 
> > This of course detracts from what we like to cite as one of D's 
> > key advantages, compilation speed, because you cannot exploit the 
> > many cores of your build machine anymore, and the 
> > edit-compile-debug cycle is slowed down because you have to build 
> > the full application every time. But this turns into a severe 
> > problem as soon as your compilation does not fit into RAM 
> > anymore, which happens quite quickly when using D's advanced 
> > features – see for example Liran Zvibel's DConf talk, where he 
> > describes that they needed machines with more than 100 GB of RAM 
> > in order to build the Weka code base.

A quick workaround could be enabling the GC for DDMD. IIRC I read
somewhere on github that the segfaulting code was actually rewritten
now and the GC might just work.

> > 
> > In any case, I hope you agree that fixing these kinds of issues 
> > that prevent D code from being compiled all is strategically 
> > important for us, since they are a deciding factor in driving 
> > widespread adoption. Sadly, the template problems didn't seem to 
> > receive a lot of attention beyond quick fixes recently, possibly 
> > because they don't occur as readily for smaller projects and if 
> > so, are easier to work around.
> > 
> > With all that out of my system, I'd like to point your attention 
> > to a particular template instantiation issue I managed to reduce 
> > recently when working with the folks at Weka. It seems like there 
> > is a fundamental oversight in the way the code tries to elide 
> > repeat code generation of instances – or, of course, I'm just 
> > missing something:
> > 
> > https://issues.dlang.org/show_bug.cgi?id=15318
> > 
> > This is one particular issue that makes writing "idiomatic D" 
> > (which is rather template-heavy) in large code bases hard, but 
> > likely not the only one. In fact, I already know about another 
> > issue where function-local imports cause semantic analysis not to 
> > be properly run on template instances, but I'm still struggling 
> > to find a minimal example for the problem. My hope would be that 
> > we can clean up this mess soon and replace the current ad-hoc 
> > patchwork with a more principled approach. In the process, we 
> > should also make sure that all of our assumptions [1] about the 
> > compilation process are clearly stated in the documentation, as 
> > that frequently leads to confusion among newcomers.
> > 
> > Best,
> > David
> > 
> > 
> > 
> > [1] That is: All imported modules must also be compiled into the 
> > executable; incremental compilation is only guaranteed to work if 
> > precisely the same subsets of modules are compiled every time.  
> 
> 
> http://forum.dlang.org/thread/n1omke$1bh5$1...@digitalmars.com
> 

Sorry, sent the wrong message ;-)

Ellery Newcomer recently reported a template emission bug where
templates are emitted twice:
http://forum.dlang.org/thread/n1omke$1bh5$1...@digitalmars.com 

I think we should really fix these issues, working separate compilation
is very important.

Re: Our template emission strategy is broken

[Johannes Pfau via Digitalmars-d]

Am Wed, 11 Nov 2015 13:56:50 +
schrieb Martin Nowak :

> On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
> wrote:
> > I think we should really fix these issues, working separate 
> > compilation is very important.  
> 
> No, it's not. Separate compilation is C++-ishly slow by design 
> (b/c all dependencies get imported over and over again, just like 
> headers).
> We should endorse compiling small (rather independent) packages 
> to static/shared libraries.
> 

And where do you draw the line? Do we expect ARM users to have 4GB of
RAM to compile phobos? Is 512MB enough? 64 MB? Even huge C++ codebases
such as GCC compile with 128MB memory.

I use docker hub to build docker images. One Dockerfile compiles one
small D tool which depends on vibe.d for JSON parsing. Without
separate compilation the build fails as docker hub kills the build
process, out of memory...

Re: Our template emission strategy is broken

[Paolo Invernizzi via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:

Hi all,

Nevertheless, I've been continually hearing complaints from 
various people with large D code bases (our commercial users 
and some of the more complex open source projects) that they 
have problems compiling their code doing anything else than an 
all-at-once build.


The same here, we are not able to compile two of our products 
without an all-at-once build.


Actually we are plagued also by that:

https://issues.dlang.org/show_bug.cgi?id=14517

That's a real pain...

---
Paolo

Re: Our template emission strategy is broken

[Johannes Pfau via Digitalmars-d]

Am Wed, 11 Nov 2015 13:08:06 +
schrieb David Nadlinger :

> Hi all,
> 
> Kenji and Walter have been working on improving the template 
> emission strategy during the last couple of releases, i.e. 
> whether a template instance is emitted to a given object file or 
> not. Nevertheless, I've been continually hearing complaints from 
> various people with large D code bases (our commercial users and 
> some of the more complex open source projects) that they have 
> problems compiling their code doing anything else than an 
> all-at-once build.
> 
> This of course detracts from what we like to cite as one of D's 
> key advantages, compilation speed, because you cannot exploit the 
> many cores of your build machine anymore, and the 
> edit-compile-debug cycle is slowed down because you have to build 
> the full application every time. But this turns into a severe 
> problem as soon as your compilation does not fit into RAM 
> anymore, which happens quite quickly when using D's advanced 
> features – see for example Liran Zvibel's DConf talk, where he 
> describes that they needed machines with more than 100 GB of RAM 
> in order to build the Weka code base.
> 
> In any case, I hope you agree that fixing these kinds of issues 
> that prevent D code from being compiled all is strategically 
> important for us, since they are a deciding factor in driving 
> widespread adoption. Sadly, the template problems didn't seem to 
> receive a lot of attention beyond quick fixes recently, possibly 
> because they don't occur as readily for smaller projects and if 
> so, are easier to work around.
> 
> With all that out of my system, I'd like to point your attention 
> to a particular template instantiation issue I managed to reduce 
> recently when working with the folks at Weka. It seems like there 
> is a fundamental oversight in the way the code tries to elide 
> repeat code generation of instances – or, of course, I'm just 
> missing something:
> 
> https://issues.dlang.org/show_bug.cgi?id=15318
> 
> This is one particular issue that makes writing "idiomatic D" 
> (which is rather template-heavy) in large code bases hard, but 
> likely not the only one. In fact, I already know about another 
> issue where function-local imports cause semantic analysis not to 
> be properly run on template instances, but I'm still struggling 
> to find a minimal example for the problem. My hope would be that 
> we can clean up this mess soon and replace the current ad-hoc 
> patchwork with a more principled approach. In the process, we 
> should also make sure that all of our assumptions [1] about the 
> compilation process are clearly stated in the documentation, as 
> that frequently leads to confusion among newcomers.
> 
> Best,
> David
> 
> 
> 
> [1] That is: All imported modules must also be compiled into the 
> executable; incremental compilation is only guaranteed to work if 
> precisely the same subsets of modules are compiled every time.


http://forum.dlang.org/thread/n1omke$1bh5$1...@digitalmars.com

Re: Our template emission strategy is broken

[Martin Nowak via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:
I think we should really fix these issues, working separate 
compilation is very important.


No, it's not. Separate compilation is C++-ishly slow by design 
(b/c all dependencies get imported over and over again, just like 
headers).
We should endorse compiling small (rather independent) packages 
to static/shared libraries.

Re: Our template emission strategy is broken

[Daniel Kozak via Digitalmars-d]

V Wed, 11 Nov 2015 13:56:50 +
Martin Nowak via Digitalmars-d  napsáno:

> On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
> wrote:
> > I think we should really fix these issues, working separate 
> > compilation is very important.  
> 
> No, it's not. Separate compilation is C++-ishly slow by design 
> (b/c all dependencies get imported over and over again, just like 
> headers).
> We should endorse compiling small (rather independent) packages 
> to static/shared libraries.
> 

This isn't a solution it's a workaround.
It seems to me like "It's not a bug it's a feature" 

Re: Our template emission strategy is broken

[Martin Nowak via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:08:08 UTC, David Nadlinger 
wrote:


We're well aware of the template problems and FWIW the most 
important step is to simplify the current implementation (not the 
rules).

http://forum.dlang.org/post/55f07e92.7040...@dawg.eu
Once that is done and we have a clearer understanding of what the 
problems are, we could rediscuss the emission rules.


Ultimatively the decision to instantiate/emit as few templates as 
possible sped up compilation a lot but basically means -c with 
multiple objects will never work reliably.
Compiling separate objects is a C++-idiom anyhow and doesn't make 
too much sense for D. Instead separating your codebase into 
packages with little interdependencies and always recompiling 
those packages when something changes should be the preferred way.

Re: Our template emission strategy is broken

[Dicebot via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:
On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:
I think we should really fix these issues, working separate 
compilation is very important.


No, it's not. Separate compilation is C++-ishly slow by design 
(b/c all dependencies get imported over and over again, just 
like headers).
We should endorse compiling small (rather independent) packages 
to static/shared libraries.


Note that many of template emission issues that affect "plain" 
separate compilation are likely to affect package-based static 
libraties as well, thus in context of this thread distinction is 
not very important.

Re: Our template emission strategy is broken

[Dicebot via Digitalmars-d]

On Wednesday, 11 November 2015 at 14:23:16 UTC, Martin Nowak 
wrote:

On Wednesday, 11 November 2015 at 14:06:43 UTC, Dicebot wrote:
Note that many of template emission issues that affect "plain" 
separate compilation are likely to affect package-based static 
libraties as well, thus in context of this thread distinction 
is not very important.


Yes, you can get similar issues with package libraries, but 
only if the packages are mutually dependent on each other. 
Think of that for a moment, no package manager allows you to 
have cycles in your dependencies.


The case where you do compile packages with circular 
dependencies seems to mostly come from artificially splitting 
packages to deal with OOM, and we better address that at it's 
root.


In general you save many troubles with a library b/c you always 
link with all objects that were compiled (so it's only 
important that a template was emitted but not into which 
object).


I was referring to D packages, not dub packages.

Re: Our template emission strategy is broken

[Martin Nowak via Digitalmars-d]

On Wednesday, 11 November 2015 at 14:15:54 UTC, Johannes Pfau 
wrote:
And where do you draw the line? Do we expect ARM users to have 
4GB of RAM to compile phobos? Is 512MB enough? 64 MB? Even huge 
C++ codebases such as GCC compile with 128MB memory.


OK let's clarify the terminology first.

separate compilation: -c single_source.d
incremental compilation: -c source_a.d source_b.d (as well as -c 
-ofsingle.o source_a.d source_b.d)

library compilation: -lib all_pkg_sources

Separate compilation works fine (and predictable) but is slow 
(due to reparsing), you do need to link all objects from all your 
sources though.

Choosing this strategy makes sense to mitigate OOM issues.

Incremental compilation doesn't work b/c templates are easily 
emitted into different objects, causing all sorts of linker 
errors. It works somewhat when you delete all objects and 
recompile all of them.


Library compilation works and recompiles a whole library whenever 
something changes.

It's fast b/c modules/semantic can be reused by the compiler.
Choose this strategy unless your running into OOM issues.

Re: Our template emission strategy is broken

[Martin Nowak via Digitalmars-d]

On Wednesday, 11 November 2015 at 14:26:28 UTC, Dicebot wrote:

I was referring to D packages, not dub packages.


Not much of a difference, compiling something separately that has 
mutual dependencies is flawed but can be made working to some 
extend.

Re: Our template emission strategy is broken

[deadalnix via Digitalmars-d]

On Wednesday, 11 November 2015 at 17:19:51 UTC, Iain Buclaw wrote:
Please no.  I've done my fair share of investigating this in 
libunwind, and it just isn't worth it.


Ok I know Walter won't listen, and I wouldn't be as radical as 
Iain, but I concur. There are thing that can be done to improve 
C++ and D interrop with Exception. For instance, SDC makes sure 
that C++ exception unwind properly through D and vice versa, but 
don't allow to catch across language.


There is a reason for that. To be able to catch C++ exceptions, 
you need to duplicate the whole RTTI mechanism from C++ + runtime 
support for it + hook oneself in the C++ runtime to signal 
catching and rethrowing.


This is definitively not impossible, but the cost benefit ratio 
is such as I wouldn't prioritize this. There are definitively 
more way to have impact without summoning a Golem when it comes 
to C++ interrop.

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/11/2015 06:06 PM, Martin Nowak wrote:

On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei Alexandrescu wrote:

A redesign of template instantiation is necessary, and Walter needs to
be fully involved in it. However, please give it time.


I'd still opt to first refactor the existing code into something that's
understandable before starting to redesign the whole topic. Maybe the
design is already fine but buried by complex to understand control flow.
Also just redesigning this on a blank sheet will easily miss important
edge cases. And last but not least if we can't come up with a better
design we'd still have a better codebase.


We have to come up with a better design. Not coming with a better design 
is not an option. Also there is ample evidence that the months-long 
process of patching things as problems are found does not work. -- Andrei

Re: Our template emission strategy is broken

[Martin Nowak via Digitalmars-d]

On Wednesday, 11 November 2015 at 17:19:31 UTC, David Nadlinger 
wrote:

Yes, there is a single cyclic inter-package dependency hidden
somewhere deep inside the dependency graph, but getting rid
of it after the fact would  have either required re-structuring
large parts of a ˜10^5 LOC code base, or would have again
involved creating logical


Understood. FWIW it'd be more of a help to dig down and fix that 
particular bug sooner rather than going back to come up with a 
new template emission design.


Cross-Posting my Bugzilla comment here.
https://issues.dlang.org/show_bug.cgi?id=15318#c1

This seems to be a case where the second rule of this issue 14431 
fix doesn't work.

http://forum.dlang.org/post/mailman.697.1440962414.13986.digitalmars-d-b...@puremagic.com

If a template is instantiated in non-root module, compiler 
usually does not have to put it in object file. But if a 
template is instantiated in both of root and non-root modules 
which mutually import each other, it needs to placed in objfile.


===

In both of the compilations there is a root and a non-root module 
instantiating bar!5. Now unfortunately the compiler decides both 
times that the non-root module should do it.
This problem would be solved by my proposal to define a global 
order for who is responsible to instantiate a template, by 
choosing the module with the lexicographically smaller module 
name.

https://github.com/D-Programming-Language/dmd/pull/4384#discussion_r29910422
https://issues.dlang.org/show_bug.cgi?id=14431#c12

This would establish a stable order between B and C, and no 
matter how you compile them, B gets to instantiate the template.


===

An intermediate workaround for your problem is to use the 
-allinst switch, even though it slows down compilation a lot. You 
can also compile each module separately in which case both B and 
C get the instance (in general this is even slower than -allinst).


===

packages so large that the memory inefficiencies (CTFE is the 
biggest problem here) and run-time scaling issues (e.g. O(n^2) 
behavior in the overall number of templates) in the front-end 
become a problem again.


There shouldn't be anything quadratic left for template 
instantiation in the frontend.

Re: Our template emission strategy is broken

[Walter Bright via Digitalmars-d]

On 11/11/2015 4:45 PM, deadalnix wrote:

Ok I know Walter won't listen,


Does anyone else hear a buzzing sound? Or is my hearing aid on the fritz again?

Re: Our template emission strategy is broken

[James Hofmann via Digitalmars-d]

On Wednesday, 11 November 2015 at 17:19:31 UTC, David Nadlinger 
wrote:
Of course, many of the problems could have probably been 
avoided if there was an iron-clad rule that the module 
dependency graph must remain acyclic from the beginning of 
development (at least at the level of units of compilation). 
But how could they have known how bad it would get otherwise? I 
don't think this is reflected in our documentation anywhere, at 
least not in a prominent place.


 — David


There is some literature about whether this kind of rule, 
enforced at compile-time, can benefit software architecture in 
general, in the context of F# projects vs. similar C# projects. 
(Answer: probably yes - there's a relationship between cyclical 
dependencies and other accidental coupling; you can make some 
case for it both theoretically and in statistics)


http://evelinag.com/blog/2014/06-09-comparing-dependency-networks/#.VkP6mL_eNYU

Looking at what D does now, at least according to "The D 
Programming Language" (2010), it tries to allow any ordering but 
throws an exception at runtime when ambiguious cases are 
detected. So there's already some precedent to avoid cyclical 
dependency simply to avoid those errors. The coupling argument 
and the compile-times argument just add more urgency to it.


Nim enforced the same dependency rule as F# the last time I 
looked, but I think that position was softening towards 
"optional" due to some community pressure. I don't think anything 
makes this architecture style actually impractical and it might 
help to have the compiler warn towards it - although the same 
kind of community pressure is going to arise if it did do so. 
After all, nobody likes to be told that they are writing Bad Code 
:)

Re: Our template emission strategy is broken

[deadalnix via Digitalmars-d]

On Wednesday, 11 November 2015 at 18:06:22 UTC, rsw0x wrote:
On Wednesday, 11 November 2015 at 13:56:51 UTC, Martin Nowak 
wrote:
On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:
I think we should really fix these issues, working separate 
compilation is very important.


No, it's not. Separate compilation is C++-ishly slow by design 
(b/c all dependencies get imported over and over again, just 
like headers).
We should endorse compiling small (rather independent) 
packages to static/shared libraries.


This would be great if cross-module inlining actually worked. 
pragma(inline,true) doesn't even work across modules afaik


It does if you compile by package with something else than DMD, 
because DMD does inlining in some very weird ways.

Re: Our template emission strategy is broken

[Walter Bright via Digitalmars-d]

On 11/11/2015 4:16 PM, Iain Buclaw via Digitalmars-d wrote:

The main problem for seamless support is having some way to generate the C++
typeinfo in D to allow work across boundaries.


I suspect that is unnecessary.

The whole point of D's C++ interface is to interface with C++ code. Defining C++ 
classes in D code that does not link to any C++ code that also defines those 
classes won't be supported. I.e. we can reply on the C++ code generating the 
necessary C++ typeinfo. The generated D code only has to link to it, i.e. 
generate an extern reference to the name of the typeinfo.


Similarly, we never need to throw a C++ exception from D code. All we need to do 
is call a library function written in C++ that throws it for us.


What D does need to do is support catching an std::exception*, or a class 
derived from std::exception. I believe we are on solid ground in not supporting 
any other C++ types being thrown.


Nor do I believe we need to catch C++ objects with catch(...). Catching C++ 
objects will need an explicit catch(std.exception*).


Over time, perhaps it may become apparent that we do need to support more kinds 
of C++ types being thrown. But it is not necessary to get this lead brick 
airborne, and we shouldn't waste time worrying about it.

Re: Our template emission strategy is broken

[Andrei Alexandrescu via Digitalmars-d]

On 11/11/2015 07:45 PM, deadalnix wrote:

This is definitively not impossible, but the cost benefit ratio is such
as I wouldn't prioritize this.


The way we see it is the benefit is all C++ interop. You can't really do 
any meaningful interop whatsoever if you can't catch std::exception. -- 
Andrei

Re: Our template emission strategy is broken

[Iain Buclaw via Digitalmars-d]

On 12 Nov 2015 7:15 am, "Walter Bright via Digitalmars-d" <
digitalmars-d@puremagic.com> wrote:
>
> On 11/11/2015 4:16 PM, Iain Buclaw via Digitalmars-d wrote:
>>
>> The main problem for seamless support is having some way to generate the
C++
>> typeinfo in D to allow work across boundaries.
>
>
> I suspect that is unnecessary.
>
> The whole point of D's C++ interface is to interface with C++ code.
Defining C++ classes in D code that does not link to any C++ code that also
defines those classes won't be supported. I.e. we can reply on the C++ code
generating the necessary C++ typeinfo. The generated D code only has to
link to it, i.e. generate an extern reference to the name of the typeinfo.
>

It is a small concern to me that C++ catch handlers work by calling some
support library code, If I recall correctly both at the start and end of
the handler.  These calls may change the state of C++'s language specific
unwind library, such as adjust any reference counters, mark the thrown
object as having been caught.

Maybe less of a concern now than last year.  But I do wonder what effect it
will incur on a long running program.

--
Regards,
Iain

Re: Our template emission strategy is broken

[Iain Buclaw via Digitalmars-d]

On 12 Nov 2015 7:15 am, "Walter Bright via Digitalmars-d" <
digitalmars-d@puremagic.com> wrote:
>
> On 11/11/2015 4:16 PM, Iain Buclaw via Digitalmars-d wrote:
>>
>> The main problem for seamless support is having some way to generate the
C++
>> typeinfo in D to allow work across boundaries.
>
>
> I suspect that is unnecessary.
>
> The whole point of D's C++ interface is to interface with C++ code.
Defining C++ classes in D code that does not link to any C++ code that also
defines those classes won't be supported. I.e. we can reply on the C++ code
generating the necessary C++ typeinfo. The generated D code only has to
link to it, i.e. generate an extern reference to the name of the typeinfo.
>
> Similarly, we never need to throw a C++ exception from D code. All we
need to do is call a library function written in C++ that throws it for us.
>

Yes, never throw a C++ exception from D, even if we manage to intercept the
object.

> What D does need to do is support catching an std::exception*, or a class
derived from std::exception. I believe we are on solid ground in not
supporting any other C++ types being thrown.
>
> Nor do I believe we need to catch C++ objects with catch(...). Catching
C++ objects will need an explicit catch(std.exception*).
>

Well, as demonstrated, that is the one thing that can be supported now for
free.  Either by allowing C++-style catch-all or using language specific
catch-all class matching.

> Over time, perhaps it may become apparent that we do need to support more
kinds of C++ types being thrown. But it is not necessary to get this lead
brick airborne, and we shouldn't waste time worrying about it.
>

Any type could be caught, again I demonstrated this, but I would have more
confidence if g++ generated the typeinfo bindings for us using
pragma(mangle) to some D-friendly symbol.  Rather than us pushing in more
frail mangling support from our end.

--
Regards,
Iain

Re: Our template emission strategy is broken

[Robert burner Schadek via Digitalmars-d]

On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:
A quick workaround could be enabling the GC for DDMD. IIRC I 
read somewhere on github that the segfaulting code was actually 
rewritten now and the GC might just work.


I just run make -f posix.mak unittest -j10 on phobos after 
removing GC.disable() from ddmd. It worked just fine. Everything 
build everything passed the tests.


(the time program told 1:41.25 for nogc and 1:49.80 with gc. This 
is with running all the tests.)


Shouldn't we at least add a command line switch to dmd to 
activate the GC.

Re: Our template emission strategy is broken

[Robert burner Schadek via Digitalmars-d]

druntime also works with GC activated

Re: Our template emission strategy is broken

[rsw0x via Digitalmars-d]

On Wednesday, 11 November 2015 at 23:44:04 UTC, Robert burner 
Schadek wrote:
On Wednesday, 11 November 2015 at 13:47:27 UTC, Johannes Pfau 
wrote:
A quick workaround could be enabling the GC for DDMD. IIRC I 
read somewhere on github that the segfaulting code was 
actually rewritten now and the GC might just work.


I just run make -f posix.mak unittest -j10 on phobos after 
removing GC.disable() from ddmd. It worked just fine. 
Everything build everything passed the tests.


(the time program told 1:41.25 for nogc and 1:49.80 with gc. 
This is with running all the tests.)


Shouldn't we at least add a command line switch to dmd to 
activate the GC.


if it works, shouldn't the runtime 
CLI(http://dlang.org/changelog/2.067.0.html#gc-options) just be 
used to manually deactivate it if needed?

Re: Our template emission strategy is broken

[Iain Buclaw via Digitalmars-d]

On 11 November 2015 at 19:02, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d@puremagic.com> wrote:

> On 11/11/2015 12:19 PM, Iain Buclaw via Digitalmars-d wrote:
>
>> On 11 November 2015 at 17:25, Andrei Alexandrescu via Digitalmars-d
>> > wrote:
>>
>> On 11/11/2015 08:08 AM, David Nadlinger wrote:
>> [snip]
>>
>> Regarding the top-level issue. Walter and I agree it's an important
>> problem, and also that plugging holes as they start leaking (which
>> is what we've been doing so far) is not going to work well in the
>> long haul.
>>
>> A redesign of template instantiation is necessary, and Walter needs
>> to be fully involved in it. However, please give it time. Walter is
>> currently working full time on catching C++ exceptions from D code,
>> and as we all know the best way of getting many things done is to do
>> one thing at a given time and do it fully. It should take him at
>> least two weeks' time to get there.
>>
>>
>> Please no.  I've done my fair share of investigating this in libunwind,
>> and it just isn't worth it.
>>
>
> Could you please provide Walter more detail? What's not worth it -
> catching C++ exceptions from D code? -- Andrei
>
>
The main problem for seamless support is having some way to generate the
C++ typeinfo in D to allow work across boundaries.  However there is
nothing stopping catch-all style exceptions from working.

Consider the following desired example (this works today with minimal
library changes in gdc's libunwind library):
*
#include 

int compute(int a, int b)
{
  if (b == 0)
throw "Division by 0 in C++!";

  return a / b;
}

void cpp_main()
{
  try {
 std::cout << compute(1, 0);
  } catch (...) {
std::cout << "Unknown exception caught in C++, rethrowing..." << "\n";
throw;
  }
}
*

Then put in bindings and main in D.
*
import std.stdio;

extern(C++) void cpp_main();

void main()
{
try {
cpp_main();
}
catch {
writeln("Unknown exception caught in D, exiting...");
}
}
*

In our unwind library, we can put in a copy of the C++ unwind
implementation:
*
static if (GNU_ARM_EABI_Unwinder)
{
  const _Unwind_Exception_Class __gxx_exception_class
  = ['G', 'N', 'U', 'C', 'C', '+', '+', '\0'];
}
else
{
  const _Unwind_Exception_Class __gxx_exception_class =
0x474e5543432b2b00UL;
}

// Structure of a C++ exception, represented as a C structure...
// See unwind-cxx.h for the full definition.
struct __cxa_exception
{
  void *exceptionType;
  void function(void *) exceptionDestructor;
  void function() unexpectedHandler;
  void function() terminateHandler;
  __cxa_exception *nextException;
  int handlerCount;

  static if (GNU_ARM_EABI_Unwinder)
  {
__cxa_exception* nextPropagatingException;
int propagationCount;
  }
  else
  {
int handlerSwitchValue;
const ubyte *actionRecord;
const ubyte *languageSpecificData;
_Unwind_Ptr catchTemp;
void *adjustedPtr;
  }

  _Unwind_Exception unwindHeader;
}
*

Ignoring the main guts of gdc's unwind implementation (should be very
similar in ldc and sdc by coincidence), let's instead focus on the exact
part where we try to match the exception object, you'd have something like
the following:
*
if (ar_filter > 0)
  {
// Positive filter values are handlers.
void *catch_type = get_ttype_entry (, ar_filter);

if (!foreign_exception)
  {
if (_d_isbaseof (xh.object.classinfo, cast(ClassInfo)catch_type))
  saw_handler = true;
  }
else
  {
// == Case A ==
// Null catch type is a catch-all handler; we can catch foreign
exceptions with this.
if (catch_type is null)
  saw_handler = true;

else if (ue_header.exception_class == __gxx_exception_class)
  {
// == Case B ==
// Typeinfo are directly compared, which might not be correct
if they aren't merged.
void *except_typeinfo = ((cast(__cxa_exception *)(ue_header +
1)) - 1).exceptionType;

if (catch_type == except_typeinfo)
  saw_handler = true;

// == Case C ==
// Typeid are compared between catch type and some specially
crafted C++ Exception class.
if (cast(ClassInfo)catch_type is typeid(UnknownCPPException))
  saw_handler = true;
  }
  }
  }
*

Now let's discuss the three cases:

- Case A:
A `null` catch type never happens in D code, because `catch { }` is always
re-written to `catch(Throwable) { }` (this might have changed since 2.066,
but I doubt it).  So any kind of `catch(...)` equivalent in D would make
this is a valid candidate for catching not just C++ exceptions, but also

Re: Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

On Wednesday, 11 November 2015 at 18:02:01 UTC, Andrei 
Alexandrescu wrote:

On 11/11/2015 12:13 PM, David Nadlinger wrote:
When was the last time you did "dmd -unittest -main $(find std 
-name
'*.d')"? On current master, dmd takes just below 9 GiB to 
complete the

build.


"Then don't do that". Seriously, I agree reducing memory 
consumed is necessary but there are always extreme cases to 
construct.


Sure. Let me point out, though, that this very thread is about a 
bug making that impossible. ;)



There is agreement that a problem exists.


I'm glad that there is now.

I recall last time you got really upset, you said something 
that was so incredibly offensive, Walter and I had to hold a 
pow-wow and decide to simply ignore it and just give you time 
to calm down.


I've had to do the same quite a few times with either of you. 
What specific occasion are you thinking of? Maybe just call me 
out on it right there, right then next time (even privately, if 
you prefer so)?



Agreed. Glad you're helping them out!


Well, part of the reason I wanted to make sure that this topic 
makes it near the center of our collective attention again is 
precisely that I won't be able to spend any more time (at least 
no significant amount) working with them anytime soon. They are 
doing some quite remarkable things with D, but unfortunately 
their computers are not quantum enough for me.


 — David

Re: Our template emission strategy is broken

[David Nadlinger via Digitalmars-d]

On Wednesday, 11 November 2015 at 15:04:01 UTC, Andrei 
Alexandrescu wrote:
Liran Zvibel […] mentioned the package-at-a-time solved 
essentially all of their build problems.


They were stuck at 2.066 until very recently, though, because of 
issues very similar to the one discussed here. Right now, they 
are using LDC 2.068.2 with a couple of local workarounds for e.g. 
the issue from the original post that are not really general 
enough to go into upstream. The idea would be to return to an 
unpatched frontend as soon as possible, though.


 — David

Re: Our template emission strategy is broken

[Martin Nowak via Digitalmars-d]

On Wednesday, 11 November 2015 at 16:25:34 UTC, Andrei 
Alexandrescu wrote:
A redesign of template instantiation is necessary, and Walter 
needs to be fully involved in it. However, please give it time.


I'd still opt to first refactor the existing code into something 
that's understandable before starting to redesign the whole 
topic. Maybe the design is already fine but buried by complex to 
understand control flow. Also just redesigning this on a blank 
sheet will easily miss important edge cases. And last but not 
least if we can't come up with a better design we'd still have a 
better codebase.