Re: Migrating dmd to D?

[H. S. Teoh]

On Wed, Feb 27, 2013 at 07:37:50PM -0500, Andrei Alexandrescu wrote:
> Hello,
> 
> Walter and I have had a long conversation about the next radical
> thing to do to improve D's standing. Like others in this community,
> we believe it's a good time to consider bootstrapping the compiler.
> Having the D compiler written in D has quite a few advantages, among
> which taking advantages of D's features and having a large codebase
> that would be its own test harness.

Aren't there already parts of a D compiler written by various community
members? IIRC, Timon has a prototype D compiler that supports quite a
good subset of D already. And I believe there are various D lexers and
parsers lying around, some of which may serve as the basis of a
bootstrapping D compiler.

Shouldn't we make use of these existing efforts instead of starting from
ground zero?


> By this we'd like to initiate a dialog about how this large project
> can be initiated and driven through completion. Our initial basic
> ideas are:
> 
> 1. Implement the dtoh standalone program that takes a D module and
> generates its corresponding C++ header.
> 
> 2. Use dtoh to initiate and conduct an incremental port of the
> compiler. At given points throughout the code D code will coexist
> and link with C++ code.
> 
> 3. At a point in the future the last C++ module will be replaced
> with a D module. Going forward there will be no more need for a C++
> compiler to build the compiler (except as a bootstrapping test).
> 
> It is essential that we get support from the larger community for
> this. This is a large project that should enjoy strong leadership
> apart from Walter himself (as he is busy with dynamic library
> support which is strategic) and robust participation from many of
> us.
[...]

How will this work with the continual stream of fixes that the current
C++-based compiler is getting? I assume we're not just going to put DMD
development on hold.

Also, wouldn't this be a good time to review some of the current designs
in DMD that may be hampering the full implementation of features that
we'd like, such as discrepancies with TDPL, etc.? Would it make sense to
redesign some of the code currently causing hard-to-fix issues as we're
porting that part of DMD into D? It seems a bit counterproductive to
simply transcribe the current buggy code into D, only to rewrite it
later when (if) we finally get round to fixing it.

Finally, I think somebody has brought up the idea of "freezing" a
particular subset of D that the D compiler can use in its own code,
preferably a reasonably simple subset that is safe from breaking changes
down the road (it would be pathetic if a breaking change causes the
compiler to be unable to compile itself, because the source code uses a
language construct that was later deemed to need redesign). As DMD is
ported over to D, it should be restricted to using only this subset of
the language, so that it does not hamper future developments of the
language unnecessarily.


T

-- 
Without geometry, life would be pointless. -- VS

Re: Migrating dmd to D?

[bearophile]

Andrei Alexandrescu:

Walter and I have had a long conversation about the next 
radical thing to do to improve D's standing. Like others in 
this community, we believe it's a good time to consider 
bootstrapping the compiler. Having the D compiler written in D 
has quite a few advantages, among which taking advantages of 
D's features and having a large codebase that would be its own 
test harness.


I think having the D front-end in D is a nice idea, but maybe 
it's better to use a back-end written by someone else, so most 
developing work will be spent on D itself.


bye,
bearophile

Re: Migrating dmd to D?

[timotheecour]

Use dtoh to initiate and conduct an incremental port of the 
compiler.


How about going the other way around?

Using swig to make existing dmd C++ code available to D, so we 
can advance with a mix of D and C++ code until all of C++ code is 
converted.
I was able to successfully convert large code bases from C++ to D 
using swig (eg: opencv, sfml, etc). It's the most hands-off way, 
with very minimal interface file that can recursively make things 
accessible with fine grained control (for opencv the interface 
file was < 200 loc).

Re: Migrating dmd to D?

[John Colvin]

On Thursday, 28 February 2013 at 00:55:44 UTC, timotheecour wrote:
Use dtoh to initiate and conduct an incremental port of the 
compiler.


How about going the other way around?

Using swig to make existing dmd C++ code available to D, so we 
can advance with a mix of D and C++ code until all of C++ code 
is converted.
I was able to successfully convert large code bases from C++ to 
D using swig (eg: opencv, sfml, etc). It's the most hands-off 
way, with very minimal interface file that can recursively make 
things accessible with fine grained control (for opencv the 
interface file was < 200 loc).


OT:

Could you perhaps detail the process you went through to get 
opencv to D? It would be a big help to me as I'm currently 
staring down the barrel of having to re-implement a chunk of 
OpenCV in D for a data processing app.

Re: Migrating dmd to D?

[Andrei Alexandrescu]

On 2/27/13 7:55 PM, timotheecour wrote:

Use dtoh to initiate and conduct an incremental port of the compiler.


How about going the other way around?

Using swig to make existing dmd C++ code available to D, so we can
advance with a mix of D and C++ code until all of C++ code is converted.
I was able to successfully convert large code bases from C++ to D using
swig (eg: opencv, sfml, etc). It's the most hands-off way, with very
minimal interface file that can recursively make things accessible with
fine grained control (for opencv the interface file was < 200 loc).


I think that's a fine idea but I also believe dtoh would be a mightily 
powerful program in and by itself. Once available, it would make 
migration of C++ projects to D possible and easy.


Andrei

Re: Migrating dmd to D?

[John Colvin]

On Thursday, 28 February 2013 at 00:37:50 UTC, Andrei 
Alexandrescu wrote:

Hello,


Walter and I have had a long conversation about the next 
radical thing to do to improve D's standing. Like others in 
this community, we believe it's a good time to consider 
bootstrapping the compiler. Having the D compiler written in D 
has quite a few advantages, among which taking advantages of 
D's features and having a large codebase that would be its own 
test harness.


By this we'd like to initiate a dialog about how this large 
project can be initiated and driven through completion. Our 
initial basic ideas are:


1. Implement the dtoh standalone program that takes a D module 
and generates its corresponding C++ header.


2. Use dtoh to initiate and conduct an incremental port of the 
compiler. At given points throughout the code D code will 
coexist and link with C++ code.


3. At a point in the future the last C++ module will be 
replaced with a D module. Going forward there will be no more 
need for a C++ compiler to build the compiler (except as a 
bootstrapping test).


It is essential that we get support from the larger community 
for this. This is a large project that should enjoy strong 
leadership apart from Walter himself (as he is busy with 
dynamic library support which is strategic) and robust 
participation from many of us.


Please chime in with ideas on how to make this happen.


Thanks,

Andrei


This is good news :)

What will this mean for licensing? Will we be able to fully free 
the backend?

Re: Migrating dmd to D?

[Andrei Alexandrescu]

On 2/27/13 7:57 PM, H. S. Teoh wrote:

On Wed, Feb 27, 2013 at 07:37:50PM -0500, Andrei Alexandrescu wrote:

Hello,

Walter and I have had a long conversation about the next radical
thing to do to improve D's standing. Like others in this community,
we believe it's a good time to consider bootstrapping the compiler.
Having the D compiler written in D has quite a few advantages, among
which taking advantages of D's features and having a large codebase
that would be its own test harness.


Aren't there already parts of a D compiler written by various community
members? IIRC, Timon has a prototype D compiler that supports quite a
good subset of D already. And I believe there are various D lexers and
parsers lying around, some of which may serve as the basis of a
bootstrapping D compiler.

Shouldn't we make use of these existing efforts instead of starting from
ground zero?


Of course. This is the purpose of this entire discussion - to leverage 
existing and new ideas, talent, and code.



By this we'd like to initiate a dialog about how this large project
can be initiated and driven through completion. Our initial basic
ideas are:

1. Implement the dtoh standalone program that takes a D module and
generates its corresponding C++ header.

2. Use dtoh to initiate and conduct an incremental port of the
compiler. At given points throughout the code D code will coexist
and link with C++ code.

3. At a point in the future the last C++ module will be replaced
with a D module. Going forward there will be no more need for a C++
compiler to build the compiler (except as a bootstrapping test).

It is essential that we get support from the larger community for
this. This is a large project that should enjoy strong leadership
apart from Walter himself (as he is busy with dynamic library
support which is strategic) and robust participation from many of
us.

[...]

How will this work with the continual stream of fixes that the current
C++-based compiler is getting? I assume we're not just going to put DMD
development on hold.


I don't have all answers. I do have some ideas though. I'm thinking we 
need a wedge first - dtoh in place, and one seed D file in the middle of 
the C++ project. For example the module containing the main function. 
Then I imagine there will be pull requests that delete entire modules 
and replace them with .d modules. We need to have a form of a protocol 
that "freezes" modules that are under translation.



Also, wouldn't this be a good time to review some of the current designs
in DMD that may be hampering the full implementation of features that
we'd like, such as discrepancies with TDPL, etc.? Would it make sense to
redesign some of the code currently causing hard-to-fix issues as we're
porting that part of DMD into D? It seems a bit counterproductive to
simply transcribe the current buggy code into D, only to rewrite it
later when (if) we finally get round to fixing it.


I think fixing while translating is difficult and should be approached 
on a case basis.



Finally, I think somebody has brought up the idea of "freezing" a
particular subset of D that the D compiler can use in its own code,
preferably a reasonably simple subset that is safe from breaking changes
down the road (it would be pathetic if a breaking change causes the
compiler to be unable to compile itself, because the source code uses a
language construct that was later deemed to need redesign). As DMD is
ported over to D, it should be restricted to using only this subset of
the language, so that it does not hamper future developments of the
language unnecessarily.


Far as I can tell the freeze is "last non-bootstrapped version is D 
2.xxx" and go from there.



Andrei

Re: Migrating dmd to D?

[Ary Borenszweig]

On 2/27/13 9:37 PM, Andrei Alexandrescu wrote:

Hello,


Walter and I have had a long conversation about the next radical thing
to do to improve D's standing. Like others in this community, we believe
it's a good time to consider bootstrapping the compiler.


If you do it, I think it's an excellent opportunity to rewrite the 
compiler *from scratch*, using features in D, and probably using a 
better design. It's probably easier to design the compiler now that all 
the features are more or less known. I also remember that DMD didn't 
have a visitor of sort for the semantic analysis.

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 4:37 PM, Andrei Alexandrescu wrote:

3. At a point in the future the last C++ module will be replaced with a D
module. Going forward there will be no more need for a C++ compiler to build the
compiler (except as a bootstrapping test).


Not exactly - the back end will not realistically be converted to D. This is a 
front end only conversion.

Re: Migrating dmd to D?

[Andrei Alexandrescu]

On 2/27/13 8:07 PM, John Colvin wrote:

What will this mean for licensing? Will we be able to fully free the
backend?


The backend will stay as is.

Andrei

Re: Migrating dmd to D?

[Adam D. Ruppe]

On Thursday, 28 February 2013 at 01:05:08 UTC, Andrei 
Alexandrescu wrote:
I think that's a fine idea but I also believe dtoh would be a 
mightily powerful program in and by itself.


I think the one I started a while ago is still sitting up on 
github.

Re: Migrating dmd to D?

[John Colvin]

On Thursday, 28 February 2013 at 01:12:06 UTC, Andrei 
Alexandrescu wrote:

On 2/27/13 8:07 PM, John Colvin wrote:
What will this mean for licensing? Will we be able to fully 
free the

backend?


The backend will stay as is.

Andrei


Ah, so we're just bootstrapping the frontend, not the whole 
compiler.


This presents a good opportunity to make the frontend completely 
backend agnostic (I don't know how close it is currently).

Re: Migrating dmd to D?

[Andrei Alexandrescu]

On 2/27/13 8:12 PM, Adam D. Ruppe wrote:

On Thursday, 28 February 2013 at 01:05:08 UTC, Andrei Alexandrescu wrote:

I think that's a fine idea but I also believe dtoh would be a mightily
powerful program in and by itself.


I think the one I started a while ago is still sitting up on github.


Did you finish it?

Andrei

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 5:07 PM, John Colvin wrote:

What will this mean for licensing? Will we be able to fully free the backend?


The backend will not be part of this conversion project in the foreseeable 
future.

And besides, a translation of the backend into D will not void its license.

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 4:57 PM, H. S. Teoh wrote:

How will this work with the continual stream of fixes that the current
C++-based compiler is getting? I assume we're not just going to put DMD
development on hold.


I've done many projects consisting of converting a medium sized code base from 
one language to another. The way that works is to do it incrementally. 
Incrementally means:


1. at each step (i.e. pull request) we will have a fully functioning D compiler 
that passes its test suite


2. there is no divergence in code bases because there is not a divergent code 
base.


Also, wouldn't this be a good time to review some of the current designs
in DMD that may be hampering the full implementation of features that
we'd like, such as discrepancies with TDPL, etc.? Would it make sense to
redesign some of the code currently causing hard-to-fix issues as we're
porting that part of DMD into D? It seems a bit counterproductive to
simply transcribe the current buggy code into D, only to rewrite it
later when (if) we finally get round to fixing it.


My experience chiming in - never ever ever attempt to refactor while 
translating. What always happens is you wind up with a mess that just doesn't work.




Finally, I think somebody has brought up the idea of "freezing" a
particular subset of D that the D compiler can use in its own code,
preferably a reasonably simple subset that is safe from breaking changes
down the road (it would be pathetic if a breaking change causes the
compiler to be unable to compile itself, because the source code uses a
language construct that was later deemed to need redesign). As DMD is
ported over to D, it should be restricted to using only this subset of
the language, so that it does not hamper future developments of the
language unnecessarily.


Experience chiming in - a successful model is that the HEAD is compiled by the 
previous official release of D.

Re: Migrating dmd to D?

[Vladimir Panteleev]

On Thursday, 28 February 2013 at 00:37:50 UTC, Andrei 
Alexandrescu wrote:

Please chime in with ideas on how to make this happen.


Not an expert on the topic, but what does this mean for 
maintainers of integration with other backends, like GDC and LDC? 
For example: are there any other frontends in GCC not written in 
C/C++?

Re: Migrating dmd to D?

[Brad Anderson]

On Thursday, 28 February 2013 at 01:12:47 UTC, Adam D. Ruppe 
wrote:
On Thursday, 28 February 2013 at 01:05:08 UTC, Andrei 
Alexandrescu wrote:
I think that's a fine idea but I also believe dtoh would be a 
mightily powerful program in and by itself.


I think the one I started a while ago is still sitting up on 
github.


Yep. https://github.com/adamdruppe/tools/blob/dtoh/dtoh.d

Re: Migrating dmd to D?

[bearophile]

Andrei Alexandrescu:

Like others in this community, we believe it's a good time to 
consider bootstrapping the compiler. Having the D compiler 
written in D has quite a few advantages, among which taking 
advantages of D's features and having a large codebase that 
would be its own test harness.


If just the front-end is written in D, then I think it can't be 
defined a bootstrapping, because you still need a C++ compiler to 
compile a complete D compiler.


Bye,
bearophile

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 5:11 PM, Ary Borenszweig wrote:

If you do it, I think it's an excellent opportunity to rewrite the compiler
*from scratch*, using features in D, and probably using a better design. It's
probably easier to design the compiler now that all the features are more or
less known. I also remember that DMD didn't have a visitor of sort for the
semantic analysis.



My experience with such things is it, while tempting, has a large probability of 
destroying the project entirely.

Re: Migrating dmd to D?

[sclytrack]

On Thursday, 28 February 2013 at 01:32:58 UTC, Vladimir Panteleev
wrote:
On Thursday, 28 February 2013 at 00:37:50 UTC, Andrei 
Alexandrescu wrote:

Please chime in with ideas on how to make this happen.


Not an expert on the topic, but what does this mean for 
maintainers of integration with other backends, like GDC and 
LDC? For example: are there any other frontends in GCC not 
written in C/C++?


gcc only started accepting c++ in 2010. So it will be really
though to get them to accept D.

Re: Migrating dmd to D?

[Ary Borenszweig]

On 2/27/13 10:58 PM, Walter Bright wrote:

ures in D, and probably using a better design. It's
probably easier to design the compiler now that all the features are
more or
less known. I also remember that DMD didn't have a visitor of sort for the
semantic analysis.


Why? What happened?

If you don't use all D features and plan the compiler for being written 
in D I don't think the compiler will be a good candidate for 
stress-testing the language.


It's also more fun to do it from scratch. What's the hurry?

Re: Migrating dmd to D?

[bearophile]

Walter Bright:

My experience with such things is it, while tempting, has a 
large probability of destroying the project entirely.


I agree.
And translating code must be done in an extremely methodical way, 
if you want one chance to see a working result :-) In such work 
taking short-cuts gives a high probability of producing trash. 
You have to go slowly, and double test every intermediate step.


Bye,
bearophile

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 6:12 PM, Ary Borenszweig wrote:

Why? What happened?


There's a lot of lore in the original code involving arcana about how things 
really work. If you refactor and translate at the same time, you don't have an 
incremental conversion you can run through the test suite at each step.


You wind up with one very large step change, and it doesn't work, and you're 
lost.

Another reason is the people doing the translation/refactoring have an 
inadequate grasp of why the code is the way it is, so they just wind up breaking 
it. The resulting frustration and finger-pointing ruins everything.


Translate. Test. Verify. *THEN* refactor. Because when the verify step fails, 
and you have a one:one correspondence to the original code that does work, you 
can quickly find out what went wrong. And believe me, things go wrong in the 
translation at every step of the process.




If you don't use all D features and plan the compiler for being written in D I
don't think the compiler will be a good candidate for stress-testing the 
language.


The point is not to use the compiler to stress test the language. NOT AT ALL. 
The point is to improve the compiler by taking advantage of what D offers.




It's also more fun to do it from scratch. What's the hurry?


We have limited resources, and we shouldn't squander them on something I have a 
lot of experience on knowing will fail.


Hey, anyone can ignore me and go ahead and do it that way. I wish you the best 
of luck - sometimes us old coots are dead wrong - but forgive me if I'm not 
going to be terribly sympathetic if you ignore my advice and things go badly!

Re: Migrating dmd to D?

[Marco Leise]

There are > 1000 open bugs and well known, expected language
features not implemented. In my opinion, the compiler should
be ported after all important language features are finalized.
I don't mean syntax, but stuff that only bearophile has the
complete list of: shared, allocators, ...
Also DMD leaks memory -> it is tempting to use the GC -> DMD
will often be a whole lot slower in the end. :D
Also Phobos is designed for safety and generic programming,
not for raw speed like many old C functions (at least that's
my experience). E.g. I have seen or written Unicode and
float/string conversion routines that perform 7x to 13x faster
than the 'obvious' way in D.

-- 
Marco

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 8:03 PM, Marco Leise wrote:

There are > 1000 open bugs and well known, expected language
features not implemented. In my opinion, the compiler should
be ported after all important language features are finalized.
I don't mean syntax, but stuff that only bearophile has the
complete list of: shared, allocators, ...
Also DMD leaks memory -> it is tempting to use the GC -> DMD
will often be a whole lot slower in the end. :D
Also Phobos is designed for safety and generic programming,
not for raw speed like many old C functions (at least that's
my experience). E.g. I have seen or written Unicode and
float/string conversion routines that perform 7x to 13x faster
than the 'obvious' way in D.



The motivation for the migration is not for fun, it's not even to "eat our own 
dogfood". The idea is to make the front end more reliable and more flexible by 
using D features that help. This should make us more productive and able to fix 
problems faster and presumably have fewer problems in the first place.


There are a long list of D things that will help.

Re: Migrating dmd to D?

[H. S. Teoh]

On Wed, Feb 27, 2013 at 09:32:08PM -0800, Walter Bright wrote:
[...]
> The motivation for the migration is not for fun, it's not even to
> "eat our own dogfood". The idea is to make the front end more
> reliable and more flexible by using D features that help. This
> should make us more productive and able to fix problems faster and
> presumably have fewer problems in the first place.
> 
> There are a long list of D things that will help.

How does this affect GDC/LDC? AFAIK, the GCC build scripts do not (yet?)
support bootstrapping D code.


T

-- 
If you compete with slaves, you become a slave. -- Norbert Wiener

Re: Migrating dmd to D?

[Iain Buclaw]

On Feb 28, 2013 1:31 AM, "Walter Bright"  wrote:
>
> On 2/27/2013 4:57 PM, H. S. Teoh wrote:
>>
>> How will this work with the continual stream of fixes that the current
>> C++-based compiler is getting? I assume we're not just going to put DMD
>> development on hold.
>
>
> I've done many projects consisting of converting a medium sized code base
from one language to another. The way that works is to do it incrementally.
Incrementally means:
>
> 1. at each step (i.e. pull request) we will have a fully functioning D
compiler that passes its test suite
>
> 2. there is no divergence in code bases because there is not a divergent
code base.
>
>
>> Also, wouldn't this be a good time to review some of the current designs
>> in DMD that may be hampering the full implementation of features that
>> we'd like, such as discrepancies with TDPL, etc.? Would it make sense to
>> redesign some of the code currently causing hard-to-fix issues as we're
>> porting that part of DMD into D? It seems a bit counterproductive to
>> simply transcribe the current buggy code into D, only to rewrite it
>> later when (if) we finally get round to fixing it.
>
>
> My experience chiming in - never ever ever attempt to refactor while
translating. What always happens is you wind up with a mess that just
doesn't work.
>
>
>
>> Finally, I think somebody has brought up the idea of "freezing" a
>> particular subset of D that the D compiler can use in its own code,
>> preferably a reasonably simple subset that is safe from breaking changes
>> down the road (it would be pathetic if a breaking change causes the
>> compiler to be unable to compile itself, because the source code uses a
>> language construct that was later deemed to need redesign)g . As DMD is
>> ported over to D, it should be restricted to using only this subset of
>> the language, so that it does not hamper future developments of the
>> language unnecessarily.
>
>
> Experience chiming in - a successful model is that the HEAD is compiled
by the previous official release of D.
>

Once HEAD is compiled by the previous release (or system D compiler), it
might be a good practice for HEAD to compile itself too. Then this compiler
built by HEAD will then build the library.

Regards
Iain

Re: Migrating dmd to D?

[Iain Buclaw]

On Feb 28, 2013 1:40 AM, "Vladimir Panteleev" 
wrote:
>
> On Thursday, 28 February 2013 at 00:37:50 UTC, Andrei Alexandrescu wrote:
>>
>> Please chime in with ideas on how to make this happen.
>
>
> Not an expert on the topic, but what does this mean for maintainers of
integration with other backends, like GDC and LDC? For example: are there
any other frontends in GCC not written in C/C++?

Before gdc, there was no such thing as a frontend written in C++ in gcc. As
far as history goes this was a huge drawback to make any progress in the
development of a gcc D compiler.

Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 01:37, Andrei Alexandrescu wrote:

Hello,


Walter and I have had a long conversation about the next radical thing
to do to improve D's standing. Like others in this community, we believe
it's a good time to consider bootstrapping the compiler. Having the D
compiler written in D has quite a few advantages, among which taking
advantages of D's features and having a large codebase that would be its
own test harness.


Now this is some great news to wake up to :)


By this we'd like to initiate a dialog about how this large project can
be initiated and driven through completion. Our initial basic ideas are:

1. Implement the dtoh standalone program that takes a D module and
generates its corresponding C++ header.

2. Use dtoh to initiate and conduct an incremental port of the compiler.
At given points throughout the code D code will coexist and link with
C++ code.

3. At a point in the future the last C++ module will be replaced with a
D module. Going forward there will be no more need for a C++ compiler to
build the compiler (except as a bootstrapping test).

It is essential that we get support from the larger community for this.
This is a large project that should enjoy strong leadership apart from
Walter himself (as he is busy with dynamic library support which is
strategic) and robust participation from many of us.


Short term goal:

I agree with what Walter has said in other posts that we need to make a 
direct translation as possible minimize translation bugs.


Long term goal:

When the translation is done we should refactor the compiler/front end 
to be a library, usable by other tools.


It would be nice to hear some comments from the GDC and LDC developers.

BTW, there's already a translation of DMD available, DDMD:

http://www.dsource.org/projects/ddmd

But this is a bit outdated. I also don't know how a direct translation 
this is. I can at least tell that it has a more one-to-one mapping of 
files and classes than DMD does.


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 01:37, Andrei Alexandrescu wrote:


Walter and I have had a long conversation about the next radical thing
to do to improve D's standing. Like others in this community, we believe
it's a good time to consider bootstrapping the compiler. Having the D
compiler written in D has quite a few advantages, among which taking
advantages of D's features and having a large codebase that would be its
own test harness.


BTW, how are we going to bootstrap the compiler on any possibly new 
platforms? Cross compiling?


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 02:05, Andrei Alexandrescu wrote:


I think that's a fine idea but I also believe dtoh would be a mightily
powerful program in and by itself. Once available, it would make
migration of C++ projects to D possible and easy.


It depends on where we want to put our time.

--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 02:22, Walter Bright wrote:


I've done many projects consisting of converting a medium sized code
base from one language to another. The way that works is to do it
incrementally. Incrementally means:

1. at each step (i.e. pull request) we will have a fully functioning D
compiler that passes its test suite

2. there is no divergence in code bases because there is not a divergent
code base.


Also, wouldn't this be a good time to review some of the current designs
in DMD that may be hampering the full implementation of features that
we'd like, such as discrepancies with TDPL, etc.? Would it make sense to
redesign some of the code currently causing hard-to-fix issues as we're
porting that part of DMD into D? It seems a bit counterproductive to
simply transcribe the current buggy code into D, only to rewrite it
later when (if) we finally get round to fixing it.


My experience chiming in - never ever ever attempt to refactor while
translating. What always happens is you wind up with a mess that just
doesn't work.



Finally, I think somebody has brought up the idea of "freezing" a
particular subset of D that the D compiler can use in its own code,
preferably a reasonably simple subset that is safe from breaking changes
down the road (it would be pathetic if a breaking change causes the
compiler to be unable to compile itself, because the source code uses a
language construct that was later deemed to need redesign). As DMD is
ported over to D, it should be restricted to using only this subset of
the language, so that it does not hamper future developments of the
language unnecessarily.


Experience chiming in - a successful model is that the HEAD is compiled
by the previous official release of D.


I agree with Walter here.

--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 07:45, Iain Buclaw wrote:


Once HEAD is compiled by the previous release (or system D compiler), it
might be a good practice for HEAD to compile itself too. Then this
compiler built by HEAD will then build the library.


This is a good idea.

--
/Jacob Carlborg

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 9:35 PM, H. S. Teoh wrote:

How does this affect GDC/LDC? AFAIK, the GCC build scripts do not (yet?)
support bootstrapping D code.


I don't know. I presume other gcc language tools are not written in C.

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 11:34 PM, Jacob Carlborg wrote:

BTW, there's already a translation of DMD available, DDMD:

http://www.dsource.org/projects/ddmd

But this is a bit outdated. I also don't know how a direct translation this is.
I can at least tell that it has a more one-to-one mapping of files and classes
than DMD does.



Curiously, there appears to be no copyright/license information.

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 04:49, Walter Bright wrote:


There's a lot of lore in the original code involving arcana about how
things really work. If you refactor and translate at the same time, you
don't have an incremental conversion you can run through the test suite
at each step.


I interpreted Ary's post as basically doing a clean room implementation. 
Not translate the existing code.



Hey, anyone can ignore me and go ahead and do it that way. I wish you
the best of luck - sometimes us old coots are dead wrong - but forgive
me if I'm not going to be terribly sympathetic if you ignore my advice
and things go badly!


There are already several people doing clean room implementations. At 
least of the front end.


* Dil
* SDC
* A couple of lexers/parsers

--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jonathan M Davis]

On Wednesday, February 27, 2013 23:44:02 Walter Bright wrote:
> On 2/27/2013 9:35 PM, H. S. Teoh wrote:
> > How does this affect GDC/LDC? AFAIK, the GCC build scripts do not (yet?)
> > support bootstrapping D code.
> 
> I don't know. I presume other gcc language tools are not written in C.

Wasn't all of that stuff written in pure C until fairly recently when they 
finally started letting C++ in? Maybe that was only the core stuff though and 
some of the language extensions aren't that strict. I don't know.

- Jonathan M Davis

Re: Migrating dmd to D?

[dennis luehring]

Am 28.02.2013 01:37, schrieb Andrei Alexandrescu:

Hello,


Walter and I have had a long conversation about the next radical thing
to do to improve D's standing. Like others in this community, we believe
it's a good time to consider bootstrapping the compiler. Having the D
compiler written in D has quite a few advantages, among which taking
advantages of D's features and having a large codebase that would be its
own test harness.

By this we'd like to initiate a dialog about how this large project can
be initiated and driven through completion. Our initial basic ideas are:

1. Implement the dtoh standalone program that takes a D module and
generates its corresponding C++ header.

2. Use dtoh to initiate and conduct an incremental port of the compiler.
At given points throughout the code D code will coexist and link with
C++ code.

3. At a point in the future the last C++ module will be replaced with a
D module. Going forward there will be no more need for a C++ compiler to
build the compiler (except as a bootstrapping test).


sounds like an very good idea, incremental stupid ports - refactoring 
comes later when the port is as bug-free as original


and this will also shape dtoh to an even better state

next question is - is there any way of semi-auto conversion of the code 
itself possible?

Re: Migrating dmd to D?

[Robert burner Schadek]

On 02/28/2013 02:11 AM, Ary Borenszweig wrote:

On 2/27/13 9:37 PM, Andrei Alexandrescu wrote:

Hello,


Walter and I have had a long conversation about the next radical thing
to do to improve D's standing. Like others in this community, we believe
it's a good time to consider bootstrapping the compiler.


If you do it, I think it's an excellent opportunity to rewrite the 
compiler *from scratch*, using features in D, and probably using a 
better design. It's probably easier to design the compiler now that 
all the features are more or less known. I also remember that DMD 
didn't have a visitor of sort for the semantic analysis.



+1

I think translating the frontend from C++ to D will yield very bad code. 
The AST Node types in dmd use struct with inheritance. This leads to the 
question whether or not to use structs or classes in the D frontend. If 
classes are used the GC will hit you from time to time, so class are not 
a good idea. Sure the GC could be disabled, but imho will lead to the 
same problems the frontend has now. Using struct will also require to 
use heap an pointers, also bad, but imho there are great benefits by 
from structs and there is a GC workaround for structs. Structs will make 
it easier to make the frontend into a library and use it from anything 
that knows C. This is a huge +1 imo. To get around the GC, the AST could 
be build using shared_ptr pointing to structs. Sure this will have a 
overhead, but I would rather see the compiler taking longer than telling 
me "no more memory available". I would also argue that a clean room D 
impl. with tons of unittests, testings all parse methodes, the lexer, 
the semantic analyzer etc, will yield a much better maintainable and bug 
free compiler.


Pro clean room with shared_ptr!(ASTNode):
 - Binding from C
 - fixes memory problem
 - more maintainable code

Re: Migrating dmd to D?

[eles]

On Thursday, 28 February 2013 at 00:55:33 UTC, bearophile wrote:
I think having the D front-end in D is a nice idea, but maybe 
it's better to use a back-end written by someone else, so most 
developing work will be spent on D itself.


Won't that hinder the gdc/gcc work?

Re: Migrating dmd to D?

[deadalnix]

On Thursday, 28 February 2013 at 00:37:50 UTC, Andrei 
Alexandrescu wrote:

Hello,


Walter and I have had a long conversation about the next 
radical thing to do to improve D's standing. Like others in 
this community, we believe it's a good time to consider 
bootstrapping the compiler. Having the D compiler written in D 
has quite a few advantages, among which taking advantages of 
D's features and having a large codebase that would be its own 
test harness.


By this we'd like to initiate a dialog about how this large 
project can be initiated and driven through completion. Our 
initial basic ideas are:


1. Implement the dtoh standalone program that takes a D module 
and generates its corresponding C++ header.


2. Use dtoh to initiate and conduct an incremental port of the 
compiler. At given points throughout the code D code will 
coexist and link with C++ code.


3. At a point in the future the last C++ module will be 
replaced with a D module. Going forward there will be no more 
need for a C++ compiler to build the compiler (except as a 
bootstrapping test).


It is essential that we get support from the larger community 
for this. This is a large project that should enjoy strong 
leadership apart from Walter himself (as he is busy with 
dynamic library support which is strategic) and robust 
participation from many of us.


Please chime in with ideas on how to make this happen.


Thanks,

Andrei


I don't think that is a good idea.

That will impair GDC and LDC quite a lot. Especially GDC as GCC 
team accept C++ only recently. This mean no inclusion in the 
official GCC collection.


This mean that porting DMD to D and making it the default 
implementation imply to stick with DMD backend (and it isn't open 
source even if almost). Not being open source can really impair 
D's popularity.


It also prevent to do actual progress on D during the translation 
process. FInally, Denis's ddmd hasn't succeeded.

Re: Migrating dmd to D?

[Walter Bright]

On 2/28/2013 2:03 AM, deadalnix wrote:

That will impair GDC and LDC quite a lot. Especially GDC as GCC team accept C++
only recently. This mean no inclusion in the official GCC collection.


Hmm. I had thought gccgo was written in go, but it is written in C++:

http://golang.org/doc/gccgo_contribute.html

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 11:49 PM, Jacob Carlborg wrote:

There are already several people doing clean room implementations. At least of
the front end.


Why? The only point would be to change the license of the front end.

Re: Migrating dmd to D?

[Martin Nowak]

On 02/28/2013 01:37 AM, Andrei Alexandrescu wrote:

Please chime in with ideas on how to make this happen.


Splitting off druntime's GC to also use it for dmd could be a great project.

Re: Migrating dmd to D?

[Walter Bright]

On 2/27/2013 11:35 PM, Jacob Carlborg wrote:

BTW, how are we going to bootstrap the compiler on any possibly new platforms?
Cross compiling?


Cross compiling.

Re: Migrating dmd to D?

[Maxim Fomin]

On Thursday, 28 February 2013 at 00:37:50 UTC, Andrei
Alexandrescu wrote:

Hello,


Walter and I have had a long conversation about the next 
radical thing to do to improve D's standing. Like others in 
this community, we believe it's a good time to consider 
bootstrapping the compiler. Having the D compiler written in D 
has quite a few advantages, among which taking advantages of 
D's features and having a large codebase that would be its own 
test harness.


D contains hundred of files, a "little" file can contain 5K of
good old buggy C++ code. A big "file" can contain up to 10K of
lines. I didn't collect statistics. How do you plan to convert it?

By this we'd like to initiate a dialog about how this large 
project can be initiated and driven through completion. Our 
initial basic ideas are:


1. Implement the dtoh standalone program that takes a D module 
and generates its corresponding C++ header.


With some kind of magical tool called 'dtoh'? Ok, it can
translate declarations. But what (maybe who) would rewrite code?

2. Use dtoh to initiate and conduct an incremental port of the 
compiler. At given points throughout the code D code will 
coexist and link with C++ code.


The fact that dmd outputs old object format at win32 would come
at help, wouldn't it? And how such code can coexists? From
straightforward POV, it is clear how D function can call C
function (forgetting about 64 C struct ABI problem), but how for
ex. D code can use C++ class and vice versa? And what about
runtime? Two runtimes: C++ and D? And how GC would treat C++
staff?

3. At a point in the future the last C++ module will be 
replaced with a D module. Going forward there will be no more 
need for a C++ compiler to build the compiler (except as a 
bootstrapping test).


It is essential that we get support from the larger community 
for this. This is a large project that should enjoy strong 
leadership apart from Walter himself (as he is busy with 
dynamic library support which is strategic) and robust 
participation from many of us.


So, you both are asking community help? It is nice to hear, but I
consider that community was in some kind of discrimination
against you in the past except in trivial cases like fixing bugs
and asking something which was badly needed. The very single
example of when you both agreed that you are wrong (after long
insisting that you are right because you are right) is bugzilla
issue on class inheritance and preconditions - whether base class
invariant should be respected or not.

So, I see this idea (and I can be rude and biased here) as "we
haven't treated you seriously in the past, please rewtite 100K
from C++ to D for us, we are to high to do the dirty job
ourselves".


Please chime in with ideas on how to make this happen.


Thanks,

Andrei


P.S. Latest passage is a sum of each small disappointment of how
D project is governed.

Re: Migrating dmd to D?

[Andrea Fontana]

On Thursday, 28 February 2013 at 07:34:11 UTC, Jacob Carlborg 
wrote:

Long term goal:

When the translation is done we should refactor the 
compiler/front end to be a library, usable by other tools.


Something like Fabrice Bellard tcc/libtcc? 
(http://bellard.org/tcc/)
If you can call dmd using an api, you can write a new range of 
application. Every program could be a custom compiler. And can 
compile and patch itself.

Re: Migrating dmd to D?

[Dicebot]

I support the intention and agree that it needs to be done 
part-by-part with no re-factoring allowed to minimize 
regressions. Probably could have even added myself to volunteers, 
but not sure, it looks like a very time-intensive project.


But issues with LDC and GDC need to be settled first. If D 
front-end in D considerably harms any of those, it is a complete 
no-no, even if porting will be perfect. Just not worth the loss.

Re: Migrating dmd to D?

[FG]

On 2013-02-28 02:03, John Colvin wrote:

Could you perhaps detail the process you went through to get opencv to D? It
would be a big help to me as I'm currently staring down the barrel of having to
re-implement a chunk of OpenCV in D for a data processing app.


I second that request! :)

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 08:48, Walter Bright wrote:


Curiously, there appears to be no copyright/license information.


Should be the same as DMD uses.

--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 11:25, Walter Bright wrote:


Why? The only point would be to change the license of the front end.


I don't know, I'm not doing it. Possibly reasons:

* Fun
* Learn
* Change the license
* DMD is not written in D
* DMD is not built/usable as a library
* DMD contains a lot of bugs

Although I don't know for sure if they're clean room implementations or 
not. They are at least not direct translations.


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 11:45, Andrea Fontana wrote:


Something like Fabrice Bellard tcc/libtcc? (http://bellard.org/tcc/)
If you can call dmd using an api, you can write a new range of
application. Every program could be a custom compiler. And can compile
and patch itself.


I have no idea about tcc, I was more thinking of like Clang and LLVM.

--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 11:03, deadalnix wrote:


I don't think that is a good idea.

That will impair GDC and LDC quite a lot. Especially GDC as GCC team
accept C++ only recently. This mean no inclusion in the official GCC
collection.

This mean that porting DMD to D and making it the default implementation
imply to stick with DMD backend (and it isn't open source even if
almost). Not being open source can really impair D's popularity.

It also prevent to do actual progress on D during the translation
process. FInally, Denis's ddmd hasn't succeeded.


They could stick with the C++ front end and fold in changes as needed. 
They can't to direct merges though.


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 11:27, Martin Nowak wrote:


Splitting off druntime's GC to also use it for dmd could be a great
project.


How do you mean. If DMD is written in D it would be using druntime 
including its GC?


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 11:35, Maxim Fomin wrote:


With some kind of magical tool called 'dtoh'? Ok, it can
translate declarations. But what (maybe who) would rewrite code?


One would translate a single file to D. Then run "dtoh" over that file 
to get a C/C++ interface to the D file. You can then link the D object 
file with the rest of the C++ code.


--
/Jacob Carlborg

Re: Migrating dmd to D?

[H. S. Teoh]

On Thu, Feb 28, 2013 at 03:05:31PM +0100, Jacob Carlborg wrote:
> On 2013-02-28 11:03, deadalnix wrote:
> 
> >I don't think that is a good idea.
> >
> >That will impair GDC and LDC quite a lot. Especially GDC as GCC team
> >accept C++ only recently. This mean no inclusion in the official GCC
> >collection.
> >
> >This mean that porting DMD to D and making it the default
> >implementation imply to stick with DMD backend (and it isn't open
> >source even if almost). Not being open source can really impair D's
> >popularity.
> >
> >It also prevent to do actual progress on D during the translation
> >process. FInally, Denis's ddmd hasn't succeeded.
> 
> They could stick with the C++ front end and fold in changes as needed.
> They can't to direct merges though.
[...]

This will be a big problem once the front end is completely written in
D. The GDC maintainers will have to translate bugfixes to the D code
back to the C++ code. In some cases this may not be possible (D-specific
features may be used in the fix, which requires non-trivial translation
to C++, which is prone to bugs not in the D code). This will be a lot of
maintenance work.

This is one of the reasons I suggested using a frozen version of D to
write the front end with. That way, we can include the C++ source for
that frozen version in GDC, and then bundle the newer D source code with
it, so during the bootstrapping process, the GCC scripts first build a
working (but older) D compiler from the C++ sources, then use that to
compile the newer D source code to produce the final compiler.


T

-- 
The trouble with TCP jokes is that it's like hearing the same joke over
and over.

Re: Migrating dmd to D?

[Ary Borenszweig]

On 2/28/13 4:49 AM, Jacob Carlborg wrote:

On 2013-02-28 04:49, Walter Bright wrote:


There's a lot of lore in the original code involving arcana about how
things really work. If you refactor and translate at the same time, you
don't have an incremental conversion you can run through the test suite
at each step.


I interpreted Ary's post as basically doing a clean room implementation.
Not translate the existing code.


Yes, exactly.

Re: Migrating dmd to D?

[Andrei Alexandrescu]

On 2/28/13 1:45 AM, Iain Buclaw wrote:

Once HEAD is compiled by the previous release (or system D compiler), it
might be a good practice for HEAD to compile itself too. Then this
compiler built by HEAD will then build the library.


Do you think there's a risk that bootstrapping causes trouble for gdc?

Andrei

Re: Migrating dmd to D?

[FG]

On 2013-02-28 11:58, Dicebot wrote:

But issues with LDC and GDC need to be settled first. If D front-end in D
considerably harms any of those, it is a complete no-no, even if porting will be
perfect. Just not worth the loss.


Indeed, but even if LDC and GDC don't stop this from happening, I'm more worried 
(as someone willing to write more of his programs in D instead of picking C++) 
about stretching resources too thin on this one project, while there are tons of 
more important things to do first (from my POV).


Let's see:
1) shared libraries (loading and being loaded),
2) GC, const refs, manual MM, containers managing their memory,
3) stop hiding AA's implementation,
4) improve libraries: bigint, xml, you name it,
...
n) rewrite the compiler's frontend.

I'm sure you can find a lot more to fit into the [5..n].
Even the infamous properties could rank higher than this migration, because, 
frankly, I don't care what language the compiler is in, as long as I don't have 
to install a JVM to use it. :)

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 16:02, FG wrote:


frankly, I don't care what language the compiler is in, as long
as I don't have to install a JVM to use it. :)


Then .Net it is :) /irony

--
/Jacob Carlborg

Re: Migrating dmd to D?

[Iain Buclaw]

On Feb 28, 2013 3:02 PM, "Andrei Alexandrescu" <
seewebsiteforem...@erdani.org> wrote:
>
> On 2/28/13 1:45 AM, Iain Buclaw wrote:
>>
>> Once HEAD is compiled by the previous release (or system D compiler), it
>> might be a good practice for HEAD to compile itself too. Then this
>> compiler built by HEAD will then build the library.
>
>
> Do you think there's a risk that bootstrapping causes trouble for gdc?
>
> Andrei

No more a risk than bootstrapping for dmd.  However my main concern is that
I'd rather see this happen at a time when we port to more architectures
other than x86 and 64bit. Leaving the cross-compiler step as a non-issue as
there is already a suitable D compiler on the targeted system.

Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Andrei Alexandrescu]

On 2/28/13 5:03 AM, deadalnix wrote:

That will impair GDC and LDC quite a lot.


Let's see what the respective project leaders say.

Andrei

Re: Migrating dmd to D?

[Andrei Alexandrescu]

On 2/28/13 5:35 AM, Maxim Fomin wrote:

So, you both are asking community help? It is nice to hear, but I
consider that community was in some kind of discrimination
against you in the past except in trivial cases like fixing bugs
and asking something which was badly needed. The very single
example of when you both agreed that you are wrong (after long
insisting that you are right because you are right) is bugzilla
issue on class inheritance and preconditions - whether base class
invariant should be respected or not.

So, I see this idea (and I can be rude and biased here) as "we
haven't treated you seriously in the past, please rewtite 100K
from C++ to D for us, we are to high to do the dirty job
ourselves".


Now that's some grudge. What happened here? Were you wronged somehow in 
the past?


Thanks,

Andrei

Re: Migrating dmd to D?

[FG]

On 2013-02-28 16:07, Jacob Carlborg wrote:

On 2013-02-28 16:02, FG wrote:


frankly, I don't care what language the compiler is in, as long
as I don't have to install a JVM to use it. :)


Then .Net it is :) /irony



I was wondering if I should have also mentioned .Net.
Now I know the answer. :)

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 16:25, FG wrote:


I was wondering if I should have also mentioned .Net.
Now I know the answer. :)


I'm sure we can find some other environment you would need to install to 
be able to run D :)


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Iain Buclaw]

On 28 February 2013 15:24, Andrei Alexandrescu <
seewebsiteforem...@erdani.org> wrote:

> On 2/28/13 5:03 AM, deadalnix wrote:
>
>> That will impair GDC and LDC quite a lot.
>>
>
> Let's see what the respective project leaders say.
>
> Andrei
>


I'll provide facts, but I'll reserve any opinion to myself.

So, feel free to send me a list of questions you want me to answer. :o)

-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Andrei Alexandrescu]

On 2/28/13 10:53 AM, Iain Buclaw wrote:

On 28 February 2013 15:24, Andrei Alexandrescu
mailto:seewebsiteforem...@erdani.org>>
wrote:

On 2/28/13 5:03 AM, deadalnix wrote:

That will impair GDC and LDC quite a lot.


Let's see what the respective project leaders say.

Andrei



I'll provide facts, but I'll reserve any opinion to myself.

So, feel free to send me a list of questions you want me to answer. :o)


"Would an initiative of porting dmd to D create difficulties for gdc?"

Andrei

Re: Migrating dmd to D?

[Shahid]

On Wed, 27 Feb 2013 17:58:27 -0800, Walter Bright wrote:

> On 2/27/2013 5:11 PM, Ary Borenszweig wrote:
>> If you do it, I think it's an excellent opportunity to rewrite the
>> compiler *from scratch*, using features in D, and probably using a
>> better design. It's probably easier to design the compiler now that all
>> the features are more or less known. I also remember that DMD didn't
>> have a visitor of sort for the semantic analysis.
>>
>>
> My experience with such things is it, while tempting, has a large
> probability of destroying the project entirely.

I wholeheartedly agree with Walter on this. I'd like to see as much 1:1 
translation as possible first, then refactoring can begin.

Re: Migrating dmd to D?

[Marco Leise]

Am Wed, 27 Feb 2013 21:32:08 -0800
schrieb Walter Bright :
 
> The motivation for the migration is not for fun, it's not even to "eat our 
> own 
> dogfood". The idea is to make the front end more reliable and more flexible 
> by 
> using D features that help. This should make us more productive and able to 
> fix 
> problems faster and presumably have fewer problems in the first place.
> 
> There are a long list of D things that will help.

In a way it means "eat your own dogfood" if you compare C++ to
D. C++ may be lacking, but you can emulate a few things and it
has good code analysis tools.
Maybe I'm too pessimistic in thinking this will take a year,
stop bug fixes and stall language design issues from being
resolved as well as slow the compiler down notably, since
you'll be writing easy to maintain code using Phobos and a GC
and that is always slower than ASM, right? :p

-- 
Marco

Re: Migrating dmd to D?

[Iain Buclaw]

On 28 February 2013 16:01, Andrei Alexandrescu <
seewebsiteforem...@erdani.org> wrote:

> On 2/28/13 10:53 AM, Iain Buclaw wrote:
>
>> On 28 February 2013 15:24, Andrei Alexandrescu
>> > > >>
>>
>> wrote:
>>
>> On 2/28/13 5:03 AM, deadalnix wrote:
>>
>> That will impair GDC and LDC quite a lot.
>>
>>
>> Let's see what the respective project leaders say.
>>
>> Andrei
>>
>>
>>
>> I'll provide facts, but I'll reserve any opinion to myself.
>>
>> So, feel free to send me a list of questions you want me to answer. :o)
>>
>
> "Would an initiative of porting dmd to D create difficulties for gdc?"
>
> Andrei
>


Gnat's frontend is written in Ada, however it does not depend on having to
call anything from the gcc backend.

We still do not know what portions of the frontend is being ported over to
D.  The way gdc is written, it takes the D Frontend, removes all C++ parts
that interface with dmd's backend - toElem; toIR; toObjFile; toSymbol;
toCtype; toDt (this latter one I am in the middle of removing from gdc) and
implements them to instead build GCC trees, the code itself also being in
C++.

These are all methods defined in D Front-End, and rely on calling and
interfacing with the gcc backend.  Re-writing these in D is not an option,
as require access to GCC macros.

See tree.h for the majority of that list:
http://gcc.gnu.org/git/?p=gcc.git;a=blob_plain;f=gcc/tree.h;hb=refs/heads/master


-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Timon Gehr]

On 02/28/2013 03:02 PM, Jacob Carlborg wrote:

On 2013-02-28 11:25, Walter Bright wrote:


Why? The only point would be to change the license of the front end.


I don't know, I'm not doing it. Possibly reasons:

* Fun


Yup.


...
* Change the license


Actually I dislike the whole licensing issue.


* DMD is not written in D


Yup.


* DMD is not built/usable as a library


Yup. There should be a sufficiently simple frontend on which libraries 
for source code analysis and manipulation tools can be built.



* DMD contains a lot of bugs



Yup, even the intention on the compiler developer side is buggy 
sometimes. DMD will keep breaking my code because the hand-wavy notion 
of a "forward reference error" somehow appears to be accepted.


* Having only one implementation harms the language quality, because 
people are more likely to be willing to accept arguably buggy or stupid 
behaviour, and there is no pressure to clarify details of the spec. (eg. 
UDA's in DMD introduce an awkward and underpowered AST macro system. 
Value range propagation is too conservative for bitwise operations, DMD 
contains undocumented features, like types as function arguments in a 
typeof, or invoking opCall via an assignment, etc.)


* wc -l reveals that the DMD front end source code is roughly 30 to 40 
times larger than it should be for what it does in my opinion. 
Refactoring it so that it shrinks by that factor in C++ is a lot harder 
than building it from scratch in D.



(* Knowing the guts of a front end means you can decide to add type 
system and syntax extensions for private use. :P)



Although I don't know for sure if they're clean room implementations or
not. They are at least not direct translations.



Mine is clean room.

Re: Migrating dmd to D?

[Arlen]

On Wed, Feb 27, 2013 at 6:37 PM, Andrei Alexandrescu <
seewebsiteforem...@erdani.org> wrote:

> Hello,
>
>
> Walter and I have had a long conversation about the next radical thing to
> do to improve D's standing. Like others in this community, we believe it's
> a good time to consider bootstrapping the compiler. Having the D compiler
> written in D has quite a few advantages, among which taking advantages of
> D's features and having a large codebase that would be its own test harness.
>
> By this we'd like to initiate a dialog about how this large project can be
> initiated and driven through completion. Our initial basic ideas are:
>
> 1. Implement the dtoh standalone program that takes a D module and
> generates its corresponding C++ header.
>
> 2. Use dtoh to initiate and conduct an incremental port of the compiler.
> At given points throughout the code D code will coexist and link with C++
> code.
>
> 3. At a point in the future the last C++ module will be replaced with a D
> module. Going forward there will be no more need for a C++ compiler to
> build the compiler (except as a bootstrapping test).
>
> It is essential that we get support from the larger community for this.
> This is a large project that should enjoy strong leadership apart from
> Walter himself (as he is busy with dynamic library support which is
> strategic) and robust participation from many of us.
>
> Please chime in with ideas on how to make this happen.
>
>
> Thanks,
>
> Andrei
>

Having ported Boost.units to D, I can attest to this being a lot of work.
I did try translating first and then refactoring the code, but that did not
go well, mainly because of all the tricks and hacks employed when doing
template meta-programming in C++ that did not translate well at all to D.
With my first attempt I pretty much ended up with C++ code that was written
in D, and that's not what I wanted.  So I had to start over, refactoring
and writing D code in D as I went.  The problem with refactoring is that
once you refactor a piece, chances are that you will need to refactor
everything that depends on the code that was refactored, and that starts a
domino effect.

Of course, things are different with DMD, so translating first and then
refactoring is probably the right way to do it. But, I don't see how we
could use D's nice features without refactoring.  So, I presume this is
going to be done in two phases:

Phase 1: direct translation to make sure everything works.
Phase 2: refactoring to use D's nice features.

And your three steps would be describing Phase 1.


Arlen

Re: Migrating dmd to D?

[js.mdnq]

I believe a complete rewrite from the ground up using a fixed 
stable dmd is needed. The reason is two fold: Many things have 
been learned about the evolution of the D language over time. 
Much of the trouble of D has been stabling the D implementation 
and spec. Second, Trying to port the C++ code to D to make a D 
compiler will only multiply the bugs in DMD. (i.e., it will 
introduce new bugs from the conversion and retain the old bugs)


Instead, I believe proper project management is needed along with 
a SOLID language specification and clear delineation of goals. If 
the language spec itself is flawed then the same things will 
occur as is with DMD.


To tie the dependence on C/C++ the D compiler would need to be 
written in the language subset of the intersection between DMD 
and the new language spec. This should not be hard to do but must 
be strictly maintained. Else one will always require dmd to 
compile the compiler.


Hence, a solid language spec for the new D compiler is needed. 
The language spec must overlap with the old spec and the D 
compiler must only be written in this overlap. (It should be 
obvious but this allows the D compiler to be compiled in DMD or 
itself, after the bootstrap one can gradually evolve the subset 
to include the newer features a few versions behind since the old 
dmd is not needed)


The problem with such an undertaking behind successful is all in 
the project management. I would say we need a solid language spec 
and the subset between it and the current spec(frozen at some 
point). I imagine they would be almost identical so actually 
little real work would be needed.


So, who's up for writing the D 3.0 spec?

Re: Migrating dmd to D?

[pjmlp]

On Thursday, 28 February 2013 at 07:58:31 UTC, Jonathan M Davis 
wrote:

On Wednesday, February 27, 2013 23:44:02 Walter Bright wrote:

On 2/27/2013 9:35 PM, H. S. Teoh wrote:
> How does this affect GDC/LDC? AFAIK, the GCC build scripts 
> do not (yet?)

> support bootstrapping D code.

I don't know. I presume other gcc language tools are not 
written in C.


Wasn't all of that stuff written in pure C until fairly 
recently when they
finally started letting C++ in? Maybe that was only the core 
stuff though and
some of the language extensions aren't that strict. I don't 
know.


- Jonathan M Davis


GNAT is written in Ada.

If I am not mistaken, many non standard frontends for Modula-2, 
Modula-3 and Pascal also use their own languages.


http://gcc.gnu.org/frontends.html

--
Paulo

Re: Migrating dmd to D?

[Iain Buclaw]

On Feb 28, 2013 9:36 PM, "pjmlp"  wrote:
>
> On Thursday, 28 February 2013 at 07:58:31 UTC, Jonathan M Davis wrote:
>>
>> On Wednesday, February 27, 2013 23:44:02 Walter Bright wrote:
>>>
>>> On 2/27/2013 9:35 PM, H. S. Teoh wrote:
>>> > How does this affect GDC/LDC? AFAIK, the GCC build scripts > do not
(yet?)
>>> > support bootstrapping D code.
>>>
>>> I don't know. I presume other gcc language tools are not written in C.
>>
>>
>> Wasn't all of that stuff written in pure C until fairly recently when
they
>> finally started letting C++ in? Maybe that was only the core stuff
though and
>> some of the language extensions aren't that strict. I don't know.
>>
>> - Jonathan M Davis
>
>
> GNAT is written in Ada.
>
> If I am not mistaken, many non standard frontends for Modula-2, Modula-3
and Pascal also use their own languages.
>
> http://gcc.gnu.org/frontends.html
>
> --
> Paulo

See my message above. The problem is not what language the frontend is
written in, the problem is not requiring to interface to the gcc backend
from the parts that are written in D.

Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Walter Bright]

On 2/28/2013 5:15 AM, Jacob Carlborg wrote:

On 2013-02-28 08:48, Walter Bright wrote:


Curiously, there appears to be no copyright/license information.


Should be the same as DMD uses.


Well, it should say. In fact, with the current copyright law which specifies the 
default as "copyrighted, and no license at all" it needs to or nobody can use it.


I know I probably come off as a ninny about this, but professional users will 
run screaming from any open source code unless it contains:


1. a copyright notice
2. a license
3. who owns the above

Re: Migrating dmd to D?

[Walter Bright]

On 2/28/2013 7:53 AM, Iain Buclaw wrote:

So, feel free to send me a list of questions you want me to answer. :o)


Would it impair having it accepted as part of gcc?

Re: Migrating dmd to D?

[Russel Winder]

Walter,

On Thu, 2013-02-28 at 14:46 -0800, Walter Bright wrote:
[…]
> I know I probably come off as a ninny about this, but professional users will 
> run screaming from any open source code unless it contains:
> 
> 1. a copyright notice
> 2. a license
> 3. who owns the above

Not ninni-ish at all, very sensible. Of course it is not the
professional users that worry about these things, it is their lawyers.
Worse there is a whole collection of misunderstanding and
misapprehensions, not to mention FUD, about the various well known
licences.

-- 
Russel.
=
Dr Russel Winder  t: +44 20 7585 2200   voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


signature.asc
Description: This is a digitally signed message part

Re: Migrating dmd to D?

[Rob T]

On Thursday, 28 February 2013 at 17:04:24 UTC, Marco Leise wrote:

Am Wed, 27 Feb 2013 21:32:08 -0800
schrieb Walter Bright :

The motivation for the migration is not for fun, it's not even 
to "eat our own dogfood". The idea is to make the front end 
more reliable and more flexible by using D features that help. 
This should make us more productive and able to fix problems 
faster and presumably have fewer problems in the first place.


There are a long list of D things that will help.


In a way it means "eat your own dogfood" if you compare C++ to
D. C++ may be lacking, but you can emulate a few things and it
has good code analysis tools.
Maybe I'm too pessimistic in thinking this will take a year,
stop bug fixes and stall language design issues from being
resolved as well as slow the compiler down notably, since
you'll be writing easy to maintain code using Phobos and a GC
and that is always slower than ASM, right? :p


The biggest benefit I predict that will come from an effort like 
this, is the productive change that comes about when you "eat 
your own dog food".


--rt

Re: Migrating dmd to D?

[Marco Leise]

Am Thu, 28 Feb 2013 18:44:31 +0100
schrieb Timon Gehr :

> * wc -l reveals that the DMD front end source code is roughly 30 to 40 
> times larger than it should be for what it does in my opinion.

That can only mean that you don't really know what it does in
my opinion. Sure such a large code base accumulates duplicates
since not everybody knows about all helper functions or copy
and paste was the least intrusive bug fix somewhere, but you
don't really believe that 94% of the front end are
unnecessary, do you?

> (* Knowing the guts of a front end means you can decide to add type 
> system and syntax extensions for private use. :P)

Ah, I see where the wind blows ;)

-- 
Marco

Re: Migrating dmd to D?

[Philippe Sigaud]

>> (* Knowing the guts of a front end means you can decide to add type
>> system and syntax extensions for private use. :P)
>
> Ah, I see where the wind blows ;)

Or, even better, an extensible FE with commonly distributed extensions
like the Haskell compiler. And, of course, an AST macro system and a
pony ;)

No, scratch that, with a good macro system, we can have the pony.

Re: Migrating dmd to D?

[Zach the Mystic]

On Thursday, 28 February 2013 at 05:32:37 UTC, Walter Bright 
wrote:

On 2/27/2013 8:03 PM, Marco Leise wrote:

There are > 1000 open bugs and well known, expected language
features not implemented. In my opinion, the compiler should
be ported after all important language features are finalized.
I don't mean syntax, but stuff that only bearophile has the
complete list of: shared, allocators, ...
Also DMD leaks memory -> it is tempting to use the GC -> DMD
will often be a whole lot slower in the end. :D
Also Phobos is designed for safety and generic programming,
not for raw speed like many old C functions (at least that's
my experience). E.g. I have seen or written Unicode and
float/string conversion routines that perform 7x to 13x faster
than the 'obvious' way in D.



The motivation for the migration is not for fun, it's not even 
to "eat our own dogfood". The idea is to make the front end 
more reliable and more flexible by using D features that help. 
This should make us more productive and able to fix problems 
faster and presumably have fewer problems in the first place.


There are a long list of D things that will help.


So you're saying some of our dogfood is actually caviar then...

I would divide the caviar into two groups, manifest and hidden. 
The manifest caviar is the easiest to sell. Hidden caviar is the 
benefits which are unexpected by at least a portion of the D 
community. Each piece of hidden caviar therefore needs one or 
more champions.


Not that this is a perfect example, but the lexer being assembled 
by Brian and Dmitri seems to have a spark of the hidden caviar 
about it, lending weight to the "clean room" camp. The politics 
of "existing" versus "clean room" must be mastered because 
there's a lot of room for resentment there if the wrong choices 
are made, it seems to me.


One thing both "clean room" and "existing" have, or should have, 
in common is the test suite, which is probably a better spec than 
the spec is. Perhaps a method can be devised which makes it easy 
to divide and conquer the test suite.

Re: Migrating dmd to D?

[Rob T]

On Thursday, 28 February 2013 at 07:34:11 UTC, Jacob Carlborg 
wrote:

[...]

Long term goal:

When the translation is done we should refactor the 
compiler/front end to be a library, usable by other tools.




Yes that would be awesome. It can make the exact same tool at 
least 10x more useful and versatile.


I'm a big fan of tools that allow plugins, so I would also like 
to see parts of the compiler be written as loadable plugins that 
can be swapped in/out with different versions, and also to allow 
the tool set to be extensible by anyone.


--rt

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 16:53, Iain Buclaw wrote:


So, feel free to send me a list of questions you want me to answer. :o)


Could the GDC front end remain in C++ and changes be folded in anyway? 
These changes do not need to be direct translation of the D code.


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 18:18, Iain Buclaw wrote:


We still do not know what portions of the frontend is being ported over
to D.  The way gdc is written, it takes the D Frontend, removes all C++
parts that interface with dmd's backend - toElem; toIR; toObjFile;
toSymbol; toCtype; toDt (this latter one I am in the middle of removing
from gdc) and implements them to instead build GCC trees, the code
itself also being in C++.

These are all methods defined in D Front-End, and rely on calling and
interfacing with the gcc backend.  Re-writing these in D is not an
option, as require access to GCC macros.


If you're removing these functions does it matter which language they're 
written in ?


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-02-28 20:28, Arlen wrote:


Having ported Boost.units to D, I can attest to this being a lot of
work.  I did try translating first and then refactoring the code, but
that did not go well, mainly because of all the tricks and hacks
employed when doing template meta-programming in C++ that did not
translate well at all to D.  With my first attempt I pretty much ended
up with C++ code that was written in D, and that's not what I wanted.
So I had to start over, refactoring and writing D code in D as I went.
The problem with refactoring is that once you refactor a piece, chances
are that you will need to refactor everything that depends on the code
that was refactored, and that starts a domino effect.


That sounds more like one needs to figure out the intent of the code and 
not just look at the exact syntax. An easy example. C++ supports 
multiple inheritance, D does not. Trying to emulate that will most 
likely cause a lot of problem. But the use case in C++ could just be 
interfaces.


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Thomas Koch]

> Walter and I have had a long conversation about the next radical thing
> to do to improve D's standing. Like others in this community, we believe
> it's a good time to consider bootstrapping the compiler. Having the D
> compiler written in D has quite a few advantages, among which taking
> advantages of D's features and having a large codebase that would be its
> own test harness.

Two points from the viewpoint of the Debian distribution:

Debian is ported to many different platforms and in average one new platform 
port started every year. A huge pain point for porters are circular (or 
self) dependencies. A lot of effort goes into breaking such circles.

So in the moment the D language is great in that it does not introduce a new 
circular dependency. It would be a pity to lose this.

The second important thing for Debian (and Fedora and others) is licensing. 
It's a pity that DMD isn't free software and I believe DMD not being in 
distros is one reason for the low popularity of D. It's hard to learn D with 
gdc while all tutorials are based on DMD.

So instead of a rewrite of D, it would rather be important (from my humble 
point of view) to replace non-free parts of DMD.

Thank you, Thomas Koch

Re: Migrating dmd to D?

[Timon Gehr]

On 03/01/2013 07:48 AM, Marco Leise wrote:

Am Thu, 28 Feb 2013 18:44:31 +0100
schrieb Timon Gehr :


* wc -l reveals that the DMD front end source code is roughly 30 to 40
times larger than it should be for what it does in my opinion.


That can only mean that you don't really know what it does in
my opinion.


I guess it (almost) implements the language. :)


Sure such a large code base accumulates duplicates
since not everybody knows about all helper functions or copy
and paste was the least intrusive bug fix somewhere, but you
don't really believe that 94% of the front end are
unnecessary, do you?



I believe it is bloated. Maybe it's partly because it is written in C++.


...

Re: Migrating dmd to D?

[Vincent]

Andrei, I cannot catch these steps:

1. Implement the dtoh standalone program that takes a D module 
and generates its corresponding C++ header.


2. Use dtoh to initiate and conduct an incremental port of the 
compiler. At given points throughout the code D code will 
coexist and link with C++ code.


Since D is written on C++, what exactly you gonna convert to "C++ 
header"??


Anyway, I offer to keep away from C++ code - it's written with 
"C++ in mind", while D offers brand new world. So I support 
rewritting D on D from scratch.
Initially frontend only + join LLVM backend. And next (if we find 
LLVM not so well) we can write our own backend.

Re: Migrating dmd to D?

[dennis luehring]

Am 01.03.2013 10:54, schrieb Vincent:

Andrei, I cannot catch these steps:


1. Implement the dtoh standalone program that takes a D module
and generates its corresponding C++ header.

2. Use dtoh to initiate and conduct an incremental port of the
compiler. At given points throughout the code D code will
coexist and link with C++ code.


Since D is written on C++, what exactly you gonna convert to "C++
header"??


it will be an incremental port using both C++ and D code
so the DMD C++ base needs headers from the D based module replacements

>Anyway, I offer to keep away from C++ code - it's written with
>"C++ in mind", while D offers brand new world. So I support
>rewritting D on D from scratch.

that will just not work for an project of that size - or better - there 
are several tries of from-scratch without success out there


an incremental port and then refactor is the best/fasted and error-free 
thing we can get


benefits:
-will help to find more bugs/missing features/tools in the C/C++ <-> D 
conversion/adaption area (which is still a big plus for D)


-keeps both the pure D and the semi-pure :) D guys (Walter,gdc,ldc 
frontend developers) in the same boat


-the dmd frontend will become the very first community driven BIG 
project that can be a much better orientation for bug-prevention 
designs/future D ideas etc. then every thing else


etc...

Re: Migrating dmd to D?

[Iain Buclaw]

On 28 February 2013 22:50, Walter Bright  wrote:

> On 2/28/2013 7:53 AM, Iain Buclaw wrote:
>
>> So, feel free to send me a list of questions you want me to answer. :o)
>>
>
> Would it impair having it accepted as part of gcc?
>
>
Not if we follow by way of example, eg: the Ada or Go model.  Where they
have separately maintained code for their front end that may be used
verbatim in multiple compilers, with the code outside the front end doing
everything related to interfacing with GCC, and only what's related to
interfacing with GCC.

The only part where this may be problematic for D is that there are still
parts of the front end that require patching for use with GCC, this is
being worked on, but would require co-operation from both code gdc, ldc and
dmd maintains to align their copies of the front end up.


Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Iain Buclaw]

On 1 March 2013 07:26, Jacob Carlborg  wrote:

> On 2013-02-28 18:18, Iain Buclaw wrote:
>
>  We still do not know what portions of the frontend is being ported over
>> to D.  The way gdc is written, it takes the D Frontend, removes all C++
>> parts that interface with dmd's backend - toElem; toIR; toObjFile;
>> toSymbol; toCtype; toDt (this latter one I am in the middle of removing
>> from gdc) and implements them to instead build GCC trees, the code
>> itself also being in C++.
>>
>> These are all methods defined in D Front-End, and rely on calling and
>> interfacing with the gcc backend.  Re-writing these in D is not an
>> option, as require access to GCC macros.
>>
>
> If you're removing these functions does it matter which language they're
> written in ?
>
> --
> /Jacob Carlborg
>


Not removed, re-written.  For example, VectorExp::toElem().

For gdc:
https://github.com/D-Programming-GDC/GDC/blob/master/gcc/d/d-elem.cc#L2375
For dmd:
https://github.com/D-Programming-Language/dmd/blob/master/src/e2ir.c#L3820


Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Iain Buclaw]

On 1 March 2013 07:28, Jacob Carlborg  wrote:

> On 2013-02-28 16:53, Iain Buclaw wrote:
>
>  So, feel free to send me a list of questions you want me to answer. :o)
>>
>
> Could the GDC front end remain in C++ and changes be folded in anyway?
> These changes do not need to be direct translation of the D code.
>
> --
> /Jacob Carlborg
>

The code that interfaces with gcc needs to be in either C or C++.  There
are C++ structs/classes defined in the D frontend that while they include
all methods required for parsing/semantic analysis of D code.  They also
include methods that are used to generate the codegen for the backend
(toElem, toIR, toSymbol, etc).  In gdc, these are gcc interfacing methods
that can't be converted to D.

-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Iain Buclaw]

On 1 March 2013 08:50, Thomas Koch  wrote:

> > Walter and I have had a long conversation about the next radical thing
> > to do to improve D's standing. Like others in this community, we believe
> > it's a good time to consider bootstrapping the compiler. Having the D
> > compiler written in D has quite a few advantages, among which taking
> > advantages of D's features and having a large codebase that would be its
> > own test harness.
>
> Two points from the viewpoint of the Debian distribution:
>
> Debian is ported to many different platforms and in average one new
> platform
> port started every year. A huge pain point for porters are circular (or
> self) dependencies. A lot of effort goes into breaking such circles.
>
>
As I understand it, the biggest pain is getting an initial system compiler
on the ported target in the first place.  One that package is in place, it
gets easier to manage the circular dependency.


Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-03-01 11:23, Iain Buclaw wrote:


Not removed, re-written.  For example, VectorExp::toElem().

For gdc:
https://github.com/D-Programming-GDC/GDC/blob/master/gcc/d/d-elem.cc#L2375
For dmd:
https://github.com/D-Programming-Language/dmd/blob/master/src/e2ir.c#L3820


Aha, I see.

--
/Jacob Carlborg

Re: Migrating dmd to D?

[Jonathan M Davis]

On Friday, March 01, 2013 09:50:13 Thomas Koch wrote:
> > Walter and I have had a long conversation about the next radical thing
> > to do to improve D's standing. Like others in this community, we believe
> > it's a good time to consider bootstrapping the compiler. Having the D
> > compiler written in D has quite a few advantages, among which taking
> > advantages of D's features and having a large codebase that would be its
> > own test harness.
> 
> Two points from the viewpoint of the Debian distribution:
> 
> Debian is ported to many different platforms and in average one new platform
> port started every year. A huge pain point for porters are circular (or
> self) dependencies. A lot of effort goes into breaking such circles.
> 
> So in the moment the D language is great in that it does not introduce a new
> circular dependency. It would be a pity to lose this.
> 
> The second important thing for Debian (and Fedora and others) is licensing.
> It's a pity that DMD isn't free software and I believe DMD not being in
> distros is one reason for the low popularity of D. It's hard to learn D with
> gdc while all tutorials are based on DMD.
> 
> So instead of a rewrite of D, it would rather be important (from my humble
> point of view) to replace non-free parts of DMD.

I don't really care about the license, but I can definitely say that part of me 
finds the idea of having a compiler compiling itself to be a bad idea, much is 
compiler folks love to do that. Recently, I had some older haskell code that I 
needed to compile, but it followed the previous haskell standard, and I 
couldn't get the current compiler to compile it even in compatability mode. 
So, I tried to compile an older version of the compiler from before the new 
standard, and it had exactly the same problems that my code did, because it 
was written in haskell using the older standard. So, I had to give up on being 
able to compile my code, because I couldn't get my hands on an old enough 
version of the compiler. If they'd just written it in C/C++, then I wouldn't 
have had that problem.

I know that it's generally touted as a great idea for a language to compile 
itself, and I'm sure that it would be great to be able to use D's features in 
the compiler, but the circular dependency that that causes is a definite 
negative IMHO.

- Jonathan M Davis

Re: Migrating dmd to D?

[Jacob Carlborg]

On 2013-03-01 11:29, Iain Buclaw wrote:


The code that interfaces with gcc needs to be in either C or C++.  There
are C++ structs/classes defined in the D frontend that while they
include all methods required for parsing/semantic analysis of D code.
They also include methods that are used to generate the codegen for the
backend (toElem, toIR, toSymbol, etc).  In gdc, these are gcc
interfacing methods that can't be converted to D.


Can you use the current toElem, toIR and toSymbol written in C++. Then 
port in changes from the version written in D as needed?


--
/Jacob Carlborg

Re: Migrating dmd to D?

[Iain Buclaw]

On 1 March 2013 10:43, Jacob Carlborg  wrote:

> On 2013-03-01 11:29, Iain Buclaw wrote:
>
>  The code that interfaces with gcc needs to be in either C or C++.  There
>> are C++ structs/classes defined in the D frontend that while they
>> include all methods required for parsing/semantic analysis of D code.
>> They also include methods that are used to generate the codegen for the
>> backend (toElem, toIR, toSymbol, etc).  In gdc, these are gcc
>> interfacing methods that can't be converted to D.
>>
>
> Can you use the current toElem, toIR and toSymbol written in C++. Then
> port in changes from the version written in D as needed?
>
> --
> /Jacob Carlborg
>

It's much more complex than that.  Think about compatibility between
calling D structs/classes from C++, and that dmd and gdc don't share the
same representation of types in the back-end that are common to the
front-end - elem, type, IRState, real_t, to name a few.


Regards
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Iain Buclaw]

On 1 March 2013 10:43, Jacob Carlborg  wrote:

> On 2013-03-01 11:29, Iain Buclaw wrote:
>
>  The code that interfaces with gcc needs to be in either C or C++.  There
>> are C++ structs/classes defined in the D frontend that while they
>> include all methods required for parsing/semantic analysis of D code.
>> They also include methods that are used to generate the codegen for the
>> backend (toElem, toIR, toSymbol, etc).  In gdc, these are gcc
>> interfacing methods that can't be converted to D.
>>
>
> Can you use the current toElem, toIR and toSymbol written in C++. Then
> port in changes from the version written in D as needed?
>
> --
> /Jacob Carlborg
>


Also, what changes dmd makes to it's back-end facing functions do not
necessarily affect gdc.  So there has never really been a direct conversion
from one to the other, however as they (should) do effectively the same
code generation, one can draw comparisons between them.

Regards,
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

Re: Migrating dmd to D?

[Timon Gehr]

On 03/01/2013 11:08 AM, dennis luehring wrote:

Am 01.03.2013 10:54, schrieb Vincent:
...
 >Anyway, I offer to keep away from C++ code - it's written with
 >"C++ in mind", while D offers brand new world. So I support
 >rewritting D on D from scratch.

that will just not work for an project of that size - or better - there
are several tries of from-scratch without success out there
...


As well as some that will likely become successful.
Anyway, imo both should be attempted.