Re: DMD front-end can be used as a library with Dub
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote: Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated. Interesting. Did you have inline ASM in your code ? From my experience it required the backend to work, and couldn't be mocked, because else some flow analysis would go wrong (e.g. it would complain about missing `return`).
Re: DMD front-end can be used as a library with Dub
On Thu, 01 Sep 2016 18:42:13 +0200, Rory McGuire via Digitalmars-d wrote: > Thanks, for the GC stub, that will be great for playing with whether or > not a little dmd app crashes after gc_annihilate(true). > > Did I understand that right? It sounds like this there's a special function `gc_annihilate` that immediately frees all memory that libdmd has allocated. It doesn't do any reference tracking or mark/sweep. libdmd disables the collector as dmdfe does. You need to keep dmd in its own DLL. Otherwise, symbols between dmd and your application will be deduplicated or something and either you get a linker error or an arbitrary implementation is selected. Another safe way of working is to use the GC as usual, but before you call into libdmd, you call GC.disable, and you wait until you've finished using the data structures it returns before calling GC.enable.
Re: DMD front-end can be used as a library with Dub
Thanks, for the GC stub, that will be great for playing with whether or not a little dmd app crashes after gc_annihilate(true). Did I understand that right? R On Thu, Sep 1, 2016 at 6:16 PM, Cauterite via Digitalmars-d < digitalmars-d@puremagic.com> wrote: > On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote: > >> >> > Because of the poor memory management in the compiler, I included a > modified GC-stub when I compiled the frontend as a library, so that it can > be used in long-running processes: > > https://gist.github.com/Cauterite/4eb74347aea040f5d371fb49054e1819 > > You can call gc_annihilate(true) to delete the entire heap. > > Obviously you need to keep the library in a separate DLL with its own > runtime, and carefully avoid holding references to GC memory across > annihilations. > > This technique is working pretty smoothly for me so far. > > Also to compile it as a DLL you either have to remove main() from mars.d > or play games with the C runtime: https://gist.github.com/Cauter > ite/b190e62891c773703d0de3a1d99df362 >
Re: DMD front-end can be used as a library with Dub
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote: Because of the poor memory management in the compiler, I included a modified GC-stub when I compiled the frontend as a library, so that it can be used in long-running processes: https://gist.github.com/Cauterite/4eb74347aea040f5d371fb49054e1819 You can call gc_annihilate(true) to delete the entire heap. Obviously you need to keep the library in a separate DLL with its own runtime, and carefully avoid holding references to GC memory across annihilations. This technique is working pretty smoothly for me so far. Also to compile it as a DLL you either have to remove main() from mars.d or play games with the C runtime: https://gist.github.com/Cauterite/b190e62891c773703d0de3a1d99df362
Re: DMD front-end can be used as a library with Dub
I didn't see your announcement, but... AWESOME!!
This could be the basis for some really good tutorials on making compiler
backends etc...
We need more little teaser examples like the one you posted in the
beginning of this thread.
PS: I did already check the code out on github because I watch
code.dlang.org (a lot).
Thanks!
On Mon, Aug 29, 2016 at 12:42 PM, Alexander Breckel via Digitalmars-d <
digitalmars-d@puremagic.com> wrote:
> I just tried to use the DMD frontend as a Dub package in my own project to
> parse and analyze D files and it ... just worked. The dub.json for dmd is
> fairly small and doesn't require any changes to the code. This might be
> common knowledge, but I was completely unprepared for this :)
>
> Please note: This is "only" the dmd frontend (lexer, parser, semantic
> passes, CTFE). Code generation will be more complicated.
>
> The following dub pre-generation hooks were necessary:
>
> - run src/idgen.d to generate src/id.d
> - create verstr.h and SYSCONFDIR.imp
> - create and link source/ddmd/ to src/
>
> The last point makes ddmd modules reside in its correct package. I'm using
> a symbolic link for this, which is the only reason this approach is
> currently limited to Linux. In the long run, I think the cleanest way would
> be to place all ddmd files in src/ddmd/ and just leave mars.d and things
> like idgen.d in the main src/ directory.
>
> For demonstration purposes, here is a dub package to play around with:
> http://code.dlang.org/packages/ddmd-experimental
>
> Here is the dub.json that was used:
> https://github.com/aneas/ddmd-experimental/blob/master/dub.json
>
> And here is an example program using this API, have fun!
> (run with $ chmod +x file.d; ./file.d)
> ---
> #!/usr/bin/env dub
> /+ dub.sdl:
> name "ddmd-example"
> dependency "ddmd-experimental" version="~>2.71.2-b2dub"
> +/
>
> import std.path;
> import std.stdio;
> import std.string;
> import ddmd.builtin;
> import ddmd.dmodule;
> import ddmd.dclass;
> import ddmd.dsymbol;
> import ddmd.expression;
> import ddmd.func;
> import ddmd.globals;
> import ddmd.id;
> import ddmd.identifier;
> import ddmd.mtype;
> import ddmd.visitor;
>
> void main(string[] args) {
> if(args.length != 2) {
> writeln("prints top-level function and class
> declarations");
> writeln("usage: ", args[0].baseName, " d-filepath");
> return;
> }
>
> // Initialization
> global._init();
> global.params.isLinux = true;
> Type._init();
> Id.initialize();
> Module._init();
> Expression._init();
> builtin_init();
>
> // Read and parse specified module
> auto id = Identifier.idPool(args[1].baseName.stripExtension);
> auto m = new Module(args[1].toStringz, id, false, false);
> m.read(Loc());
> m.parse();
>
> // Output
> m.accept(new class Visitor {
> extern(C++):
> alias visit = Visitor.visit;
> override void visit(Module m) {
> foreach(i; 0 .. m.members.dim)
> (*m.members)[i].accept(this);
> }
> override void visit(Dsymbol s) {
> }
> override void visit(FuncDeclaration fd) {
> writeln("function ", fd.ident.toString);
> }
> override void visit(ClassDeclaration cd) {
> writeln("class ", cd.ident.toString);
> }
> });
> }
>
>
Re: DMD front-end can be used as a library with Dub
On 29/08/2016 11:54 PM, Alexander Breckel wrote: On Monday, 29 August 2016 at 11:31:58 UTC, Seb wrote: On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote: I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) [...] Nice :) How about submitting this as PR? Then you get updates for free and more people can work on tweaking the DUB file. I'm afraid this is not yet ready for upstream. Dmd would first have to move all ddmd files into src/ddmd/ to have a clean package namespace. And someone smarter than me should probably adjust the dub.json to somehow emphasize that the library contains only the front-end. With the help of sourceFiles you should be fine. You will also want sourcePaths to be set to [""] to ensure it doesn't have anything e.g. src in it. I would recommend you have dub pregenerate some import files and add them via importPaths.
Re: DMD front-end can be used as a library with Dub
On Monday, 29 August 2016 at 11:31:58 UTC, Seb wrote: On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote: I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) [...] Nice :) How about submitting this as PR? Then you get updates for free and more people can work on tweaking the DUB file. I'm afraid this is not yet ready for upstream. Dmd would first have to move all ddmd files into src/ddmd/ to have a clean package namespace. And someone smarter than me should probably adjust the dub.json to somehow emphasize that the library contains only the front-end.
Re: DMD front-end can be used as a library with Dub
On Monday, 29 August 2016 at 11:27:44 UTC, Basile B. wrote: On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote: I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated. Thx, that's interesting from an IDE developer perspective. However since DMD is a "single shot" program, using it as library could introduce memory problems. Also string handling is C style, unlike libdparse, which is truly written in D. However libdparse does nothing more than producing the AST, which becomes problematic with "static if" expressions, mixins, etc. Right. With these limitations, it might also be useful for writing static analysis tools, coding guidelines checkers, ... If someone wants to write a tool that tells me which functions and methods are not reachable from main(), or could be declared pure nothrow @nogc, i would be more than happy to use it :)
Re: DMD front-end can be used as a library with Dub
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote: I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) [...] Nice :) How about submitting this as PR? Then you get updates for free and more people can work on tweaking the DUB file.
Re: DMD front-end can be used as a library with Dub
On Monday, 29 August 2016 at 10:42:23 UTC, Alexander Breckel wrote: I just tried to use the DMD frontend as a Dub package in my own project to parse and analyze D files and it ... just worked. The dub.json for dmd is fairly small and doesn't require any changes to the code. This might be common knowledge, but I was completely unprepared for this :) Please note: This is "only" the dmd frontend (lexer, parser, semantic passes, CTFE). Code generation will be more complicated. Thx, that's interesting from an IDE developer perspective. However since DMD is a "single shot" program, using it as library could introduce memory problems. Also string handling is C style, unlike libdparse, which is truly written in D. However libdparse does nothing more than producing the AST, which becomes problematic with "static if" expressions, mixins, etc.
DMD front-end can be used as a library with Dub
I just tried to use the DMD frontend as a Dub package in my own
project to parse and analyze D files and it ... just worked. The
dub.json for dmd is fairly small and doesn't require any changes
to the code. This might be common knowledge, but I was completely
unprepared for this :)
Please note: This is "only" the dmd frontend (lexer, parser,
semantic passes, CTFE). Code generation will be more complicated.
The following dub pre-generation hooks were necessary:
- run src/idgen.d to generate src/id.d
- create verstr.h and SYSCONFDIR.imp
- create and link source/ddmd/ to src/
The last point makes ddmd modules reside in its correct package.
I'm using a symbolic link for this, which is the only reason this
approach is currently limited to Linux. In the long run, I think
the cleanest way would be to place all ddmd files in src/ddmd/
and just leave mars.d and things like idgen.d in the main src/
directory.
For demonstration purposes, here is a dub package to play around
with:
http://code.dlang.org/packages/ddmd-experimental
Here is the dub.json that was used:
https://github.com/aneas/ddmd-experimental/blob/master/dub.json
And here is an example program using this API, have fun!
(run with $ chmod +x file.d; ./file.d)
---
#!/usr/bin/env dub
/+ dub.sdl:
name "ddmd-example"
dependency "ddmd-experimental" version="~>2.71.2-b2dub"
+/
import std.path;
import std.stdio;
import std.string;
import ddmd.builtin;
import ddmd.dmodule;
import ddmd.dclass;
import ddmd.dsymbol;
import ddmd.expression;
import ddmd.func;
import ddmd.globals;
import ddmd.id;
import ddmd.identifier;
import ddmd.mtype;
import ddmd.visitor;
void main(string[] args) {
if(args.length != 2) {
writeln("prints top-level function and class declarations");
writeln("usage: ", args[0].baseName, " d-filepath");
return;
}
// Initialization
global._init();
global.params.isLinux = true;
Type._init();
Id.initialize();
Module._init();
Expression._init();
builtin_init();
// Read and parse specified module
auto id = Identifier.idPool(args[1].baseName.stripExtension);
auto m = new Module(args[1].toStringz, id, false, false);
m.read(Loc());
m.parse();
// Output
m.accept(new class Visitor {
extern(C++):
alias visit = Visitor.visit;
override void visit(Module m) {
foreach(i; 0 .. m.members.dim)
(*m.members)[i].accept(this);
}
override void visit(Dsymbol s) {
}
override void visit(FuncDeclaration fd) {
writeln("function ", fd.ident.toString);
}
override void visit(ClassDeclaration cd) {
writeln("class ", cd.ident.toString);
}
});
}
