Re: (Oh My) Gentool 0.3.0 released
On Monday, 10 May 2021 at 07:11:26 UTC, evilrat wrote: Can't say I hate that idea, but it has same issues as SWIG, writing any non trivial rule becomes next to impossible as there is practically zero examples and very poor documentation, the whole process becomes trial and error marathon without chance to win, and it is basically write-only code that is as worse as C++ templates. But it is definitely better than SWIG in that regard as you can get type and functions information using dir() and help() and your trusty IDE with debugger. Yeah, this is true. I actually ran into this issue already -- you have extreme devspeed + flexibility with Python but you're essentially "blind" as to the API and it's a lot of trial-and-error if it's not a very trivial C++ library. I think the end goal of being able to provide some way to easily hook into the codegen pipeline so that users can make tweaks easily & contribute them back, or customize it for personal use is probably more important than the "how". This was just the best "how" I could come up with at the time, hah. My current plan though is to provide predefined pre-generate and post-generate rules that is applied declaratively in project config, for example `ignoreDecls *::new[]` that will ignore all new operator overloads in any namespace, or one of any other existing rules that deals with specific patterns, and at some point later allow users to write their own rules like you described. After all this is binding/translator tool, not an universal one-for-all code generator. What about adding `cling` for interpreted JIT C++ without handicaps? https://blog.llvm.org/posts/2021-03-25-cling-beyond-just-interpreting-cpp/ This would let both you and users rapidly develop/iterate-on the codegen part without needing to recompile the entire solution. The main application hosts the `cling` process, and you don't do anything except bring the codegen bits in as JIT-interpreted files. It's not like performance is critical here, you know?😅 The end result is that you (or anyone else) can modify the actual codegen code, and re-run the compiled binary without needing to do anything else. `cling` will take care of acting as the engine to sort out `#include`'s, template instatiations, global context, etc. I dunno, I don't really know that much about this area at all. So there's also a good chance this idea couldn't/wouldn't work, and is not a good one -- but in my naive head it sounds good! No way, Docker is too heavy and not very user friendly or even CI friendly. It is ok to have builds optionally packed in container, but not as the only way to distribute. It might work for cppyy/cling because they rely on specific dynamic library properties or fork process on *NIX that is not there on Windows. Yeah, so the issue there was that `cppyy` needs dynamic libs to be able to work. LibTooling hilariously enough can't even be compiled as a dynamic lib on Windows, since there's a limit of ~65,000 exported visible symbols and `LINK.exe` will error. So between wanting the user to have a fairly recent, self-compiled version of LLVM, and not working on Windows at all, the easiest thing seemed to be to ask "Hey, just do `docker run` and that'll run an Ubuntu image with LLVM 13 built with LLVM_ENABLE_SHARED_LIBS and Python 3 and all of this set up for you." Not ideal but also I think maybe the only way to make the `cppyy` thing work haha.
Re: (Oh My) Gentool 0.3.0 released
On Sunday, 9 May 2021 at 19:35:52 UTC, Gavin Ray wrote:
**However, I had an idea which I haven't seen tried yet, and
have been prototyping:**
- Using `cppyy` in Python (which uses `cling`) for runtime
bindings to C++ and ability to write raw C++ code in Python
strings and JIT compile it.
- Allow users to write "drivers"/"clients" in Python which do
the codegen. Since Python isn't compiled, this means you can
realtime tweak and visualize your output much faster than
manually recompiling a C++ based LibTooling application.
I am thinking of some kind of API where you can declare rules
using annotations for AST nodes above functions for handling
them. Something like:
```py
class DCodegen:
# "t" here is a LibTooling AST node and we can use all of
Clang/LibTooling's AST API
@rule(lambda t: t.is_pointer() or t.is_reference() and \
t.pointee().is_record_indirection())
def input(cls, t, args):
return f"{{interm}} = &{c_util.struct_cast(t,
'{inp}')};"
@rule(lambda t: t.is_pointer() or t.is_reference())
def input(cls, t, args):
raise ValueError("unsupported input pointer/reference
type {}".format(t))
```
This would allow people to contribute or tweak the codegen to
their liking very rapidly.
It's all libclang under the hood though. Never looked at the
sources of cppyy or cling, but very likely this works for them
because of libs such as pybind11 that wraps C++ stuff in C++,
while the other tools trying to rely on C API(very limited) or
keep fighting with C++ API, you've already seen that all that
tools from that list except gentool is using C API and what
capabilities each provides.
Can't say I hate that idea, but it has same issues as SWIG,
writing any non trivial rule becomes next to impossible as there
is practically zero examples and very poor documentation, the
whole process becomes trial and error marathon without chance to
win, and it is basically write-only code that is as worse as C++
templates. But it is definitely better than SWIG in that regard
as you can get type and functions information using dir() and
help() and your trusty IDE with debugger.
My current plan though is to provide predefined pre-generate and
post-generate rules that is applied declaratively in project
config,
for example
`ignoreDecls *::new[]`
that will ignore all new operator overloads in any namespace, or
one of any other existing rules that deals with specific
patterns, and at some point later allow users to write their own
rules like you described.
After all this is binding/translator tool, not an universal
one-for-all code generator.
For distribution, it could be done in an Ubuntu Docker
container that comes with LLVM and Python in it, and the
scripts, then mapped to local filesystem for read access + also
if you want to edit the `DCodegen` script or supply your own
Python file as the driver.
No way, Docker is too heavy and not very user friendly or even CI
friendly. It is ok to have builds optionally packed in container,
but not as the only way to distribute. It might work for
cppyy/cling because they rely on specific dynamic library
properties or fork process on *NIX that is not there on Windows.
Re: (Oh My) Gentool 0.3.0 released
On Friday, 7 May 2021 at 18:15:47 UTC, Jacob Carlborg wrote:
On 2021-05-05 12:01, user1234 wrote:
Is it possible to use libclang and more generally LLVM c++ api
[directly in D](https://dlang.org/spec/cpp_interface.html) or
the Cpp interface is too limited ?
Was this an option, have you tried ?
Yes, it's possible to use libclang. DStep [1] is using that and
it fully written in D. Although DStep cannot create bindings
for C++ yet so I cannot guarantee that using only libclang will
work for C++ code.
[1] https://github.com/jacob-carlborg/dstep
I don't think using libclang is a good idea for C++. I think it
works very well and is the ideal choice (due to lack of
dependencies and infra) for C, but C++ has a nightmarish AST.
Have been investigating doing this with LibTooling using the C++
API, however a bit differently than has been done before (will
explain in a minute below).
The reasoning is that, when I spoke with LLVM developers, and
developers who had built successful codegen tooling for C++,
every one of them cautioned against using libclang, said that
they regretted the choice, and that the API and AST info it
exposes is not sufficient.
_I had a chance to ask Atila this same thing, last Beerconf,
and that was also his stance -- roughly that a large part of
the hurdles with dpp (beyond C++ being a nightmare to start)
were that it was built on libclang._
---
I have a large personal interest in building/helping to build a
viable C++-header-to-D-`extern (C++)` generator because I think
that is what would unlock a vast amount of potential + power for
D. Quick, no-hassle direct bindings to any (or even most) C++
libraries and tools.
But I've been doing my homework both on how to approach this and
what the state of C++ bindgen is in D. I've tried:
- D++
- Dstep
- Ohmygentool
- SWIG, with and without "Directors" feature enabled
- CPP2D
Of those, I've had the best success C++ using Ohmygentool by a
fairly large margin. It can often do several-thousand line
projects with non-trivial `#includes` mostly automatically.
---
**However, I had an idea which I haven't seen tried yet, and have
been prototyping:**
- Using `cppyy` in Python (which uses `cling`) for runtime
bindings to C++ and ability to write raw C++ code in Python
strings and JIT compile it.
- Allow users to write "drivers"/"clients" in Python which do the
codegen. Since Python isn't compiled, this means you can realtime
tweak and visualize your output much faster than manually
recompiling a C++ based LibTooling application.
I am thinking of some kind of API where you can declare rules
using annotations for AST nodes above functions for handling
them. Something like:
```py
class DCodegen:
# "t" here is a LibTooling AST node and we can use all of
Clang/LibTooling's AST API
@rule(lambda t: t.is_pointer() or t.is_reference() and \
t.pointee().is_record_indirection())
def input(cls, t, args):
return f"{{interm}} = &{c_util.struct_cast(t, '{inp}')};"
@rule(lambda t: t.is_pointer() or t.is_reference())
def input(cls, t, args):
raise ValueError("unsupported input pointer/reference
type {}".format(t))
```
This would allow people to contribute or tweak the codegen to
their liking very rapidly.
For distribution, it could be done in an Ubuntu Docker container
that comes with LLVM and Python in it, and the scripts, then
mapped to local filesystem for read access + also if you want to
edit the `DCodegen` script or supply your own Python file as the
driver.
---
**What do you all think of this idea?**
- Would love to hear feedback/opinions.
- I unfortunately have neither the D nor C++ expertise to
properly write the translation rules, so it would take some
help/collaboration from the community.
---
Here's an example of loading LibTooling in Python and building an
AST from some code.
You can see the C++ object in the `print()` output in the top
right:
- 
- 
Re: (Oh My) Gentool 0.3.0 released
On Friday, 7 May 2021 at 18:17:36 UTC, Jacob Carlborg wrote: On 2021-05-05 13:54, user1234 wrote: Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar. If I understand correctly, the Zig compiler is implemented partially in Zig. It use the LLVM C API and some wrappers C around the C++ API where the C API is not sufficient. And it quickly becomes insufficient using only the C API as feature complexity increases. No idea if Zig has to deal with C++ compiler (clang) or all it needs is pure LLVM, because the latter should have more or less feature rich C API, unlike clang that deals with C++ and has ever "unstable" API. Making wrappers for missing parts still will be a PITA, as having to pass around smart pointers definitely does not makes it easier. Anyway like I said, for bootstrap goal it is probably easier to re-purpose the tool to make thin wrappers & stubs on C++ side, pretty much just like SWIG does. Side note: But all this does not compares to what potential D to nextgen-language bindings making process would look like, as D feature set makes it even harder to translate, esp. stuff like templates and CTFE, now add static if's to that and it becomes a real mess.
Re: (Oh My) Gentool 0.3.0 released
On 2021-05-05 13:54, user1234 wrote: Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar. If I understand correctly, the Zig compiler is implemented partially in Zig. It use the LLVM C API and some wrappers C around the C++ API where the C API is not sufficient. -- /Jacob Carlborg
Re: (Oh My) Gentool 0.3.0 released
On 2021-05-05 12:01, user1234 wrote: Is it possible to use libclang and more generally LLVM c++ api [directly in D](https://dlang.org/spec/cpp_interface.html) or the Cpp interface is too limited ? Was this an option, have you tried ? Yes, it's possible to use libclang. DStep [1] is using that and it fully written in D. Although DStep cannot create bindings for C++ yet so I cannot guarantee that using only libclang will work for C++ code. [1] https://github.com/jacob-carlborg/dstep -- /Jacob Carlborg
Re: (Oh My) Gentool 0.3.0 released
On Wednesday, 5 May 2021 at 11:54:51 UTC, user1234 wrote: On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote: Maybe some time in future, but for now there is a lot more priority stuff to do before even attempting this. Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar. Yeah, but unlike LDC, for a tool that translates C++ to D it is a goal being able to also translate itself from C++ to D. And reaching this goal would be a huge milestone!
Re: (Oh My) Gentool 0.3.0 released
On Wednesday, 5 May 2021 at 11:51:27 UTC, evilrat wrote: On Wednesday, 5 May 2021 at 10:35:23 UTC, Dominikus Dittes Scherkl wrote: On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote: ... To answer both: clang has lots of templates, sometimes not so trivial ones, its code base filled with C++ constructs that does not have nice one to one translation(or simply a C++ specific detail like alignment), and the most annoying part - it has various 'tables' generated as a build step that is a mix of external tools and macros. Basically every template will need some care, likely it will be easier to just emit C++ stubs that will force compiler to emit actual code to link with than trying to translate them. So no, not currently possible. Definitely not an unpaid job, well I'll still reject it even if it is paid one. Maybe some time in future, but for now there is a lot more priority stuff to do before even attempting this. Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar.
Re: (Oh My) Gentool 0.3.0 released
On Wednesday, 5 May 2021 at 10:35:23 UTC, Dominikus Dittes Scherkl wrote: On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote: I have a technical question about the tool itself. It is mostly written in cpp. Oh dear! Isn't it possible to use it to translate itself into D? To answer both: clang has lots of templates, sometimes not so trivial ones, its code base filled with C++ constructs that does not have nice one to one translation(or simply a C++ specific detail like alignment), and the most annoying part - it has various 'tables' generated as a build step that is a mix of external tools and macros. Basically every template will need some care, likely it will be easier to just emit C++ stubs that will force compiler to emit actual code to link with than trying to translate them. So no, not currently possible. Definitely not an unpaid job, well I'll still reject it even if it is paid one. Maybe some time in future, but for now there is a lot more priority stuff to do before even attempting this.
Re: (Oh My) Gentool 0.3.0 released
On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote: I have a technical question about the tool itself. It is mostly written in cpp. Oh dear! Isn't it possible to use it to translate itself into D?
Re: (Oh My) Gentool 0.3.0 released
On Wednesday, 5 May 2021 at 06:50:29 UTC, evilrat wrote: (Oh My) Gentool - Yet another C/C++ binding generator. It is a tool to convert C/C++ code to D usable form. It takes JSON config, basically all C++ compiler flags and switches, and outputs extern(C++) declarations, (hopefully) in usable form D! ... Source https://github.com/Superbelko/ohmygentool ... I have a technical question about the tool itself. It is mostly written in cpp. Is it possible to use libclang and more generally LLVM c++ api [directly in D](https://dlang.org/spec/cpp_interface.html) or the Cpp interface is too limited ? Was this an option, have you tried ?
(Oh My) Gentool 0.3.0 released
(Oh My) Gentool - Yet another C/C++ binding generator. It is a tool to convert C/C++ code to D usable form. It takes JSON config, basically all C++ compiler flags and switches, and outputs extern(C++) declarations, (hopefully) in usable form D! It can already process (dear) imgui library (immediate mode GUI popular in game development and various graphics related tools and demos) without manual fixes! It can process recastnavigation (navmesh generation and pathfinding library) with just a few manual edits. Please note that it is still in its early stage and may contain bugs and missing language features, as well as lack of conversion for certain language constructs. It is still hard to use it directly in the build process on a real libraries due to many syntax and semantics issues, however it is already a valuable tool for making thin wrappers on C++ side to quickly bring them to your D code, given that your wrapper headers does not contains complex bodies or templates, or direct inclusions of other libraries headers such as Boost(ok, no STL too). That's it, even if it produces incomplete translation this could reduce bindings making process from hours down to minutes! Who wants to spent 10 hours manually making bindings for entire PhysX when it can be reduced to just 30 minutes? Absolutely no one! Grab one today and stay ahead of your competitors with regular updates! How to start https://github.com/Superbelko/ohmygentool/wiki/QuickStart Source https://github.com/Superbelko/ohmygentool Windows build https://github.com/Superbelko/ohmygentool/releases/tag/v0.3.0
