Wim, Thanks for the detailed and thoughtful reply. I will digest and respond when I am properly back in circulation.
On 15 September 2017 at 07:43, <wlavrij...@lbl.gov> wrote: > Shaheed, > >> Ah, I had not realised rootcling existed. I've seen that I can invoke >> it using Python version-specific paths...is this the correct way to >> invoke it: >> >> ROOTCLING=/usr/local/lib/python3.6/dist-packages/cppyy_backend >> LD_LIBRARY_PATH=$ROOTCLING/lib $ROOTCLING/bin/rootcling -h > > > Yes, and here's a description of the LinkDef.h format: > > > https://root.cern.ch/root/html/guides/users-guide/AddingaClass.html#the-linkdef.h-file > >> or is there a recommended wrapper? > > > No, but I'm going to add one for pip, same as I did for genreflex. I've > been fleshing out the backend generation, taken over from Anto: > > https://bitbucket.org/wlav/cppyy-backend > > where all that can live. I'm told that I'll need rootcling anyway for > use of modules (see below). > >> I actually get some warnings and then the error: > > > Add this set of exclusions to the selection.xml: > > <exclusion> > <class pattern="*thread_mutex*" /> > <class pattern="*new_allocator*" /> > <class pattern="*Alloc_hider*" /> > </exclusion> > > Of course, the larger problem of pulling in these standard libs over and > over again is that it is a waste of cpu and memory, so I do want to see > the file_name attribute fixed. As it stands, I'd simply exclude: > > <class pattern="std::*" /> > <class pattern="__gnu_cxx::*" /> > > especially since they are already available by default. Note that those two > rules cover the ones needed for new_allocator and Alloc_hider. > > However, there is a more efficient approach that is right around the corner > (and has been right about the corner for a long time, so don't hold me to > that). Next release now seems likely though. > > The long term goal has always been to use modules: > > http://clang.llvm.org/docs/Modules.html > > but the original drivers (Apple, Google, and the C++ standards committee) > have been going back and forth on it. Now, things are finally falling into > place. Here's Google: > > https://www.youtube.com/watch?v=dHFNpBfemDI > > And here's ROOT: > > > https://indico.cern.ch/event/643728/contributions/2612822/attachments/1494074/2323893/ROOTs_C_modules_status_report.pdf > > The big deal is that C++ developers have an incentive to deploy modules, so > being able to patch into that should be a huge time saver (and where they > don't, rootcling will soon be able to create modules from headers). Note > that modules don't come for free: it will require some ambiguity resolution, > but that is typically a Good Thing (code-quality wise). > > Modules allow deserialization of only the piece of the AST that is actually > being requested, saving memory. This as opposed to header files (whether or > not precompiled) which pull in everything before them. See the status report > above for the improvements in memory usage. > > And with modules, of course, selection becomes unnecessary (markup for > automatic streamers may still be useful, but that is not relevant for > bindings generation). > >> I did wonder if I was missing some "-isystem" includes, and tried >> adding them but the --debug output from genreflex seemed to suggest >> they were being ignored. > > > Some flags are ignored as no-one was using them (so far). Some others > are definitely obsolete by now. > >> What is interesting, and might possibly throw light on the selection >> filter issue, is that the file name for the classes in >> kjsinterpreter.h itself is always the empty string ''. Classes that >> come from included files return non-empty strings such as >> 'kjsobject.h' for 'KJSObject'. > > > That's after the fact (i.e. what is stored); I don't see the rule being > respected/used at all. > >> BTW, the reason for doing this is that lots of KDE code has multiple >> classes and even namespaces in a single header file. Now, for >> discoverability of the loaded objects, I find the incremental "pop >> into cppyy,gbl on demand" somewhat limiting and I wanted to play about >> with that. I could also workaround the filter issue if I precomputed >> the needed names in a precursor pass. > > > The issue here is the memory cost of loading things that won't get used > in the end. This is why a functional dir() (which needs nothing but > strings, after all), in conjunction with lazy loading/creation when a > real access happens work well. LLVM is fully lookup based, btw. There > is a custom layer on top of Cling to make enumeration possible. > >> Finally, and most importantly given the fidelity with which cppyy >> renders the C++ code, I'm think about how Pythonisation customisation >> might be handled: e.g. a Python wrapper layer to allow a >> pointer-plus-size to render as a Python list/tuple, or generate a dict >> mapping fora QSet, and so on. (I'm dimly aware of the >> boost-recognition logic you have alluded to, this is specifically more >> about Qt-specific patterns and ad-hoc scenarios). > > > In 2015, a GSoC student fleshed this out. I never put it into PyPy b/c of > a lack of test coverage, but did put in in PyROOT. Here's an example of > the "pointer-plus-size" pythonization (from ROOT.py): > > # python side pythonizations (should live in their own file, if we get > many) > def set_size(self, buf): > buf.SetSize(self.GetN()) > return buf > > # TODO: add pythonization API to pypy-c > if not PYPY_CPPYY_COMPATIBILITY_FIXME: > cppyy.add_pythonization( > cppyy.compose_method("^TGraph(2D)?$|^TGraph.*Errors$", > "GetE?[XYZ]$", set_size)) > > The functions selected by the regexps return naked pointers, but the object > can be queried for the size (all have a consistent GetN() function). So the > method composer patches up the return value, making it a sized array, > instead of an "open-ended" one. > > I'm sitting on some patches as I wanted to tweak his APIs a bit. There > was some ordering that I felt didn't compose well, but that is minor. > > Similarly, there's code to apply ownership rules, mapping exceptions, > the new C++11 smartptrs, controlling auto-casting, handling the GIL, making > properties, and adding overloads. All driven by regexp matching of patterns. > See here: > > > https://bitbucket.org/wlav/cppyy/src/4d14ba325e494f13cc11f3f11cbb87b44048b256/python/cppyy/_pythonization.py?at=master > > (plus further support inside the bindings layer itself). > > Of course, one can hook up completely custom functions, and he made it so > that that is per C++ namespace, so nicely self-contained. > > Again, this is currently only partly available, as I need to write a lot > more tests for PyPy (which are bound to unearth some problems along the > way). And then there is documentation to be written ... > >> P.S. Please note that after today, I'll likely not have much Internet >> access for a couple of weeks, so any responses may be limited. > > > I'll make sure I have at least all my local changes pushed by then. :) > > > Best regards, > Wim > -- > wlavrij...@lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
