Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

I'm afraid that there has been no progress as I am buried in "startup"
mode. I'm not sure when that might change.

On Mon, 5 Nov 2018, 14:02 Philipp A.  Hi Shaheed!
>
> The year is nearing its end, and I wonder if there has been any progress
> and/or if you people need help with the bindings!
>
> I’d really like to revive my IPython console in Kate :D
>
> Best, Philipp
>
> Shaheed Haque  schrieb am Sa., 13. Jan. 2018 um
> 19:06 Uhr:
>
>> Thanks to some upstream fixes, I have the cppyy-based bindings for KF5
>> and also Qt5 (see below) showing signs of life. Notes:
>>
>>
>>1. The packaging has advanced to the point where I think ECM-based
>>framework-by-framework bindings are a real possibility, with both Py2 and
>>Py3. AFAICS, this addresses the main feedback received to date.
>>2. With reference to the remark about tracking dependencies between
>>frameworks, apologies for the delayed response as I somehow missed
>>the email. I note that the dependencies currently in CMake often seem
>>incomplete. I'll bring that to the community separately.
>>3. There is one issue still open upstream (
>>
>> https://bitbucket.org/wlav/cppyy/issues/16/pragma-link-defined_in-seems-to-select).
>>However, I don't consider this to be a showstopper...we might even be able
>>to live with it as is.
>>4. For me, the jury is still out on PyQt versus a new set of
>>cppyy-based Qt bindings. Clearly PyQt is solid and mature, but the
>>limitations really concern me (if anybody wants to know more, I'm happy to
>>discuss, but let's do that in another thread please). Now, given that 
>> there
>>are examples in the wild of interoperating cppyy/cling/ROOT with PyQt, I'm
>>going to sidestep this question but am playing with a cppyy-based 
>> approach.
>>At this point, all of Qt has basic cppyy-based bindings, and the next step
>>is to tackle things like finding a way to express the object
>>ownership/destruction rules in a more-or-less systematic way.
>>5. On the P2/P3 question, I'm presently still committed to both P2
>>and P3. I *have* had a couple of minor occasions where P3-only might have
>>been nice *for my code*, but if I do find an issue that tips the balance,
>>or I find some serious benefit *for the bindings*, I'll drop P2. One
>>possible such benefit would be if I can see a sane way to address PEP484
>>type hints.
>>
>> To get here, I had to build a subset of the tooling I previously had
>> developed for the SIP-based approach. The big difference is the absence of
>> any need to support customisation of the generated bindings. I am hopeful
>> that in the worst case, there might be some minimal customisation (known as
>> Pythonisations in cppyy parlance) such as for #4 above, but nothing like
>> the scale needed for SIP.
>>
>> The core tooling is not specific to KF5 or KDE or Qt5, and is developed
>> in upstream cppyy over on bitbucket.org. The core tooling is built
>> around CMake, notably for the generation phase and the C++ library build.
>>
>> The PoC extends the core tooling with Pythonic packaging and installation
>> using pip/wheels, also from CMake. As before I would look for help to get
>> an ECM equivalent, possibly based on the same approach but perhaps
>> including CI and distribution via PyPi.
>>
>> Finally, now would be a good time for anybody else who wants to get
>> involved to step up, especially as a new job limits my free time.
>>
>> Thanks, Shaheed
>>
>> P.S. Not to stoke the the P2/P3 wars unnecessarily, but while I know that
>> upstream Clang just added P3 support in the clang 5.0 release, current
>> Ubuntu only packages it for 2.7.14. So I won't be moving yet...
>>
>> On 5 November 2017 at 13:23, Boudewijn Rempt  wrote:
>>
>>> On Sat, 4 Nov 2017, Chris Burel wrote:
>>>
>>> > I think this is a remarkably short sighted statement. It assumes that
>>> people that would use these bindings have no existing Python codebase at
>>> all, and can afford to start a brand new project. The reality is much
>>> different.
>>> >
>>> > Let's take a specific example. I have 6 years experience writing
>>> Python for the visual effects industry. We have a 10 year old Python 2
>>> codebase. We also use an application from Autodesk called Maya. It has been
>>> a Qt 4 application with Python 2 embedded since 2012. In 2016 they jumped
>>> to qt 5 and p

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Hi Luca,

On 15 January 2018 at 08:24, Luca Beltrame  wrote:

> Il giorno Sat, 13 Jan 2018 18:05:45 +
> Shaheed Haque  ha scritto:
>
> Hello Shaheed,
>
> >1. The packaging has advanced to the point where I think ECM-based
> >framework-by-framework bindings are a real possibility, with both
>
> That's absolutely fantastic. Thanks for the continued effort!
>

It is a relief to get to this point after what, about 2 years? Also, a lot
of credit has to go to the support of upstream (Wim, but also not
forgettting Phil on the the SIP side of things). Still, it is not quite
time to put out the bunting...

>
> >4. For me, the jury is still out on PyQt versus a new set of
> > cppyy-based Qt bindings. Clearly PyQt is solid and mature, but the
>
> How do the bindings work? I suppose you'll need cppyy-based Qt to run
> your bindings, correct?
>

Technically, there seem to be example in the wild of people (at CERN and
possibly elsewhere?) creating applications with ROOT (the immediate
precursor of cppyy) and PyQt. And since everything is, at one level, just
Python, I don't see any bar to using PyQT. However, recall that I only went
down the cppyy route after pouring a LOT of effort into the SIP route, and
then concluding both the customisation effort and the limitations were too
great for my liking. Basically, I expect it would take ongoing large-scale
investment from somebody like me to keep the bindings going.

So, given the current cppyy Qt bindings required NO customisation effort
(but see below) and should free us of the limitations of the SIP-generated
approach, I'm putting my effort into the cppyy-based approach.

> like finding a way to express the object ownership/destruction rules
>
> How are you dealing with Python's GC? A major point (and PITA) when
> developing with PyQt and PyKDE4 was that a single slip in tracking
> objects meant the C++ object would get GC'd under your feet,
> causing crashes. This was particularly bad with the Plasma bindings
> because they would crash the whole desktop.
>

As you will understand, there is no magic bullet for this. Both SIP and
cppyy need to be told how to handle this. With SIP, part of the extensive
customisation framework I created allowed me to say things like:

-for all constructors that match a set of regexps, and which have one
argument called "parent" of type "QWidget *", declare that Python does not
own the object

There is a similar capability in cppyy, and I just have to create a way to
pass that hint.


> > benefit *for the bindings*, I'll drop P2. One possible such benefit
>
> I don't mind either approach: FYI though, most major distros are
> phasing out Py2-as-default (Ubuntu, Arch, and openSUSE are the ones I'm
> aware of).
>
> > minimal customisation (known as Pythonisations in cppyy parlance)
> > such as for #4 above, but nothing like the scale needed for SIP.
>
> That would be great. It was one of the major issues when keeping PyKDE4
> up to date.
>

Precisely. In marked contrast to SIP, I am hopeful that this ownership
thingy may in fact be the ONLY customisation needed (though I/we need to
think about Qt meta object stuff like signals/slots too).


> > look for help to get an ECM equivalent, possibly based on the same
> > approach but perhaps including CI and distribution via PyPi.
>
> Can you link the repository with the tools?
>

Not sure what you mean. It is all on PyPI (and bitbucket):
https://pypi.python.org/pypi?%3Aaction=search&term=cppyy&submit=search.

Let me know if I missed your point.

Thanks, Shaheed


> --
> Luca Beltrame, Ph.D.
> Translational Genomics Unit, Department of Oncology
> IRCCS Istituto di Ricerche Farmacologiche "Mario Negri"
>

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Thanks to some upstream fixes, I have the cppyy-based bindings for KF5 and
also Qt5 (see below) showing signs of life. Notes:


   1. The packaging has advanced to the point where I think ECM-based
   framework-by-framework bindings are a real possibility, with both Py2 and
   Py3. AFAICS, this addresses the main feedback received to date.
   2. With reference to the remark about tracking dependencies between
   frameworks, apologies for the delayed response as I somehow missed the
   email. I note that the dependencies currently in CMake often seem
   incomplete. I'll bring that to the community separately.
   3. There is one issue still open upstream (
   
https://bitbucket.org/wlav/cppyy/issues/16/pragma-link-defined_in-seems-to-select).
   However, I don't consider this to be a showstopper...we might even be able
   to live with it as is.
   4. For me, the jury is still out on PyQt versus a new set of cppyy-based
   Qt bindings. Clearly PyQt is solid and mature, but the limitations really
   concern me (if anybody wants to know more, I'm happy to discuss, but let's
   do that in another thread please). Now, given that there are examples in
   the wild of interoperating cppyy/cling/ROOT with PyQt, I'm going to
   sidestep this question but am playing with a cppyy-based approach. At this
   point, all of Qt has basic cppyy-based bindings, and the next step is to
   tackle things like finding a way to express the object
   ownership/destruction rules in a more-or-less systematic way.
   5. On the P2/P3 question, I'm presently still committed to both P2 and
   P3. I *have* had a couple of minor occasions where P3-only might have been
   nice *for my code*, but if I do find an issue that tips the balance, or I
   find some serious benefit *for the bindings*, I'll drop P2. One possible
   such benefit would be if I can see a sane way to address PEP484 type hints.

To get here, I had to build a subset of the tooling I previously had
developed for the SIP-based approach. The big difference is the absence of
any need to support customisation of the generated bindings. I am hopeful
that in the worst case, there might be some minimal customisation (known as
Pythonisations in cppyy parlance) such as for #4 above, but nothing like
the scale needed for SIP.

The core tooling is not specific to KF5 or KDE or Qt5, and is developed in
upstream cppyy over on bitbucket.org. The core tooling is built around
CMake, notably for the generation phase and the C++ library build.

The PoC extends the core tooling with Pythonic packaging and installation
using pip/wheels, also from CMake. As before I would look for help to get
an ECM equivalent, possibly based on the same approach but perhaps
including CI and distribution via PyPi.

Finally, now would be a good time for anybody else who wants to get
involved to step up, especially as a new job limits my free time.

Thanks, Shaheed

P.S. Not to stoke the the P2/P3 wars unnecessarily, but while I know that
upstream Clang just added P3 support in the clang 5.0 release, current
Ubuntu only packages it for 2.7.14. So I won't be moving yet...

On 5 November 2017 at 13:23, Boudewijn Rempt  wrote:

> On Sat, 4 Nov 2017, Chris Burel wrote:
>
> > I think this is a remarkably short sighted statement. It assumes that
> people that would use these bindings have no existing Python codebase at
> all, and can afford to start a brand new project. The reality is much
> different.
> >
> > Let's take a specific example. I have 6 years experience writing Python
> for the visual effects industry. We have a 10 year old Python 2 codebase.
> We also use an application from Autodesk called Maya. It has been a Qt 4
> application with Python 2 embedded since 2012. In 2016 they jumped to qt 5
> and pyside2. Now Autodesk knows that companies have built large codebase
> around their product that requires Python 2. What would've happened if
> pyside2 did not support Python 2.7? They'd be stuck either forcing all
> their customers to move to Python 3 and risk people not wanting the new
> version of the software, or they'd be prevented from moving to Qt 5.
> >
>
> You will have to switch to Python 3 by 2019, since that's what the VFX
> Reference Platform says. If you haven't started on the migration yet,
> you're very late. And the VFX Refernece Platform is basically Autodesk
> telling the rest of the industry what to use, including their weird
> patchset for Qt...
>
> > So no, Python 2 is not dead. Not by a long shot.
>
> For VFX, it will be dead in 2019. See http://www.vfxplatform.com/
>
>
> --
> Boudewijn Rempt | http://www.krita.org, http://www.valdyas.org
>

D8780: Try `llvm-config` to find `libclang`

[Shaheed Haque]

shaheed added inline comments.

INLINE COMMENTS

> turbov wrote in FindPythonModuleGeneration.cmake:217
> As for me, I would remove whole this part of "brute forcing" available clang 
> version. I can't see how it could work w/o additional hints due a  location 
> of `libclang` is not a standard directory (typically `/usr/lib/llvm-X.Y/`) 
> and no `HINTS` given to `find_library`...
> 
> Also this code is really lack of flexibility: what if I have few versions of 
> `clang`, but want to use only particular one... (which is not a newest one)?? 
> -- Nowadays, this scenario not handled at all...
> 
> Instead, I propose to be not so aggressive:
> 
> - If a user has `clang` installed, let's ask the version of it (`clang 
> --version`), then "guess" a library dir
> - if nothing found, then OK... just skip the generation of SIP files...
> - If a user has some private installation of Clang (possibly built somewhere 
> in `/home/john/work/tools/clang/...`) and really want to build Python 
> bindings, let's give him an option to provide a libdir via CMake CLI (like 
> `-DGPB_LIBCLANG_PATH=...`)
> - If a user wants a particular version, he always can set a hint variable to 
> the desired path...

I have found a new, potentially less horrid, approach to finding libclang. It 
turns out that the Python bindings expose some APIs which I had not noticed 
before, meaning that - AFAICS - the bindings are more or less able to find the 
library they need by themselves. Also the relevant support code appears to be 
platform sensitive, so stands some chance of working on Macs and Windows.

You may wonder why, given this, we need to do anything at all...I can only 
guess the authors wanted to give users the ability to override the default 
library choice. The resulting Python code can be completely self-contained, 
with the main reason to need CMake support being to support any such override:

  from clang.cindex import Config
  if args.libclang:
  Config.set_library_file(args.libclang)
  lib = Config().lib

However, there is a secondary reason too: Python3 support. The problem is that 
the even though upstream has, as of libclang-5.0 shipped Python3 support, the 
distros mostly have not yet picked this up (I've filed requests for both Debian 
and Ubuntu). Until then, the pPython bindings are only usable from Python2. I 
have a FindLibClang.cmake which does the necessary work here:

https://bitbucket.org/ShaheedHaque/cppyy-backend/commits/d585e46cf60a866b0d6649960a53a2bd40af626b?at=feature_cmake_macros#chg-cling/python/cppyy_backend/cmake/FindLibClang.cmake

Good luck.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D8780

To: turbov
Cc: cgiboudeaux, shaheed, #frameworks, #build_system

How is symbol visibility set in KF5 and KDE?

[Shaheed Haque]

Hi all,

I just realised that the Python binding effort is not setting the default
visibility for symbols using the -fvisibility=xxx option when processing
the header files [1]. Of course I can see the export macros set by the
likes of attica_exports.h, but I don't see where the compiler default is
set. Can somebody kindly point that out?

Thanks, Shaheed

[1] I'm also a bit mystified by the fact that I am deliberately querying
CMake for the COMPILE_FLAGS to use, but I have not seen -fvisibility
anywhere...

D8780: Try `llvm-config` to find `libclang`

[Shaheed Haque]

shaheed added inline comments.

INLINE COMMENTS

> shaheed wrote in FindPythonModuleGeneration.cmake:212
> The original code had a minimum requirement of either 3.8 or 3.9 (I don't 
> recall exactly, but the code here implies 3.8 is the minimum). Newer versions 
> are fine.

I remembered as soon as I submitted...

3.8 is the minimum requirement. 3.9 introduced extra enumeration values which 
the code as-is does not use,but my not-to-be-merged branch does.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D8780

To: turbov
Cc: cgiboudeaux, shaheed, #frameworks, #build_system

D8780: Try `llvm-config` to find `libclang`

[Shaheed Haque]

shaheed added inline comments.

INLINE COMMENTS

> turbov wrote in FindPythonModuleGeneration.cmake:212
> What distribution is this? Why only 3.8 and 3.9?? Why not 4.0 or 5.0???
> 
> I've just checked Ubuntu 14.04 (Clang 3.3 to 3.9 available) and 17.10 (up to 
> 5.0 available)... installing `libclang-X.Y` (or `libclang-X.Y-devel` 
> obviously) always bring `libclang.so` library (which is of cause a symlink to 
> a versioned name), but it always here... Also, I've checked CentOS 7 w/ DTS7 
>  (Clang 4.0) 
> -- same here...
> 
> so the question is "why to search a versioned name?" -- I don't know any 
> distro where `libclang.so` is not available... do you?

The original code had a minimum requirement of either 3.8 or 3.9 (I don't 
recall exactly, but the code here implies 3.8 is the minimum). Newer versions 
are fine.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D8780

To: turbov
Cc: cgiboudeaux, shaheed, #frameworks, #build_system

D8780: Try `llvm-config` to find `libclang`

[Shaheed Haque]

shaheed added a comment.


  I'm afraid your's truly might take a bit of the blame. The basis of the logic 
was likely my original Python code, which did just enough to make Ubuntu work.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D8780

To: turbov
Cc: shaheed, #frameworks, #build_system

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

I have made an attempt to get roughly all the bindings I was previously
attempting with SIP to be (a) generated and (b) built.

As of now, we have:

- Customisations with a diffstat that reads "21 files changed, 20
insertions(+), 48 deletions(-)".
- All but 5-6 of the formal tier 1, 2 and 3 frameworks [1] building.
- Of the wider set of 111 informal frameworks [2] I have all but 14
building.

Most of the customisations are to workaround missing dependencies in the
CMake descriptions.Most of the build issues are compile-issues (e.g.
incomplete #includes, or incorrect -I settings), but there are at least 2
that look like cppyy issues. This is pretty solid proof that building atop
technology that is based on LLVM and Cling can trivially process at least
these KDE sources (including templates, overloads, operators, and the rest).

Oh, and it has taken about a month to get here. As a baseline, the
SIP-based attempt took about 18 months, several thousand lines of custom
rules (including a lot of deleting overloads, operators and templates that
could not be handled) to get a little over half the bindings through steps
a and b. The build takes about 10 minutes on a single core, versus about
3-4 hours with the SIP-based attempt.

More anon...

Thanks, Shaheed

[1] https://api.kde.org/frameworks/
[2] More liberally defined than [1], corresponding to what Ubuntu installs
as "libkf5xxx" packages. This include [1].

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Philipp,

On 5 November 2017 at 14:48, Philipp A.  wrote:

> Hi Shaheed, Chris,
>
> Shaheed Haque  schrieb am Sa., 4. Nov. 2017 um
> 18:35 Uhr:
>
>> FWIW, I already tried that (types.ModuleType is itself a perfectly
>> subclassable class!) […]
>>
>> Now, none of that may be a limiting factor in the plan you seem to be
>> discussing, but it was part of what made me think "here be dragons"...
>
>
> For me that sounds like a perfectly acceptable clearly-defined instruction:
>
>- If you can, don’t use a __init__.py for namespace packages (because
>it’s simple).
>- If you need an __init__.py, only use one or use identical ones.
>
> Reasons for needing a __init__.py include the need for toplevel
> constants/classes/functions, or supporting Python 2. The docs also say that
> you can mix __init__.py omission and (identical) __init__.pys.
>
> I created an example here: https://github.com/flying-sheep/namespace-test
>

As I said, I know how this works, and am using it. As I also said, I'm
happy for others to pursue ECM-based bindings because I don't plan to.

My branch in pykde5.git is now at the point where the packaging now works
with wheels. With any luck, the combination of CMake and pip/setuptools I
have
brewed there will facilitate any ECM-based bindings. Note: the packaging
code depends very much on the evolving PR at
https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-experimental-version-of-a.
Neither branch is ready to merge or formally review.

I plan to continue to push forward on the remaining functional TODOs.

Thanks, Shaheed

You can see that by using a __init__.py in exactly one of the merged
> packages, you can define toplevel constants/classes/functions like BASE in
> the example.
>
> If we need, we can also define toplevel constants and so on in multiple
> distributions, like this (this specific version requires all distributions
> to know about all others, but that’s automatable):
>
> *namespace-test-1/namespace_test/namespace_test_1_init.py*:
> FOO = 1
> *namespace-test-2/namespace_test/**namespace_test_2_init.py*:
> BAR= 1
> *namespace-test-{1,2}/namespace_test/**__init__.py*:
> from pkgutil import extend_path
> __path__ = extend_path(__path__, __name__)
> try:
> from .namespace_test_1_init import *
> except ImportError:
> pass
> try:
> from .namespace_test_2_init import *
> except ImportError:
> pass
>
> Chris Burel  schrieb am So., 5. Nov. 2017 um
> 02:49 Uhr:
>
>> I think this is a remarkably short sighted statement. It assumes that
>> people that would use these bindings have no existing Python codebase at
>> all, and can afford to start a brand new project. The reality is much
>> different. […]
>
>
> No, because you’re missing something here: There’s no KF5 bindings. So
> every project that’ll use Shaheed’s new cool KF5 bindings will be a new
> project.
>
> Therefore the only thing Python 2 KF5 bindings would accomplish is to make
> people think that starting a *new* Python 2 project in 2018 was still a
> good idea. Which it isn’t.
>

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

There is a POC-quality implementation of the integration with KDE here:

https://cgit.kde.org/pykde5.git/tree/?h=srhaque-cppyy-bindings&id=19a94fb3ae2b40a985913ed4e49400e02df56dc2

This contains examples of bindings for Akonadi and KDcraw. My next
steps will be to do a few more, and then move on to the TODOs listed
earlier.

On 4 November 2017 at 17:35, Shaheed Haque  wrote:
> Wim, Philipp,
>
> On 4 November 2017 at 16:45, Philipp A.  wrote:
>> Hi Wim!
>>
>>> So now I have a (C++) namespace 'A' that bears no relationship to anything
>>> to do with the file system or any type of Python packaging: it exists only
>>> in memory for the duration of the python session.
>>
>>
>> Yeah, cool, so we just use a path hook and are ready to go right?
>>
>> https://www.python.org/dev/peps/pep-0302/#specification-part-2-registering-hooks
>>
>> Every framework gets its own path hook that handles imports from that
>> framework.
>>
>> Frameworks with shared namespaces will just handle their own specific paths,
>> right?
>>
>>> The renaming does not actually scare me that much from a performance point
>>> of view: already, C++ has typedefs for classes and aliases for namespaces.
>>> There is always the problem of "leakage" as mentioned above, where the
>>> end-user sees both at some point, but internally, aliasing will work fine:
>>> it's just another reference to the same object.
>>
>>
>> Cool, so this might be possible after all! Certainly not as a top priority
>> before getting things to work, but still!
>>
>>> It is to attach a meta class for a __getattr__, the use of properties, and
>>> to allow pickling. The module type does not support meta classes.
>>
>>
>> Can't we replace the framework modules’ type with a subclass of both
>> “module” and “type”?
>
> FWIW, I already tried that (types.ModuleType is itself a perfectly
> subclassable class!) in the case where the framework had the form
>  and that worked fine. I did not pursue this for the case
> where the framework name if "KF5.", i.e. the case where
> the shared namespace comes into the picture because of these
> statements on 
> https://packaging.python.org/guides/packaging-namespace-packages/
> under the native Python3-only case:
>
> - "It is extremely important that every distribution that uses the
> namespace package omits the __init__.py or uses a pkgutil-style
> __init__.py. If any distribution does not, it will cause the namespace
> logic to fail and the other sub-packages will not be importable."
>
> and under the Python2/3 compatible pkg-util case:
>
> - "The __init__.py file for the namespace package needs to contain
> only the following:
>
> __path__ = __import__('pkgutil').extend_path(__path__, __name__)
>
> Every distribution that uses the namespace package must include an
> identical __init__.py. If any distribution does not, it will cause the
> namespace logic to fail and the other sub-packages will not be
> importable. Any additional code in __init__.py will be inaccessible."
>
> Now, none of that may be a limiting factor in the plan you seem to be
> discussing, but it was part of what made me think "here be dragons"...
>
> Shaheed
>
>> Thank you for explaining,
>> Best regards, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Wim, Philipp,

On 4 November 2017 at 16:45, Philipp A.  wrote:
> Hi Wim!
>
>> So now I have a (C++) namespace 'A' that bears no relationship to anything
>> to do with the file system or any type of Python packaging: it exists only
>> in memory for the duration of the python session.
>
>
> Yeah, cool, so we just use a path hook and are ready to go right?
>
> https://www.python.org/dev/peps/pep-0302/#specification-part-2-registering-hooks
>
> Every framework gets its own path hook that handles imports from that
> framework.
>
> Frameworks with shared namespaces will just handle their own specific paths,
> right?
>
>> The renaming does not actually scare me that much from a performance point
>> of view: already, C++ has typedefs for classes and aliases for namespaces.
>> There is always the problem of "leakage" as mentioned above, where the
>> end-user sees both at some point, but internally, aliasing will work fine:
>> it's just another reference to the same object.
>
>
> Cool, so this might be possible after all! Certainly not as a top priority
> before getting things to work, but still!
>
>> It is to attach a meta class for a __getattr__, the use of properties, and
>> to allow pickling. The module type does not support meta classes.
>
>
> Can't we replace the framework modules’ type with a subclass of both
> “module” and “type”?

FWIW, I already tried that (types.ModuleType is itself a perfectly
subclassable class!) in the case where the framework had the form
 and that worked fine. I did not pursue this for the case
where the framework name if "KF5.", i.e. the case where
the shared namespace comes into the picture because of these
statements on https://packaging.python.org/guides/packaging-namespace-packages/
under the native Python3-only case:

- "It is extremely important that every distribution that uses the
namespace package omits the __init__.py or uses a pkgutil-style
__init__.py. If any distribution does not, it will cause the namespace
logic to fail and the other sub-packages will not be importable."

and under the Python2/3 compatible pkg-util case:

- "The __init__.py file for the namespace package needs to contain
only the following:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

Every distribution that uses the namespace package must include an
identical __init__.py. If any distribution does not, it will cause the
namespace logic to fail and the other sub-packages will not be
importable. Any additional code in __init__.py will be inaccessible."

Now, none of that may be a limiting factor in the plan you seem to be
discussing, but it was part of what made me think "here be dragons"...

Shaheed

> Thank you for explaining,
> Best regards, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Hi Philipp,

On 3 November 2017 at 14:09, Philipp A.  wrote:
> Hi Shaheed,
>
> Shaheed Haque  schrieb am Fr., 3. Nov. 2017 um 14:16
> Uhr:
>>
>> Philipp,
>>
>> - my overall understanding of this technique is that it may end up
>> being fragile, especially given the difference between P2 and P3.
>
>
> Python 2? I’m sure we shouldn’t include into our decision making an obsolete
> language that has a final (yes, really this time!) expiration date 2 years
> in the future. 2020 is the end of the line, I certainly don’t bother anymore
> to write a single line accomodating to it, and would suggest you do the same
> for a new project.

The key issue for me has only ever been to get to usable bindings for
Kate, via the necessary stepping stone of KF5. My observation is that
*nobody* is likely to help with that problem: the framework owners did
nothing obvious to either keep PyKDE4 going (out of tree) or to help
Steve with my earlier SIP based efforts (in tree).

In that context, this is a low level issue I'm not especially excited by.

>> - using it further down might not work as expected especially if there
>> are "accidental" collisions in the directory/namespace names.
>> - it is also not clear if the technique can be used multiple times.
>
>
> We should check if this is the case. Who’s a Python guru who knows the ins
> and outs of the module system?

To be clear, I'm not that guru.

Having said that, the comment about Python2 is a missing the the key
point that the separated namespace stuff relies on what seems to me to
be an intrinsically fragile name-of-directory based scheme. Arguably,
the P3 version is even more fragile in this respect.

Worse, the (limited) commentary on the web suggests that the core
Python devs not only got this "wrong" in Python2, there is an
awareness that the Python3 solution has its own issues. That's enough
to scare me away from building a deep dependency on it!

My usage of the for the KF5 namespace level is based on the low
probability of a collision on that name, and the only reason I
mentioned Python2 is that I chose the scheme that supports both P2 and
P3. If you are right, and P2 is never an issue, it should be fine. I
do however prefer to maintain compatibility until the burden of doing
so presents an insurmountable issue.

>> - cppyy gives us classes. (Actually, in conversation with Wim, CC'd
>> here, it turns out that the choice of using classes is not arbitrary,
>> but we were pondering simple modules at that point, for other
>> reasons).
>
>
> I’d be interested in why. Usually using classes as namespaces is only done
> for reasons of cuteness (callable namespaces, namespaces usable as context
> managers, …) or so.

Feel free to discuss with Wim (it has something to do with
serialisation of classes and there are some interesting grottos to do
with __module__ naming to consider too). I don't propose to follow
this up. FWIW, this came up because of some substantive issues to do
with __module__, and the approach I now have in hand, based on
discussions with Wim, will be largely decoupled from this topic, and
so the way is open for anybody who does care to take this forward.

> Even in this case, it’s possible to replace the module’s class with a module
> subclass that has the necessary capabilities (modules are objects that have
> a class, too)

Yes, I'm well aware of some of those possibilities as I've tried
several of them :-).

>> Bottom line: I'm not keen on using Python namespace modules here for
>> the reasons listed.
>
>
> I’m not entirely convinced. We should only do a final decision once we know
> if either your suspicions of multiple levels not working turn out to be
> true, or the reason for using classes is important.

There is no need for a "final decision" from me. I would suggest that
the first question for anybody that cares is to assess the scope of
the issue. Unfortunately, i have other more fundamental issues to fry.

>> That's likey to be a bad idea because of the potential impact on arbitrary
>> round trips between C++ and Python (remember that everything is based on
>> named-based lookups). In addition, we already have problems like "gpgme++",
>> and the use of capitalisation to separate legacy and forwarding headers in
>> KDE: further case-based confusion is the last thing that is needed!
>
>
> I see, makes sense. I guess the allcaps example here is not very common
> anyway, and most namespaces will be UpperCamelCase like Qt’s, right?

Ack. So far (and based on my experiences with the SIP work), I started
with something very simple (KDCRAW), and moved onto one of the worst
cases (Akonadi). I won't be in a position to gain a wider perspective
until these two are working solidly.

Best regards, Shaheed

>> Thanks for the review/remarks, Shaheed
>
>
> NP!
> Best, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Philipp,

On 3 November 2017 at 12:52, Philipp A.  wrote:
> Hi Shaheed,
>
> Thank you so much for all your work!
>
>> a framework-by-framework integration of the binding generation logic (as
>> previously pioneered by Steve) probably cannot work in general because there
>> are cases where multiple frameworks contribute to to the same C++ namespace
>> […]
>>
>> The problem is that the Python implementation of these namespaces is a
>> class, and so treating these frameworks […] as separate would result in
>> multiple colliding Python class definitions.
>
>
> Am I missing something? Namespaces should be Python modules, not classes. If
> we can do represent them this way, the problem is solveable:
> https://packaging.python.org/guides/packaging-namespace-packages/

First, I am using this technique at the KF5 level (the pkg-util
version). My concerns about using it to address the namespace problem
are that:

- my overall understanding of this technique is that it may end up
being fragile, especially given the difference between P2 and P3.
- using it further down might not work as expected especially if there
are "accidental" collisions in the directory/namespace names.
- it is also not clear if the technique can be used multiple times.
- cppyy gives us classes. (Actually, in conversation with Wim, CC'd
here, it turns out that the choice of using classes is not arbitrary,
but we were pondering simple modules at that point, for other
reasons).

Bottom line: I'm not keen on using Python namespace modules here for
the reasons listed.

>> The original TODOs and bugs have been resolved, and there is the
>> beginnings of support for packaging frameworks under a Python namespace as
>> in "KF5.KDCRAW".
>
>
> Once they’re modules, we should probably respect that Python modules are by
> convention lowercase. It would be best if we named them kf5.kdcraw and so
> on.

That's likey to be a bad idea because of the potential impact on
arbitrary round trips between C++ and Python (remember that everything
is based on named-based lookups). In addition, we already have
problems like "gpgme++", and the use of capitalisation to separate
legacy and forwarding headers in KDE: further case-based confusion is
the last thing that is needed!

Thanks for the review/remarks, Shaheed

> Thank you again,
> Best, Philipp

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Albert,

On 2 November 2017 at 21:43, Albert Astals Cid  wrote:
> El dijous, 2 de novembre de 2017, a les 18:22:38 CET, Shaheed Haque va
> escriure:
>> A progress update...
>>
>> On 24 October 2017 at 13:05, Shaheed Haque  wrote:
>> > Hi all,
>> >
>> > I have a preliminary version of the Cppyy bindings generator CMake
>> >
>> > support available here:
>> > https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-ex
>> > perimental-version-of-a/diff>
>> > There are some TODOs yet to be addressed,
>>
>> The original TODOs and bugs have been resolved, and there is the
>> beginnings of support for packaging frameworks under a Python
>> namespace as in "KF5.KDCRAW". Also, as a significant datapoint, I'm
>> close [1] to being able to generate a *complete* set of bindings for
>> all of Akonadi driven from CMake with just 2-3 lines of custom logic.
>> This contrasts with the 549 SLOC of customisation needed to produce a
>> substantially slash-and-burned subset of Akonadi [2] with the
>> SIP-based approach.
>>
>> > but I would appreciate
>> > feedback on how easy it would be to integrate this with KDE's
>> > buildsystem, especially for the frameworks. I'm a CMake noob, but the
>> > basic idea I have is that the packager of some_framework might do
>> > something like this:
>> >
>> > find_package(cppyy)
>> > CPPYY_ADD_BINDINGS(
>> >
>> > ...
>> > LINK_LIBRARIES some_framework_LIBRARIES
>> > H_DIR some_framework_INCLUDE_DIRS
>> > H_FILES )
>>
>> In the course of working through the "KF5" namespace implementation,
>> it has become apparent to me that a framework-by-framework integration
>> of the binding generation logic (as previously pioneered by Steve)
>> probably cannot work in general because there are cases where multiple
>> frameworks contribute to to the same C++ namespace, for example:
>>
>> $ grep -r '^namespace Akonadi' /usr/include/KF5/Akonadi*
>> /usr/include/KF5/AkonadiAgentBase/resourcesettings.h:namespace Akonadi
>> ..
>> /usr/include/KF5/AkonadiCore/agentfilterproxymodel.h:namespace Akonadi
>> ..
>> /usr/include/KF5/AkonadiSearch/Debug/akonadisearchdebugsearchpathcombobox.h:
>> namespace Akonadi
>> ..
>> /usr/include/KF5/AkonadiWidgets/agenttypedialog.h:namespace Akonadi
>> ..
>> /usr/include/KF5/AkonadiXml/xmldocument.h:namespace Akonadi
>> ..
>>
>> The problem is that the Python implementation of these namespaces is a
>> class, and so treating these frameworks (let's not quibble over
>> whether KF5Akonadi* are truly KF5 frameworks, the point is more
>> general) as separate would result in multiple colliding Python class
>> definitions. The only solution I can see would be to bundle all of
>> KF5Akonadi* into a single set of bindings, e.g. KF5.Akonadi, and
>> AFAICS, this can only be done out of tree from the individual
>> frameworks, say in kde-bindings.git [3].
>
> Sincerely if you go with out ot tree bindings they will always be broken
> because people working on the frameworks won't even see them, if the problem
> is that frameworks have to be self-contained, this seems like a good general
> rule to me and maybe what we should do is fix the libs?

Thanks for the input, but...

Firstly, we have existence proofs in the form of PyQt and the PyKDE4
bindings that (even with the SIP support tax), that out of tree
bindings are possible, and I am hopeful that things will be much
easier with cppyy.

Also, I wonder how easy that will be given the need to keep source
compatibility? I've hardly done a comprehensive survey, but if the
Akonadi example is anything to go by, I don't think there is a short
term prospect of seeing a solution to this problem. If there is
developer effort available from framework owners, I'd much rather see
that put to productive use (there are plenty of point issues like
#includes broken-from-my-POV and the like).

Finally, not to put to fine a point on it, I've been working on this
for the best part of two years, and I am very keen to get to something
workable. (Not least because I hope to get a job at some point...and
then intensive effort from me at least might be much trickier to offer
up).

Cheers, Shaheed

> Cheers,
>   Albert
>
>>
>> The work to date attempts to maintain a clean separation such that all
>> C++ builds are done from CMake, and all Python builds are done using
>> setuptools/pip.
>>
>> Apart from working through bugs [1], the remaining work items I can

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

A progress update...

On 24 October 2017 at 13:05, Shaheed Haque  wrote:
> Hi all,
>
> I have a preliminary version of the Cppyy bindings generator CMake
> support available here:
>
> 
> https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-experimental-version-of-a/diff
>
> There are some TODOs yet to be addressed,

The original TODOs and bugs have been resolved, and there is the
beginnings of support for packaging frameworks under a Python
namespace as in "KF5.KDCRAW". Also, as a significant datapoint, I'm
close [1] to being able to generate a *complete* set of bindings for
all of Akonadi driven from CMake with just 2-3 lines of custom logic.
This contrasts with the 549 SLOC of customisation needed to produce a
substantially slash-and-burned subset of Akonadi [2] with the
SIP-based approach.

> but I would appreciate
> feedback on how easy it would be to integrate this with KDE's
> buildsystem, especially for the frameworks. I'm a CMake noob, but the
> basic idea I have is that the packager of some_framework might do
> something like this:
>
> find_package(cppyy)
> CPPYY_ADD_BINDINGS(
> ...
> LINK_LIBRARIES some_framework_LIBRARIES
> H_DIR some_framework_INCLUDE_DIRS
> H_FILES )

In the course of working through the "KF5" namespace implementation,
it has become apparent to me that a framework-by-framework integration
of the binding generation logic (as previously pioneered by Steve)
probably cannot work in general because there are cases where multiple
frameworks contribute to to the same C++ namespace, for example:

$ grep -r '^namespace Akonadi' /usr/include/KF5/Akonadi*
/usr/include/KF5/AkonadiAgentBase/resourcesettings.h:namespace Akonadi
...
/usr/include/KF5/AkonadiCore/agentfilterproxymodel.h:namespace Akonadi
...
/usr/include/KF5/AkonadiSearch/Debug/akonadisearchdebugsearchpathcombobox.h:namespace
Akonadi
...
/usr/include/KF5/AkonadiWidgets/agenttypedialog.h:namespace Akonadi
...
/usr/include/KF5/AkonadiXml/xmldocument.h:namespace Akonadi
...

The problem is that the Python implementation of these namespaces is a
class, and so treating these frameworks (let's not quibble over
whether KF5Akonadi* are truly KF5 frameworks, the point is more
general) as separate would result in multiple colliding Python class
definitions. The only solution I can see would be to bundle all of
KF5Akonadi* into a single set of bindings, e.g. KF5.Akonadi, and
AFAICS, this can only be done out of tree from the individual
frameworks, say in kde-bindings.git [3].

The work to date attempts to maintain a clean separation such that all
C++ builds are done from CMake, and all Python builds are done using
setuptools/pip.

Apart from working through bugs [1], the remaining work items I can
think of are, as before:

>> - Need to look into the exact usage of Qt-specifics: signals/slots and
>> interoperability with SIP-based PyQt
>> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glue-ing-applications,
>> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)
>>
>> - Need to figure out how any customisations which *are* required
>> should be handled.

plus:

- I'm working with upstream on how to support discovery (e.g. via
autocompletion in Python3). There is some POC-level hackery in git as
above, but there is work ongoing with upstream to find a robust
solution.

- Flesh out how to make one set of bindings depend on another (e.g.
tier 2 framework bindings might depend on tier 1 bindings, or maybe it
is better to avoid PyQt and just produce cppyy-based bindings for Qt
and depend on those).

As always, comments/ideas/suggestions are welcome.

Thanks, Shaheed

[1] There is a bug with namespaced externs being worked on with upstream.

[2] 
https://github.com/ShaheedHaque/extra-cmake-modules/blob/shaheed_master/find-modules/module_generation/PyKF5/Akonadi.py

[3] I attach an example CMakeLists.txt which shows now this can be
driven from CMake for the case of KF5Akonadi*...the implementation is
intended to serve as the basis for a generic solution usable across
KF5 at least.

> On 16 October 2017 at 16:16, Shaheed Haque  wrote:
>> As promised, here is an interim update on the investigation into the
>> use of cppyy-based bindings for KF5 (and more...) instead of SIP-based
>> bindings.
>>
>> The first thing is that the underlying technology of cppyy,
>> cling/ROOT, has been under development at CERN for quite a while. It
>> directly reads regular C++ files (there is no intermediate format like
>> SIP).
>>
>> The bindings it generates from Python to C++ seem far more complete
>> and automatic than SIP. For example:
>>
>> - Template instantiation is done on the fly as needed.
>>
>> - Since it uses C++ directly, there is none the 

Re: Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

Hi all,

I have a preliminary version of the Cppyy bindings generator CMake
support available here:


https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-experimental-version-of-a/diff

There are some TODOs yet to be addressed, but I would appreciate
feedback on how easy it would be to integrate this with KDE's
buildsystem, especially for the frameworks. I'm a CMake noob, but the
basic idea I have is that the packager of some_framework might do
something like this:

find_package(cppyy)
CPPYY_ADD_BINDINGS(
...
LINK_LIBRARIES some_framework_LIBRARIES
H_DIR some_framework_INCLUDE_DIRS
H_FILES )

Thanks, Shaheed

On 16 October 2017 at 16:16, Shaheed Haque  wrote:
> As promised, here is an interim update on the investigation into the
> use of cppyy-based bindings for KF5 (and more...) instead of SIP-based
> bindings.
>
> The first thing is that the underlying technology of cppyy,
> cling/ROOT, has been under development at CERN for quite a while. It
> directly reads regular C++ files (there is no intermediate format like
> SIP).
>
> The bindings it generates from Python to C++ seem far more complete
> and automatic than SIP. For example:
>
> - Template instantiation is done on the fly as needed.
>
> - Since it uses C++ directly, there is none the effort required to
> decollide SIP's notion of forward and duplicate declarations.
>
> - Function overloads are cleanly handled, as are most (all?) operators.
>
> The net result is that so far, there is about 3 days work and
> approximately [1] no "customisation" required in order to get to
> roughly where the SIP based bindings were after 18 months. Without the
> need for customisations on a mass scale, I suspect that we might get
> away without anything like the tooling I had to create to SIP, and
> just integrate with CMake
> (https://root.cern.ch/how/integrate-root-my-project-cmake).
>
> This all sounds pretty amazing, right? Well, there are a few caveats...
>
> - The packaging is pretty new, and is evolving pretty rapidly. We
> are/will be an early adopter (https://bitbucket.org/wlav/cppyy/ and
> https://bitbucket.org/wlav/cppyy-backend). Packaging is via PyPI and
> pip/pip3.
>
> - There is a lot of documentation around for the system overall, but
> frankly, it has  been/still is a struggle to understand how the
> different parts relate to each other as some parts are obsolete, and
> other parts have yet to be built out to their intended end-state.
>
> - There are bugs [1], [2]. The upstream dev has been very responsive,
> and the overall quality approach looks sound. IIUC, the vast bulk of
> the code seems to be in daily use at CERN (and is based on LLVM).
>
> - Need to look into the exact usage of Qt-specifics: signals/slots and
> interoperability with SIP-based PyQt
> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glue-ing-applications,
> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)
>
> - Need to figure out how any customisations which *are* required
> should be handled.
>
> These seem like perfectly tractable issues, and so I conclude that
> using cppyy is definitely the way to go. With luck and a bit of
> effort, I am hopeful that we can get to some REALLY
> easy-to-develop-and-maintain bindings.
>
> [1] There is a bug with the binding producing stuff for private definitions.
>
> [2] There is a bug with missing globals.
>
>
[snip]

Python bindings using cppyy (was: An update on Python bindings)

[Shaheed Haque]

As promised, here is an interim update on the investigation into the
use of cppyy-based bindings for KF5 (and more...) instead of SIP-based
bindings.

The first thing is that the underlying technology of cppyy,
cling/ROOT, has been under development at CERN for quite a while. It
directly reads regular C++ files (there is no intermediate format like
SIP).

The bindings it generates from Python to C++ seem far more complete
and automatic than SIP. For example:

- Template instantiation is done on the fly as needed.

- Since it uses C++ directly, there is none the effort required to
decollide SIP's notion of forward and duplicate declarations.

- Function overloads are cleanly handled, as are most (all?) operators.

The net result is that so far, there is about 3 days work and
approximately [1] no "customisation" required in order to get to
roughly where the SIP based bindings were after 18 months. Without the
need for customisations on a mass scale, I suspect that we might get
away without anything like the tooling I had to create to SIP, and
just integrate with CMake
(https://root.cern.ch/how/integrate-root-my-project-cmake).

This all sounds pretty amazing, right? Well, there are a few caveats...

- The packaging is pretty new, and is evolving pretty rapidly. We
are/will be an early adopter (https://bitbucket.org/wlav/cppyy/ and
https://bitbucket.org/wlav/cppyy-backend). Packaging is via PyPI and
pip/pip3.

- There is a lot of documentation around for the system overall, but
frankly, it has  been/still is a struggle to understand how the
different parts relate to each other as some parts are obsolete, and
other parts have yet to be built out to their intended end-state.

- There are bugs [1], [2]. The upstream dev has been very responsive,
and the overall quality approach looks sound. IIUC, the vast bulk of
the code seems to be in daily use at CERN (and is based on LLVM).

- Need to look into the exact usage of Qt-specifics: signals/slots and
interoperability with SIP-based PyQt
(https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glue-ing-applications,
https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html)

- Need to figure out how any customisations which *are* required
should be handled.

These seem like perfectly tractable issues, and so I conclude that
using cppyy is definitely the way to go. With luck and a bit of
effort, I am hopeful that we can get to some REALLY
easy-to-develop-and-maintain bindings.

[1] There is a bug with the binding producing stuff for private definitions.

[2] There is a bug with missing globals.


On 7 September 2017 at 10:23, Shaheed Haque  wrote:
> [ Copying in kde-frameworks-devel as suggested ]
>
> On 5 September 2017 at 22:12, Shaheed Haque  wrote:
>> A lot of progress has been made in the last 18 months or so:
>>
>> THE TOOLING
>> ===
>>
>> We have:
>>
>> - A pretty powerful KDE-independent automatic binding generation capability.
>>
>> - Supplemented by a powerful/fine-grained manual override "rule" capability.
>>
>> - Comprehensive (rule-based) support for the main Qt templates (QList,
>> QVector, QHash, QSet and QFlags), some selected std:: and boost::
>> templates support, multi-dimensional arrays and lots more.
>>
>> - CMake-based portability (the frontend is solid enough to "read" KDE,
>> only the final C++ compilation remains to be moved to either CMake or
>> a Python-centric packaging form [2]).
>>
>> The code can be seen here:
>>
>> https://github.com/ShaheedHaque/extra-cmake-modules/tree/shaheed_master/find-modules/module_generation
>>
>> The KDE framework bindings
>> =
>>
>> For 120 out of 167 KDE 5 frameworks (I'm using the term loosely, see
>> [6]), I have created all the manual override "rules" to get the
>> bindings through the SIP intermediate tooling and the g++ compiler.
>>
>> These rules mentioned are in the PyKF5 subdirectory, and it should be
>> made clear that they are in proof-of-concept form, and some hopefully
>> modest work would be needed (per framework, for some frameworks) to
>> get actual useful bindings.
>>
>> Also, note that the bindings have not actually been run [1] because
>> the focus till now has been to ensure the feasibility of the approach.
>> The whole point of automation is of course that any fixes needed to
>> get them going would be easy to apply (think any fixes needed for
>> templates, arrays, unions, exceptions, threading etc etc).
>>
>> THE GOOD NEWS
>> ==
>>
>> The good news is that the approach appears to have lived up to my hope
>> in that the amount of rule code for PyKF5 needed appears to be
>> minimal:
>>
>> -

D7736: Not-to-be-merged review of Python bindings generator

[Shaheed Haque]

shaheed added a comment.


  In https://phabricator.kde.org/D7736#154948, @jasontibbitts wrote:
  
  > I hate to add a ping without any useful review, but I'm quite interested in 
this effort as I have a pykde4-based application which I would really like to 
get ported to the modern frameworks.  Current distro packages of pykde4 are 
increasingly broken, and coverage of the existing bindings isn't sufficient to 
get the stuff I'm using (which isn't much, really) running.
  >
  > Is there anything that someone like me can do to help with this?
  
  
  Well, I only really got into this because I wanted to revive the Python 
support I added to Kate!
  
  In all seriousness, I strongly doubt that a SIP-based solution for PyKF5 can 
ever be sustained without a dedicated maintainer, and my starting premise for 
my work was this was not a viable prospect. As hinted on the mailing list I've 
now spent the last 2 days with cppyy, and once I have made a bit more progress 
will post an update on that.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D7736

To: shaheed, lbeltrame
Cc: jasontibbitts, #frameworks, #build_system

D7736: Not-to-be-merged review of Python bindings generator

[Shaheed Haque]

shaheed added a comment.


  My suggestion would be to focus any review efforts in this order:
  
  1. files in the top level directory.
  2. possibly files the ./templates subdirectory.
  3. __init__.py and CMakelists.txt in the ./PyKF5 subdirectory. Ignore all 
other files therein.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D7736

To: shaheed, lbeltrame
Cc: #frameworks, #build_system

D7736: Not-to-be-merged review of Python bindings generator

[Shaheed Haque]

shaheed added inline comments.

INLINE COMMENTS

> lbeltrame wrote in CMakeLists.txt:49
> Don't use `find_package(FOO REQUIRED)`. Use `find_package(FOO)` then 
> `set_package_properties(FOO TYPE REQUIRED...`. There are many examples in KDE 
> git you can use. You will need to include FeatureSummary for this to work 
> properly.
> 
> Why do I say this? Because now CMake will fail *immediately* at the first 
> error. Otherwise, it will collect and print all missing packages after 
> configuration. It is much easier on the packagers.

Good point. I even helped update the Techbase docs for this, but then forgot to 
implement it :-).

> lbeltrame wrote in CMakeLists.txt:53
> Any specific reason for this SIP version?

Yes, this contains needed patches. Though it turns out that even this will not 
be sufficient because I submitted further patches for more issues. 
Unfortunately, upstream decided to implement some of those patches using 
Python3 only and I'm still stuck on Python2 because of the CLang dependency.

CLang Python3 support was shipped yesterday.

When it is available, a newer version of SIP that contains the support for 
scoped enums as per 
https://www.riverbankcomputing.com/pipermail/pyqt/2017-August/039494.html will 
need to be integrated (the core logic here may or may not need to change 
slightly).

> lbeltrame wrote in FindClang.cmake:52
> This won't work on all distros. Can you either:
> 
> a. Take inspiration from KDevelop's FindClang (which works already nicely)
> b. Use upstream Clang module to find it and its version?
> 
> The same comment goes for FindLibClang.

I'm well aware of the issue. Unfortunately, upstream CLang was not available to 
me, and the KDevelop one is not sufficiently complete to do what I need.

Hence this would need addressing in due course.

> lbeltrame wrote in HOWTO:12
> Before distro-specific instructions, put a list of all the dependencies with 
> their upstream names. This ensures that packagers of non Ubuntu/Debian 
> distros can also look them up.

Yup. I'd likely need a bit of help at that point, but I would hope that using 
KDE's CI system would be a good starting point.

Another issue that would need to be addressed is the fact that the newer 
versions of SIP will only support Python3 as noted above.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D7736

To: shaheed, lbeltrame
Cc: #frameworks, #build_system

D7736: Not-to-be-merged review of Python bindings generator

[Shaheed Haque]

shaheed added a comment.


  My comments here are phrased as if this SIP-based approach was the solution 
eventually adopted (cppyy might be different). With that said...
  
  This...
  
  > (this thing is huge).
  
  and this...
  
  > One question: would it be possible to have the bindings per-framework, 
rather than a single, long list? This is also what made PyKDE4 unwieldy. IOW, 
each Framework should ship their (optional) bindings.
  
  are related. Though the diff is huge, much of that is due to the 98 files in 
the PyKF5 directory (as opposed to the 10 that actually contain the crown 
jewels, plus the 5 template-related files). Let me explain...
  
  Those 98 files are effectively two things:
  
  1. The basis of the regression test suite for the tooling. In this sense, 
these rules (more or less) do just enough work to get beyond the 
fail-on-first-error semantics of the SIP compiler, and so are needed to allow 
enough of the syntax of the KF5 suite in the long list to be processed for me 
to feel comfortable that the tooling is "complete".
  2. The starting point for the real per-framework rules that would be used in 
real life. In real life, I would expect the bindings to be packaged 
per-framework via ECM as you say, with each framework owner starting from the 
set of rules here.
  
  > We can put the tooling in ECM so that everything is in place for all the 
Frameworks. (This is how the current approach works):
  
  The tooling in ECM would likely also host the shared rules (including the 5 
files worth of template support) much as the current ECM does.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D7736

To: shaheed, lbeltrame
Cc: #frameworks, #build_system

D7736: Not-to-be-merged review of Python bindings generator

[Shaheed Haque]

shaheed edited the summary of this revision.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D7736

To: shaheed, lbeltrame
Cc: #frameworks, #build_system

D7736: Not-to-be-merged review of Python bindings generator

[Shaheed Haque]

shaheed retitled this revision from "Not-to-be-merged review fo Python bindings 
generator" to "Not-to-be-merged review of Python bindings generator".

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D7736

To: shaheed, lbeltrame
Cc: #frameworks, #build_system

D7736: Not-to-be-merged review fo Python bindings generator

[Shaheed Haque]

shaheed edited the summary of this revision.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D7736

To: shaheed, lbeltrame
Cc: #frameworks, #build_system

D7736: Not-to-be-merged review fo Python bindings generator

[Shaheed Haque]

shaheed created this revision.
Restricted Application added projects: Frameworks, Build System.
Restricted Application added subscribers: Build System, Frameworks.

REVISION SUMMARY
  As per https://marc.info/?l=kde-core-devel&m=150464598710128&w=2, this is a 
snapshot of the current state of the SIP-based Python binding generator 
intended for KF5, KDE, and more generally.

TEST PLAN
  n/a

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D7736

AFFECTED FILES
  find-modules/module_generation/.gitignore
  find-modules/module_generation/CMakeLists.txt
  find-modules/module_generation/COPYING.LIB
  find-modules/module_generation/FindClang.cmake
  find-modules/module_generation/FindLibClang.cmake
  find-modules/module_generation/FindSIP.cmake
  find-modules/module_generation/HOWTO
  find-modules/module_generation/HOWTO.WRITE_RULES
  find-modules/module_generation/PyKF5/.gitignore
  find-modules/module_generation/PyKF5/Akonadi.py
  find-modules/module_generation/PyKF5/Attica.py
  find-modules/module_generation/PyKF5/Baloo.py
  find-modules/module_generation/PyKF5/BluezQt.py
  find-modules/module_generation/PyKF5/CMakeLists.txt
  find-modules/module_generation/PyKF5/CalendarSupport.py
  find-modules/module_generation/PyKF5/FindPyQt.cmake
  find-modules/module_generation/PyKF5/FollowupReminder.py
  find-modules/module_generation/PyKF5/IncidenceEditor.py
  find-modules/module_generation/PyKF5/KActivitiesStats.py
  find-modules/module_generation/PyKF5/KAlarmCal.py
  find-modules/module_generation/PyKF5/KArchive.py
  find-modules/module_generation/PyKF5/KAuth.py
  find-modules/module_generation/PyKF5/KBlog.py
  find-modules/module_generation/PyKF5/KBookmarks.py
  find-modules/module_generation/PyKF5/KCMUtils.py
  find-modules/module_generation/PyKF5/KCalCore.py
  find-modules/module_generation/PyKF5/KCalUtils.py
  find-modules/module_generation/PyKF5/KCodecs.py
  find-modules/module_generation/PyKF5/KCompletion.py
  find-modules/module_generation/PyKF5/KConfigCore.py
  find-modules/module_generation/PyKF5/KConfigGui.py
  find-modules/module_generation/PyKF5/KConfigWidgets.py
  find-modules/module_generation/PyKF5/KContacts.py
  find-modules/module_generation/PyKF5/KCoreAddons.py
  find-modules/module_generation/PyKF5/KCrash.py
  find-modules/module_generation/PyKF5/KDEWebKit.py
  find-modules/module_generation/PyKF5/KDGantt2.py
  find-modules/module_generation/PyKF5/KDeclarative.py
  find-modules/module_generation/PyKF5/KEmoticons.py
  find-modules/module_generation/PyKF5/KExiv2.py
  find-modules/module_generation/PyKF5/KF5KDEGames.py
  find-modules/module_generation/PyKF5/KFace.py
  find-modules/module_generation/PyKF5/KGAPI.py
  find-modules/module_generation/PyKF5/KGeoMap.py
  find-modules/module_generation/PyKF5/KGlobalAccel.py
  find-modules/module_generation/PyKF5/KHtml.py
  find-modules/module_generation/PyKF5/KI18n.py
  find-modules/module_generation/PyKF5/KIMAP.py
  find-modules/module_generation/PyKF5/KIO.py
  find-modules/module_generation/PyKF5/KIOFileWidgets.py
  find-modules/module_generation/PyKF5/KIOWidgets.py
  find-modules/module_generation/PyKF5/KIPI.py
  find-modules/module_generation/PyKF5/KIconThemes.py
  find-modules/module_generation/PyKF5/KIdentityManagement.py
  find-modules/module_generation/PyKF5/KIdleTime.py
  find-modules/module_generation/PyKF5/KItemModels.py
  find-modules/module_generation/PyKF5/KItemViews.py
  find-modules/module_generation/PyKF5/KJS.py
  find-modules/module_generation/PyKF5/KJobWidgets.py
  find-modules/module_generation/PyKF5/KLDAP.py
  find-modules/module_generation/PyKF5/KMbox.py
  find-modules/module_generation/PyKF5/KMime.py
  find-modules/module_generation/PyKF5/KNewStuff3.py
  find-modules/module_generation/PyKF5/KNotifications.py
  find-modules/module_generation/PyKF5/KNotifyConfig.py
  find-modules/module_generation/PyKF5/KPIMTextEdit.py
  find-modules/module_generation/PyKF5/KPackage.py
  find-modules/module_generation/PyKF5/KParts.py
  find-modules/module_generation/PyKF5/KPlotting.py
  find-modules/module_generation/PyKF5/KPty.py
  find-modules/module_generation/PyKF5/KRunner.py
  find-modules/module_generation/PyKF5/KScreen.py
  find-modules/module_generation/PyKF5/KService.py
  find-modules/module_generation/PyKF5/KSieveUi.py
  find-modules/module_generation/PyKF5/KStyle.py
  find-modules/module_generation/PyKF5/KTNEF.py
  find-modules/module_generation/PyKF5/KTextEditor.py
  find-modules/module_generation/PyKF5/KTextWidgets.py
  find-modules/module_generation/PyKF5/KUnitConversion.py
  find-modules/module_generation/PyKF5/KWidgetsAddons.py
  find-modules/module_generation/PyKF5/KWindowSystem.py
  find-modules/module_generation/PyKF5/KXmlGui.py
  find-modules/module_generation/PyKF5/Kaddressbook.py
  find-modules/module_generation/PyKF5/KdepimDBusInterfaces.py
  find-modules/module_generation/PyKF5/KontactInterface.py
  find-modules/module_generation/PyKF5/Kross.py
  find-modules/module_generation/PyKF5/Libkdepim.

Re: An update on Python bindings (Re: A new attempt on PyKDE5 binding generation)

[Shaheed Haque]

[ Copying in kde-frameworks-devel as suggested ]

On 5 September 2017 at 22:12, Shaheed Haque  wrote:
> A lot of progress has been made in the last 18 months or so:
>
> THE TOOLING
> ===
>
> We have:
>
> - A pretty powerful KDE-independent automatic binding generation capability.
>
> - Supplemented by a powerful/fine-grained manual override "rule" capability.
>
> - Comprehensive (rule-based) support for the main Qt templates (QList,
> QVector, QHash, QSet and QFlags), some selected std:: and boost::
> templates support, multi-dimensional arrays and lots more.
>
> - CMake-based portability (the frontend is solid enough to "read" KDE,
> only the final C++ compilation remains to be moved to either CMake or
> a Python-centric packaging form [2]).
>
> The code can be seen here:
>
> https://github.com/ShaheedHaque/extra-cmake-modules/tree/shaheed_master/find-modules/module_generation
>
> The KDE framework bindings
> =
>
> For 120 out of 167 KDE 5 frameworks (I'm using the term loosely, see
> [6]), I have created all the manual override "rules" to get the
> bindings through the SIP intermediate tooling and the g++ compiler.
>
> These rules mentioned are in the PyKF5 subdirectory, and it should be
> made clear that they are in proof-of-concept form, and some hopefully
> modest work would be needed (per framework, for some frameworks) to
> get actual useful bindings.
>
> Also, note that the bindings have not actually been run [1] because
> the focus till now has been to ensure the feasibility of the approach.
> The whole point of automation is of course that any fixes needed to
> get them going would be easy to apply (think any fixes needed for
> templates, arrays, unions, exceptions, threading etc etc).
>
> THE GOOD NEWS
> ==
>
> The good news is that the approach appears to have lived up to my hope
> in that the amount of rule code for PyKF5 needed appears to be
> minimal:
>
> - A *LOT* of the bindings are completely automatically generated.
>
> - Very few rules are anything more than 1-2 lines of code.
>
> - A *LOT* of the rules are merely invocations of a small set of
> pre-provided rule helpers. There is a HOWTO/FAQ that should provide
> most of the needed idiomatic knowledge to help move things along.
>
> THE BAD NEWS
> =
>
> While SIP provides a huge amount of capability, and my tooling covers
> many of the gaps, there are a bunch of things that are going to be
> hard to deal with. Some of this is on a one off basis [3], others on
> an ongoing basis [4], and yet others simply have no obvious solution
> [5].
>
> (Notice however, that SIP does seemingly successfully handle something
> as complex as Qt).
>
> NEXT STEPS
> ===
>
> Based on THE GOOD NEWS it would be easy to conclude that continuing
> down the current route would result in usable bindings, assuming (say)
> that I would facilitate a couple of hours of effort/consultancy per
> framework with ,
> and consolidate any needed info as Techbase documentation.
>
> However, I'm also concerned that THE BAD NEWS means this will result
> in a set of bindings which are just different enough from the C++ to
> be vaguely irritating, and run the risk of a similar fate as befell
> PyKDE4.
>
> There is a possible alternative way forward:
> https://pypi.python.org/pypi/cppyy. This seems to offer a significant
> step forward by (a) dispensing with a highly "opinionated"
> intermediate layer like SIP and (b) using a "C++ interpreter". (Note,
> IIUC, the resulting bindings will depend on the presence of the Cling
> interpreter to function).
>
> Now, I have no doubt that there will still be issues with things like
> overloaded functions in C++ mapping to a single function in Python,
> but I suspect the issues will be a strict subset of the issues with
> the SIP approach (for example, it looks as though the "native"
> approach to the evolution of C++, templates etc will be a big help).
> AFAIK, there should be no issue in using the resulting bindings with
> PyQt (except that PyQt is tied to CPython, whereas cppyy also supports
> PyPy).
>
> On balance, as despite the work that has gone into the SIP approach, I
> propose to explore the cppyy option.
>
> Comments, thoughts?
>
> Thanks, Shaheed
>
> [1] I'm ignoring the 5-6 bindings which Stephen Kelly's fork of the
> code got going. As far as I know, Stephen has not continued with that
> work.
>
> [2] Stephen's solution to this only addressed the inner layer of the
> tooling. The outer layer, needed for much of the SIP workaround logic,
> was not addre

D5289: Import Find{Clang,LLVM} from KDevelop for Python bindings generation

[Shaheed Haque]

shaheed added a comment.


  I did some negative testing, and from what I can see, 3.8 might well be OK
  for the ECM fork. My version depends on 3.9 (for example, there are some
  new constants defined by clang which I use), but I've no idea if this will
  ever be merged.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D5289

To: heikobecker, #frameworks, #build_system, skelly, kfunk
Cc: rdieter, shaheed, kde-buildsystem, lbeltrame

D5289: Import Find{Clang,LLVM} from KDevelop for Python bindings generation

[Shaheed Haque]

shaheed added subscribers: kde-buildsystem, shaheed.
shaheed added a comment.


  Fwiw, I think that a specific version check may not be needed. The original
  code I wrote, which I assume Steve may have simply carried forward in the
  cmake ecm logic, DID have a version check but only because the python
  bindings were newish circa libclang 3.8.
  
  Afaik, there is no logic which is version-specifc in the code. (Otoh, I
  have not specifically tested for this).

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D5289

To: heikobecker, #frameworks, #build_system, skelly, kfunk
Cc: shaheed, kde-buildsystem, lbeltrame

[Differential] [Commented On] D4510: Python bindings: Restore handling of deprecated constructs.

[Shaheed Haque]

shaheed added inline comments.

INLINE COMMENTS

> skelly wrote in sip_generator.py:172
> It was possible to handle exports without looking for the text EXPORT in the 
> MACRO NAME. Why is deprecated different?

Because the expansion of the attribute in this case contains not a string, but 
a compiler reserved value

  __deprecated__

in the case of clang. I am not aware of a portable way to know what the 
expansion is, so the choice was between a string compare of the clang spelling, 
or the original. The latter seemed somewhat more reliable to me. IIRC, my 
original code did the same for the EXPORT stuff (I know you changed it and to 
be honest, I think that was suboptimal for the same reason, but at least the 
text appears as "hidden" not an opaque reserved value).

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D4510

EMAIL PREFERENCES
  https://phabricator.kde.org/settings/panel/emailpreferences/

To: shaheed, skelly, #build_system, #frameworks
Cc: #frameworks, #build_system

[Differential] [Commented On] D4509: Python bindings: Some comment-only tidyups and PEP-8 fixes.

[Shaheed Haque]

shaheed added inline comments.

INLINE COMMENTS

> skelly wrote in rules_engine.py:494
> Does the docs match the code if you add this here? Why is this in this commit?

Yes. Why not?

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D4509

EMAIL PREFERENCES
  https://phabricator.kde.org/settings/panel/emailpreferences/

To: shaheed, #build_system, #frameworks, skelly
Cc: #frameworks, #build_system

[Differential] [Commented On] D4509: Python bindings: Some comment-only tidyups and PEP-8 fixes.

[Shaheed Haque]

shaheed added a comment.


  I think the scope of the changes in this commit are perfectly reasonable. I 
don't think it is unreasonable to group them either.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D4509

EMAIL PREFERENCES
  https://phabricator.kde.org/settings/panel/emailpreferences/

To: shaheed, #build_system, #frameworks, skelly
Cc: #frameworks, #build_system

[Differential] [Commented On] D4509: Python bindings: Some comment-only tidyups and PEP-8 fixes.

[Shaheed Haque]

shaheed added inline comments.

INLINE COMMENTS

> skelly wrote in rules_engine.py:58
> I don't think this should be here.

I disagree. Having this in the code suppresses the false positives which are 
caused by the "_" used by gettext.

> skelly wrote in rules_engine.py:704
> I don't think the comments are valuable.

I think they are perfectly reasonable.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D4509

EMAIL PREFERENCES
  https://phabricator.kde.org/settings/panel/emailpreferences/

To: shaheed, #build_system, #frameworks, skelly
Cc: #frameworks, #build_system

[Differential] [Commented On] D4509: Python bindings: Some comment-only tidyups and PEP-8 fixes.

[Shaheed Haque]

shaheed added a comment.


  The PEP-8 changes are some blank line changes.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D4509

EMAIL PREFERENCES
  https://phabricator.kde.org/settings/panel/emailpreferences/

To: shaheed, #build_system, #frameworks, skelly
Cc: #frameworks, #build_system

[Differential] [Request, 20 lines] D4510: Python bindings: Restore handling of deprecated constructs.

[Shaheed Haque]

shaheed created this revision.
shaheed set the repository for this revision to R240 Extra CMake Modules.
Restricted Application added projects: Frameworks, Build System.
Restricted Application added subscribers: Build System, Frameworks.

TEST PLAN
  Add to the unit tests and verify the result is marked with the expected 
annotations:
  
$ tail  ./tests/GenerateSipBindings/sip/PyTest/CppLib/cpplib.sip
...
class DeprecatedClass /Deprecated/
{
%TypeHeaderCode
#include 
%End
public:
void deprecatedFn(int bar) /Deprecated/;
};

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D4510

AFFECTED FILES
  find-modules/sip_generator.py
  tests/GenerateSipBindings/cpplib.h

EMAIL PREFERENCES
  https://phabricator.kde.org/settings/panel/emailpreferences/

To: shaheed
Cc: #frameworks, #build_system

[Differential] [Updated] D4509: Python bindings: Some comment-only tidyups and PEP-8 fixes.

[Shaheed Haque]

shaheed retitled this revision from "Some comment-only tidyups and PEP-8 
fixes." to "Python bindings: Some comment-only tidyups and PEP-8 fixes.".

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D4509

EMAIL PREFERENCES
  https://phabricator.kde.org/settings/panel/emailpreferences/

To: shaheed
Cc: #frameworks, #build_system

[Differential] [Request, 15 lines] D4509: Some comment-only tidyups and PEP-8 fixes.

[Shaheed Haque]

shaheed created this revision.
shaheed set the repository for this revision to R240 Extra CMake Modules.
Restricted Application added projects: Frameworks, Build System.
Restricted Application added subscribers: Build System, Frameworks.

TEST PLAN
  No code changes.

REPOSITORY
  R240 Extra CMake Modules

REVISION DETAIL
  https://phabricator.kde.org/D4509

AFFECTED FILES
  find-modules/rules_engine.py

EMAIL PREFERENCES
  https://phabricator.kde.org/settings/panel/emailpreferences/

To: shaheed
Cc: #frameworks, #build_system

Re: Review Request 129760: Fixup handling of KFontUtils::adaptFontSize's flags' default value.

[Shaheed Haque]

I've actually switched all activity to KDE/master (I'm presently on
a5f3a76e14799c68b5e8f74e375baa5f6f6ab4dc), so I'll check again when I've
got your latest updates. BTW, one other thing I noticed was this, if I
"make clean" in kguiaddons, src/sip/PyKF5/KGuiAddons/KGuiAddonsmod.sip is
not removed. It is only ever created when cmake is run.

Is this as expected?

On 18 January 2017 at 21:16, Stephen Kelly  wrote:

> On 01/18/2017 12:41 PM, Shaheed Haque wrote:
> > Hi Steve,
> >
> > I have verified this works on KDE/master, and will close the review.
> >
> > I did see a "funny" in that I had to run CMake twice in a fresh
> > directory before it would run clean. You might want to look into that
> > as it was not clear to me what it CMake is trying to do. Anyway, here
> > is the first unsuccessful run:
>
> Hi Shaheed,
>
> I somehow made my frameworks-bindings repo point to incorrect old sha1s
> for frameworks. I've fixed that now, so you should be able to build all
> of them with a clean build.
>
> Thanks,
>
> Steve.
>
>
>

Re: Review Request 129760: Fixup handling of KFontUtils::adaptFontSize's flags' default value.

[Shaheed Haque]

---
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/129760/
---

(Updated Jan. 18, 2017, 12:45 p.m.)


Status
--

This change has been discarded.


Review request for KDE Frameworks.


Repository: kguiaddons


Description
---

This depends on a fix in extra-cmake-module to actually emit the
default value for the flags parameter.


Diffs
-

  autotests/pythontest.py c93e75397f87ba4a84f834dd248d671614ac89dd 
  cmake/rules_PyKF5.py PRE-CREATION 
  src/CMakeLists.txt 63e7598d13fa1c4d9d67e2f99edcc13e0269920e 

Diff: https://git.reviewboard.kde.org/r/129760/diff/


Testing
---

Without this change, once the code the depends-on review is committed, 
kguiaddons will fail to compile as follows:

In file included from /home/.../PyKF5/KGuiAddons/unifiedKGuiAddons.cpp:1:0:
/home/.../PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp: In function ‘PyObject* 
meth_KFontUtils_adaptFontSize(PyObject*, PyObject*)’:
/home/...d/PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp:30:50: error: ‘NoFlags’ 
was not declared in this scope
 KFontUtils::AdaptFontSizeOptions a6def = NoFlags;

With the change, it compiles and the tests run:

$ ctest
Test project /home/srhaque/kdebuild/kguiaddons
Start 1: appstreamtest
1/6 Test #1: appstreamtest    Passed0.02 sec
Start 2: Py3Test
2/6 Test #2: Py3Test ..   Passed0.13 sec
Start 3: Py2Test
3/6 Test #3: Py2Test ..   Passed0.11 sec
Start 4: kwordwraptest
4/6 Test #4: kwordwraptest    Passed0.16 sec
Start 5: kcolorutilstest
5/6 Test #5: kcolorutilstest ..   Passed0.53 sec
Start 6: kiconutilstest
6/6 Test #6: kiconutilstest ...   Passed0.08 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) =   1.03 sec


Thanks,

Shaheed Haque

Re: Review Request 129760: Fixup handling of KFontUtils::adaptFontSize's flags' default value.

[Shaheed Haque]

Hi Steve,

I have verified this works on KDE/master, and will close the review.

I did see a "funny" in that I had to run CMake twice in a fresh directory
before it would run clean. You might want to look into that as it was not
clear to me what it CMake is trying to do. Anyway, here is the first
unsuccessful run:

==
$ rm -r *
$ cmake ~/kdedev/frameworks-bindings/kguiaddons/;
-- The C compiler identification is GNU 6.2.0
-- The CXX compiler identification is GNU 6.2.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- 

-- Looking for __GLIBC__
-- Looking for __GLIBC__ - found
-- Performing Test _OFFT_IS_64BIT
-- Performing Test _OFFT_IS_64BIT - Success
-- Looking for XOpenDisplay in
/usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so
-- Looking for XOpenDisplay in
/usr/lib/x86_64-linux-gnu/libX11.so;/usr/lib/x86_64-linux-gnu/libXext.so -
found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Found X11: /usr/lib/x86_64-linux-gnu/libX11.so
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Found XCB_XCB: /usr/lib/x86_64-linux-gnu/libxcb.so (found version
"1.11.1")
-- Found XCB: /usr/lib/x86_64-linux-gnu/libxcb.so (found version "1.11.1")
found components:  XCB
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
CMake Error at
/usr/local/share/ECM/find-modules/FindPythonModuleGeneration.cmake:256
(message):
  Failed to find clang driver corresponding to
  /usr/lib/x86_64-linux-gnu/libclang-3.9.so
Call Stack (most recent call first):
  /usr/local/share/ECM/find-modules/FindPythonModuleGeneration.cmake:301
(_compute_implicit_include_dirs)
  src/CMakeLists.txt:94 (ecm_generate_python_binding)


-- Configuring incomplete, errors occurred!
See also "/home/srhaque/kdebuild/kguiaddons/CMakeFiles/CMakeOutput.log".


and then the second successful run:


$ cmake ~/kdedev/frameworks-bindings/kguiaddons/;
-- 

-- 
-- The following OPTIONAL packages have been found:

 * X11
 * PkgConfig
 * XCB , X protocol C-language Binding , <http://xcb.freedesktop.org>
 * Qt5X11Extras (required version >= 5.5.0)
 * PythonModuleGeneration

-- The following REQUIRED packages have been found:

 * ECM (required version >= 5.30.0) , Extra CMake Modules. , <
https://projects.kde.org/projects/kdesupport/extra-cmake-modules>
 * Qt5Gui (required version >= 5.5.0)
 * Qt5Test
 * Qt5Widgets
 * Qt5 (required version >= 5.5.0)

-- Configuring done
-- Generating done
-- Build files have been written to: /home/srhaque/kdebuild/kguiaddons



After this, "make; make test" ran with no issues, generating the .sip files
and so on.

Thanks, Shaheed



On 14 January 2017 at 13:07, Shaheed Haque  wrote:

> Thanks for the update, I'll take a look!
>
> On 13 January 2017 at 19:15, Stephen Kelly  wrote:
>
>> This is an automatically generated e-mail. To reply, visit:
>> https://git.reviewboard.kde.org/r/129760/
>>
>> On January 5th, 2017, 10:56 p.m. UTC, *Stephen Kelly* wrote:
>>
>> I've added the change to the unit test. It already passes, so it's not clear 
>> to me what else is needed from this review request.
>>
>> On January 5th, 2017, 11:06 p.m. UTC, *Shaheed Haque* wrote:
>>
>> What version of SIP compiler and C++ compiler are you using? I'm on:
>>
>> $ sip -V
>> 4.18.1
>> $ g++ --version
>> g++ (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
>> Copyright (C) 2016 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>
>> In any case, unless the change breaks thing for you, I would prefer to merge 
>> the complete change as it is definitely needed here.
>>
>> On January 5th, 2017, 11:11 p.m. UTC, *Stephen Kelly* wrote:
>>
>> I have
>>
>> $ sip -V
>&

Re: Review Request 129760: Fixup handling of KFontUtils::adaptFontSize's flags' default value.

[Shaheed Haque]

Thanks for the update, I'll take a look!

On 13 January 2017 at 19:15, Stephen Kelly  wrote:

> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/129760/
>
> On January 5th, 2017, 10:56 p.m. UTC, *Stephen Kelly* wrote:
>
> I've added the change to the unit test. It already passes, so it's not clear 
> to me what else is needed from this review request.
>
> On January 5th, 2017, 11:06 p.m. UTC, *Shaheed Haque* wrote:
>
> What version of SIP compiler and C++ compiler are you using? I'm on:
>
> $ sip -V
> 4.18.1
> $ g++ --version
> g++ (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
> Copyright (C) 2016 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> In any case, unless the change breaks thing for you, I would prefer to merge 
> the complete change as it is definitely needed here.
>
> On January 5th, 2017, 11:11 p.m. UTC, *Stephen Kelly* wrote:
>
> I have
>
> $ sip -V
> 4.17
> $ g++ --version
> g++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
> Copyright (C) 2015 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> $ cat /etc/issue
> Ubuntu 16.04.1 LTS \n \l
>
> Can you ensure that when you test this that you are using the clean master 
> branch of ECM and the master branch of KGuiAddons? If you have
>
> https://git.reviewboard.kde.org/r/129759/
>
> applied or any other patches, that could affect what you get when you run the 
> test.
>
> On January 6th, 2017, 8:54 a.m. UTC, *Shaheed Haque* wrote:
>
> As per my reviewboard markings, this change depends on '759. The point is 
> that without '759, the SIP compiler bails out before this issue can be seen. 
> So the real question is why/how you were able to manage without '759...I 
> would guess that the SIP compiler version delta explains this.
>
> Please see https://bugs.kde.org/show_bug.cgi?id=374801 and the commits 
> associated with it.
>
> In particular: your compiles were failing, but bindings were being created 
> anyway, which mostly worked for you with the exception of this enum issue. 
> This enum issue failed because it relies on being able to parse the QFlags 
> properly. Your compilations were failing because you are using Qt 5.7 or 
> later which requires -std=c++11 or later. That was not being added to the 
> flags when parsing the headers with libclang. ECM commit 
> 8aa6843404f9c6faef66cb9c76358158eafc1af1 fixed the parse failure, and 
> ed1b9ce2bb2a2e51410e0a1754a72c110010a6a0 fixed the logging bug.
>
> Please verify that you can build kguiaddons master with ECM master.
>
>
> - Stephen
>
> On January 3rd, 2017, 12:47 p.m. UTC, Shaheed Haque wrote:
> Review request for KDE Frameworks.
> By Shaheed Haque.
>
> *Updated Jan. 3, 2017, 12:47 p.m.*
> *Repository: * kguiaddons
> Description
>
> This depends on a fix in extra-cmake-module to actually emit the
> default value for the flags parameter.
>
> Testing
>
> Without this change, once the code the depends-on review is committed, 
> kguiaddons will fail to compile as follows:
>
> In file included from /home/.../PyKF5/KGuiAddons/unifiedKGuiAddons.cpp:1:0:
> /home/.../PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp: In function 
> ‘PyObject* meth_KFontUtils_adaptFontSize(PyObject*, PyObject*)’:
> /home/...d/PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp:30:50: error: 
> ‘NoFlags’ was not declared in this scope
>  KFontUtils::AdaptFontSizeOptions a6def = NoFlags;
>
> With the change, it compiles and the tests run:
>
> $ ctest
> Test project /home/srhaque/kdebuild/kguiaddons
> Start 1: appstreamtest
> 1/6 Test #1: appstreamtest    Passed0.02 sec
> Start 2: Py3Test
> 2/6 Test #2: Py3Test ..   Passed0.13 sec
> Start 3: Py2Test
> 3/6 Test #3: Py2Test ..   Passed0.11 sec
> Start 4: kwordwraptest
> 4/6 Test #4: kwordwraptest    Passed0.16 sec
> Start 5: kcolorutilstest
> 5/6 Test #5: kcolorutilstest ..   Passed0.53 sec
> Start 6: kiconutilstest
> 6/6 Test #6: kiconutilstest ...   Passed0.08 sec
>
> 100% tests passed, 0 tests failed out of 6
>
> Total Test time (real) =   1.03 sec
>
> Diffs
>
>- autotests/pythontest.py (c93e75397f87ba4a84f834dd248d671614ac89dd)
>- cmake/rules_PyKF5.py (PRE-CREATION)
>- src/CMakeLists.txt (63e7598d13fa1c4d9d67e2f99edcc13e0269920e)
>
> View Diff <https://git.reviewboard.kde.org/r/129760/diff/>
>

Re: Review Request 129760: Fixup handling of KFontUtils::adaptFontSize's flags' default value.

[Shaheed Haque]

> On Jan. 5, 2017, 10:56 p.m., Stephen Kelly wrote:
> > I've added the change to the unit test. It already passes, so it's not 
> > clear to me what else is needed from this review request.
> 
> Shaheed Haque wrote:
> What version of SIP compiler and C++ compiler are you using? I'm on:
> 
> $ sip -V
> 4.18.1
> $ g++ --version
> g++ (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
> Copyright (C) 2016 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR 
> PURPOSE.
> 
> In any case, unless the change breaks thing for you, I would prefer to 
> merge the complete change as it is definitely needed here.
> 
> Stephen Kelly wrote:
> I have
> 
> $ sip -V
> 4.17
> $ g++ --version
> g++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
> Copyright (C) 2015 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR 
> PURPOSE.
> 
> $ cat /etc/issue
> Ubuntu 16.04.1 LTS \n \l
> 
> 
> Can you ensure that when you test this that you are using the clean 
> master branch of ECM and the master branch of KGuiAddons? If you have 
> 
>  https://git.reviewboard.kde.org/r/129759/
>  
> applied or any other patches, that could affect what you get when you run 
> the test.

As per my reviewboard markings, this change depends on '759. The point is that 
without '759, the SIP compiler bails out before this issue can be seen. So the 
real question is why/how you were able to manage without '759...I would guess 
that the SIP compiler version delta explains this.


- Shaheed


---
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/129760/#review101827
---


On Jan. 3, 2017, 12:47 p.m., Shaheed Haque wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/129760/
> ---
> 
> (Updated Jan. 3, 2017, 12:47 p.m.)
> 
> 
> Review request for KDE Frameworks.
> 
> 
> Repository: kguiaddons
> 
> 
> Description
> ---
> 
> This depends on a fix in extra-cmake-module to actually emit the
> default value for the flags parameter.
> 
> 
> Diffs
> -
> 
>   autotests/pythontest.py c93e75397f87ba4a84f834dd248d671614ac89dd 
>   cmake/rules_PyKF5.py PRE-CREATION 
>   src/CMakeLists.txt 63e7598d13fa1c4d9d67e2f99edcc13e0269920e 
> 
> Diff: https://git.reviewboard.kde.org/r/129760/diff/
> 
> 
> Testing
> ---
> 
> Without this change, once the code the depends-on review is committed, 
> kguiaddons will fail to compile as follows:
> 
> In file included from /home/.../PyKF5/KGuiAddons/unifiedKGuiAddons.cpp:1:0:
> /home/.../PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp: In function 
> ‘PyObject* meth_KFontUtils_adaptFontSize(PyObject*, PyObject*)’:
> /home/...d/PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp:30:50: error: 
> ‘NoFlags’ was not declared in this scope
>  KFontUtils::AdaptFontSizeOptions a6def = NoFlags;
> 
> With the change, it compiles and the tests run:
> 
> $ ctest
> Test project /home/srhaque/kdebuild/kguiaddons
> Start 1: appstreamtest
> 1/6 Test #1: appstreamtest    Passed0.02 sec
> Start 2: Py3Test
> 2/6 Test #2: Py3Test ..   Passed0.13 sec
> Start 3: Py2Test
> 3/6 Test #3: Py2Test ..   Passed0.11 sec
> Start 4: kwordwraptest
> 4/6 Test #4: kwordwraptest    Passed    0.16 sec
> Start 5: kcolorutilstest
> 5/6 Test #5: kcolorutilstest ..   Passed0.53 sec
> Start 6: kiconutilstest
> 6/6 Test #6: kiconutilstest ...   Passed0.08 sec
> 
> 100% tests passed, 0 tests failed out of 6
> 
> Total Test time (real) =   1.03 sec
> 
> 
> Thanks,
> 
> Shaheed Haque
> 
>

Re: Review Request 129760: Fixup handling of KFontUtils::adaptFontSize's flags' default value.

[Shaheed Haque]

> On Jan. 5, 2017, 10:56 p.m., Stephen Kelly wrote:
> > I've added the change to the unit test. It already passes, so it's not 
> > clear to me what else is needed from this review request.

What version of SIP compiler and C++ compiler are you using? I'm on:

$ sip -V
4.18.1
$ g++ --version
g++ (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

In any case, unless the change breaks thing for you, I would prefer to merge 
the complete change as it is definitely needed here.


- Shaheed


---
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/129760/#review101827
---


On Jan. 3, 2017, 12:47 p.m., Shaheed Haque wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://git.reviewboard.kde.org/r/129760/
> ---
> 
> (Updated Jan. 3, 2017, 12:47 p.m.)
> 
> 
> Review request for KDE Frameworks.
> 
> 
> Repository: kguiaddons
> 
> 
> Description
> ---
> 
> This depends on a fix in extra-cmake-module to actually emit the
> default value for the flags parameter.
> 
> 
> Diffs
> -
> 
>   autotests/pythontest.py c93e75397f87ba4a84f834dd248d671614ac89dd 
>   cmake/rules_PyKF5.py PRE-CREATION 
>   src/CMakeLists.txt 63e7598d13fa1c4d9d67e2f99edcc13e0269920e 
> 
> Diff: https://git.reviewboard.kde.org/r/129760/diff/
> 
> 
> Testing
> ---
> 
> Without this change, once the code the depends-on review is committed, 
> kguiaddons will fail to compile as follows:
> 
> In file included from /home/.../PyKF5/KGuiAddons/unifiedKGuiAddons.cpp:1:0:
> /home/.../PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp: In function 
> ‘PyObject* meth_KFontUtils_adaptFontSize(PyObject*, PyObject*)’:
> /home/...d/PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp:30:50: error: 
> ‘NoFlags’ was not declared in this scope
>  KFontUtils::AdaptFontSizeOptions a6def = NoFlags;
> 
> With the change, it compiles and the tests run:
> 
> $ ctest
> Test project /home/srhaque/kdebuild/kguiaddons
> Start 1: appstreamtest
> 1/6 Test #1: appstreamtest    Passed0.02 sec
> Start 2: Py3Test
> 2/6 Test #2: Py3Test ..   Passed0.13 sec
> Start 3: Py2Test
> 3/6 Test #3: Py2Test ..   Passed0.11 sec
> Start 4: kwordwraptest
> 4/6 Test #4: kwordwraptest    Passed0.16 sec
> Start 5: kcolorutilstest
> 5/6 Test #5: kcolorutilstest ..   Passed0.53 sec
> Start 6: kiconutilstest
> 6/6 Test #6: kiconutilstest ...   Passed0.08 sec
> 
> 100% tests passed, 0 tests failed out of 6
> 
> Total Test time (real) =   1.03 sec
> 
> 
> Thanks,
> 
> Shaheed Haque
> 
>

Review Request 129760: Fixup handling of KFontUtils::adaptFontSize's flags' default value.

[Shaheed Haque]

---
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/129760/
---

Review request for KDE Frameworks.


Repository: kguiaddons


Description
---

This depends on a fix in extra-cmake-module to actually emit the
default value for the flags parameter.


Diffs
-

  autotests/pythontest.py c93e75397f87ba4a84f834dd248d671614ac89dd 
  cmake/rules_PyKF5.py PRE-CREATION 
  src/CMakeLists.txt 63e7598d13fa1c4d9d67e2f99edcc13e0269920e 

Diff: https://git.reviewboard.kde.org/r/129760/diff/


Testing
---

Without this change, once the code the depends-on review is committed, 
kguiaddons will fail to compile as follows:

In file included from /home/.../PyKF5/KGuiAddons/unifiedKGuiAddons.cpp:1:0:
/home/.../PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp: In function ‘PyObject* 
meth_KFontUtils_adaptFontSize(PyObject*, PyObject*)’:
/home/...d/PyKF5/KGuiAddons/sipKGuiAddonsKFontUtils.cpp:30:50: error: ‘NoFlags’ 
was not declared in this scope
 KFontUtils::AdaptFontSizeOptions a6def = NoFlags;

With the change, it compiles and the tests run:

$ ctest
Test project /home/srhaque/kdebuild/kguiaddons
Start 1: appstreamtest
1/6 Test #1: appstreamtest    Passed0.02 sec
Start 2: Py3Test
2/6 Test #2: Py3Test ..   Passed0.13 sec
Start 3: Py2Test
3/6 Test #3: Py2Test ..   Passed0.11 sec
Start 4: kwordwraptest
4/6 Test #4: kwordwraptest    Passed0.16 sec
Start 5: kcolorutilstest
5/6 Test #5: kcolorutilstest ..   Passed0.53 sec
Start 6: kiconutilstest
6/6 Test #6: kiconutilstest ...   Passed0.08 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) =   1.03 sec


Thanks,

Shaheed Haque