A progress update... On 24 October 2017 at 13:05, Shaheed Haque <srha...@theiet.org> wrote: > Hi all, > > I have a preliminary version of the Cppyy bindings generator CMake > support available here: > > > https://bitbucket.org/wlav/cppyy-backend/pull-requests/6/an-interim-experimental-version-of-a/diff > > There are some TODOs yet to be addressed,
The original TODOs and bugs have been resolved, and there is the beginnings of support for packaging frameworks under a Python namespace as in "KF5.KDCRAW". Also, as a significant datapoint, I'm close [1] to being able to generate a *complete* set of bindings for all of Akonadi driven from CMake with just 2-3 lines of custom logic. This contrasts with the 549 SLOC of customisation needed to produce a substantially slash-and-burned subset of Akonadi [2] with the SIP-based approach. > but I would appreciate > feedback on how easy it would be to integrate this with KDE's > buildsystem, especially for the frameworks. I'm a CMake noob, but the > basic idea I have is that the packager of some_framework might do > something like this: > > find_package(cppyy) > CPPYY_ADD_BINDINGS( > ... > LINK_LIBRARIES some_framework_LIBRARIES > H_DIR some_framework_INCLUDE_DIRS > H_FILES <list_of_h_files>) In the course of working through the "KF5" namespace implementation, it has become apparent to me that a framework-by-framework integration of the binding generation logic (as previously pioneered by Steve) probably cannot work in general because there are cases where multiple frameworks contribute to to the same C++ namespace, for example: $ grep -r '^namespace Akonadi' /usr/include/KF5/Akonadi* /usr/include/KF5/AkonadiAgentBase/resourcesettings.h:namespace Akonadi ... /usr/include/KF5/AkonadiCore/agentfilterproxymodel.h:namespace Akonadi ... /usr/include/KF5/AkonadiSearch/Debug/akonadisearchdebugsearchpathcombobox.h:namespace Akonadi ... /usr/include/KF5/AkonadiWidgets/agenttypedialog.h:namespace Akonadi ... /usr/include/KF5/AkonadiXml/xmldocument.h:namespace Akonadi ... The problem is that the Python implementation of these namespaces is a class, and so treating these frameworks (let's not quibble over whether KF5Akonadi* are truly KF5 frameworks, the point is more general) as separate would result in multiple colliding Python class definitions. The only solution I can see would be to bundle all of KF5Akonadi* into a single set of bindings, e.g. KF5.Akonadi, and AFAICS, this can only be done out of tree from the individual frameworks, say in kde-bindings.git [3]. The work to date attempts to maintain a clean separation such that all C++ builds are done from CMake, and all Python builds are done using setuptools/pip. Apart from working through bugs [1], the remaining work items I can think of are, as before: >> - Need to look into the exact usage of Qt-specifics: signals/slots and >> interoperability with SIP-based PyQt >> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glue-ing-applications, >> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html) >> >> - Need to figure out how any customisations which *are* required >> should be handled. plus: - I'm working with upstream on how to support discovery (e.g. via autocompletion in Python3). There is some POC-level hackery in git as above, but there is work ongoing with upstream to find a robust solution. - Flesh out how to make one set of bindings depend on another (e.g. tier 2 framework bindings might depend on tier 1 bindings, or maybe it is better to avoid PyQt and just produce cppyy-based bindings for Qt and depend on those). As always, comments/ideas/suggestions are welcome. Thanks, Shaheed [1] There is a bug with namespaced externs being worked on with upstream. [2] https://github.com/ShaheedHaque/extra-cmake-modules/blob/shaheed_master/find-modules/module_generation/PyKF5/Akonadi.py [3] I attach an example CMakeLists.txt which shows now this can be driven from CMake for the case of KF5Akonadi*...the implementation is intended to serve as the basis for a generic solution usable across KF5 at least. > On 16 October 2017 at 16:16, Shaheed Haque <srha...@theiet.org> wrote: >> As promised, here is an interim update on the investigation into the >> use of cppyy-based bindings for KF5 (and more...) instead of SIP-based >> bindings. >> >> The first thing is that the underlying technology of cppyy, >> cling/ROOT, has been under development at CERN for quite a while. It >> directly reads regular C++ files (there is no intermediate format like >> SIP). >> >> The bindings it generates from Python to C++ seem far more complete >> and automatic than SIP. For example: >> >> - Template instantiation is done on the fly as needed. >> >> - Since it uses C++ directly, there is none the effort required to >> decollide SIP's notion of forward and duplicate declarations. >> >> - Function overloads are cleanly handled, as are most (all?) operators. >> >> The net result is that so far, there is about 3 days work and >> approximately [1] no "customisation" required in order to get to >> roughly where the SIP based bindings were after 18 months. Without the >> need for customisations on a mass scale, I suspect that we might get >> away without anything like the tooling I had to create to SIP, and >> just integrate with CMake >> (https://root.cern.ch/how/integrate-root-my-project-cmake). >> >> This all sounds pretty amazing, right? Well, there are a few caveats... >> >> - The packaging is pretty new, and is evolving pretty rapidly. We >> are/will be an early adopter (https://bitbucket.org/wlav/cppyy/ and >> https://bitbucket.org/wlav/cppyy-backend). Packaging is via PyPI and >> pip/pip3. >> >> - There is a lot of documentation around for the system overall, but >> frankly, it has been/still is a struggle to understand how the >> different parts relate to each other as some parts are obsolete, and >> other parts have yet to be built out to their intended end-state. >> >> - There are bugs [1], [2]. The upstream dev has been very responsive, >> and the overall quality approach looks sound. IIUC, the vast bulk of >> the code seems to be in daily use at CERN (and is based on LLVM). >> >> - Need to look into the exact usage of Qt-specifics: signals/slots and >> interoperability with SIP-based PyQt >> (https://root.cern.ch/root/htmldoc/guides/users-guide/PythonRuby.html#glue-ing-applications, >> https://root.cern.ch/doc/v606_ORIG/guide/ROOTandQt.html) >> >> - Need to figure out how any customisations which *are* required >> should be handled. >> >> These seem like perfectly tractable issues, and so I conclude that >> using cppyy is definitely the way to go. With luck and a bit of >> effort, I am hopeful that we can get to some REALLY >> easy-to-develop-and-maintain bindings. >> >> [1] There is a bug with the binding producing stuff for private definitions. >> >> [2] There is a bug with missing globals. >> >> > [snip]
cmake_minimum_required(VERSION 3.9) find_package(Cppyy) find_package(ECM REQUIRED NO_MODULE) set(CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR} ${ECM_MODULE_PATH}) include(FeatureSummary) include(FindPkgConfig) include(CMakeFindDependencyMacro) # # Find the targets and dependencies for a KDE component. # set(_DEPENDENCIES) function(get_kf5_info component) find_dependency(${component}) set(real_targets) set(real_dependencies) # # Loop over all cmake files. # set(file_glob ${${component}_DIR}/*.cmake) file(GLOB files ${file_glob}) foreach(f ${files}) # # Targets. # file(STRINGS ${f} matches REGEX "^ *add_library\\(.*\\) *") if(NOT matches STREQUAL "") foreach(target ${matches}) string(REGEX REPLACE " *add_library\\(([^ \\)]+).*" "\\1" target ${target}) if(TARGET ${target}) list(APPEND real_targets ${target}) list(REMOVE_DUPLICATES real_targets) else() message(STATUS "Ignoring invalid target \"${target}\" for ${component} in ${f}") endif() endforeach() endif() # # Dependencies. # file(STRINGS ${f} matches REGEX "^ *find_dependency\\(.*\\) *") if(NOT matches STREQUAL "") foreach(dependency ${matches}) string(REGEX REPLACE " *find_dependency\\(([^ \\)]+).*" "\\1" dependency ${dependency}) if(NOT ${dependency} STREQUAL "") list(APPEND real_dependencies ${dependency}) list(REMOVE_DUPLICATES real_dependencies) # # Recurse...if we have not been here before. # string(FIND ${dependency} "KF5" found) if(found EQUAL 0 AND NOT dependency IN_LIST _DEPENDENCIES) get_kf5_info(${dependency}) list(APPEND real_dependencies ${dependencies}) list(REMOVE_DUPLICATES real_dependencies) endif() else() message(STATUS "Ignoring invalid dependency \"${dependency}\" for ${component} in ${f}") endif() endforeach() endif() endforeach() set(targets "${real_targets}" PARENT_SCOPE) set(dependencies "${real_dependencies}" PARENT_SCOPE) endfunction(get_kf5_info) # # Find the targets and dependencies for a Qt component. # function(get_qt5_info component) find_dependency(${component} NO_MODULE) # # Targets. # string(REPLACE "Qt5" "Qt5::" target ${component}) if(NOT TARGET ${target}) message(STATUS "Ignoring invalid target \"${target}\" for ${component}") set(target "") endif() set(targets "${target}" PARENT_SCOPE) # # Dependencies. # set(dependencies "" PARENT_SCOPE) endfunction(get_qt5_info) # # Fetch a target property, recursing if necessary. # function(get_target_property_recursive target property) set(result) get_target_property(values ${target} ${property}) if(values STREQUAL "values-NOTFOUND") # Skip # message(STATUS "Warning: Target ${target} has no property ${property}") else() foreach(value ${values}) string(FIND ${value} "$<TARGET_PROPERTY:" found) if(found EQUAL 0) # # Recurse. The format is: # # $<TARGET_PROPERTY:KF5::WebKit,INTERFACE_INCLUDE_DIRECTORIES> # string(REGEX REPLACE "\\$<TARGET_PROPERTY:(.*),(.*)>" "\\1" nested_tgt ${value}) string(REGEX REPLACE "\\$<TARGET_PROPERTY:(.*),(.*)>" "\\2" nested_prop ${value}) get_target_property_recursive(${nested_tgt} ${nested_prop}) list(APPEND result ${get_target_property_recursive_result}) else() list(APPEND result ${value}) endif() endforeach() endif() set(get_target_property_recursive_result "${result}" PARENT_SCOPE) endfunction(get_target_property_recursive) # # Find the includes, libraries etc. for a component. # function(get_targets_info component targets) if(targets STREQUAL "") message(STATUS "Warning: No targets for ${component}") return() endif() # # Make a combined list of includes, libraries etc. # # There is a potential impedence mismatch between the directory-centric # Pythonic notion of a package, and the possibility that the the multiple # targets *might* have conflicting options. Luckily, this seems not to be # a problem in KF5. # set(libraries) set(includes) set(compile_flags) foreach(target ${targets}) if(TARGET ${target}) get_target_property(tmp ${target} LOCATION) list(APPEND libraries ${tmp}) get_target_property_recursive(${target} INTERFACE_INCLUDE_DIRECTORIES) list(APPEND includes ${get_target_property_recursive_result}) get_target_property_recursive(${target} INTERFACE_COMPILE_DEFINITIONS) foreach(definition ${get_target_property_recursive_result}) if(${definition} MATCHES ".*QT_NO_DEBUG>") # # Qt uses the formulation "$<$<NOT:$<CONFIG:Debug>>:QT_NO_DEBUG>". # elseif(${definition} MATCHES "QT_.*_LIB") # # Qt uses the formulation "QT_CORE_LIB" even for INTERFACE_COMPILE_FLAGS. # else() list(APPEND compile_flags "-D${definition}") endif() endforeach() get_target_property_recursive(${target} INTERFACE_COMPILE_OPTIONS) list(APPEND compile_flags ${get_target_property_recursive_result}) else() message(STATUS "Warning: Ignoring invalid target \"${target}\" in ${f}") endif() endforeach() # # De-duplicate and write results. # if(DEFINED includes) list(REMOVE_DUPLICATES includes) # # Not sure why the headers seem to include this. # list(REMOVE_ITEM includes "/usr/include") endif() if(DEFINED compile_flags) list(REMOVE_DUPLICATES compile_flags) endif() set(libraries "${libraries}" PARENT_SCOPE) set(includes "${includes}" PARENT_SCOPE) set(compile_flags "${compile_flags}" PARENT_SCOPE) endfunction(get_targets_info) # # Find the includes, libraries etc. for a pkg-config component. # function(get_pkgconfig_info component) set(libraries) set(includes ${${component}_INCLUDEDIR}) set(compile_flags ${${component}_CFLAGS}) foreach(tmp ${${component}_LIBRARIES}) find_library(lib${tmp} NAMES ${tmp} PATHS ${${component}_LIBRARIES}) list(APPEND libraries ${lib${tmp}}) endforeach() set(libraries "${libraries}" PARENT_SCOPE) set(includes "${includes}" PARENT_SCOPE) set(compile_flags "${compile_flags}" PARENT_SCOPE) endfunction(get_pkgconfig_info) # # Return the information required to create the bindings for a set of KF5 components. # # get_kf5_binding_info( # COMPONENTS components # DEPENDENCIES extras) # # Arguments and options: # # COMPONENTS component # The CMake packages to include in the bindings. # # DEPENDENCIES dependency # Any CMake packages not detected by the automatic # dependency extraction logic. # function(get_kf5_binding_info) cmake_parse_arguments( ARG "" "" "COMPONENTS;DEPENDENCIES" ${ARGN}) if(NOT "${ARG_UNPARSED_ARGUMENTS}" STREQUAL "") message(SEND_ERROR "Unexpected arguments specified '${ARG_UNPARSED_ARGUMENTS}'") endif() if("${ARG_COMPONENTS}" STREQUAL "") message(SEND_ERROR "No COMPONENTS specified") endif() # # Find dependencies and other info. # set(_H_DIRS) set(_H_FILES) set(_COMPILE_OPTIONS) set(_INCLUDE_DIRS) set(_LINK_LIBRARIES) foreach(component IN LISTS ARG_COMPONENTS) get_kf5_info(${component}) # # Automatic dependencies. # list(APPEND _DEPENDENCIES ${dependencies}) list(REMOVE_DUPLICATES _DEPENDENCIES) # # Other info. # get_targets_info(${component} ${targets}) list(APPEND _H_DIRS ${includes}) list(APPEND _LINK_LIBRARIES ${libraries}) list(APPEND _COMPILE_OPTIONS "${compile_flags}") list(REMOVE_DUPLICATES _H_DIRS) list(REMOVE_DUPLICATES _LINK_LIBRARIES) list(REMOVE_DUPLICATES _COMPILE_OPTIONS) endforeach(component) # # Find all header files. # foreach(h_dir IN LISTS _H_DIRS) file(GLOB tmp ${h_dir}/*.h) list(APPEND _H_FILES ${tmp}) list(REMOVE_DUPLICATES _H_FILES) endforeach(h_dir) list(FILTER _H_FILES EXCLUDE REGEX ".*_version.h") # # Add dependencies. # foreach(component IN LISTS _DEPENDENCIES ARG_DEPENDENCIES) string(FIND ${component} "KF5" found_kf5) string(FIND ${component} "Qt5" found_qt5) if(component MATCHES "^KF5") get_kf5_info(${component}) elseif(component MATCHES "^Qt5") get_qt5_info(${component}) endif() get_targets_info(${component} "${targets}") list(APPEND _INCLUDE_DIRS ${includes}) list(APPEND _LINK_LIBRARIES ${libraries}) list(APPEND _COMPILE_OPTIONS "${compile_flags}") list(REMOVE_DUPLICATES _INCLUDE_DIRS) list(REMOVE_DUPLICATES _LINK_LIBRARIES) list(REMOVE_DUPLICATES _COMPILE_OPTIONS) endforeach(component) # # Find the version from the first component. # list(GET ARG_COMPONENTS 1 first_component) include(${${first_component}_DIR}/${first_component}ConfigVersion.cmake) # # Return results. # set(version ${PACKAGE_VERSION} PARENT_SCOPE) set(h_dirs ${_H_DIRS} PARENT_SCOPE) set(h_files ${_H_FILES} PARENT_SCOPE) set(include_dirs ${_INCLUDE_DIRS} PARENT_SCOPE) set(compile_options ${_COMPILE_OPTIONS} PARENT_SCOPE) set(link_libraries ${_LINK_LIBRARIES} PARENT_SCOPE) #message("version=${PACKAGE_VERSION}") #message("h_dirs=${_H_DIRS}") #message("h_files=${_H_FILES}") #message("include_dirs=${_INCLUDE_DIRS}") #message("compile_options=${_COMPILE_OPTIONS}") #message("link_libraries=${_LINK_LIBRARIES}") endfunction(get_kf5_binding_info) # # Main code. # get_kf5_binding_info( COMPONENTS KF5Akonadi KF5AkonadiCalendar KF5AkonadiContact KF5AkonadiMime KF5AkonadiNotes KF5AkonadiSearch DEPENDENCIES KF5Konq) list(FILTER h_files EXCLUDE REGEX ".*qtest_akonadi.h") list(FILTER h_files EXCLUDE REGEX ".*/KF5/[^/]+.h") CPPYY_ADD_BINDINGS( "KF5.Akonadi" "${version}" "Shaheed" "srha...@theiet.org" LANGUAGE_STANDARD "14" GENERATE_OPTIONS "-D__PIC__;-Wno-macro-redefined" INCLUDE_DIRS ${include_dirs} LINK_LIBRARIES ${link_libraries} H_DIRS ${h_dirs} H_FILES ${h_files})