Cross compiling KDE / Nokia N900
On Tuesday 22 December 2009, Thiago Macieira wrote: ... We've been successful at getting Qt to cross compile for the N900. It works much faster and requires no VM. In fact, it works with build farms too, so get all the benefits of a normal build. Any chance we can get an effort started to make KDE cross-compileable? Yes, it's also on my todo, but it would be really nice if somebody else could actually do the work and I would just guide throught the necessary steps. There are mainly three things which lead to problems when cross compiling: 1) try to build and run executable during the configure check 2) running executables during the configure step to find out information 3) running executables during the build step to do something. 1) is the hardest one. If somebody does a try_run (check_c_source_runs()) in a cross compile, it will build the executable, but it will not try to run it (since it can't, in general). Instead cmake will provide a template cmake script file prepared so that the expected values from the try_run() can be entered there manually. Also the compiled executable is made available, so that the developer can take it, run it on the target and in this way figure out the result which have to go into the prepared cmake script file. This file than has to be preloaded into cmake using -C on the next cmake run. Doing this for all try_run()s is actual work. 2) We do this e.g. for Qt (FindQt4.cmake calls qmake), we query kde-config4, some find-modules call pkg-config (I'm not sure how to handle pkg-config when cross compiling, some env.vars could be set so that it finds only in the target root dir, ...). All these steps could be in some way done differently when cross compiling (e.g. only try to find qmake and then expect that the headers will be in the qmake_dir/../include etc. This can be guarded in if(CMAKE_CROSSCOMPILING) ... endif(CMAKE_CROSSCOMPILING) 3) execute tools build during the build. This is not a real problem, it's just a bit of work. We already export our libraries in kdelibs (KDELibs4LibraryTargets.cmake), and also separate from this our executable targets (KDELibs4ToolsTargets.cmake). For cross compiling, these executable targets have to be available from a native build, and then this file can be loaded into the cross-compiling kdelibs and used there in the add_custom_commands/add_custom_targets(). Again this needs some if (CMAKE_CROSSCOMPILING) guards, most probably mainly in KDE4Macros.cmake, but then it should work. PS: maybe move this discussion to kde-buildsystem? Hereby I give my ok to move my part of the thread out of ev-membership. Done :-) Alex ___ Kde-buildsystem mailing list Kde-buildsystem@kde.org https://mail.kde.org/mailman/listinfo/kde-buildsystem
Re: Cross compiling KDE / Nokia N900
On Wednesday 23 December 2009, Pino Toscano wrote: Hi, Alle mercoledì 23 dicembre 2009, Alexander Neundorf ha scritto: Yes, it's also on my todo, but it would be really nice if somebody else could actually do the work and I would just guide throught the necessary steps. There are mainly three things which lead to problems when cross compiling: 1) try to build and run executable during the configure check [...] 1) is the hardest one. If somebody does a try_run (check_c_source_runs()) in a cross compile, it will build the executable, but it will not try to run it (since it can't, in general). Instead cmake will provide a template cmake script file prepared so that the expected values from the try_run() can be entered there manually. Also the compiled executable is made available, so that the developer can take it, run it on the target and in this way figure out the result which have to go into the prepared cmake script file. This file than has to be preloaded into cmake using -C on the next cmake run. Doing this for all try_run()s is actual work. I think this point could be solved if cmake would have the distinction between host (the platform it is building for) and build (the platform it is building on) compilers (of course, in case of no crosscompilation, host and build stuff would be the same). This is the case since cmake 2.6.0 :-) You can check if (CMAKE_CROSSCOMPILING) ... and you have CMAKE_SYSTEM_NAME (target host) and CMAKE_HOST_SYSTEM_NAME (build host), if not cross compiling both are the same. For example, take the check_c_source_runs() example, and assume I'm crosscompiling on i386 (build gcc: cc) for arm (host cc: arm-linux-cc): check_c_source_runs() would be compiled using the build compiler (cc), so it can be run later. Do you mean that even when cross compiling the i386 gcc should be used for check_c_source_runs() ? This wouldn't make sense. The sources must be built using the target compiler (arm_linux-cc), since -host and target compiler may be completely different compilers (think gcc and sdcc, or msvc and gcc) -being able to compile and link the source with the host (i386) compiler doesn't tell you anything about whether it would also link on the target system -running it in the build host environment doesn't tell you anything about the target environment This would have the advantage I could force the use of the build compiler for intermediate build steps, for example a custom target which has to compile some executable and run it to get the input files for other targets. This proposal would also avoid the template cmake script with the expected values and using CMAKE_CROSSCOMPILING in such situations. Would it feasible to do? No, I don't think so. It must be done with the target compiler. Something what might be feasible would be to add special support for executing executables via some wrapper (e.g. upload it to the target, execute it and return results, run it via wine, etc.). Alex ___ Kde-buildsystem mailing list Kde-buildsystem@kde.org https://mail.kde.org/mailman/listinfo/kde-buildsystem
Re: Cross compiling KDE / Nokia N900
Alle mercoledì 23 dicembre 2009, Alexander Neundorf ha scritto: On Wednesday 23 December 2009, Pino Toscano wrote: Hi, Alle mercoledì 23 dicembre 2009, Alexander Neundorf ha scritto: Yes, it's also on my todo, but it would be really nice if somebody else could actually do the work and I would just guide throught the necessary steps. There are mainly three things which lead to problems when cross compiling: 1) try to build and run executable during the configure check [...] 1) is the hardest one. If somebody does a try_run (check_c_source_runs()) in a cross compile, it will build the executable, but it will not try to run it (since it can't, in general). Instead cmake will provide a template cmake script file prepared so that the expected values from the try_run() can be entered there manually. Also the compiled executable is made available, so that the developer can take it, run it on the target and in this way figure out the result which have to go into the prepared cmake script file. This file than has to be preloaded into cmake using -C on the next cmake run. Doing this for all try_run()s is actual work. I think this point could be solved if cmake would have the distinction between host (the platform it is building for) and build (the platform it is building on) compilers (of course, in case of no crosscompilation, host and build stuff would be the same). This is the case since cmake 2.6.0 :-) You can check if (CMAKE_CROSSCOMPILING) ... and you have CMAKE_SYSTEM_NAME (target host) and CMAKE_HOST_SYSTEM_NAME (build host), if not cross compiling both are the same. I know cmake can do cross compiling this way (I do that at work), but it's far from being pain-free. For example, take the check_c_source_runs() example, and assume I'm crosscompiling on i386 (build gcc: cc) for arm (host cc: arm-linux-cc): check_c_source_runs() would be compiled using the build compiler (cc), so it can be run later. Do you mean that even when cross compiling the i386 gcc should be used for check_c_source_runs() ? This wouldn't make sense. The sources must be built using the target compiler (arm_linux-cc), since -host and target compiler may be completely different compilers (think gcc and sdcc, or msvc and gcc) -being able to compile and link the source with the host (i386) compiler doesn't tell you anything about whether it would also link on the target system -running it in the build host environment doesn't tell you anything about the target environment Those points are valid, although there cold be situations where you want to do that anyway: for example, a small program that #include some version.h-like header, and outputs different stuff depending on different #if VERSION_MAJOR 4 ... #elif VERSION_MAJOR 2 ... etc. This is so simple no compiler/toolchain could get it wrong. Furthermore, there's still the other case of usefulness for having build and host compilers separated: This would have the advantage I could force the use of the build compiler for intermediate build steps, for example a custom target which has to compile some executable and run it to get the input files for other targets. -- Pino Toscano signature.asc Description: This is a digitally signed message part. ___ Kde-buildsystem mailing list Kde-buildsystem@kde.org https://mail.kde.org/mailman/listinfo/kde-buildsystem
Re: Cross compiling KDE / Nokia N900
On Wednesday 23 December 2009, Pino Toscano wrote: Alle mercoledì 23 dicembre 2009, Alexander Neundorf ha scritto: ... and you have CMAKE_SYSTEM_NAME (target host) and CMAKE_HOST_SYSTEM_NAME (build host), if not cross compiling both are the same. I know cmake can do cross compiling this way (I do that at work), but it's far from being pain-free. Can you please share your experiences, what works nice, what could need improvement, what doesn't work ? ... Those points are valid, although there cold be situations where you want to do that anyway: for example, a small program that #include some version.h-like header, and outputs different stuff depending on different #if VERSION_MAJOR 4 ... #elif VERSION_MAJOR 2 ... etc. This is so simple no compiler/toolchain could get it wrong. Yes, there are probably situations where it would work, but since it wouldn't work in general, cmake AFAIK will not support this, since every case where it then does not work may get reported as a bug. (something which doesn't always work but only in most cases is considered broken by Kitware) Furthermore, there's still the other case of usefulness for having build and host compilers separated: This would have the advantage I could force the use of the build compiler for intermediate build steps, for example a custom target which has to compile some executable and run it to get the input files for other targets. Yes. The issue with this is that right now it is one of the basic assumptions of cmake that you have _one_ compiler doing one build. E.g. the CMakeCache.txt is valid for just one compiler. Everything contained in it can potentially be wrong for a second compiler for the same language. So, adding this capability to cmake would be AFAICT a _lot_ of work. Alex ___ Kde-buildsystem mailing list Kde-buildsystem@kde.org https://mail.kde.org/mailman/listinfo/kde-buildsystem
Re: Cross compiling KDE / Nokia N900
Em Quarta-feira 23. Dezembro 2009, às 11.06.18, Alexander Neundorf escreveu: Yes, it's also on my todo, but it would be really nice if somebody else could actually do the work and I would just guide throught the necessary steps. There are mainly three things which lead to problems when cross compiling: 1) try to build and run executable during the configure check 2) running executables during the configure step to find out information 3) running executables during the build step to do something. 1) is the hardest one. If somebody does a try_run (check_c_source_runs()) in a cross compile, it will build the executable, but it will not try to run it (since it can't, in general). Instead cmake will provide a template cmake script file prepared so that the expected values from the try_run() can be entered there manually. Also the compiled executable is made available, so that the developer can take it, run it on the target and in this way figure out the result which have to go into the prepared cmake script file. This file than has to be preloaded into cmake using -C on the next cmake run. Doing this for all try_run()s is actual work. Indeed. Autoconf's solution to the problem is that an AC_CHECK_RUNS macro has three possible outcomes: 1) it runs 2) it doesn't run 3) it is being cross-compiled This tri-state is what we need in CMake and what we need to fix in KDE (all of the checks). Maybe CMake could also learn of scratchbox-like builds: i.e. cross-compiling, but the target executable can be run (maybe with an additional tool). For example, suppose we cross-compile for Windows using mingw32-g++. It is possible to run the resulting executable with Wine. For the N900, it's possible to execute the ARM executable with qemu / scratchbox. I don't know how difficult this would be, though. Note that this is still a cross-compilation build, since non-native executables may have a performance penalty. The following point is important: 2) We do this e.g. for Qt (FindQt4.cmake calls qmake), we query kde-config4, some find-modules call pkg-config (I'm not sure how to handle pkg-config when cross compiling, some env.vars could be set so that it finds only in the target root dir, ...). All these steps could be in some way done differently when cross compiling (e.g. only try to find qmake and then expect that the headers will be in the qmake_dir/../include etc. This can be guarded in if(CMAKE_CROSSCOMPILING) ... endif(CMAKE_CROSSCOMPILING) You have to distinguish between host tools and target tools. The output of a Qt cross-compilation is that qmake, moc, uic, uic3 are host executables, whereas the libraries are target binaries (designer, assistant, etc. are not built, but if they were, they'd be target too). Interestingly, qdbusxml2cpp and qdbuscpp2xml are host tools, but qdbus and qdbusviewer are target ones. So we need to be able to run host executables even in a cross-compilation. And we need to be able to create host executables for our own builds too, and run them. As for pkg-config, it's also a host tool. The difference is you set some environment variables telling pkg-config where the cross-compilation root is, so that it may find the correct .pc files. Note, however, that the latest version of pkg-config (0.23) has a bug that makes it unusable for cross- compilation. Upstream has fixed it already, but no new release has happened. You need to ensure your distribution has packaged the patch. So the rule of thumb is: would this tool be in the main package or in the - devel/-dev package for my library? 3) execute tools build during the build. This is not a real problem, it's just a bit of work. We already export our libraries in kdelibs (KDELibs4LibraryTargets.cmake), and also separate from this our executable targets (KDELibs4ToolsTargets.cmake). For cross compiling, these executable targets have to be available from a native build, and then this file can be loaded into the cross-compiling kdelibs and used there in the add_custom_commands/add_custom_targets(). Again this needs some if (CMAKE_CROSSCOMPILING) guards, most probably mainly in KDE4Macros.cmake, but then it should work. Same as above. -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Senior Product Manager - Nokia, Qt Development Frameworks PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 signature.asc Description: This is a digitally signed message part. ___ Kde-buildsystem mailing list Kde-buildsystem@kde.org https://mail.kde.org/mailman/listinfo/kde-buildsystem
Re: Cross compiling KDE / Nokia N900
On Wednesday 23 December 2009, Thiago Macieira wrote: Em Quarta-feira 23. Dezembro 2009, às 11.06.18, Alexander Neundorf escreveu: Yes, it's also on my todo, but it would be really nice if somebody else could actually do the work and I would just guide throught the necessary steps. There are mainly three things which lead to problems when cross compiling: 1) try to build and run executable during the configure check 2) running executables during the configure step to find out information 3) running executables during the build step to do something. 1) is the hardest one. If somebody does a try_run (check_c_source_runs()) in a cross compile, it will build the executable, but it will not try to run it (since it can't, in general). Instead cmake will provide a template cmake script file prepared so that the expected values from the try_run() can be entered there manually. Also the compiled executable is made available, so that the developer can take it, run it on the target and in this way figure out the result which have to go into the prepared cmake script file. This file than has to be preloaded into cmake using -C on the next cmake run. Doing this for all try_run()s is actual work. Indeed. Autoconf's solution to the problem is that an AC_CHECK_RUNS macro has three possible outcomes: 1) it runs 2) it doesn't run 3) it is being cross-compiled This tri-state is what we need in CMake and what we need to fix in KDE (all of the checks). It does that basically. CMake always builds the executable, if not cross compiling, it tries to execute it, which fails or succeeds, if cross compiling, it doesn't try to execute it but leaves the result to the user/developer. Maybe CMake could also learn of scratchbox-like builds: i.e. cross-compiling, but the target executable can be run (maybe with an additional tool). For example, suppose we cross-compile for Windows using mingw32-g++. It is possible to run the resulting executable with Wine. For the N900, it's possible to execute the ARM executable with qemu / scratchbox. I don't know how difficult this would be, though. Note that this is still a cross-compilation build, since non-native executables may have a performance penalty. The following point is important: 2) We do this e.g. for Qt (FindQt4.cmake calls qmake), we query kde-config4, some find-modules call pkg-config (I'm not sure how to handle pkg-config when cross compiling, some env.vars could be set so that it finds only in the target root dir, ...). All these steps could be in some way done differently when cross compiling (e.g. only try to find qmake and then expect that the headers will be in the qmake_dir/../include etc. This can be guarded in if(CMAKE_CROSSCOMPILING) ... endif(CMAKE_CROSSCOMPILING) You have to distinguish between host tools and target tools. The output of a Qt cross-compilation is that qmake, moc, uic, uic3 are host executables, whereas the libraries are target binaries (designer, assistant, etc. are not built, but if they were, they'd be target too). Interestingly, qdbusxml2cpp and qdbuscpp2xml are host tools, but qdbus and qdbusviewer are target ones. So we need to be able to run host executables even in a cross-compilation. Yes, this is possible. With cmake you can specifiy a CMAKE_FIND_ROOT_PATH, which is used as prefix to all search paths, i.e. it should point to the location where the target environment is installed: http://www.cmake.org/Wiki/CMake_Cross_Compiling#Searching_and_finding_external_software You can specifiy for the find_xxx() calls whether they should look in the ROOT_PATH or in the normal dirs. For executables the default is to look in the normal dirs, for libs and header the default is to look in the ROOT_PATH dirs. This can be specified for each find_xxx() call separately. And we need to be able to create host executables for our own builds too, and run them. This has to be done in a separate build tree with cmake (or reuse an existing build host installation). As for pkg-config, it's also a host tool. The difference is you set some environment variables telling pkg-config where the cross-compilation root is, so that it may find the correct .pc files. Note, however, that the Locally I have a patch for FindPkgConfig.cmake, which sets PKG_CONFIG_LIBDIR and PKG_CONFIG_PATH when cross compiling to point only inside the ROOT_PATH (i.e. the target environment). Does that sound reasonable or would this be wrong ? Alex ___ Kde-buildsystem mailing list Kde-buildsystem@kde.org https://mail.kde.org/mailman/listinfo/kde-buildsystem
Re: Cross compiling KDE / Nokia N900
Em Quarta-feira 23. Dezembro 2009, às 14.29.48, Alexander Neundorf escreveu: And we need to be able to create host executables for our own builds too, and run them. This has to be done in a separate build tree with cmake (or reuse an existing build host installation). That's a bit too much work. When cross-compiling, sometimes we need to compile a program to do extra work. Sometimes that program is installed, other times it isn't (example: meinproc). So we need host targets as well as cross-compilation targets. I agree this is much more complex, as many tests would need to be duplicated. It's like qmake's debug-and-release mode, which basically runs everything in two separate passes. As for pkg-config, it's also a host tool. The difference is you set some environment variables telling pkg-config where the cross-compilation root is, so that it may find the correct .pc files. Note, however, that the Locally I have a patch for FindPkgConfig.cmake, which sets PKG_CONFIG_LIBDIR and PKG_CONFIG_PATH when cross compiling to point only inside the ROOT_PATH (i.e. the target environment). Does that sound reasonable or would this be wrong ? My script for cross-compiling sets this: PKG_CONFIG_LIBDIR=/opt/maemo/arm-none-linux-gnueabi/libc/usr/lib/pkgconfig PKG_CONFIG_SYSROOT_DIR=/opt/maemo/arm-none-linux-gnueabi/libc PATH=/opt/teambuilder2/bin:/opt/maemo/bin:$PATH export PKG_CONFIG_LIBDIR PKG_CONFIG_SYSROOT_DIR PATH It's the SYSROOT variable that is necessary but triggers the bug. -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Senior Product Manager - Nokia, Qt Development Frameworks PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 signature.asc Description: This is a digitally signed message part. ___ Kde-buildsystem mailing list Kde-buildsystem@kde.org https://mail.kde.org/mailman/listinfo/kde-buildsystem
Re: Cross compiling KDE / Nokia N900
On Wednesday 23 December 2009, Thiago Macieira wrote:
Em Quarta-feira 23. Dezembro 2009, às 14.29.48, Alexander Neundorf escreveu:
And we need to be able to create host executables for our own builds
too, and run them.
This has to be done in a separate build tree with cmake (or reuse an
existing build host installation).
That's a bit too much work. When cross-compiling, sometimes we need to
compile a program to do extra work. Sometimes that program is installed,
other times it isn't (example: meinproc).
So we need host targets as well as cross-compilation targets.
s/we need/it would be nice to have/.
There are two options:
- we could just require that an installed kdelibs is required for cross
compiling KDE
- we can introduce a custom target build_tools or something like that which
depends on all the tools required for the cross-build. Then everybody would
need to create a native build tree and build just this target there (make
build_tools), and then point the cross-build to this native build-tree (via
a cmake or environment variable or maybe it can be found automatically).
Both options can work and are real-world tested, and both options are maybe
more work than having separate target platforms in one build tree.
I agree this is much more complex, as many tests would need to be
duplicated. It's like qmake's debug-and-release mode, which basically runs
everything in two separate passes.
But practically this does not really matter, since AFAIK adding support for
having multiple toolchains in one buildtree is not on the near-term roadmap
for CMake. There are two ways to change this:
1) submit a patch which adds this functionality (well, patch, I think it
would be a major piece of work), which passes all tests, works in all cases,
etc.
2) get Kitware to implement it, easiest way is a payed contract
As for pkg-config, it's also a host tool. The difference is you set
some environment variables telling pkg-config where the
cross-compilation root is, so that it may find the correct .pc files.
Note, however, that the
Locally I have a patch for FindPkgConfig.cmake, which sets
PKG_CONFIG_LIBDIR and PKG_CONFIG_PATH when cross compiling to point only
inside the ROOT_PATH (i.e. the target environment).
Does that sound reasonable or would this be wrong ?
My script for cross-compiling sets this:
PKG_CONFIG_LIBDIR=/opt/maemo/arm-none-linux-gnueabi/libc/usr/lib/pkgconfig
PKG_CONFIG_SYSROOT_DIR=/opt/maemo/arm-none-linux-gnueabi/libc
PATH=/opt/teambuilder2/bin:/opt/maemo/bin:$PATH
export PKG_CONFIG_LIBDIR PKG_CONFIG_SYSROOT_DIR PATH
It's the SYSROOT variable that is necessary but triggers the bug.
This is what my patch currently does:
+# When crosscompiling, pkg_config should not search for packages of the host
+# system, but for the target system. To do this, there is the environment
+# variable PKG_CONFIG_LIBDIR, which was added to pkg_config explicitely for
+# crosscompiling. Additionally PKG_CONFIG_PATH is set empty, otherwise these
+# directories are searched before PKG_CONFIG_LIBDIR (at least here with
+# version 0.21).
+# So set this here to point inside the CMAKE_FIND_ROOT_PATH so it can find
+# only packages for the target system when crosscompiling. Alex
+set(pkgconfigLibDir)
+if(CMAKE_CROSSCOMPILING AND CMAKE_FIND_ROOT_PATH)
+ foreach(currentFindRoot ${CMAKE_FIND_ROOT_PATH})
+ set(pkgconfigLibDir ${pkgconfigLibDir}:
${currentFindRoot}/usr/lib/pkgconfig:
${currentFindRoot}/usr/local/lib/pkgconfig )
+ endforeach(currentFindRoot)
+ set(ENV{PKG_CONFIG_LIBDIR} ${pkgconfigLibDir} )
+ set(ENV{PKG_CONFIG_PATH} )
+endif(CMAKE_CROSSCOMPILING AND CMAKE_FIND_ROOT_PATH)
(and it also messes around with setting a prefix so the directories are
correct for the host system)
Alex
___
Kde-buildsystem mailing list
Kde-buildsystem@kde.org
https://mail.kde.org/mailman/listinfo/kde-buildsystem
