KRandom broken?
Hi, it seems as if KRandom is currently broken. https://bugs.kde.org/show_bug.cgi?id=362161 Given the potential impact, is there something to be done (besides fixing it) wrt distro packagers? Regards
Moving CI server documentation to community.ko? (was: Re: KDE CI enforcing ECMEnableSanitizers.cmake/KDECompilerSettings.cmake?)
Am Montag, 25. April 2016, 23:26:45 CEST schrieb Albert Astals Cid: > El dilluns, 25 d’abril de 2016, a les 10:59:13 CEST, Friedrich W. H. > Kossebau > va escriure: > > Hi, > > > > can anyone shed full light on KDE CI and the usage of ASan? > > > > Currently e.g. all tests of Marble are failing, with the message > > "ASan runtime does not come first in initial library list; you should > > either link runtime to your application or manually preload it with > > LD_PRELOAD." > > > > https://build.kde.org/job/marble%20master%20kf5-qt5/ > > PLATFORM=Linux,compiler=gcc/26/testReport/ > > > > As Marble tries to be optionally Qt-only by tradition, also the current > > Qt5(/ KF5)-based version has lots of custom CMake logic, for full > > independence, and only falls back to using ECM macros/settings when it > > comes to Plasma & other KDE system integration code. > > > > Which currently also means all tests are not influenced anywhere by ECM > > macros. Incl. KDECompilerSettings.cmake (and thus > > ECMEnableSanitizers.cmake). So the code from ECMEnableSanitizers.cmake > > which handles the > > "ECM_ENABLE_SANITIZERS" cmake argument is never run. > > > > Where currently -DECM_ENABLE_SANITIZERS='address' seems to be passed in > > the > > kf5-qt5 configuration: > > https://quickgit.kde.org/?p=sysadmin%2Fci-builder-tools.git&a=blob&f=build > > %2Fkf5-qt5.cfg > > > > Is this the reason that all the Marble tests fail on KDE CI, with the > > given > > error? > > > > So do any projects which are build on KDE CI need to have > > ECMEnableSanitizers.cmake included? > > > > Would it make sense to have ASan as an option to be turned off? > > It's compile time, it's off for your project, but you're linking against KF5 > libraries that have ASAN compiled in. > > > And is that possible, or is ASan usage viral (if deps built on CI have it, > > it needs to be used)? > > Yes ASan usage is viral-ish, if you're using a library that was compiled > with ASAN you either need to compile your binary with asan too or pass the > LD_PRELOAD as the error says, that may be tricky since it needs a full > path. > > While using ASan seems to be useful for improved test coverage, this > > requirement still would need to be explained and documented somewhere, > > please. > > Where would you document it? Right now AFAIK the only documentation about CI is at https:// sysadmin.kde.org/services/continuous-integration/ Though perhaps it makes sense to move that over to somewhere below https:// community.kde.org/Infrastructure, both to improve finding it and to lower the burden for non-sysadmin people to maintain the pages (possibly still need higher edit restrictions to minimize wrong information there). Sysadmin & else, what do you think? Cheers Friedrich
Re: KDE CI enforcing ECMEnableSanitizers.cmake/KDECompilerSettings.cmake?
Am Montag, 25. April 2016, 23:26:45 CEST schrieb Albert Astals Cid: > El dilluns, 25 d’abril de 2016, a les 10:59:13 CEST, Friedrich W. H. > Kossebau > va escriure: > > Hi, > > > > can anyone shed full light on KDE CI and the usage of ASan? > > > > Currently e.g. all tests of Marble are failing, with the message > > "ASan runtime does not come first in initial library list; you should > > either link runtime to your application or manually preload it with > > LD_PRELOAD." > > > > https://build.kde.org/job/marble%20master%20kf5-qt5/ > > PLATFORM=Linux,compiler=gcc/26/testReport/ > > > > As Marble tries to be optionally Qt-only by tradition, also the current > > Qt5(/ KF5)-based version has lots of custom CMake logic, for full > > independence, and only falls back to using ECM macros/settings when it > > comes to Plasma & other KDE system integration code. > > > > Which currently also means all tests are not influenced anywhere by ECM > > macros. Incl. KDECompilerSettings.cmake (and thus > > ECMEnableSanitizers.cmake). So the code from ECMEnableSanitizers.cmake > > which handles the > > "ECM_ENABLE_SANITIZERS" cmake argument is never run. > > > > Where currently -DECM_ENABLE_SANITIZERS='address' seems to be passed in > > the > > kf5-qt5 configuration: > > https://quickgit.kde.org/?p=sysadmin%2Fci-builder-tools.git&a=blob&f=build > > %2Fkf5-qt5.cfg > > > > Is this the reason that all the Marble tests fail on KDE CI, with the > > given > > error? > > > > So do any projects which are build on KDE CI need to have > > ECMEnableSanitizers.cmake included? > > > > Would it make sense to have ASan as an option to be turned off? > > It's compile time, it's off for your project, but you're linking against KF5 > libraries that have ASAN compiled in. > > > And is that possible, or is ASan usage viral (if deps built on CI have it, > > it needs to be used)? > > Yes ASan usage is viral-ish, if you're using a library that was compiled > with ASAN you either need to compile your binary with asan too or pass the > LD_PRELOAD as the error says, that may be tricky since it needs a full > path. Given we have stable setting on CI, the path might be known perhaps. What needs to be done with LD_PRELOAD here exactly? Perhaps that could be done optionally based on a flag in the build setup in the future. Requiring ECMEnableSanitizers.cmake (or equivalent for non-cmake-based buildsystem) for any projects on KDE CI to have tests being run properly (if using other KDE CI products) feels like a small obstacle. If possible without too much pain, it might be nice to avoid that. (For Marble nevertheless I have a patch up for review to optionally use ECMEnableSanitizers.cmake) Cheers Friedrich
Re: KDE CI enforcing ECMEnableSanitizers.cmake/KDECompilerSettings.cmake?
El dimarts, 26 d’abril de 2016, a les 18:50:55 CEST, Friedrich W. H. Kossebau va escriure: > Am Montag, 25. April 2016, 23:26:45 CEST schrieb Albert Astals Cid: > > El dilluns, 25 d’abril de 2016, a les 10:59:13 CEST, Friedrich W. H. > > Kossebau > > > > va escriure: > > > Hi, > > > > > > can anyone shed full light on KDE CI and the usage of ASan? > > > > > > Currently e.g. all tests of Marble are failing, with the message > > > "ASan runtime does not come first in initial library list; you should > > > either link runtime to your application or manually preload it with > > > LD_PRELOAD." > > > > > > https://build.kde.org/job/marble%20master%20kf5-qt5/ > > > PLATFORM=Linux,compiler=gcc/26/testReport/ > > > > > > As Marble tries to be optionally Qt-only by tradition, also the current > > > Qt5(/ KF5)-based version has lots of custom CMake logic, for full > > > independence, and only falls back to using ECM macros/settings when it > > > comes to Plasma & other KDE system integration code. > > > > > > Which currently also means all tests are not influenced anywhere by ECM > > > macros. Incl. KDECompilerSettings.cmake (and thus > > > ECMEnableSanitizers.cmake). So the code from ECMEnableSanitizers.cmake > > > which handles the > > > "ECM_ENABLE_SANITIZERS" cmake argument is never run. > > > > > > Where currently -DECM_ENABLE_SANITIZERS='address' seems to be passed in > > > the > > > kf5-qt5 configuration: > > > https://quickgit.kde.org/?p=sysadmin%2Fci-builder-tools.git&a=blob&f=bui > > > ld > > > %2Fkf5-qt5.cfg > > > > > > Is this the reason that all the Marble tests fail on KDE CI, with the > > > given > > > error? > > > > > > So do any projects which are build on KDE CI need to have > > > ECMEnableSanitizers.cmake included? > > > > > > Would it make sense to have ASan as an option to be turned off? > > > > It's compile time, it's off for your project, but you're linking against > > KF5 libraries that have ASAN compiled in. > > > > > And is that possible, or is ASan usage viral (if deps built on CI have > > > it, > > > it needs to be used)? > > > > Yes ASan usage is viral-ish, if you're using a library that was compiled > > with ASAN you either need to compile your binary with asan too or pass the > > LD_PRELOAD as the error says, that may be tricky since it needs a full > > path. > > Given we have stable setting on CI, the path might be known perhaps. > > What needs to be done with LD_PRELOAD here exactly? Perhaps that could be > done optionally based on a flag in the build setup in the future. You set it to the asan library before running your binary. > Requiring ECMEnableSanitizers.cmake (or equivalent for non-cmake-based > buildsystem) for any projects on KDE CI to have tests being run properly (if > using other KDE CI products) feels like a small obstacle. > If possible without too much pain, it might be nice to avoid that. It's hard to avoid unless you compile all libraries twice both with and without ASAN. It is not a requirement for "any projects on KDE CI". As pointed you can just LD_PRELOAD the ASAN library if you have troubles. Also that is only needed if you use other projects that are linked against ASAN. That is, you're trying not to use ECM (or ECMEnableSanitizers.cmake) but you depend on KF5 stuff that depends on ECM/ECMEnableSanitizers.cmake, maybe you're trying to be too fine in not needing ECM. Cheers, Albert > > (For Marble nevertheless I have a patch up for review to optionally use > ECMEnableSanitizers.cmake) > > Cheers > Friedrich
Re: KRandom broken?
El dilluns, 25 d’abril de 2016, a les 21:38:29 CEST, Frederik Schwarzer va escriure: > Hi, > > it seems as if KRandom is currently broken. > https://bugs.kde.org/show_bug.cgi?id=362161 > > Given the potential impact, is there something to be done (besides > fixing it) wrt distro packagers? Send an email to the distro packagers mailing list and let them decide if they care or not :D Cheers, Albert > > Regards
Re: KDE CI enforcing ECMEnableSanitizers.cmake/KDECompilerSettings.cmake?
Am Dienstag, 26. April 2016, 23:21:40 CEST schrieb Albert Astals Cid: > El dimarts, 26 d’abril de 2016, a les 18:50:55 CEST, Friedrich W. H. > Kossebau > va escriure: > > Am Montag, 25. April 2016, 23:26:45 CEST schrieb Albert Astals Cid: > > > El dilluns, 25 d’abril de 2016, a les 10:59:13 CEST, Friedrich W. H. > > > Kossebau > > > > > > va escriure: > > > > So do any projects which are build on KDE CI need to have > > > > ECMEnableSanitizers.cmake included? > > > > > > > > Would it make sense to have ASan as an option to be turned off? > > > > > > It's compile time, it's off for your project, but you're linking against > > > KF5 libraries that have ASAN compiled in. > > > > > > > And is that possible, or is ASan usage viral (if deps built on CI have > > > > it, > > > > it needs to be used)? > > > > > > Yes ASan usage is viral-ish, if you're using a library that was compiled > > > with ASAN you either need to compile your binary with asan too or pass > > > the > > > LD_PRELOAD as the error says, that may be tricky since it needs a full > > > path. > > > > Given we have stable setting on CI, the path might be known perhaps. > > > > What needs to be done with LD_PRELOAD here exactly? Perhaps that could be > > done optionally based on a flag in the build setup in the future. > > You set it to the asan library before running your binary. Okay, that sounds doable. "Just" needs someone to add support for that :) Filed a feature request on phabricator for that, anyone interested to earn some laurels on that? https://phabricator.kde.org/T2366 > > Requiring ECMEnableSanitizers.cmake (or equivalent for non-cmake-based > > buildsystem) for any projects on KDE CI to have tests being run properly > > (if using other KDE CI products) feels like a small obstacle. > > If possible without too much pain, it might be nice to avoid that. > > It's hard to avoid unless you compile all libraries twice both with and > without ASAN. Double builds ideally can be avoided. And after all ASAN is very much useful on CI, that's why you(?) pushed it there, for some good catches already (myself could also harvest some bugs from it already) :) > It is not a requirement for "any projects on KDE CI". > > As pointed you can just LD_PRELOAD the ASAN library if you have troubles. > Also that is only needed if you use other projects that are linked against > ASAN. As I understood it so far, almost anything built on KDE CI is built with ASAN, incl. 3rd-party dependencies like Qt5, no? As said above, it only makes sense to me to do it. But we need to also have support for the non-ECM-using projects, that's what I am here for and try to understand how things are done, to see what could be done. > That is, you're trying not to use ECM (or ECMEnableSanitizers.cmake) but you > depend on KF5 stuff that depends on ECM/ECMEnableSanitizers.cmake, maybe > you're trying to be too fine in not needing ECM. While I personally would also see to use ECM & CMake where possible, we still need to consider at least: a) KF5 products do not require CMake as the buildsystem for any of its consumers. There is extra dance for supporting qmake/pkg-config buildsystems at least. So other potential KDE projects which use KF5 but not CMake cannot use ECM. Ideally there would be similar helpers like ECMEnableSanitizers.cmake one day for them, sure, but those still would be extra dependencies, and any extra dependency increases burdens, so would be nice to have it special settings for KDE CI optional if possible without a big deal. b) Qt5 on CI is also build with ASAN (AFAIK, that is why the all Marble tests, which are Qt5-only, are failing, unless I missed something). So Qt5-only projects (as in: ECM/KF5-less) still would need to have separate support by/ for the KDE CI. Cheers Friedrich
Re: KDE CI enforcing ECMEnableSanitizers.cmake/KDECompilerSettings.cmake?
El dimecres, 27 d’abril de 2016, a les 0:25:30 CEST, Friedrich W. H. Kossebau va escriure: > Am Dienstag, 26. April 2016, 23:21:40 CEST schrieb Albert Astals Cid: > > El dimarts, 26 d’abril de 2016, a les 18:50:55 CEST, Friedrich W. H. > > Kossebau > > > > va escriure: > > > Am Montag, 25. April 2016, 23:26:45 CEST schrieb Albert Astals Cid: > > > > El dilluns, 25 d’abril de 2016, a les 10:59:13 CEST, Friedrich W. H. > > > > Kossebau > > > > > va escriure: > > > > > > > So do any projects which are build on KDE CI need to have > > > > > ECMEnableSanitizers.cmake included? > > > > > > > > > > Would it make sense to have ASan as an option to be turned off? > > > > > > > > It's compile time, it's off for your project, but you're linking > > > > against > > > > KF5 libraries that have ASAN compiled in. > > > > > > > > > And is that possible, or is ASan usage viral (if deps built on CI > > > > > have > > > > > it, > > > > > it needs to be used)? > > > > > > > > Yes ASan usage is viral-ish, if you're using a library that was > > > > compiled > > > > with ASAN you either need to compile your binary with asan too or pass > > > > the > > > > LD_PRELOAD as the error says, that may be tricky since it needs a full > > > > path. > > > > > > Given we have stable setting on CI, the path might be known perhaps. > > > > > > What needs to be done with LD_PRELOAD here exactly? Perhaps that could > > > be > > > done optionally based on a flag in the build setup in the future. > > > > You set it to the asan library before running your binary. > > Okay, that sounds doable. "Just" needs someone to add support for that :) > Filed a feature request on phabricator for that, anyone interested to earn > some laurels on that? > https://phabricator.kde.org/T2366 > > > > Requiring ECMEnableSanitizers.cmake (or equivalent for non-cmake-based > > > buildsystem) for any projects on KDE CI to have tests being run properly > > > (if using other KDE CI products) feels like a small obstacle. > > > If possible without too much pain, it might be nice to avoid that. > > > > It's hard to avoid unless you compile all libraries twice both with and > > without ASAN. > > Double builds ideally can be avoided. And after all ASAN is very much useful > on CI, that's why you(?) pushed it there, for some good catches already > (myself could also harvest some bugs from it already) :) > > > It is not a requirement for "any projects on KDE CI". > > > > As pointed you can just LD_PRELOAD the ASAN library if you have troubles. > > Also that is only needed if you use other projects that are linked against > > ASAN. > > As I understood it so far, almost anything built on KDE CI is built with > ASAN, incl. 3rd-party dependencies like Qt5, no? "only" the kf5-qt5 jobs that use cmake get built with ASAN > As said above, it only > makes sense to me to do it. But we need to also have support for the > non-ECM-using projects, that's what I am here for and try to understand how > things are done, to see what could be done. > > > That is, you're trying not to use ECM (or ECMEnableSanitizers.cmake) but > > you depend on KF5 stuff that depends on ECM/ECMEnableSanitizers.cmake, > > maybe you're trying to be too fine in not needing ECM. > > While I personally would also see to use ECM & CMake where possible, we > still need to consider at least: > > a) KF5 products do not require CMake as the buildsystem for any of its > consumers. There is extra dance for supporting qmake/pkg-config buildsystems > at least. So other potential KDE projects which use KF5 but not CMake > cannot use ECM. Ideally there would be similar helpers like > ECMEnableSanitizers.cmake one day for them, sure, but those still would be > extra dependencies, and any extra dependency increases burdens, so would be > nice to have it special settings for KDE CI optional if possible without a > big deal. > > b) Qt5 on CI is also build with ASAN (AFAIK, that is why the all Marble > tests, which are Qt5-only, are failing, unless I missed something). No, Qt5 is not built with ASAN on CI. It is strange that your Qt5-only tests are failing, may it be that they are loading some plugin that is linked against some KF5 lib? > So > Qt5-only projects (as in: ECM/KF5-less) still would need to have separate > support by/ for the KDE CI. Qt5 projects that don't use ECM nor KF5 should be fine. Cheers, Albert > > Cheers > Friedrich
Re: KDE CI enforcing ECMEnableSanitizers.cmake/KDECompilerSettings.cmake?
On Wednesday, 27 April 2016 01:21:22 CEST, Albert Astals Cid wrote: It is strange that your Qt5-only tests are failing, may it be that they are loading some plugin that is linked against some KF5 lib? Qt guesses what platform one is running on in order to blend with it. In KDE and under the Plasma desktop, this involves loading plugins/platformthemes/KDEPlatformTheme.so which belongs to KF5's frameworkintegration. Is the KDE CI setting some variables which might trigger loading of these variables? Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: KDE CI enforcing ECMEnableSanitizers.cmake/KDECompilerSettings.cmake?
Am Mittwoch, 27. April 2016, 01:29:06 CEST schrieb Jan Kundrát: > On Wednesday, 27 April 2016 01:21:22 CEST, Albert Astals Cid wrote: > > No, Qt5 is not built with ASAN on CI. Okay, good to know. (next to the failing tests I also remembered to have seen ASAN_OPTIONS detect_leaks=0 as env setting with the qt5 builds on CI. So that is just default env setup and those vars ignored otherwise, okay). Next to documenting things, can we start with some rule what gets or should be built with ASAN, so people know what to expect? I would assume: any KDE software based on C++/C. And then there might be a few exceptions, for whatever reasons (built screwed, incompatible, etc). > > It is strange that your Qt5-only tests are failing, may it be that they > > are > > loading some plugin that is linked against some KF5 lib? For Marble plugins only if something is not like it should be: for one, the current build on CI even gets "-DQTONLY=TRUE" passed, which is turned into the cmake var "set(WITH_KF5 FALSE)", and all KF5 deps are looked for with "macro_optional_find_package(KF5 QUIET COMPONENTS something)", so should always yield NOT_FOUND (looking at it I found a bug which still made KF5DocTools to be found, but now fixed and should not have any effect on linking to KF5 libs). More, the only things being explicitely linked to KF5 libs are KF5 thumbnailer plugin, KRunner plugin, and the KF5-enhanced Marble app variant. So nothing which should be used in the tests. Ah, Phonon! That is linked by at least the RoutingPlugin. And Phonon gets instrumented with ASAN: https://build.kde.org/job/phonon%20master%20kf5-qt5/ PLATFORM=Linux,compiler=gcc/5/console That might explain what we see currently. > Qt guesses what platform one is running on in order to blend with it. In > KDE and under the Plasma desktop, this involves loading > plugins/platformthemes/KDEPlatformTheme.so which belongs to KF5's > frameworkintegration. > > Is the KDE CI setting some variables which might trigger loading of these > plugins [edited]? Good idea, that might be indeed other intrusion path for ASAN deps. @Scarlett, can you tell? Cheers Friedrich