Re: Unit tests all pass in Jenkins on Linux
On Mon, Mar 14, 2022 at 4:40 AM David Faure wrote: > After the recent discussions on state of CI, I fixed the last unittest > failures (kio, purpose... + apol fixed ECM) so that > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/ > is all green^H^Hblue again. > Please keep it that way! > Thanks for looking into and fixing all of these David. > > Note however that > > * kwayland has a flaky test: > > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/ > > FAIL! : TestDataDevice::testReplaceSource() Compared values are not the > same >Actual (selectionOfferedSpy.count()): 1 >Expected (2) : 2 >Loc: [autotests/client/test_datadevice.cpp(557)] > > Who can look at this one? git log mostly shows Martin Flöser < > mgraess...@kde.org> > who I think isn't active anymore? > Not sure if it applies to KWayland as well, but I know that KWin has load sensitive tests (which is why the Gitlab .kde-ci.yml files support the flag tests-load-sensitive) If this test appears to be flaky, then it is quite possible that it is load sensitive as well. > > * krunner has a flaky test [2] because it measures time spent and expects > small values like 65ms > (I changed that one to 100ms), 250ms, 300ms. With only 10% safety margins. > On a busy CI system, > this is bound to fail regularly, even with bigger safety margins. In my > experience this kind of test > is just not possible (we're not running on a real time OS), I vote for > removing the test. > CC'ing Eduardo. > > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/ Yes, that will definitely fail more often than not - your only way to make sure tests like this pass on our CI system is to set tests-load-sensitive=True (in Gitlab CI) Note however that option should be avoided where possible as it means your build will stop and wait for load to fall to low levels before proceeding with running tests - which blocks a CI worker slot from being used by another project. I'd also be in favour of removing this test. > > -- > David Faure, fa...@kde.org, http://www.davidfaure.fr > Working on KDE Frameworks 5 > > > > Cheers, Ben
Re: Unit tests all pass in Jenkins on Linux
On Mon, Mar 14, 2022 at 4:44 PM Eduardo de Souza Cruz < eduardo.c...@kdemail.net> wrote: > Hi, > > Regarding the krunner timer-based test which I authored: > > We could just delete the few lines that are enforcing those upper-limit > timeouts and add some comments explaining that in real life those timeouts > should have been met, I don't think we need to delete the entire test. I > was explicitly asked to write a test when I submitted the MR to enforce > that this functionally wouldn't regress in the future. > > Also, I'm wondering... if we happen to have a compilation directive that's > #defined in this CI environment only, or some framework function that > returns weather we are in the CI environment or not, we could put some > #ifndef (or if's) on those few upper-limit timeout lines and avoid > compiling/running just those lines in the CI environment. I'm not familiar > with this environment so I don't know if there is such a thing, I'm just > wondering... > Not at this time i'm afraid. There are some environment variables you can look to but those differ between our Jenkins and Gitlab setups. > > I'm available to submit a quick MR if you'd like me to, just give me some > directions for what you'd like me to do. > > [], > Eduardo > Cheers, Ben > > ------ > *From:* Ben Cooksley > *Sent:* Sunday, March 13, 2022 1:53 PM > *To:* KDE Frameworks > *Cc:* eduardo.c...@kdemail.net > *Subject:* Re: Unit tests all pass in Jenkins on Linux > > On Mon, Mar 14, 2022 at 4:40 AM David Faure wrote: > > After the recent discussions on state of CI, I fixed the last unittest > failures (kio, purpose... + apol fixed ECM) so that > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/ > is all green^H^Hblue again. > Please keep it that way! > > > Thanks for looking into and fixing all of these David. > > > > Note however that > > * kwayland has a flaky test: > > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/ > > FAIL! : TestDataDevice::testReplaceSource() Compared values are not the > same >Actual (selectionOfferedSpy.count()): 1 >Expected (2) : 2 >Loc: [autotests/client/test_datadevice.cpp(557)] > > Who can look at this one? git log mostly shows Martin Flöser < > mgraess...@kde.org> > who I think isn't active anymore? > > > Not sure if it applies to KWayland as well, but I know that KWin has load > sensitive tests (which is why the Gitlab .kde-ci.yml files support the flag > tests-load-sensitive) > If this test appears to be flaky, then it is quite possible that it is > load sensitive as well. > > > > * krunner has a flaky test [2] because it measures time spent and expects > small values like 65ms > (I changed that one to 100ms), 250ms, 300ms. With only 10% safety margins. > On a busy CI system, > this is bound to fail regularly, even with bigger safety margins. In my > experience this kind of test > is just not possible (we're not running on a real time OS), I vote for > removing the test. > CC'ing Eduardo. > > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/ > > > Yes, that will definitely fail more often than not - your only way to make > sure tests like this pass on our CI system is to > set tests-load-sensitive=True (in Gitlab CI) > Note however that option should be avoided where possible as it means your > build will stop and wait for load to fall to low levels before proceeding > with running tests - which blocks a CI worker slot from being used by > another project. > > I'd also be in favour of removing this test. > > > > -- > David Faure, fa...@kde.org, http://www.davidfaure.fr > Working on KDE Frameworks 5 > > > > > Cheers, > Ben >
Re: Unit tests all pass in Jenkins on Linux
Hi, Regarding the krunner timer-based test which I authored: We could just delete the few lines that are enforcing those upper-limit timeouts and add some comments explaining that in real life those timeouts should have been met, I don't think we need to delete the entire test. I was explicitly asked to write a test when I submitted the MR to enforce that this functionally wouldn't regress in the future. Also, I'm wondering... if we happen to have a compilation directive that's #defined in this CI environment only, or some framework function that returns weather we are in the CI environment or not, we could put some #ifndef (or if's) on those few upper-limit timeout lines and avoid compiling/running just those lines in the CI environment. I'm not familiar with this environment so I don't know if there is such a thing, I'm just wondering... I'm available to submit a quick MR if you'd like me to, just give me some directions for what you'd like me to do. [], Eduardo From: Ben Cooksley Sent: Sunday, March 13, 2022 1:53 PM To: KDE Frameworks Cc: eduardo.c...@kdemail.net Subject: Re: Unit tests all pass in Jenkins on Linux On Mon, Mar 14, 2022 at 4:40 AM David Faure mailto:fa...@kde.org>> wrote: After the recent discussions on state of CI, I fixed the last unittest failures (kio, purpose... + apol fixed ECM) so that https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/ is all green^H^Hblue again. Please keep it that way! Thanks for looking into and fixing all of these David. Note however that * kwayland has a flaky test: https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/kwayland/job/kf5-qt5%20SUSEQt5.15/171/testReport/junit/projectroot.autotests/client/kwayland_testDataDevice/ FAIL! : TestDataDevice::testReplaceSource() Compared values are not the same Actual (selectionOfferedSpy.count()): 1 Expected (2) : 2 Loc: [autotests/client/test_datadevice.cpp(557)] Who can look at this one? git log mostly shows Martin Flöser mailto:mgraess...@kde.org>> who I think isn't active anymore? Not sure if it applies to KWayland as well, but I know that KWin has load sensitive tests (which is why the Gitlab .kde-ci.yml files support the flag tests-load-sensitive) If this test appears to be flaky, then it is quite possible that it is load sensitive as well. * krunner has a flaky test [2] because it measures time spent and expects small values like 65ms (I changed that one to 100ms), 250ms, 300ms. With only 10% safety margins. On a busy CI system, this is bound to fail regularly, even with bigger safety margins. In my experience this kind of test is just not possible (we're not running on a real time OS), I vote for removing the test. CC'ing Eduardo. https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/job/krunner/job/kf5-qt5%20SUSEQt5.15/325/testReport/junit/projectroot/autotests/runnermanagertest/ Yes, that will definitely fail more often than not - your only way to make sure tests like this pass on our CI system is to set tests-load-sensitive=True (in Gitlab CI) Note however that option should be avoided where possible as it means your build will stop and wait for load to fall to low levels before proceeding with running tests - which blocks a CI worker slot from being used by another project. I'd also be in favour of removing this test. -- David Faure, fa...@kde.org<mailto:fa...@kde.org>, http://www.davidfaure.fr Working on KDE Frameworks 5 Cheers, Ben
Re: Unit tests all pass in Jenkins on Linux
On dimanche 13 mars 2022 17:53:13 CET Ben Cooksley wrote: > On Mon, Mar 14, 2022 at 4:40 AM David Faure wrote: > > After the recent discussions on state of CI, I fixed the last unittest > > failures (kio, purpose... + apol fixed ECM) so that > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/ > > is all green^H^Hblue again. > > Please keep it that way! > > Thanks for looking into and fixing all of these David. Now I'd like to fix the remaining unittest failures on FreeBSD. I just fixed kcrash by reading the unittest code. However for the remaining ones, I need to actually debug on FreeBSD. Is there a FreeBSD virtual machine with the full setup already done for building KDE Frameworks, that I could either run locally or log into? -- David Faure, fa...@kde.org, http://www.davidfaure.fr Working on KDE Frameworks 5
Re: Unit tests all pass in Jenkins on Linux
On Mon, Mar 21, 2022 at 9:43 AM David Faure wrote: > On dimanche 13 mars 2022 17:53:13 CET Ben Cooksley wrote: > > On Mon, Mar 14, 2022 at 4:40 AM David Faure wrote: > > > After the recent discussions on state of CI, I fixed the last unittest > > > failures (kio, purpose... + apol fixed ECM) so that > > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/ > > > is all green^H^Hblue again. > > > Please keep it that way! > > > > Thanks for looking into and fixing all of these David. > > Now I'd like to fix the remaining unittest failures on FreeBSD. > > I just fixed kcrash by reading the unittest code. > However for the remaining ones, I need to actually debug on FreeBSD. > Is there a FreeBSD virtual machine with the full setup already done for > building KDE Frameworks, that I could either run locally or log into? > I believe Tobias may have some instructions on how to assemble a machine (as these are what he used to assemble the VM images currently running the FreeBSD environment on Gitlab and Jenkins). > > -- > David Faure, fa...@kde.org, http://www.davidfaure.fr > Working on KDE Frameworks 5 > > Cheers, Ben
Re: Unit tests all pass in Jenkins on Linux
On Sonntag, 13. März 2022 16:40:54 CET David Faure wrote: > After the recent discussions on state of CI, I fixed the last unittest > failures (kio, purpose... + apol fixed ECM) so that > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/ is all > green^H^Hblue again. > Please keep it that way! Great, thank you! https://invent.kde.org/sysadmin/ci-utilities/-/merge_requests/15 hopefully helps a bit keeping it that way. signature.asc Description: This is a digitally signed message part.
Re: Unit tests all pass in Jenkins on Linux
On Sunday, 20 March 2022 21:43:58 CET David Faure wrote: > On dimanche 13 mars 2022 17:53:13 CET Ben Cooksley wrote: > > On Mon, Mar 14, 2022 at 4:40 AM David Faure wrote: > > > After the recent discussions on state of CI, I fixed the last unittest > > > failures (kio, purpose... + apol fixed ECM) so that > > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/ > > > is all green^H^Hblue again. > > > Please keep it that way! > > > > Thanks for looking into and fixing all of these David. > > Now I'd like to fix the remaining unittest failures on FreeBSD. > > I just fixed kcrash by reading the unittest code. > However for the remaining ones, I need to actually debug on FreeBSD. > Is there a FreeBSD virtual machine with the full setup already done for > building KDE Frameworks, that I could either run locally or log into? I suppose the CI-builders would qualify, but that's probably not a good idea in general. I tried to put together a VM image for KDE development back in 2020: https://euroquis.nl/freebsd/2020/01/11/freebsd.html . An 8GB image is a bit much; I might be able to do a frameworks-development one with no graphical capability in a lot less space, but don't immediately have a place to host it. [ade] (argh, yeah, kmail) signature.asc Description: This is a digitally signed message part.
Re: Unit tests all pass in Jenkins on Linux
On Montag, 21. März 2022 17:31:42 CET Volker Krause wrote: > On Sonntag, 13. März 2022 16:40:54 CET David Faure wrote: > > After the recent discussions on state of CI, I fixed the last unittest > > failures (kio, purpose... + apol fixed ECM) so that > > https://build.kde.org/job/Frameworks/view/Platform%20-%20SUSEQt5.15/ is > > all > > green^H^Hblue again. > > Please keep it that way! > > Great, thank you! > > https://invent.kde.org/sysadmin/ci-utilities/-/merge_requests/15 hopefully > helps a bit keeping it that way. This has been deployed. It's now possible to specify the set of platforms on which we expect unit tests to pass, failing tests will then fail the CI run in the same way compile failures do now. This can be configured both globally (currently an empty list) and per repository in the .kde-ci.yml file. Example for the .kde-ci.yml settings: Options: require-passing-tests-on: [ 'Linux', 'FreeBSD' ] Note that this does not allow a distinction between Linux Qt5 and Linux Qt6 builds at this point. Regards, Volker signature.asc Description: This is a digitally signed message part.
