On Mon, Feb 7, 2022 at 10:56 PM Christoph Cullmann (cullmann.io) < christ...@cullmann.io> wrote:
> On 2022-02-07 10:35, Ben Cooksley wrote: > > On Sun, Feb 6, 2022 at 10:40 PM Friedrich W. H. Kossebau > > <kosse...@kde.org> wrote: > > > >> Am Montag, 24. Januar 2022, 01:06:40 CET schrieb Friedrich W. H. > >> Kossebau: > >>> Hi, > >>> > >>> since a long time there are lots of failing unit tests across > >> multiple > >>> repositories. Could the Windows platform maintainers/stakeholders > >> please > >>> look soonish into either fixing those tests or properly marking > >> them as > >>> expected to fail, so the resources the KDE CI spends on running > >> the tests > >>> every hour, day and week make some sense again, as well as having > >> something > >>> usable to diff results again, to notice any new regressions? > >>> > >>> Please see > >>> > >>> > >> > > > https://build.kde.org/job/Frameworks/view/Platform%20-%20WindowsMSVCQt5.15/ > >>> (best sort by "S" build status to get a list what need > >> > >> And those who believe in the broken windows theory also would claim > >> this > >> slacking now resulted in the regressions in the openSUSE builds, > >> where 5 > >> modules now have failing unit tests at time of release tagging, when > >> it once > >> was 0 thanks to hard work of David F. and others. :( > >> > >> Is it time to remove > >> https://community.kde.org/Frameworks/ > >> Policies#Frameworks_CI_failures_are_treated_as_stop_the_line_events > >> because seemingly this is just old dead pixels on a web page and not > >> the > >> spirit these days? > > > > Not sure that is the ideal outcome here - preferrably our tests would > > continue to all pass. > > > > I know some tests on certain platforms have been flaky and switch > > between failing/passing - are we sure that isn't the driver of people > > ignoring the results? > > One thing that sometimes lead me to ignore stuff with KTextEditor is > that the UI > tests are often very unstable. > > e.g. they just fail for me locally but then work perfectly in the CI or > the other > way around. > > Not sure how to improve that. > Interesting to note that it is GUI/UI tests that are causing issues. I would have thought that the setup on the CI for those would be a carbon copy almost every time which makes these failures interesting. Out of curiosity, what are the tests trying to accomplish and where is it failing? > > For all non-UI tests naturally no such problems exist for KTextEditor > and they are easy to keep > working. > > KSyntaxHighlighting only has non-UI test and that is very easy to keep > in a consistent shape. > > Greetings > Christoph > Thanks, Ben > > > > > Cheers, > > Ben > > > >> Friedrich > > -- > Ignorance is bliss... > https://cullmann.io | https://kate-editor.org >