Re: [gentoo-dev] Stabilizations and src_test
On Sun, 12 Apr 2020 10:43:07 +0200 Agostino Sarubbo wrote: > In case of 'yes', the arch team member must compile with FEATURE="test" and > he > is allowed to block the stabilization in case of test-failure. > > In case there will be a test-failure there are two choices: > 1) The maintainer fixes the test failure; > 2) The maintainer does not want to take care, so he can simply remove the > blocker and set "src_test pass" to no. > > Both of the above are fine for me. > > Obviously, if there are already test-failure bugs open, the flag "src_test > pass" should be set to 'no' when the stabilization bugs is filled. > > I'm sure that this way would help to avoid wasting time on both sides. > > What do you think about? Agree mostly, but this problem is in two places. Ebuilds themselves need more mechanisms to communicate this, as if an ebuild has problematic tests, then they're likely to be problematic every time, and remembering to set the appropriate bugzilla flags becomes a weakness. In this vein, there's lots of similar inadequacies you find if you follow this thread. Like for instance, I maintain a lot of packages where the ability for the tests to work is *environment specific*, and there's ultimately different *kinds* of testing, and the audiences who run each varies. Take perl packages for instance. There's a few where the test suite *cannot* work, because they need either network IO or access to physical devices to function. But simply disabling tests because of this produces a gross weakness, because Perl code cannot even be considered to be compatible with the target perl until you actually execute it, because those incompatibilities are ultimately "compiler errors", and you only see them when you spin up a runtime, unlike code with a dedicated compile stage. Subsequently, I've taken to bolting on ebuild-internal hacks that simply enumerate the modules and forces a perl instance to try loading them, to at least provide *some* degree of assurance that the code *probably* works. But ultimately this means the way I see it, every ebuild could have *multiple* test targets, and could have a control variable indicating which targets to execute by default, and consumer knobs could expand, or reduce, the testing based on preference. We could even have something like say: TEST_TARGETS="+build integration" TEST_RESTRICT="network-sandbox? ( !root? ( integration ))" Things like say, mysql, which has a standard test suite that takes days to run, might be well served by having controls like this, so consumers who want to have *some* basic assurance code works, without paying the price for a comprehensive smoke, can do so, instead of opting to choose "tests are hard, lets not do them". Just I've been cautious about talking about this, because this is such a can of worms, and leads into the debate about TDEPEND, and how we stuff everything into USE flags... And I really would *hate* for testing control variables to be USE flags, where --changed-use rampantly introduces unnecessary havoc ( And the eventual fact will be there will be code that tries to depend on specific values of these test control flags, and code that wants to omit dependencies based on them, and portage complexity goes cray cray ) pgpctmi0XWqfF.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Stabilizations and src_test
On Sun, 12 Apr 2020 11:21:29 +0200 Michał Górny wrote: > On Sun, 2020-04-12 at 10:43 +0200, Agostino Sarubbo wrote: > > Hello all, > > > > If you work on the stabilization workflow you may have noticed that: > > > > - There are people that rant if you don't run src_test against their > > packages; > > - There are people that rant if you open a test failure bug against their > > packages and you block the stabilization. > > > > So, unless there will be a clear policy about that, I'd suggest to > > introduce a > > way to establish if a package is expected to pass src_test or not. > > > > A simple way to achieve this goal would be: > > introduce a new bugzilla custom flag (like "Runtime testing required" we > > already have) maybe called "src_test pass" or similar, that, by default is > > yes > > and the maintainer should set it to no when needed. > > > > In case of 'yes', the arch team member must compile with FEATURE="test" and > > he > > is allowed to block the stabilization in case of test-failure. > > > > In case there will be a test-failure there are two choices: > > 1) The maintainer fixes the test failure; > > 2) The maintainer does not want to take care, so he can simply remove the > > blocker and set "src_test pass" to no. > > > > Both of the above are fine for me. > > > > Obviously, if there are already test-failure bugs open, the flag "src_test > > pass" should be set to 'no' when the stabilization bugs is filled. > > > > This is not a problem that can be solved by a binary flag. > > If package's test suite is entirely broken and unmaintained, the package > should use RESTRICT=test and not silently ask arch teams to ignore it. > > If package's test suite is only slightly broken, then I'd prefer saying > 'please report but ignore *these* test failures' because I can't fix > them right now but they don't seem major. Skipping the test suite > entirely is not a solution because it doesn't disambiguate between 'few > tests fail' and 'every single test explodes'. > I agree with this, the metric for marking an ebuild as "tests don't work, please ignore them is RESTRICT=test". Usually for packages that have a *few* tests that fail and don't seem major, I would suggest trying to patch out the tests, or ask the test harness to skip the known bad tests. This way the tinderbox will (hopefully) still succeed.
Re: [gentoo-dev] Stabilizations and src_test
On Sunday, 12 April 2020 11:33:38 CEST Thomas Deutschmann wrote: > On Sunday, 12 April 2020 10:43:07 CEST Agostino Sarubbo wrote: > > - There are people that rant if you open a test failure bug against their > > packages and you block the stabilization. > > Maybe start ignoring those packages until people learn that > stabilization is a lot of effort/work. Really, if you call for > stabilization and haven't tested your own package you are offloading > work to others which is not nice. I also dislike maintainers who simply > restrict tests on first failure. But in the end it's at least a strong > signal about package quality and state in Gentoo. :) Not so fast. Let's not forget that so many tests are fragile and not even easy to reproduce on a different system. And then there are those that randomly break by a dependency bump after the fact, we don't have continuous build CI so no one is to blame. It sends more of a signal about upstream package quality than Gentoo's, or that they just do not care to make tests work for packaging environments, and it is not a good use of our time to make it so. I agree then that it is better to simply restrict if we know the tests fail, rather than introduce a new mechanism. And when I look at the amount of foreign (to me) packages I need to call stabilisation for because their maintainer didn't, then that work has already been offloaded to me, so we're in the same boat. People may rant when their stabilisation is blocked by failing tests, but let me recommend to not take that personally when it most likely is just a big sigh at the circumstances/package in question. Regards, Andreas
Re: [gentoo-dev] Stabilizations and src_test
On Sun, 12 Apr 2020 10:43:07 +0200 Agostino Sarubbo wrote: > If you work on the stabilization workflow you may have noticed that: > > - There are people that rant if you don't run src_test against their packages; > - There are people that rant if you open a test failure bug against their > packages and you block the stabilization. > > So, unless there will be a clear policy about that, I'd suggest to introduce > a > way to establish if a package is expected to pass src_test or not. > > A simple way to achieve this goal would be: > introduce a new bugzilla custom flag (like "Runtime testing required" we > already have) maybe called "src_test pass" or similar, that, by default is > yes > and the maintainer should set it to no when needed. > > In case of 'yes', the arch team member must compile with FEATURE="test" and > he > is allowed to block the stabilization in case of test-failure. > > In case there will be a test-failure there are two choices: > 1) The maintainer fixes the test failure; > 2) The maintainer does not want to take care, so he can simply remove the > blocker and set "src_test pass" to no. > > Both of the above are fine for me. > > Obviously, if there are already test-failure bugs open, the flag "src_test > pass" should be set to 'no' when the stabilization bugs is filled. > > I'm sure that this way would help to avoid wasting time on both sides. Expecting the package to be marked stable while the tests fail seems like a very odd stance to take. I would politely request that you flag this up and if you get pushback but understandably don't want to fight it then just add RESTRICT="test" yourselves, maybe filtered on a particular USE flag if it only happens in specific situations. I think the QA team would back you up here? I understand the comments about only a few minor tests failing but if the maintainer doesn't want to skip the entire suite then it is their responsibility to limit the set of tests run by whatever means. Unfortunately there is no universal method of doing this. -- James Le Cuirot (chewi) Gentoo Linux Developer pgpS9YffxZvr0.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Stabilizations and src_test
On 2020-04-12 11:21, Michał Górny wrote: > This is not a problem that can be solved by a binary flag. > > If package's test suite is entirely broken and unmaintained, the package > should use RESTRICT=test and not silently ask arch teams to ignore it. > > If package's test suite is only slightly broken, then I'd prefer saying > 'please report but ignore *these* test failures' because I can't fix > them right now but they don't seem major. Skipping the test suite > entirely is not a solution because it doesn't disambiguate between 'few > tests fail' and 'every single test explodes'. ACK. I also see no need for any new mechanism. > - There are people that rant if you open a test failure bug against their > packages and you block the stabilization. Maybe start ignoring those packages until people learn that stabilization is a lot of effort/work. Really, if you call for stabilization and haven't tested your own package you are offloading work to others which is not nice. I also dislike maintainers who simply restrict tests on first failure. But in the end it's at least a strong signal about package quality and state in Gentoo. :) -- Regards, Thomas Deutschmann / Gentoo Linux Developer C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Stabilizations and src_test
On Sun, 2020-04-12 at 10:43 +0200, Agostino Sarubbo wrote: > Hello all, > > If you work on the stabilization workflow you may have noticed that: > > - There are people that rant if you don't run src_test against their packages; > - There are people that rant if you open a test failure bug against their > packages and you block the stabilization. > > So, unless there will be a clear policy about that, I'd suggest to introduce > a > way to establish if a package is expected to pass src_test or not. > > A simple way to achieve this goal would be: > introduce a new bugzilla custom flag (like "Runtime testing required" we > already have) maybe called "src_test pass" or similar, that, by default is > yes > and the maintainer should set it to no when needed. > > In case of 'yes', the arch team member must compile with FEATURE="test" and > he > is allowed to block the stabilization in case of test-failure. > > In case there will be a test-failure there are two choices: > 1) The maintainer fixes the test failure; > 2) The maintainer does not want to take care, so he can simply remove the > blocker and set "src_test pass" to no. > > Both of the above are fine for me. > > Obviously, if there are already test-failure bugs open, the flag "src_test > pass" should be set to 'no' when the stabilization bugs is filled. > This is not a problem that can be solved by a binary flag. If package's test suite is entirely broken and unmaintained, the package should use RESTRICT=test and not silently ask arch teams to ignore it. If package's test suite is only slightly broken, then I'd prefer saying 'please report but ignore *these* test failures' because I can't fix them right now but they don't seem major. Skipping the test suite entirely is not a solution because it doesn't disambiguate between 'few tests fail' and 'every single test explodes'. -- Best regards, Michał Górny signature.asc Description: This is a digitally signed message part
Re: [gentoo-dev] Stabilizations and src_test
On 2020-04-12 09:43, Agostino Sarubbo wrote: > If you work on the stabilization workflow you may have noticed that: > > - There are people that rant if you don't run src_test against their packages; > - There are people that rant if you open a test failure bug against their > packages and you block the stabilization. > > So, unless there will be a clear policy about that, I'd suggest to introduce > a > way to establish if a package is expected to pass src_test or not. My first reaction was "not against this, even though I personally very much try (as in, that I occasionally fail or forget) to make sure even unstable ebuilds I push into the tree pass all tests on amd64", having thought about it more however it feels like lowering quality standards for stable arch trees. Test failures in ~arch are one thing but I would be somewhat suspicious of stabilising anything which fails tests unless it is a known (as in, known to the upstream) issue with no easy solution. There is also the slight image problem which could arise from a user running tests themselves and reporting the failure back to us, with the "I thought this was supposed to be *stable*, guys?" undertone. In summary, I am not entirely decided but leaning towards continuing to always run tests during stabilisation. -- Marecki signature.asc Description: OpenPGP digital signature
[gentoo-dev] Stabilizations and src_test
Hello all, If you work on the stabilization workflow you may have noticed that: - There are people that rant if you don't run src_test against their packages; - There are people that rant if you open a test failure bug against their packages and you block the stabilization. So, unless there will be a clear policy about that, I'd suggest to introduce a way to establish if a package is expected to pass src_test or not. A simple way to achieve this goal would be: introduce a new bugzilla custom flag (like "Runtime testing required" we already have) maybe called "src_test pass" or similar, that, by default is yes and the maintainer should set it to no when needed. In case of 'yes', the arch team member must compile with FEATURE="test" and he is allowed to block the stabilization in case of test-failure. In case there will be a test-failure there are two choices: 1) The maintainer fixes the test failure; 2) The maintainer does not want to take care, so he can simply remove the blocker and set "src_test pass" to no. Both of the above are fine for me. Obviously, if there are already test-failure bugs open, the flag "src_test pass" should be set to 'no' when the stabilization bugs is filled. I'm sure that this way would help to avoid wasting time on both sides. What do you think about? -- Agostino
