Re: Could we disable "AutoQA" until it can produce usable results?
On Tue, 2015-03-24 at 22:44 +, Richard W.M. Jones wrote: > Witness (a): > https://admin.fedoraproject.org/updates/libguestfs-1.29.31-1.fc22 > > The log file in this case is 3452 lines long. > > Witness (b): > https://admin.fedoraproject.org/updates/ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22 > > The log file in this case is 3452 lines long (not coincidentally - > it is exactly the same file as above). > > And in both cases the log files are a morass of randomness. It's not really *that* hard to read the results, you can just search for the package name and/or the string 'fail'. I've pasted the relevant bits below. Unfortunately, though, both these are a type of false failure that comes up quite often. Packages which are multiarch and have internal dependencies - that is, multiple intra-dependent subpackages - seem to cause bogus failures. I don't think there's actually an issue with either of your updates, so it'd be fine to re-enable the auto-push. We have a ticket tracking this issue here: https://phab.qadevel.cloud.fedoraproject.org/T366 As for disabling Taskotron (it hasn't been AutoQA for a while now), well, it's a tricky call, because while it does get things wrong sometimes, it also gets things right and saves us from bad updates sometimes too. The ideal solution would be to fix the bug... not ok - depcheck for Bodhi update ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22 # FAIL --- arch: x86_64 details: output: |- Build ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22 failed depcheck package ocaml-camlp4-devel-4.02.0-0.9.git87c6a6b0.fc22.i686 requires ocaml-camlp4(x86-32) = 4.02.0-0.9.git87c6a6b0.fc22, but none of the providers can be installed ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22.i686 has inferior architecture ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22.i686 has inferior architecture package ocaml-camlp4-devel-4.02.0-0.9.git87c6a6b0.fc22.i686 requires ocaml-camlp4(x86-32) = 4.02.0-0.9.git87c6a6b0.fc22, but none of the providers can be installed ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22.i686 has inferior architecture ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22.i686 has inferior architecture item: ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22 outcome: FAILED summary: ocaml-camlp4-4.02.0-0.9.git87c6a6b0.fc22 into F22 testing type: bodhi_update not ok - depcheck for Bodhi update libguestfs-1.29.31-1.fc22# FAIL --- arch: x86_64 details: output: |- Build libguestfs-1.29.31-1.fc22 failed depcheck package libguestfs-tools-1:1.29.31-1.fc22.noarch requires libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture package libguestfs-tools-1:1.29.31-1.fc22.noarch requires libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture package libguestfs-gobject-devel-1:1.29.31-1.fc22.i686 requires libguestfs-gobject = 1:1.29.31-1.fc22, but none of the providers can be installed package libguestfs-gobject-1:1.29.31-1.fc22.i686 requires libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed package libguestfs-gobject-1:1.29.31-1.fc22.i686 requires libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture package libguestfs-devel-1:1.29.31-1.fc22.i686 requires libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture package ocaml-libguestfs-1:1.29.31-1.fc22.i686 requires libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture package ocaml-libguestfs-devel-1:1.29.31-1.fc22.i686 requires ocaml-libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed package ocaml-libguestfs-1:1.29.31-1.fc22.i686 requires libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed package ocaml-libguestfs-1:1.29.31-1.fc22.i686 requires libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture package libguestfs-man-pages-ja-1:1.29.31-1.fc22.noarch requires libguestfs = 1:1.29.31-1.fc22, but none of the providers can be installed libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture libguestfs-1:1.29.31-1.fc22.i686 has inferior architecture
Re: Exit status for runtask should be 1 if task outcome is FAILED?
Dear Kamil, thanks for your prompt reply. On Tue, Mar 24, 2015 at 05:21:44AM -0400, Kamil Paral wrote: > > Hi, > > > > I'd like to enquire if it's desired to let runtask exit with '1' if the > > task in question has a 'FAILED' outcome. > > > > Maybe there was a conscious decision of not providing this feature and > > I'd like to hear about it. > > > > I've joined Taskotron on phabricator. If there are no obvious > > resentments against such an idea, I'm happy to create a task and > > implement it. > > > > Kind Regards, > > -- > > Róman Joost > Moved this up, since I have not explained my motivation in my initial post: > Why exactly do you need this? For local testing? runtask is just an execution > wrapper, so you should be able to run your `./script some args` the same way > as `runtask -i item -t type script.yml`. Except for some set up steps (like > 'koji' or 'bodhi' directive), so if you use some of these, I can see a reason > why you would want to run it through runtask. That's exactly why I thought about runtask. To be more precise, we're currently working on rpmgrill. It ships it's own fetch-build script which is currently tied to Fedoras Koji. IMHO rpmgrill shouldn't be concerned about where the builds come from. If I can use 'runtask' to fetch the builds, let rpmgrill handle the analysis and report back to bodhi or any other similar system that would be very beneficial. Furthermore that can run easily in a Jenkins and the exit status would provide a way for the CI to figure if the package is not shipping with nasty problems. > Hey Róman, > > thanks for joining the project and welcome :) > > We have never really thought about the feature you're talking about. We > currently return non-zero in case the execution fails - ctrl+c, an exception > thrown. If we wanted to extend it to the FAILED task result, I see the > following issues: > > 1. We have other outcomes defined at the moment, like NEEDS_INSPECTION: > https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/library.html#libtaskotron.check.CheckDetail > What should happen in that case? Theoretically it'll be an exit code 1 as well. What I don't know tho is, if it's practical to change it to 1. Perhaps it would mean that currently all packages which are under test would basically 'fail'. You might have a better insight here what the consequences of that change would be. > 2. Some tasks do not check just a single item. For example, depcheck and > upgradepath check everything that is currently proposed for stable or testing > updates. They can return a dozen test results. What should happen when there > are mixed results? I'd assume to default to the worst result. So if any one of them has a a FAILED state it means exit with 1. What I'm not sure about is if it's currently easy to accumulate the worst result case with the current code. Might be easier thought than done. > 3. At the moment, it's technically possible to leave out the 'resultsdb' > directive the from task recipe. I cannot imagine a use case for this, because > in development mode, this only reports to stdout, but it's possible if you > need it. If we wanted to inspect task outcomes, we need to do it from TAP, > therefore in the 'resultsdb' directive. It might be confusing to people that > if you commented out that directive, runtask would stop returning "correct" > return codes. IMHO that's the beauty of Taskotron being decoupled from the execution and the result reporting. My main idea was just using libtaskotron for task execution and perhaps in the future to report to other aggregation systems inside Red Hat (the equivalent to Bodhi). So I thought from a developer point of view, it allows to run build analysers against your package in your CI using Taskotron without the need to report the results back into a results database. I hope I'm not twisting the intention of the whole Taskotron idea. > One further thought, instead of implementing this by default, we can > also create a new directive or variable which will allow task creators > to set the exit code, according to TAP or any other logic they want. That sounds like a good idea as well. Based on that, what we could also do is, add an argument to runtask like: --make-fatal which means if the result of the runtask is not PASSED or INFO it'll exit with 1. Less flexible than your idea, but perhaps less work involved. Kind Regards, -- Róman Joost Software Engineer, PnT DevOps - Developer (Brisbane) email: rjo...@redhat.com | tz: UTC+10 | GPG ID: 0xBE2B559D at pgp.mit.edu irc: #pnt-devops #rpmdiff pgppPErIpEtMq.pgp Description: PGP signature ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
Re: Exit status for runtask should be 1 if task outcome is FAILED?
> Hi, > > I'd like to enquire if it's desired to let runtask exit with '1' if the > task in question has a 'FAILED' outcome. > > Maybe there was a conscious decision of not providing this feature and > I'd like to hear about it. > > I've joined Taskotron on phabricator. If there are no obvious > resentments against such an idea, I'm happy to create a task and > implement it. > > Kind Regards, > -- > Róman Joost Hey Róman, thanks for joining the project and welcome :) We have never really thought about the feature you're talking about. We currently return non-zero in case the execution fails - ctrl+c, an exception thrown. If we wanted to extend it to the FAILED task result, I see the following issues: 1. We have other outcomes defined at the moment, like NEEDS_INSPECTION: https://docs.qadevel.cloud.fedoraproject.org/libtaskotron/latest/library.html#libtaskotron.check.CheckDetail What should happen in that case? 2. Some tasks do not check just a single item. For example, depcheck and upgradepath check everything that is currently proposed for stable or testing updates. They can return a dozen test results. What should happen when there are mixed results? 3. At the moment, it's technically possible to leave out the 'resultsdb' directive the from task recipe. I cannot imagine a use case for this, because in development mode, this only reports to stdout, but it's possible if you need it. If we wanted to inspect task outcomes, we need to do it from TAP, therefore in the 'resultsdb' directive. It might be confusing to people that if you commented out that directive, runtask would stop returning "correct" return codes. One further thought, instead of implementing this by default, we can also create a new directive or variable which will allow task creators to set the exit code, according to TAP or any other logic they want. Why exactly do you need this? For local testing? runtask is just an execution wrapper, so you should be able to run your `./script some args` the same way as `runtask -i item -t type script.yml`. Except for some set up steps (like 'koji' or 'bodhi' directive), so if you use some of these, I can see a reason why you would want to run it through runtask. Kamil ___ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
