Re: Future of Test Cases and their Management in Fedora QA
On 6/22/22 14:13, Adam Williamson wrote: On Wed, 2022-06-22 at 08:57 -0600, Tim Flink wrote: 3. Continue Using WikiTCMS Advantages: - Doesn't require any changes to how we're doing things - Is very flexible - can do pretty much whatever we want to without a ton of code - Doesn't require much maintenance or custom code beyond wikitcms Another advantage of the current system that I'd like to highlight is that it allows us to synthesize manual and automated test results. That is, in the current wiki results pages, you see the results from openQA and human testers together. I would definitely want any replacement for it to preserve this advantage - i.e., I would want it to be able to track results from both human testers and automated systems. Ideally, this would be achieved by resultsdb integration - it's a bit difficult to imagine technically how this could work, but it would be nice for results from both human testers and automated systems to be in resultsdb, and for any human- readable representation of the validation state to pull both sets of results from resultsdb. For what it's worth, Kiwi does have a pretty comprehensive API when it comes to stuff like reading or posting test results. When it comes to storing results for automated vs. manual testing, Kiwi does seem to focus more on having test cases which are marked as automated rather than having users marked as bots. As far as I know, there is no way to mark a Kiwi user as a bot. Personally, I think there are arguments for both approaches but changing over to Kiwi would likely require a change to how we've been doing things. wikitcms gives us the integration of the two types of result, but it does not currently forward results from human testers to resultsdb. So we only have the results from automated systems in resultsdb. openQA reports its results to *both* resultsdb and the wiki, so the wiki is the only canonical store of *both* human and bot results. I've toyed in the past with the idea of having `relval report-results` also submit the result to resultsdb, but it requires solving authentication and we still wouldn't capture human test results submitted by editing the wiki directly. Another option would be a wikitcms-based bot which monitored result pages for edits and forwarded any 'new results' to resultsdb... At some point, we'd end up with enough chewing gum and bailing wire that we might as well roll our own. I'm not sure we're there yet but if we start down the path of an alternate interface to post results and/or a bot to shuffle results between the wiki and resultsdb, I do think that would start approaching the point where we'd be better off writing our own solution. Disadvantages: - More difficult to analyze data contained in the wiki - Only one person can report results in a given matrix at a time - QA is the only group still using the Fedora wiki This kinda started a while back when CPE started asking about whether we still needed the wiki or not. The current Fedora wiki is a source of frustration for them due to the amount of spam and bot issues that they have to deal with. Seeing as how QA is one of the only (if not the only) groups left using the wiki, it's worth asking whether we really still need it or not. It does sound like most of CPE's concerns could be satisfied with a new wiki instance which could be locked down to only allow pages with a certain format but I don't believe that conversation has even been started with them. If they do get to the point where they really, really want to turn the wiki off, I'm willing to try and handle this (setting up a reduced wiki instance specifically for this purpose, with restrictions to allowable page names and so on). ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Fedora-Rawhide-20220623.n.0 compose check report
Missing expected images: Minimal raw-xz armhfp Compose PASSES proposed Rawhide gating check! All required tests passed Failed openQA tests: 15/161 (aarch64), 15/231 (x86_64) New failures (same test not failed in Fedora-Rawhide-20220621.n.3): ID: 1305350 Test: aarch64 Server-dvd-iso server_cockpit_basic@uefi URL: https://openqa.fedoraproject.org/tests/1305350 ID: 1305452 Test: x86_64 universal install_blivet_software_raid@uefi URL: https://openqa.fedoraproject.org/tests/1305452 Old failures (same test failed in Fedora-Rawhide-20220621.n.3): ID: 1305253 Test: x86_64 Workstation-live-iso desktop_login URL: https://openqa.fedoraproject.org/tests/1305253 ID: 1305268 Test: x86_64 KDE-live-iso apps_startstop URL: https://openqa.fedoraproject.org/tests/1305268 ID: 1305273 Test: x86_64 KDE-live-iso desktop_login URL: https://openqa.fedoraproject.org/tests/1305273 ID: 1305282 Test: x86_64 Silverblue-dvd_ostree-iso eog URL: https://openqa.fedoraproject.org/tests/1305282 ID: 1305285 Test: x86_64 Silverblue-dvd_ostree-iso evince URL: https://openqa.fedoraproject.org/tests/1305285 ID: 1305289 Test: x86_64 Silverblue-dvd_ostree-iso gnome_text_editor URL: https://openqa.fedoraproject.org/tests/1305289 ID: 1305316 Test: aarch64 Server-dvd-iso install_btrfs_preserve_home@uefi URL: https://openqa.fedoraproject.org/tests/1305316 ID: 1305319 Test: aarch64 Server-dvd-iso install_repository_hd_variation@uefi URL: https://openqa.fedoraproject.org/tests/1305319 ID: 1305371 Test: aarch64 Workstation-raw_xz-raw.xz desktop_terminal@uefi URL: https://openqa.fedoraproject.org/tests/1305371 ID: 1305384 Test: aarch64 Workstation-raw_xz-raw.xz gnome_text_editor@uefi URL: https://openqa.fedoraproject.org/tests/1305384 ID: 1305385 Test: aarch64 Workstation-raw_xz-raw.xz desktop_update_graphical@uefi URL: https://openqa.fedoraproject.org/tests/1305385 ID: 1305397 Test: x86_64 Workstation-upgrade desktop_login URL: https://openqa.fedoraproject.org/tests/1305397 ID: 1305405 Test: x86_64 Workstation-upgrade desktop_fprint URL: https://openqa.fedoraproject.org/tests/1305405 ID: 1305425 Test: aarch64 Workstation-upgrade desktop_browser@uefi URL: https://openqa.fedoraproject.org/tests/1305425 ID: 1305427 Test: aarch64 Workstation-upgrade desktop_printing@uefi URL: https://openqa.fedoraproject.org/tests/1305427 ID: 1305429 Test: aarch64 Workstation-upgrade gnome_text_editor@uefi URL: https://openqa.fedoraproject.org/tests/1305429 ID: 1305462 Test: x86_64 universal install_arabic_language URL: https://openqa.fedoraproject.org/tests/1305462 ID: 1305513 Test: aarch64 universal install_asian_language@uefi URL: https://openqa.fedoraproject.org/tests/1305513 ID: 1305516 Test: aarch64 universal install_arabic_language@uefi URL: https://openqa.fedoraproject.org/tests/1305516 ID: 1305566 Test: x86_64 Server-dvd-iso install_updates_nfs URL: https://openqa.fedoraproject.org/tests/1305566 ID: 1305571 Test: aarch64 Server-dvd-iso install_updates_nfs@uefi URL: https://openqa.fedoraproject.org/tests/1305571 ID: 1305572 Test: x86_64 universal support_server URL: https://openqa.fedoraproject.org/tests/1305572 ID: 1305574 Test: x86_64 universal install_kickstart_nfs URL: https://openqa.fedoraproject.org/tests/1305574 ID: 1305575 Test: x86_64 universal install_pxeboot URL: https://openqa.fedoraproject.org/tests/1305575 ID: 1305576 Test: x86_64 universal install_pxeboot@uefi URL: https://openqa.fedoraproject.org/tests/1305576 ID: 1305577 Test: aarch64 universal support_server@uefi URL: https://openqa.fedoraproject.org/tests/1305577 ID: 1305578 Test: aarch64 universal install_pxeboot@uefi URL: https://openqa.fedoraproject.org/tests/1305578 ID: 1305579 Test: aarch64 universal install_kickstart_nfs@uefi URL: https://openqa.fedoraproject.org/tests/1305579 Soft failed openQA tests: 9/161 (aarch64), 11/231 (x86_64) (Tests completed, but using a workaround for a known bug) New soft failures (same test not soft failed in Fedora-Rawhide-20220621.n.3): ID: 1305383 Test: aarch64 Workstation-raw_xz-raw.xz eog@uefi URL: https://openqa.fedoraproject.org/tests/1305383 ID: 1305394 Test: aarch64 Cloud_Base-qcow2-qcow2 cloud_autocloud@uefi URL: https://openqa.fedoraproject.org/tests/1305394 Old soft failures (same test soft failed in Fedora-Rawhide-20220621.n.3): ID: 1305245 Test: x86_64 Workstation-live-iso eog URL: https://openqa.fedoraproject.org/tests/1305245 ID: 1305291 Test: x86_64 Silverblue-dvd_ostree-iso desktop_browser URL: https://openqa.fedoraproject.org/tests/1305291 ID: 1305302 Test: x86_64 Cloud_Base-qcow2-qcow2 cloud_autocloud@uefi URL: https://openqa.fedoraproject.org/tests/1305302 ID: 1305428 Test: aarch64 Workstation-upgrade eog@uefi URL: https://openqa.fedoraproject.org/tests/1305428 ID: 1305474 Test: x86_64 universal upgrade_2_desktop_64bit URL: https://openqa.fedoraproject.org/tests/1305474 ID: 1305475
Fedora rawhide compose report: 20220623.n.0 changes
OLD: Fedora-Rawhide-20220621.n.3 NEW: Fedora-Rawhide-20220623.n.0 = SUMMARY = Added images:1 Dropped images: 0 Added packages: 6 Dropped packages:2 Upgraded packages: 137 Downgraded packages: 0 Size of added packages: 2.33 MiB Size of dropped packages:3.05 MiB Size of upgraded packages: 3.05 GiB Size of downgraded packages: 0 B Size change of upgraded packages: 39.26 MiB Size change of downgraded packages: 0 B = ADDED IMAGES = Image: Xfce raw-xz aarch64 Path: Spins/aarch64/images/Fedora-Xfce-Rawhide-20220623.n.0.aarch64.raw.xz = DROPPED IMAGES = = ADDED PACKAGES = Package: fedora-appstream-metadata-37-0.2.fc37 Summary: Operating System AppStream Metadata for Fedora Linux RPMs:fedora-appstream-metadata Size:8.78 KiB Package: libkrun-1.2.2-1.fc37 Summary: Dynamic library providing Virtualization-based process isolation capabilities RPMs:libkrun libkrun-devel Size:1.89 MiB Package: perl-MooseX-Role-WarnOnConflict-0.01-2.fc37 Summary: Warn if classes override role methods without excluding them RPMs:perl-MooseX-Role-WarnOnConflict Size:18.11 KiB Package: perl-MouseX-Types-Common-0.001000-2.fc37 Summary: Set of commonly-used type constraints RPMs:perl-MouseX-Types-Common Size:15.15 KiB Package: python-elastic-transport-8.1.2-1.fc37 Summary: Transport classes and utilities shared among Python Elastic RPMs:python3-elastic-transport Size:105.22 KiB Package: sfsexp-1.4.0-2.fc37 Summary: Small Fast S-Expression Library RPMs:sfsexp sfsexp-devel Size:309.20 KiB = DROPPED PACKAGES = Package: kup-0.3.6-11.fc36 Summary: Kernel.org Uploader RPMs:kup kup-server kup-utils Size:80.29 KiB Package: rust-fedora-coreos-pinger-0.0.4-14.fc37 Summary: Telemetry service for Fedora CoreOS RPMs:fedora-coreos-pinger Size:2.98 MiB = UPGRADED PACKAGES = Package: Lmod-8.7.5-1.fc37 Old package: Lmod-8.7.4-1.fc37 Summary: Environmental Modules System in Lua RPMs: Lmod Size: 916.91 KiB Size change: 969 B Changelog: * Thu Jun 23 2022 Orion Poplawski - 8.7.5-1 - Update to 8.7.5 Package: ansible-6.0.0-1.fc37 Old package: ansible-5.9.0-2.fc37 Summary: Curated set of Ansible collections included in addition to ansible-core RPMs: ansible Size: 36.97 MiB Size change: -3.48 MiB Changelog: * Wed Jun 22 2022 Maxwell G - 6.0.0~rc1-1 - Update to 6.0.0~rc1. - Stop duplicating docs and licenses. - Don't remove tests in %prep that are now handled by setup.py. - Hardlink duplicated files and fix rpmlint errors * Wed Jun 22 2022 Maxwell G - 6.0.0-1 - Update to 6.0.0. Package: ansible-core-2.13.1-1.fc37 Old package: ansible-core-2.12.6-2.fc37 Summary: A radically simple IT automation system RPMs: ansible-core ansible-core-doc Size: 4.47 MiB Size change: -66.74 KiB Changelog: * Thu Jun 16 2022 Maxwell G - 2.13.0-1 - Update to 2.13.0. - Re-enable tests that work with newer pytest - Patch out python3-mock - Manually build manpages to workaround upstream issue. - Remove unneeded BRs and switch to pyproject-rpm-macros. - Make ansible-base* Obsoletes/Provides compliant with Packaging Guidelines - Remove python3-jmespath dependency. json_query is part of community.general. - Correct licensing - Generate shell completions * Wed Jun 22 2022 Maxwell G - 2.13.1-1 - Update to 2.13.1 (rhbz#2096312). Package: avr-gcc-1:12.1.0-1.fc37 Old package: avr-gcc-1:11.3.0-1.fc37 Summary: Cross Compiling GNU GCC targeted at avr RPMs: avr-gcc avr-gcc-c++ Size: 177.32 MiB Size change: 35.23 MiB Changelog: * Tue Jun 21 2022 Michal Hlavinka - 1:12.1.0-1 - updated to 12.1.0 Package: bluedevil-5.25.1-1.fc37 Old package: bluedevil-5.25.0-1.fc37 Summary: Bluetooth stack for KDE RPMs: bluedevil Size: 1.76 MiB Size change: -757 B Changelog: * Tue Jun 21 2022 Marc Deop - 5.25.1-1 - 5.25.1 Package: boost-1.78.0-4.fc37 Old package: boost-1.78.0-3.fc37 Summary: The free peer-reviewed portable C++ source libraries RPMs: boost boost-atomic boost-b2 boost-build boost-chrono boost-container boost-context boost-contract boost-coroutine boost-date-time boost-devel boost-doc boost-doctools boost-examples boost-fiber boost-filesystem boost-graph boost-graph-mpich boost-graph-openmpi boost-iostreams boost-json boost-locale boost-log boost-math boost-mpich boost-mpich-devel boost-mpich-python3 boost-mpich-python3-devel boost-nowide boost-numpy3 boost-openmpi boost-openmpi-devel boost-openmpi-python3 boost-openmpi-python3-devel boost-program-options boost-python3 boost-random boost-regex boost-serialization boost-stacktrace boost-static boost-system boost-test boost-thread boost-timer boost-type_erasure boost-wave Size: 286.58 MiB Size change: 22.62 KiB Changelog: * Tue Jun 21 2022 Jonathan Wakely
Fedora-Cloud-35-20220623.0 compose check report
No missing expected images. Soft failed openQA tests: 1/8 (x86_64), 1/8 (aarch64) (Tests completed, but using a workaround for a known bug) Old soft failures (same test soft failed in Fedora-Cloud-35-20220622.0): ID: 1305106 Test: x86_64 Cloud_Base-qcow2-qcow2 cloud_autocloud URL: https://openqa.fedoraproject.org/tests/1305106 ID: 1305114 Test: aarch64 Cloud_Base-qcow2-qcow2 cloud_autocloud@uefi URL: https://openqa.fedoraproject.org/tests/1305114 Passed openQA tests: 7/8 (x86_64), 7/8 (aarch64) -- Mail generated by check-compose: https://pagure.io/fedora-qa/check-compose ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Fedora-Cloud-36-20220623.0 compose check report
No missing expected images. Soft failed openQA tests: 1/8 (x86_64), 1/8 (aarch64) (Tests completed, but using a workaround for a known bug) Old soft failures (same test soft failed in Fedora-Cloud-36-20220622.0): ID: 1304977 Test: x86_64 Cloud_Base-qcow2-qcow2 cloud_autocloud URL: https://openqa.fedoraproject.org/tests/1304977 ID: 1304985 Test: aarch64 Cloud_Base-qcow2-qcow2 cloud_autocloud@uefi URL: https://openqa.fedoraproject.org/tests/1304985 Passed openQA tests: 7/8 (x86_64), 7/8 (aarch64) -- Mail generated by check-compose: https://pagure.io/fedora-qa/check-compose ___ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
