Hi folks! I thought this might be of interest to the FreeIPA community, so I thought I'd write it up here in case anyone missed it elsewhere.
I work on the Fedora QA team, and we have been using the openQA automated test system (developed by our friends at SUSE) to run various functional tests on Fedora composes for the last couple of years. As FreeIPA is considered a critical part of Fedora Server, we run a few tests that exercise FreeIPA. The tests set up a FreeIPA server, run some basic checks on it, and also enrol two systems as clients of the domain, one using the 'realm join' command directly, one using Cockpit. The client tests do some basic client functionality testing (getent, logging in as a domain user, changing passwords, etc.) and also test the web UI to some extent. Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). The tests are automatically run on any critical path package, and as of today, they are also run on any update containing any of a manually-tended list of FreeIPA-related packages: 389-ds 389-ds-base bind bind-dyndb-ldap certmonger ding-libs freeipa krb5-server pki-core sssd tomcat cockpit This means that for any Fedora update containing one of these or any critical path package, Fedora's openQA FreeIPA tests should run, and you should see the results in the Fedora update system (Bodhi). You can see the results in Bodhi by clicking the Automated Updates tab for any update. For instance, here's a recent 389-ds-base update for Fedora 26: https://bodhi.fedoraproject.org/updates/FEDORA-2017-15e2a038b2 If you look at the Automated Tests tab, you can see passes for: update.server_role_deploy_domain_controller update.realmd_join_cockpit update.realmd_join_sssd indicating that this update didn't cause any problems for FreeIPA. Clicking on any test result will take you to the openQA page for the test, where you can diagnose failures and so on (explaining how to do this is a bit beyond the scope of this mail, please do ask me if you're interested!) I hope this stuff will help us avoid shipping updates that break FreeIPA (and other key components). If you have any questions, concerns, comments, or suggestions, please do ask! To anticipate one question: you can cause *all* the tests for an update to be re-run by editing the update in any way (you don't have to change the package loadout, just changing a single character in the description or something will do). If you think just one test result is bogus and want it re-run, currently, you'll have to ask someone with the necessary power - either me or Jan Sedlak (garretraziel on IRC). I'm in North America and he's in Europe, so we should have most timezones covered between us. We're hoping to set up a better mechanism for this in future. Note, if you're interested in the results for the nightly Fedora distribution composes, an email summary of the results for those is sent each time they're run to the Fedora test@ and devel@ lists, look for mails with "compose check report" in the subject. Any time any of the FreeIPA tests fails, the failure will be listed in the mail (passed tests are not specifically listed, just a count of them). I usually keep an eye on those results and analyze failures and file bugs, though. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
