On 03/31/2015 02:47 PM, Martin Koci wrote: > Hi all, > I'd like to open discussion on test categorization into tiers and > acceptance testing, respectively test tagging which should help us to > accomplish following goals: > > 1) Acceptance test - other FreeIPA partner projects (389/DS/PKI) should > be able to have an "Acceptance test" that would run basic *stable* test > suite that would check if anything significant broke. It should be fast > enough so that the projects can run it in a Jenkins CI after commits. > > If we also have tags @dogtag or @sssd, the projects could simply run > just the tests affecting the projects -> faster execution. > > 2) FreeIPA test run optimization. Currently, all FreeIPA tests are > running when new commit is pushed. This takes lot of resources. It would > be nice to at least be able to NOT run Tier 2 tests if Tier1 tests are > failing. Or it would be nice to not run some very expensive tests after > each commit, but maybe once per day/week. > > *TIERS* > So after discussions with couple of developers and QE's we have created > and summarized following proposal for sorting current IPA tests into > tiers. > > Currently used tests reside in freeipa/ipatests. From these the only > unit tests (tier 0 candidate) are test_{ipalib,ipapython} with the > exception of test_ipalib/test_rpc.py which requires kerberos. > > The rest of the tests either require ipa/lite-server or are an > integration test. The rest of the tests (majority XML RPC, UI > tests, ...) then fall under the definition of Tier 1 test, as they > require at least running IPA instance and admin TGT. > > As for the tagging of the test cases, pytest's capabilities can be used > [2]. Though pytest.mark currently does not work with declarative tests > (it marks all of them), when the test is an ordinary function/method the > marking works as expected. The declarative tests could be rewritten in > the future to more pytest specific form, e.g. > test_xmlrpc/test_host_plugin.py > > Official guideline for this categorization will be created on the > upstream wiki once we agree on that. > > > *ACCEPTANCE TESTING* > As for the acceptance testing Similar to `Test categorization into > tiers` (1) proposal, there is a need to define a subset of freeipa tests > that could be run by other projects or users to find out whether or not > their changes (e.g. new build, feature) works with IPA. > > This run could be composed from tier {0,1} execution followed by a > subset of integration tests test cases. The proposed mechanism for this > is the same as in [4], using pytest.mark to select the classes/tests to > run in this context. > > What I'd like to ask you here is to share any ideas on the form of the > acceptance run as well as to help me identify the areas (and tests) that > are considered important and should be a part of this test set. > > *TAGGING* > Tagging the actual tests classes with pytest decorator > (http://pytest.org/latest/mark.html). would be better than let > developers manually maintain lists of tests for different projects. The > benefit for pytest mark kept in the code is that whatever we do with the > test class (rename, move, merge), the tag goes with it, not extra list > needs to be maintained. > > As for tagging itself, the original idea which Martin Kosek was > proposing was to use just the "acceptance" tag for marking the base T2 > tests that would be part of FreeIPA acceptance tests. > > However, it seems there is a value in tagging the tests that exercise > also certain sub-component of FreeIPA - SSSD, Dogtag. As long as we do > not get too wild with the tags, it should be OK. > > So we could agreed on followings tags: > - tier0, tier1, tier2 > - acceptance > - sssd > - dogtag > > This would lead to e.g. > > @pytest.mark.dogtag > @pytest.mark.acceptance > @pytest.mark.tier2 > class TestExternalCA(IntegrationTest): > ... > > or simpler > > @dogtag > @acceptance > @tier2 > class TestExternalCA(IntegrationTest): > > Hope it's not too long and that it makes sense.
It makes a lot of sense to me (it should, since I contributed to this proposal too). So I will be looking forward to other developers thought on this. If there are no objections, we could start with the actual patches and have them properly reviewed. > Can I get your thoughts on this, please? > Thank you. > > Regards, > /koca > > *[1] - https://fedorahosted.org/freeipa/ticket/4922 > *[2] - http://pytest.org/latest/mark.html -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code