On 01/28/2015 11:58 PM, Nathaniel McCallum wrote: > Attached is a new version which fixes most of the issues found. > > Comments below.
Thanks! > > On Mon, 2015-01-12 at 15:53 +0100, Petr Viktorin wrote: >> On 01/06/2015 03:26 AM, Nathaniel McCallum wrote: >>> On Thu, 2014-11-20 at 11:13 -0500, Nathaniel McCallum wrote: >>>>> This tests the general workflow for OTP including most possible >>>>> token combinations. This includes 5872 tests. Further optimization >>>>> is possible to reduce the number of duplicate tests run. >>>>> >>>>> Things not yet tested: >>>>> * ipa-kdb >>>>> * ipa-otpd >>>>> * otptoken-sync >>>>> * RADIUS proxy >>>>> * token self-management >>>>> * type specific attributes >>> Attached is the latest iteration of the OTP test work. This now includes >>> all major cases except token self-management and RADIUS proxy (which >>> will come in its own patch). Token self-management is held up by the >>> fact that I can't get alternate ccaches to work with the API. I have >>> tried kinit-ing to an independent ccache and exporting KRB5CCNAME, but >>> this doesn't work for some reason I can't figure out. >>> >>> I ended up creating my own fixture mechanism. I'm not in love with it, >>> but it is simple and at least gets the scoping correct. It also >>> generates individual tests for each parameterized state, so the output >>> is both correct and obvious. >>> >>> I also implemented OTP myself. This isn't much code, but pyotp has a >>> major bug and is dead upstream. I'd like to migrate to >>> python-cryptography when it lands as a dependency of FreeIPA. But due to >>> timing issues, we can't land it now. This will be a small patch in the >>> future. >>> >>> Even with the caveats above, I feel like the test coverage provided by >>> this test is worth review/merge. As a rough estimate, I think this is >>> about 70% code coverage. Of the remaining coverage, I see: >>> * RADIUS proxy - 10% >>> * token self-management - 10% >>> * misc testable - 5% >>> * misc untestable - 5% >>> >>> All tests in this patch succeed on 4.1.2. >> >> It doesn't work for me on the 4.2 branch, see the attached errors. Not >> sure what I'm doing wrong. > > This doesn't work at all on the 4.2 branch. This is because nose, like > the upstream test harnesses, executes yielded tests as they are yielded. > However, pytest does not. It collects all the yielded tests into a > single bundle and then runs them after the state of their fixtures is > already destroyed. This complete breaks the fixturize() decorator. > > Why are we using pytest again? It may offer nifty features, but it sucks > at integration testing. While right now, I am mostly concerned about OTP testing in FreeIPA 4.1.* as this is the release going in downstream releases, the next step after successfully reviewing&testing of ipa-4-1 incarnation of the packages will be update in master branch, i.e. making them work with our favorite testing framework - pytest :-). Martin _______________________________________________ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel