#276: Support for independent test development
-------------------------+--------------------------------------------------
Reporter: jlaska | Owner:
Type: enhancement | Status: new
Priority: major | Milestone: 0.5.0
Component: core | Resolution:
Keywords: |
-------------------------+--------------------------------------------------
Comment (by jlaska):
Thanks for the suggestion! Anything pypi is well outside my current
knowledge, but I'm definitely open to understanding how it can solve this
problem.
I don't want to radically change how we define tests in this ticket, but I
do want to allow maintainers to develop/maintain their tests without
having to commit into autoqa and wait for a new autoqa build. As it
stands, we have the installer tests that have been waiting since December
for a new build so those new tests will run. I know we can improve that
situation.
1. One quick'n'dirty approach would be to continue with our current
autotest control file format, but run a cronjob on the autoqa server that
pulls down the latest content from git and uses *just* the tests. This
could have some side-effects if incomplete tests are pushed into master.
But for some reason that really doesn't bother me much. We tend to keep
master fairly clean anyway.
2. Another quick'n'dirty, add support so *select* packages can move their
test wrapper and control files into their dist-git checkout. The test
directory structure would remain the same, we are just moving some to a
different git repo. Again, some cronjob (or similar) service would be
needed to keep the tests updated on the autoqa server.
While the above approaches aren't pretty, they do scratch the most
immediate itch. Other approaches seem to lean more support for maintainer
contributed tests which requires support for disposable test systems,
which we can't yet offer. For example ...
1. A more long-term approach (and probably another ticket) involves
allowing maintainers to maintain/contribute tests within their
[https://fedoraproject.org/wiki/Using_Fedora_GIT own dist-git checkout].
I've heard several requests for this support already, but we'd need to
think about an appropriate format for their tests and how we would
schedule them.
* Are tests scheduled by the maintainer via 'make test' in the dist-git
checkout?
* Do we continue with the current test directory format (test wrapper,
control files, test driver/script).
* Do we try to hide some of that so they don't need to worry about it?
For example, do they really need to know about the control files?
* Do we follow the beakerlib example and assume the test driver exit
code is the test result?
* What requirements can we assert on all maintainer contributed tests?
Meaning, should the test driver always be python, what about
bash/perl/ruby/c? Do we assume that any filename matching the pattern
runtest.{py,pl,sh,o} is the test driver?
So for now, anything that can help anaconda maintain/update their existing
autoqa tests without being tied to the autoqa release schedule ... *and*
avoiding the rats nest above seems like a good candidate. What am I
missing?
--
Ticket URL: <https://fedorahosted.org/autoqa/ticket/276#comment:2>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project
_______________________________________________
autoqa-devel mailing list
[email protected]
https://fedorahosted.org/mailman/listinfo/autoqa-devel
