On 06/25/2015 11:41 AM, bertagaz wrote: > I've prepared a blueprint to start this discussion and take notes of the > decisions: > https://tails.boum.org/blueprint/automated_builds_and_tests/automated_tests_specs/
Looks great! For the record, I looked at the spec as of commit e70f8e7. > # Facts > > Running the full test suite on 1 isotester hosted on Lizard takes > around 8 hours. FYI, with the test/6094-improved-snapshots branch (previously test/wip-improved-snapshots) a full run on lizard takes ~350 minutes, so a bit less than six hours. On my laptop it takes around half that, at 180 minutes per full run. It seems > We intend to run 4 isotesters, so at the moment we would be able > to run 12 full test suites per day. So with test/6094-improved-snapshots we can run 24/6 * 4 = 16 full runs per day! :) OTOH, we'll keep on adding more tests, so these figures will eventually becomes obsolete. Until the end of 2015 (when we "probably" get more hardware) I think we can assume that the run time of a full test suite will not increase by more than 33% due to new tests, but if we assume this worst case scenario, then with the test/6094-improved-snapshots branch we end up at your 12 runs per day number again. > Scenario 1 : reviewer vs. > Scenario 2 : developer The current proposal seems to be to only start the automated test run of a feature branch when it is marked "Ready for QA". This has overloaded the meaning of this status so it no longer alone means that the branch is ready for review'n'merge; the reviewer also has to wait until the automated tester posts the result to the ticket. We could get rid of this ambiguity by splitting "Ready for QA" into "Ready for (automated) testing" (RFT) and "Ready for review" (RFR). Example: Let's say I have created a new feature branch and think I'm done. I assign it to intrigeri (who I want to review it in the end) and mark it RFT. And automated build is triggered, and then a full run of the test suite. Let's say the test suite fails. Then it gets reassigned to me and marked "Dev needed". I fix the issue (which probably was easier to spot for me than it would be for intrigeri, since I did the implementation) assign it to intrigeri and mark it RFT again. This time the test suite passes, and then the ticket is marked RFR automatically. Also, if I run the test suite myself and see it pass, then I can just mark it RFR directly. I do not know if this complicates the current plans too much, so perhaps this could be a future improvement. Or perhaps the current plan will be good enough in practice. I suppose it's bad in itself to introduce yet another QA status, complicating things. Also a question: > And the developer who proposed the merge must be notified > And the ticket should be reassigned to the branch submitter Is deciding "the developer who proposed the merge" and "the branch submitter" actually easy? We do not automatically get a mapping between Git authors and Redmine users. And parsing the redmine ticket history (e.g. who marked the ticket a certain QA status last?) also seems hairy, but perhaps it's "easy" via some API (SQL?) that I'm unaware of? Cheers! _______________________________________________ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.