Hello all!
The proposed system have become more stable lately - and I think I've solved a
few sources of flakiness.
To be really usable I also wanted to add a way to dynamically enable/disable a set of tests (for example the replication tests take ~7 hours to execute from the total of 24
hours - and they are also a bit unstable, so not running them when not neccesary would be beneficial in multiple ways) - but to do this the best would be to throw in
junit5; unfortunately the current ptest installation uses maven 3.0.5 which doesn't like these kind of things - so instead of hacking a fix for that ....I've removed it
from the dev branch for now.
I would like to propose to start an evaluation phase of the new test
procedures(INFRA-20269)
The process would look something like this:
* someone opens a PR - the tests will be run on the changes
* on every active branches the tests will run from time to time
* this will produce a bunch of test runs on the master branch as well ; which
will show how well the tests behave on the master branch without any patches
* runs on branches (PRs or active development branches(eg:master)) will be rate
limited to 5 builds/day
* at most ~4 builds at a time - to maximize resource usage
* turnaround time for a build is right now 2 hours - which I feel like a
balanced choice between speed/response time
Possible future benefits:
* toggle features using github tags
* optional testgroups (metastore/replication) tests
* ability to run the metastore verification tests
* possibility to add smoke tests
To enable this I will have to finish the HIVE-22942 ticket - beyond the new
Jenkinsfile which defines the full logic;
although I've sinked a lot of time into fixing all kind of flaky tests I would
would like to disable around ~25 tests.
I also would like to propose a method to verify the stability of a single test:
run it a 100 times in series at the same place where the precommit tests are
running.
This will put the bar high enough that only totally stable tests could satisfy
it (a 99% stable test has 36% chance to pass this without being caught :D)
After this will be in service it could be used to: validate that an existing test is unstable (before disabling it) - and then used again to prove that it got fixed during
re-enabling it.
Please let me know what you think!
cheers,
Zoltan
On 4/29/20 4:28 PM, Zoltan Haindrich wrote:
Hey All!
I was planning to replace the ptest stuff with something less complex for a
while now - I see that we struggle a lot because of ptest is more complicated
than it should be...
It would be much better if it would be constructed from well made existing CI
piece. - because of that I've started working on [1] a few months ago.
It has it's pros and cons...but it's not the same as the existing ptest stuff.
I've collected some infos about how it compares against the existing one - but
it became too long so I've moved it into a google docs document at [3].
It's not yet ready... I still have some remaining problems/concerns/etc
* what do you think about changing to a github PR based workflow?
* it will not support at all things like "isolation" - so we will have to make
our tests work with eachother without bending the rules...
* I've tried to overcommit the cpu resources which creates a more noisy environment for the actual tests - this squeezes out some new problems which should be fixed before
this could be enabled.
* for every PR the first run is somewhat sub-optimal...there are some reasons for this - the actually used resources are the same; but the overall execution time is not
optimal; I could accept this as a compromise because right now I wait >24 hours for a precommit run.
It's deployed at [2] and anyone can start a testrun on it:
* merge my HIVE-22942-ptest-alt branch from [4] into your branch
* open a PR against my hive repo on github [5]
cheers,
Zoltan
[1] https://issues.apache.org/jira/browse/HIVE-22942
[2] http://34.66.156.144:8080/job/hive-precommit
[3]
https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit?usp=sharing
[4] https://github.com/kgyrtkirk/hive/tree/HIVE-22942-ptest-alt
[5] https://github.com/kgyrtkirk/hive/
