Hello all!
The proposed system have become more stable lately - and I think I've
solved a few sources of flakiness.
To be really usable I also wanted to add a way to dynamically
enable/disable a set of tests (for example the replication tests take ~7
hours to execute from the total of 24
hours - and they are also a bit unstable, so not running them when not
neccesary would be beneficial in multiple ways) - but to do this the best
would be to throw in
junit5; unfortunately the current ptest installation uses maven 3.0.5
which doesn't like these kind of things - so instead of hacking a fix for
that ....I've removed it
from the dev branch for now.
I would like to propose to start an evaluation phase of the new test
procedures(INFRA-20269)
The process would look something like this:
* someone opens a PR - the tests will be run on the changes
* on every active branches the tests will run from time to time
* this will produce a bunch of test runs on the master branch as well ;
which will show how well the tests behave on the master branch without any
patches
* runs on branches (PRs or active development branches(eg:master)) will be
rate limited to 5 builds/day
* at most ~4 builds at a time - to maximize resource usage
* turnaround time for a build is right now 2 hours - which I feel like a
balanced choice between speed/response time
Possible future benefits:
* toggle features using github tags
* optional testgroups (metastore/replication) tests
* ability to run the metastore verification tests
* possibility to add smoke tests
To enable this I will have to finish the HIVE-22942 ticket - beyond the
new Jenkinsfile which defines the full logic;
although I've sinked a lot of time into fixing all kind of flaky tests I
would would like to disable around ~25 tests.
I also would like to propose a method to verify the stability of a single
test: run it a 100 times in series at the same place where the precommit
tests are running.
This will put the bar high enough that only totally stable tests could
satisfy it (a 99% stable test has 36% chance to pass this without being
caught :D)
After this will be in service it could be used to: validate that an
existing test is unstable (before disabling it) - and then used again to
prove that it got fixed during
re-enabling it.
Please let me know what you think!
cheers,
Zoltan
On 4/29/20 4:28 PM, Zoltan Haindrich wrote:
Hey All!
I was planning to replace the ptest stuff with something less complex
for a while now - I see that we struggle a lot because of ptest is more
complicated than it should be...
It would be much better if it would be constructed from well made
existing CI piece. - because of that I've started working on [1] a few
months ago.
It has it's pros and cons...but it's not the same as the existing ptest
stuff.
I've collected some infos about how it compares against the existing one
- but it became too long so I've moved it into a google docs document at
[3].
It's not yet ready... I still have some remaining problems/concerns/etc
* what do you think about changing to a github PR based workflow?
* it will not support at all things like "isolation" - so we will have
to make our tests work with eachother without bending the rules...
* I've tried to overcommit the cpu resources which creates a more noisy
environment for the actual tests - this squeezes out some new problems
which should be fixed before
this could be enabled.
* for every PR the first run is somewhat sub-optimal...there are some
reasons for this - the actually used resources are the same; but the
overall execution time is not
optimal; I could accept this as a compromise because right now I wait
24 hours for a precommit run.
It's deployed at [2] and anyone can start a testrun on it:
* merge my HIVE-22942-ptest-alt branch from [4] into your branch
* open a PR against my hive repo on github [5]
cheers,
Zoltan
[1] https://issues.apache.org/jira/browse/HIVE-22942
[2] http://34.66.156.144:8080/job/hive-precommit
[3]
https://docs.google.com/document/d/1dhL5B-eBvYNKEsNV3kE6RrkV5w-LtDgw5CtHV5pdoX4/edit?usp=sharing
[4] https://github.com/kgyrtkirk/hive/tree/HIVE-22942-ptest-alt
[5] https://github.com/kgyrtkirk/hive/
