During the last 5 weeks (since the separation of hadoop-ozone
repository) about 177 commits has been added to the master branch of Ozone.
This means ~7 commits per working day.
With this development velocity it's very important to keep the master
branch as stable as possible.
Any potential CI failure can hide additional problems which makes either
harder or more risky to commit/merge any PR. Multiple new test failures
are introduced because the problem was not visible due to an other,
earlier problem.
I think we need to do additional steps to keep the master branch more
stable. I think there are two main approaches.
1. Revert immediately the commits after any kind of failures.
2. Merge to the master with a script only if CI build passes
There are multiple options to implement 2nd: for example after the
review it would be enough to add a "ready-to-merge" label and an
additional job would execute all the tests and merge the PR to the
master only if all the tests are passed.
This second approach has some hard limitations:
* Commits can't be signed by the committer if the merge is organized
by a script.
* It requires tricky github actions logic as ASF INFRA doesn't allow
to use any external services which require write permission to the repo
(such as mergify)
THEREFORE I PROPOSE to follow the 1st approach:
1. Revert any commit IMMEDIATELY if new failures are introduced. (And
communicate everywhere that we need all of the new commits and potential
reverts are not against the code, just an integration step for the
stability.)
2. In case of a flaky test where the source of the flakiness is
unknown, ignore it immediately with a commit (without specific review!).
The @Ignore annotation should include the JIRA where the problem is
described/tracked.
What do you think?
Marton
PS: Right now both acceptance tests and unit tests are in bad shape.
Acceptance tests were broken by a commit two weeks ago, unit tests were
broken by a commit 4 days ago. In both cases the problem was not fully
visible due to other existing problems.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
