+1 this is a truly champion effort. On Tue, Sep 7, 2021 at 11:51 AM Sivabalan <[email protected]> wrote:
> Really great job Raymond! Good to see improvements on CI infra. Definitely > helps developer experience a lot better. > > > On Mon, Sep 6, 2021 at 9:19 PM vino yang <[email protected]> wrote: > > > awesome! Great job! > > > > Thanks for driving and landing this big infra improvement! > > > > Best, > > Vino > > > > Raymond Xu <[email protected]> 于2021年9月4日周六 上午9:42写道： > > > > > Hi all, > > > > > > As you may have noticed, we have been running Azure Pipelines for the > > tests > > > for some time and have recently retired Travis CI in this PR > > > <https://github.com/apache/hudi/pull/3556>. > > > > > > Background > > > > > > It was a pain for the CI process in the past with Travis, which from > time > > > to time queued up CI jobs forever. This severely affects the developer > > > experience for making contributions, and also the release process. > > > > > > The New Setup > > > > > > Thanks to the Flink community, who pioneered the CI setup, and MS > Azure, > > > who provided the free resources, we are able to mirror the repo and PRs > > to > > > a separate GitHub organization <https://github.com/apachehudi-ci/> and > > run > > > the tests in Azure Pipelines. Hudi's ci-bot > > > <https://github.com/apachehudi-ci/ci-bot> (forked from Flink's ci-bot > > > <https://github.com/flink-ci/ci-bot>) runs on a GCP server and > > > periodically > > > scans recently changed PRs for CI submission. CI results are commented > > back > > > to the PR by hudi-bot <https://github.com/hudi-bot>. Full details > about > > > the > > > setup are documented in this > > > < > > > > > > https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure > > > > > > > page > > > < > > > > > > https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure > > > > > > > . > > > > > > Azure Pipelines provides 10 free managed parallel jobs. CI tests are > > split > > > into 5 jobs. We have dedicated resources to test 2 PRs in parallel. > > > > > > - master builds: > > > > > > > > > https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=3 > > > - branch builds: > > > > > > > > > https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=5 > > > > > > Note: PRs against asf-site (website updates) will be ignored by this > > setup. > > > > > > Additionally, we make use of GitHub Actions to build against different > > > Spark and Scala versions. GitHub Actions jobs also provide fast > feedback > > > for compliance like checkstyle and apache-rat. > > > > > > For PR Owners and Reviewers > > > > > > With these changes, PR owners and reviewers should pay attention to the > > > following: > > > > > > - CI results are indicated in hudi-bot's comment > > > - A new commit in the same PR will trigger a new build and cancel > any > > > existing build > > > - Comment `@hudi-bot run azure` to manually trigger a new build > > > - GitHub Actions jobs will show as checks in the PR > > > - Minimum conditions to merge: > > > - Azure CI report shows success, and > > > - GitHub Actions jobs passed > > > - For website update PRs (for asf-site branch), owners post > > screenshots > > > to show the changes in lieu of CI tests. > > > > > > > > > Hope this contributes towards a more seamless developer experience. > > Please > > > reach out to the community for CI issues or further questions. > > > > > > > > > Best, > > > Raymond > > > > > > > > -- > Regards, > -Sivabalan >
