Really great job Raymond! Good to see improvements on CI infra. Definitely helps developer experience a lot better.
On Mon, Sep 6, 2021 at 9:19 PM vino yang <[email protected]> wrote: > awesome! Great job! > > Thanks for driving and landing this big infra improvement! > > Best, > Vino > > Raymond Xu <[email protected]> 于2021年9月4日周六 上午9:42写道： > > > Hi all, > > > > As you may have noticed, we have been running Azure Pipelines for the > tests > > for some time and have recently retired Travis CI in this PR > > <https://github.com/apache/hudi/pull/3556>. > > > > Background > > > > It was a pain for the CI process in the past with Travis, which from time > > to time queued up CI jobs forever. This severely affects the developer > > experience for making contributions, and also the release process. > > > > The New Setup > > > > Thanks to the Flink community, who pioneered the CI setup, and MS Azure, > > who provided the free resources, we are able to mirror the repo and PRs > to > > a separate GitHub organization <https://github.com/apachehudi-ci/> and > run > > the tests in Azure Pipelines. Hudi's ci-bot > > <https://github.com/apachehudi-ci/ci-bot> (forked from Flink's ci-bot > > <https://github.com/flink-ci/ci-bot>) runs on a GCP server and > > periodically > > scans recently changed PRs for CI submission. CI results are commented > back > > to the PR by hudi-bot <https://github.com/hudi-bot>. Full details about > > the > > setup are documented in this > > < > > > https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure > > > > > page > > < > > > https://cwiki.apache.org/confluence/display/HUDI/Guide+on+CI+infrastructure > > > > > . > > > > Azure Pipelines provides 10 free managed parallel jobs. CI tests are > split > > into 5 jobs. We have dedicated resources to test 2 PRs in parallel. > > > > - master builds: > > > > > https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=3 > > - branch builds: > > > > > https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build?definitionId=5 > > > > Note: PRs against asf-site (website updates) will be ignored by this > setup. > > > > Additionally, we make use of GitHub Actions to build against different > > Spark and Scala versions. GitHub Actions jobs also provide fast feedback > > for compliance like checkstyle and apache-rat. > > > > For PR Owners and Reviewers > > > > With these changes, PR owners and reviewers should pay attention to the > > following: > > > > - CI results are indicated in hudi-bot's comment > > - A new commit in the same PR will trigger a new build and cancel any > > existing build > > - Comment `@hudi-bot run azure` to manually trigger a new build > > - GitHub Actions jobs will show as checks in the PR > > - Minimum conditions to merge: > > - Azure CI report shows success, and > > - GitHub Actions jobs passed > > - For website update PRs (for asf-site branch), owners post > screenshots > > to show the changes in lieu of CI tests. > > > > > > Hope this contributes towards a more seamless developer experience. > Please > > reach out to the community for CI issues or further questions. > > > > > > Best, > > Raymond > > > -- Regards, -Sivabalan
