Hi all, Currently we have a couple for different and independent workflows for CI against the main/release branches and PRs: - 1 workflow for Gradle based CI - 1 workflow for the Helm charts - 1 workflow for the Python Client - 1 workflow for Regression Tests - 1 workflow for Spark Plugin Regression Tests - 1 workflow to check Markdown links
The .asf.yaml file contains references to most but not all jobs in all these workflows, which is a bit hard to maintain. During PR checks, it sometimes happens that a couple of these workflows fail for intermediate remote reasons. Re-running those workflows requires opening about 3-5 different workflow runs and to click on "rerun failed jobs" in each of those. I started an experiment to unify all CI workflows and also simplify the .asf.yaml file in this PR [3625]. A screenshot or how a PR workflow run looks like is in the first comment of that PR. You may notice that there are actually three workflows: - the ci.yml workflow with all the CI related jobs - a new ci-pr.yml workflow and - a new ci-main.yml workflow The latter two call the first one. The difference of both is the definition of their concurrency-group: for PRs, the concurrency-group cancels previous CI workflow runs. For main + release/* branches the workflow runs are not cancelled. For PRs, this means less useless GitHub runner usage, when changes are pushed to a PR while CI jobs are still running. (We all use one limited pool of runners in the whole Apache organization.) For main + release/* branches the concurrency-group ensures that the latest saved Gradle cache state is the most recent one, so eventually less work due to caching for following CI runs. You may also notice a "fake" 'Required Checks' job. This is a "trick" so that we only need to reference one check from the .asf.yaml file and would never have to care about the correct references from .asf.yaml again. The 'Required Checks' dummy job also references all jobs (the admin-tool tests and python-client-tests are currently not referenced as required checks). The unification of all these workflows allowed to simplify & shorten a couple of job definitions. The whole ci.yml file is < 400 lines including the license header. The jobs themselves still do the same thing as before. Except for the Spark Plugin Regtest one, which has been appended to the "normal" Regtest one, because the setup for both is exactly the same. WDYT? I'm curious to hear your thoughts! Robert [3625] https://github.com/apache/polaris/pull/3625 [2] https://github.com/apache/polaris/actions
