Jungtaek Lim created SPARK-58538:
------------------------------------
Summary: Add branch-4.3 CI scheduler and release integration
Key: SPARK-58538
URL: https://issues.apache.org/jira/browse/SPARK-58538
Project: Spark
Issue Type: Improvement
Components: Project Infra
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
{{branch-4.3}} was cut from {{branch-4.x}} for the 4.3.0 release, but the CI
and release plumbing that {{master}} maintains for each active release branch
has not been added for it yet. This mirrors SPARK-56710 (which set up
{{branch-4.2}} and {{branch-4.x}}) and SPARK-54143 (which set up
{{branch-4.1}}).
h3. Background
The per-branch build workflows live on the branch itself, and {{branch-4.3}}
already inherited a full set of them from {{branch-4.x}} when it was cut
({{build_maven.yml}}, {{build_java17/21/25.yml}}, {{build_non_ansi.yml}},
{{build_python_*.yml}}, etc.). What is missing is on {{master}}: the scheduler
that dispatches those workflows against {{--ref branch-4.3}}, plus the places
that enumerate active branches.
h3. Work items
On {{master}}:
* Add {{.github/workflows/branch43_scheduler.yml}}, modelled on
{{branch42_scheduler.yml}}: cron entries plus a {{workflow_dispatch}}
{{target}} choice, dispatching each build workflow with {{gh workflow run
<workflow>.yml --repo ${{ github.repository }} --ref branch-4.3}}. Cron times
should be staggered against the existing schedulers to spread runner load.
* Add {{branch-4.3}} to the branch list in
{{.github/workflows/publish_snapshot.yml}} (both the {{workflow_dispatch}}
default and the matrix default), which is currently {{'["master", "branch-4.x",
"branch-4.2", "branch-4.1", "branch-4.0", "branch-3.5"]'}}.
* Add a {{branch-4.3}} badge row to {{README.md}}, alongside the existing
{{branch-4.x}} / {{branch-4.2}} / {{branch-4.1}} rows.
* Check whether {{dev/create-release/release-util.sh}} needs any update; it
currently excludes {{master}} and {{branch-4.x}} when enumerating release
branches, and {{branch-4.3}} should be picked up automatically, but this is
worth confirming.
On {{branch-4.3}}:
* The inherited workflows are still titled for {{branch-4.x}} — e.g.
{{build_maven.yml}} reads {{name: "Build / Maven (branch-4.x, Scala 2.13,
Hadoop 3, JDK 17)"}} and {{build_java17.yml}} reads {{name: "Build / Java17
(branch-4.x, ...)"}}. These names appear in the Actions UI and check-run names,
so they should be retitled to {{branch-4.3}} to avoid confusion with the
{{branch-4.x}} runs.
h3. Dependency
The Maven workflows inherited by {{branch-4.3}} carry the bug tracked
separately: they call {{maven_test.yml}} without passing {{branch:}}, and that
workflow's default is {{master}}, so they build {{master}} instead of the
branch. That should be fixed first (or at the same time), otherwise
{{branch-4.3}}'s new scheduler will faithfully dispatch three Maven workflows
that silently test the wrong branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
