[
https://issues.apache.org/jira/browse/SPARK-58538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-58538:
---------------------------------
Description:
{{branch-4.3}} was cut from {{branch-4.x}} for the 4.3.0 release, but the CI
and release plumbing that {{master}} maintains for each active release branch
has not been added for it yet.
The current model comes from the {{branch-4.2}} work, which happened in two
phases: SPARK-58180 (#57319) prepared the build workflows on {{branch-4.2}} so
they are self-contained and dispatchable, then SPARK-58242 (#57408) added a
single {{branch42_scheduler.yml}} on {{master}} and removed the old per-build
{{build_branch42_*.yml}} files from {{master}}. That superseded the earlier
SPARK-56710 approach, in which the per-branch build files lived on {{master}}.
Only scheduled workflows on the default branch fire, so the scheduler must live
on {{master}} while everything it triggers lives on the branch.
h3. Work items
On {{master}}:
* Add {{.github/workflows/branch43_scheduler.yml}}, modelled on
{{branch42_scheduler.yml}}: cron entries plus a {{workflow_dispatch}}
{{target}} choice, dispatching each build workflow with {{gh workflow run
<workflow>.yml --repo ${{ github.repository }} --ref branch-4.3}}. Cron times
should be staggered against the existing schedulers to spread runner load.
* Add {{branch-4.3}} to the branch list in
{{.github/workflows/publish_snapshot.yml}} (both the {{workflow_dispatch}}
default and the matrix default), which is currently {{'["master", "branch-4.x",
"branch-4.2", "branch-4.1", "branch-4.0", "branch-3.5"]'}}.
* Add a {{branch-4.3}} badge row to {{README.md}}, following the current
release-branch form: badges point at the self-contained workflows filtered by
{{?branch=branch-4.3}} (including {{build_main.yml}}), as SPARK-58242 reworked
the {{branch-4.2}} row to do.
The eight workflows the scheduler dispatches are {{build_java17}},
{{build_java21}}, {{build_java25}}, {{build_maven}}, {{build_maven_java21}},
{{build_non_ansi}}, {{build_python_3.11}} and {{build_python_3.14}}, matching
the {{branch-4.2}} and {{branch-4.x}} schedulers. Note
{{build_maven_java25.yml}} exists on the branch but is deliberately not
dispatched by any existing scheduler, so it is left out here too.
On {{branch-4.3}} (the SPARK-58180 equivalent):
* Remove the dormant {{schedule:}} triggers from the nine inherited
{{build_*.yml}} workflows, leaving {{workflow_dispatch:}}. Scheduled runs never
fire from a non-default branch, so these crons are inert, and the scheduler on
{{master}} is what triggers them. {{branch-4.2}} has {{workflow_dispatch:}}
only.
* Drop the branch token from the workflow titles. The inherited names read e.g.
{{name: "Build / Maven (branch-4.x, Scala 2.13, Hadoop 3, JDK 17)"}}, which
both collides confusingly with the real {{branch-4.x}} runs in the Actions UI
and check-run names, and is the wrong convention: release branches use generic,
non-branch-tagged names, e.g. {{branch-4.2}} has {{name: "Build / Maven (Scala
2.13, Hadoop 3, JDK 17)"}}. The branch is implied by where the file lives. So
the token should be removed, not changed to {{branch-4.3}}.
* Point the {{branch}} input default of both reusable workflows at this branch:
{{build_and_test.yml}} (inherited as {{branch-4.x}}) and {{maven_test.yml}}
(inherited as {{master}}) both to {{branch-4.3}}, so the callers need no
explicit {{branch:}}. On {{branch-4.2}} and {{branch-4.1}} both default to
their own branch.
h3. Verified as needing no change
* {{dev/create-release/release-util.sh}} derives the latest release branch by
excluding {{master}} and {{branch-4.x}} and taking the highest remaining.
Checked against the real remote: it already resolves to {{branch-4.3}}.
* {{dev/merge_spark_pr.py}} was generalized by SPARK-56710 to handle any
{{branch-4.N}} / {{branch-M.x}}; it hardcodes no branch list and its 70
doctests pass unchanged.
h3. Dependency
The {{maven_test.yml}} default on {{branch-4.3}} is the {{branch-4.3}} instance
of SPARK-58537: five callers rely on it ({{build_maven}},
{{build_maven_java21}}, {{build_maven_java25}}, {{build_maven_java21_arm}} and
{{build_maven_java21_macos26}}), so until it is fixed the Maven workflows build
{{master}} rather than the branch. It must be fixed before or with the new
scheduler, otherwise the scheduler will faithfully dispatch Maven workflows
that silently test the wrong branch.
was:
{{branch-4.3}} was cut from {{branch-4.x}} for the 4.3.0 release, but the CI
and release plumbing that {{master}} maintains for each active release branch
has not been added for it yet. This mirrors SPARK-56710 (which set up
{{branch-4.2}} and {{branch-4.x}}) and SPARK-54143 (which set up
{{branch-4.1}}).
h3. Background
The per-branch build workflows live on the branch itself, and {{branch-4.3}}
already inherited a full set of them from {{branch-4.x}} when it was cut
({{build_maven.yml}}, {{build_java17/21/25.yml}}, {{build_non_ansi.yml}},
{{build_python_*.yml}}, etc.). What is missing is on {{master}}: the scheduler
that dispatches those workflows against {{--ref branch-4.3}}, plus the places
that enumerate active branches.
h3. Work items
On {{master}}:
* Add {{.github/workflows/branch43_scheduler.yml}}, modelled on
{{branch42_scheduler.yml}}: cron entries plus a {{workflow_dispatch}}
{{target}} choice, dispatching each build workflow with {{gh workflow run
<workflow>.yml --repo ${{ github.repository }} --ref branch-4.3}}. Cron times
should be staggered against the existing schedulers to spread runner load.
* Add {{branch-4.3}} to the branch list in
{{.github/workflows/publish_snapshot.yml}} (both the {{workflow_dispatch}}
default and the matrix default), which is currently {{'["master", "branch-4.x",
"branch-4.2", "branch-4.1", "branch-4.0", "branch-3.5"]'}}.
* Add a {{branch-4.3}} badge row to {{README.md}}, alongside the existing
{{branch-4.x}} / {{branch-4.2}} / {{branch-4.1}} rows.
* Check whether {{dev/create-release/release-util.sh}} needs any update; it
currently excludes {{master}} and {{branch-4.x}} when enumerating release
branches, and {{branch-4.3}} should be picked up automatically, but this is
worth confirming.
On {{branch-4.3}}:
* The inherited workflows are still titled for {{branch-4.x}} — e.g.
{{build_maven.yml}} reads {{name: "Build / Maven (branch-4.x, Scala 2.13,
Hadoop 3, JDK 17)"}} and {{build_java17.yml}} reads {{name: "Build / Java17
(branch-4.x, ...)"}}. These names appear in the Actions UI and check-run names,
so they should be retitled to {{branch-4.3}} to avoid confusion with the
{{branch-4.x}} runs.
h3. Dependency
The Maven workflows inherited by {{branch-4.3}} carry the bug tracked
separately: they call {{maven_test.yml}} without passing {{branch:}}, and that
workflow's default is {{master}}, so they build {{master}} instead of the
branch. That should be fixed first (or at the same time), otherwise
{{branch-4.3}}'s new scheduler will faithfully dispatch three Maven workflows
that silently test the wrong branch.
> Add branch-4.3 CI scheduler and release integration
> ---------------------------------------------------
>
> Key: SPARK-58538
> URL: https://issues.apache.org/jira/browse/SPARK-58538
> Project: Spark
> Issue Type: Improvement
> Components: Project Infra
> Affects Versions: 4.3.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> {{branch-4.3}} was cut from {{branch-4.x}} for the 4.3.0 release, but the CI
> and release plumbing that {{master}} maintains for each active release branch
> has not been added for it yet.
> The current model comes from the {{branch-4.2}} work, which happened in two
> phases: SPARK-58180 (#57319) prepared the build workflows on {{branch-4.2}}
> so they are self-contained and dispatchable, then SPARK-58242 (#57408) added
> a single {{branch42_scheduler.yml}} on {{master}} and removed the old
> per-build {{build_branch42_*.yml}} files from {{master}}. That superseded the
> earlier SPARK-56710 approach, in which the per-branch build files lived on
> {{master}}. Only scheduled workflows on the default branch fire, so the
> scheduler must live on {{master}} while everything it triggers lives on the
> branch.
> h3. Work items
> On {{master}}:
> * Add {{.github/workflows/branch43_scheduler.yml}}, modelled on
> {{branch42_scheduler.yml}}: cron entries plus a {{workflow_dispatch}}
> {{target}} choice, dispatching each build workflow with {{gh workflow run
> <workflow>.yml --repo ${{ github.repository }} --ref branch-4.3}}. Cron times
> should be staggered against the existing schedulers to spread runner load.
> * Add {{branch-4.3}} to the branch list in
> {{.github/workflows/publish_snapshot.yml}} (both the {{workflow_dispatch}}
> default and the matrix default), which is currently {{'["master",
> "branch-4.x", "branch-4.2", "branch-4.1", "branch-4.0", "branch-3.5"]'}}.
> * Add a {{branch-4.3}} badge row to {{README.md}}, following the current
> release-branch form: badges point at the self-contained workflows filtered by
> {{?branch=branch-4.3}} (including {{build_main.yml}}), as SPARK-58242
> reworked the {{branch-4.2}} row to do.
> The eight workflows the scheduler dispatches are {{build_java17}},
> {{build_java21}}, {{build_java25}}, {{build_maven}}, {{build_maven_java21}},
> {{build_non_ansi}}, {{build_python_3.11}} and {{build_python_3.14}}, matching
> the {{branch-4.2}} and {{branch-4.x}} schedulers. Note
> {{build_maven_java25.yml}} exists on the branch but is deliberately not
> dispatched by any existing scheduler, so it is left out here too.
> On {{branch-4.3}} (the SPARK-58180 equivalent):
> * Remove the dormant {{schedule:}} triggers from the nine inherited
> {{build_*.yml}} workflows, leaving {{workflow_dispatch:}}. Scheduled runs
> never fire from a non-default branch, so these crons are inert, and the
> scheduler on {{master}} is what triggers them. {{branch-4.2}} has
> {{workflow_dispatch:}} only.
> * Drop the branch token from the workflow titles. The inherited names read
> e.g. {{name: "Build / Maven (branch-4.x, Scala 2.13, Hadoop 3, JDK 17)"}},
> which both collides confusingly with the real {{branch-4.x}} runs in the
> Actions UI and check-run names, and is the wrong convention: release branches
> use generic, non-branch-tagged names, e.g. {{branch-4.2}} has {{name: "Build
> / Maven (Scala 2.13, Hadoop 3, JDK 17)"}}. The branch is implied by where the
> file lives. So the token should be removed, not changed to {{branch-4.3}}.
> * Point the {{branch}} input default of both reusable workflows at this
> branch: {{build_and_test.yml}} (inherited as {{branch-4.x}}) and
> {{maven_test.yml}} (inherited as {{master}}) both to {{branch-4.3}}, so the
> callers need no explicit {{branch:}}. On {{branch-4.2}} and {{branch-4.1}}
> both default to their own branch.
> h3. Verified as needing no change
> * {{dev/create-release/release-util.sh}} derives the latest release branch by
> excluding {{master}} and {{branch-4.x}} and taking the highest remaining.
> Checked against the real remote: it already resolves to {{branch-4.3}}.
> * {{dev/merge_spark_pr.py}} was generalized by SPARK-56710 to handle any
> {{branch-4.N}} / {{branch-M.x}}; it hardcodes no branch list and its 70
> doctests pass unchanged.
> h3. Dependency
> The {{maven_test.yml}} default on {{branch-4.3}} is the {{branch-4.3}}
> instance of SPARK-58537: five callers rely on it ({{build_maven}},
> {{build_maven_java21}}, {{build_maven_java25}}, {{build_maven_java21_arm}}
> and {{build_maven_java21_macos26}}), so until it is fixed the Maven workflows
> build {{master}} rather than the branch. It must be fixed before or with the
> new scheduler, otherwise the scheduler will faithfully dispatch Maven
> workflows that silently test the wrong branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
