[
https://issues.apache.org/jira/browse/CALCITE-7892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CALCITE-7892:
------------------------------------
Labels: pull-request-available (was: )
> CI test jobs cost about 230 runner-minutes per commit and test the same
> configurations on every run
> ---------------------------------------------------------------------------------------------------
>
> Key: CALCITE-7892
> URL: https://issues.apache.org/jira/browse/CALCITE-7892
> Project: Calcite
> Issue Type: Improvement
> Components: ci
> Reporter: Vladimir Sitnikov
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> One commit of [PR #5350|https://github.com/apache/calcite/pull/5350] used
> about 230 runner-minutes, and its Linux jobs waited 2 hours in the ASF queue.
> Most of that comes from the 11 test jobs in {{.github/workflows/main.yml}}:
> * Windows: JDK 8, 17, 21
> * Linux: JDK 8 with the oldest Guava, JDK 8 with the latest Guava, JDK 11,
> 17, 21, 24
> * Linux: JDK 11 with Avatica main
> * macOS: JDK 21
> Each job is a copy of another with one or two values changed. The set has
> drifted:
> * It tests JDK 24, a non-LTS release that no longer gets updates, and does
> not test JDK 25.
> * It tests only two Guava versions, the oldest ({{21.0}}) and the latest
> ({{33.4.8-jre}}).
> * Every job uses the Zulu JDK and an English locale.
> * Every commit runs the same 11 combinations, so a combination outside them
> is never tested.
> Adding a JDK or a Guava version means copying a 30-line job and editing it by
> hand.
> h3. Proposal
> Generate the test jobs with
> [github-actions-random-matrix|https://github.com/vlsi/github-actions-random-matrix].
> A short script lists the axes (OS, the LTS JDKs and the latest GA JDK, Guava
> versions from {{21.0}} to the latest, time zone, locale, JDK vendor,
> {{-XX:hashCode=2}}, {{-XX:ActiveProcessorCount=1}}) and the rows every run
> must include: Windows, JDK 8, the latest JDK, the oldest Guava, and the
> latest Guava. The library fills the remaining jobs with random rows, choosing
> rows that cover value pairs no earlier row covers.
> The static analysis jobs (ErrorProne, Checker Framework) and the Avatica job
> stay as they are.
> Apache JMeter, pgjdbc, and TestNG generate their CI matrix with this library.
> h3. Acceptance criteria
> * A commit to {{main}} runs 5 or 6 generated test jobs and the Avatica job
> instead of today's 11 test jobs, and uses about half the runner-minutes of
> today's set (estimate: 110–125).
> * Every run includes Windows, JDK 8, the latest JDK, the oldest Guava, and
> the latest Guava.
> * Each job name lists its parameters, and the log prints the seed that
> generated the matrix.
> * A pull request keeps the same rows across pushes. Each commit to {{main}}
> draws new rows.
> * Adding a JDK or a Guava version is a one-line change.
> h3. Out of scope
> * The static analysis jobs and the Avatica job.
> * The ASF-wide runner queue.
> * Running CI only once for a pull request from a branch in apache/calcite;
> that is CALCITE-7884.
> h3. Prior discussion
> * dev@ thread "Calcite CI pending delays" (October 2026):
> https://lists.apache.org/thread/3vglxtojsqh2m03fhcdv76s3v1lsqzmf
> * Proposed on dev@ in 2021:
> https://lists.apache.org/thread.html/gw7klmm4000y6lnbh3yjqj1g1ljswgt5
> * CALCITE-4790 (2021): Alessandro Solimando suggested adding time zones as a
> CI axis.
> * CALCITE-4259 (2020): Julian Hyde wrote that he tested each JDK against
> random Guava versions every night.
> * CALCITE-4816 (2021): Julian Hyde started running tests in various time
> zones and Guava versions on Travis.
> * CALCITE-6580 (2024): Julian Hyde: "the best way to test in multiple locales
> is a CI run with {{-Duser.country=CA -Duser.language=fr}}".
> I maintain the library and can prepare the pull request.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
