[
https://issues.apache.org/jira/browse/IGNITE-28823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Vinogradov updated IGNITE-28823:
--------------------------------------
Description:
h2. Problem
The \{{check-java}} job ("Check java code on JDK 17") in
\{{.github/workflows/commit-check.yml}}
is the long pole of the "Code Style, Abandoned Tests, Javadocs" workflow: it
takes ~8.9 min
while every other job in the workflow (.NET, ducktape) finishes in under 1 min,
so the whole
workflow is gated by it.
The job runs three sequential Maven invocations:
|| Step || Time || Command ||
| codestyle + licenses | ~5.4 min | \{{mvnw test-compile
-Pall-java,licenses,lgpl,checkstyle,examples,scala,check-licenses}} |
| abandoned tests | ~2.4 min | \{{mvnw test
-Pcheck-test-suites,all-java,scala}} |
| javadocs | ~0.8 min | \{{mvnw ... install ... && mvnw initialize -Pjavadoc}} |
Three inefficiencies:
# The build is single-threaded (no \{{-T}}), although GitHub \{{ubuntu-latest}}
runners have 4 vCPUs.
# The "abandoned tests" step recompiles the entire reactor a second time.
maven-compiler-plugin
3.15 logs _"Recompiling the module because of changed dependency"_ for 28
modules (including
ignite-core, 3322 source files). Because this second invocation uses a
different profile set than
the codestyle step, the plugin's incremental engine treats reactor dependencies
as changed and
cascades a full recompile. As a result ignite-core and ~30 modules are compiled
twice per job.
# The local Maven repository is not cached, so the codestyle step downloads
~343 MB
(2074 artifacts) from Maven Central on every run.
h2. Change
{code:none}
- name: Setup java
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: ${\{ matrix.java }}
+ cache: 'maven'
- name: Run codestyle and licenses checks
run: |
- ./mvnw test-compile -P...,check-licenses -B -V
+ ./mvnw test-compile -P...,check-licenses -B -V -T 1C
- name: Run abandoned tests checks.
+ # Reuse classes from the previous step; the differing profiles
otherwise trigger a full reactor recompile.
run : |
- ./mvnw test -Pcheck-test-suites,all-java,scala -B -V
+ ./mvnw test -Pcheck-test-suites,all-java,scala -B -V
-Dmaven.compiler.useIncrementalCompilation=false
{code}
* \{{cache: 'maven'}} on \{{setup-java}} — cache \{{~/.m2/repository}} between
runs.
* \{{-T 1C}} — parallel reactor build, applied to the codestyle step only.
* \{{-Dmaven.compiler.useIncrementalCompilation=false}} — switches the compiler
plugin to
timestamp-based staleness so the "abandoned tests" step reuses classes produced
by the codestyle
step instead of recompiling the whole reactor.
The "abandoned tests" step is intentionally *not* parallelized: the orphan-test
collection writes
to a shared file finalized by the last reactor module, so \{{-T}} would race.
h2. Why not merge the two steps into one invocation
Merging would pull \{{examples}}/\{{lgpl}} modules into the orphan-test check
(today it runs only
over \{{-Pall-java,scala}}) and could newly fail it. The chosen flag keeps the
orphan-check scope
identical: profile-conditional source roots (\{{java-lgpl}}) exist only in the
\{{examples}} module,
which is not part of the "abandoned tests" reactor.
h2. Local verification (full reactor)
|| || before || after ||
| step 2 recompiled modules | 30 (28 "changed dependency") | 2 (0 "changed
dependency") |
| step 1 under \{{-T 1C}} | n/a | 47 modules SUCCESS, 0 failures, checkstyle
included |
The reproduced cascade (30 modules / 28 "changed dependency") matches the CI
logs exactly, and the
flag eliminates it. The \{{~/.m2}} cache helps once it is populated (master
pushes and subsequent PR
runs); the first cold run still pays the download. Exact before/after
wall-clock is visible on the
PR's own CI run, since the change modifies \{{commit-check.yml}} itself.
h2. PR
[apache/ignite#13281|https://github.com/apache/ignite/pull/13281]
was:
h2. Problem
The \{{check-java}} job ("Check java code on JDK 17") in
\{{.github/workflows/commit-check.yml}}
is the long pole of the "Code Style, Abandoned Tests, Javadocs" workflow: it
takes ~8.9 min
while every other job in the workflow (.NET, ducktape) finishes in under 1 min,
so the whole
workflow is gated by it.
The job runs three sequential Maven invocations:
|| Step || Time || Command ||
| codestyle + licenses | ~5.4 min | \{{mvnw test-compile
-Pall-java,licenses,lgpl,checkstyle,examples,scala,check-licenses}} |
| abandoned tests | ~2.4 min | \{{mvnw test
-Pcheck-test-suites,all-java,scala}} |
| javadocs | ~0.8 min | \{{mvnw ... install ... && mvnw initialize -Pjavadoc}} |
Two inefficiencies:
# The build is single-threaded (no \{{-T}}), although GitHub \{{ubuntu-latest}}
runners have 4 vCPUs.
# The "abandoned tests" step recompiles the entire reactor a second time.
maven-compiler-plugin
3.15 logs _"Recompiling the module because of changed dependency"_ for 28
modules (including
ignite-core, 3322 source files). Because this second invocation uses a
different profile set than
the codestyle step, the plugin's incremental engine treats reactor dependencies
as changed and
cascades a full recompile. As a result ignite-core and ~30 modules are compiled
twice per job.
h2. Proposed change
{code:none}
- name: Run codestyle and licenses checks
run: |
- ./mvnw test-compile -P...,check-licenses -B -V
+ ./mvnw test-compile -P...,check-licenses -B -V -T 1C
- name: Run abandoned tests checks.
+ # Reuse classes from the previous step; the differing profiles
otherwise trigger a full reactor recompile.
run : |
- ./mvnw test -Pcheck-test-suites,all-java,scala -B -V
+ ./mvnw test -Pcheck-test-suites,all-java,scala -B -V
-Dmaven.compiler.useIncrementalCompilation=false
{code}
* \{{-T 1C}} — parallel reactor build, applied to the codestyle step only.
* \{{-Dmaven.compiler.useIncrementalCompilation=false}} — switches the compiler
plugin to
timestamp-based staleness so the "abandoned tests" step reuses classes produced
by the codestyle
step instead of recompiling the whole reactor.
The "abandoned tests" step is intentionally *not* parallelized: the orphan-test
collection writes
to a shared file finalized by the last reactor module, so \{{-T}} would race.
h2. Why not merge the two steps into one invocation
Merging would pull \{{examples}}/\{{lgpl}} modules into the orphan-test check
(today it runs only
over \{{-Pall-java,scala}}) and could newly fail it. The chosen flag keeps the
orphan-check scope
identical: profile-conditional source roots (\{{java-lgpl}}) exist only in the
\{{examples}} module,
which is not part of the "abandoned tests" reactor.
h2. Local verification (full reactor)
|| || before || after ||
| step 2 recompiled modules | 30 (28 "changed dependency") | 2 (0 "changed
dependency") |
| step 1 under \{{-T 1C}} | n/a | 47 modules SUCCESS, 0 failures, checkstyle
included |
The reproduced cascade (30 modules / 28 "changed dependency") matches the CI
logs exactly, and the
flag eliminates it. Exact before/after wall-clock will be visible on this
issue's own PR run, since
the change modifies \{{commit-check.yml}}.
h2. Out of scope
Caching \{{~/.m2/repository}} (e.g. via \{{actions/setup-java}} \{{cache:
maven}}) would further cut
dependency resolution in both steps — left for a separate change.
> Reduce duration of the "Check java code" GitHub Actions job (commit-check.yml)
> ------------------------------------------------------------------------------
>
> Key: IGNITE-28823
> URL: https://issues.apache.org/jira/browse/IGNITE-28823
> Project: Ignite
> Issue Type: Task
> Reporter: Anton Vinogradov
> Assignee: Anton Vinogradov
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Problem
> The \{{check-java}} job ("Check java code on JDK 17") in
> \{{.github/workflows/commit-check.yml}}
> is the long pole of the "Code Style, Abandoned Tests, Javadocs" workflow: it
> takes ~8.9 min
> while every other job in the workflow (.NET, ducktape) finishes in under 1
> min, so the whole
> workflow is gated by it.
> The job runs three sequential Maven invocations:
> || Step || Time || Command ||
> | codestyle + licenses | ~5.4 min | \{{mvnw test-compile
> -Pall-java,licenses,lgpl,checkstyle,examples,scala,check-licenses}} |
> | abandoned tests | ~2.4 min | \{{mvnw test
> -Pcheck-test-suites,all-java,scala}} |
> | javadocs | ~0.8 min | \{{mvnw ... install ... && mvnw initialize
> -Pjavadoc}} |
> Three inefficiencies:
> # The build is single-threaded (no \{{-T}}), although GitHub
> \{{ubuntu-latest}} runners have 4 vCPUs.
> # The "abandoned tests" step recompiles the entire reactor a second time.
> maven-compiler-plugin
> 3.15 logs _"Recompiling the module because of changed dependency"_ for 28
> modules (including
> ignite-core, 3322 source files). Because this second invocation uses a
> different profile set than
> the codestyle step, the plugin's incremental engine treats reactor
> dependencies as changed and
> cascades a full recompile. As a result ignite-core and ~30 modules are
> compiled twice per job.
> # The local Maven repository is not cached, so the codestyle step downloads
> ~343 MB
> (2074 artifacts) from Maven Central on every run.
> h2. Change
> {code:none}
> - name: Setup java
> uses: actions/setup-java@v4
> with:
> distribution: 'temurin'
> java-version: ${\{ matrix.java }}
> + cache: 'maven'
> - name: Run codestyle and licenses checks
> run: |
> - ./mvnw test-compile -P...,check-licenses -B -V
> + ./mvnw test-compile -P...,check-licenses -B -V -T 1C
> - name: Run abandoned tests checks.
> + # Reuse classes from the previous step; the differing profiles
> otherwise trigger a full reactor recompile.
> run : |
> - ./mvnw test -Pcheck-test-suites,all-java,scala -B -V
> + ./mvnw test -Pcheck-test-suites,all-java,scala -B -V
> -Dmaven.compiler.useIncrementalCompilation=false
> {code}
> * \{{cache: 'maven'}} on \{{setup-java}} — cache \{{~/.m2/repository}}
> between runs.
> * \{{-T 1C}} — parallel reactor build, applied to the codestyle step only.
> * \{{-Dmaven.compiler.useIncrementalCompilation=false}} — switches the
> compiler plugin to
> timestamp-based staleness so the "abandoned tests" step reuses classes
> produced by the codestyle
> step instead of recompiling the whole reactor.
> The "abandoned tests" step is intentionally *not* parallelized: the
> orphan-test collection writes
> to a shared file finalized by the last reactor module, so \{{-T}} would race.
> h2. Why not merge the two steps into one invocation
> Merging would pull \{{examples}}/\{{lgpl}} modules into the orphan-test check
> (today it runs only
> over \{{-Pall-java,scala}}) and could newly fail it. The chosen flag keeps
> the orphan-check scope
> identical: profile-conditional source roots (\{{java-lgpl}}) exist only in
> the \{{examples}} module,
> which is not part of the "abandoned tests" reactor.
> h2. Local verification (full reactor)
> || || before || after ||
> | step 2 recompiled modules | 30 (28 "changed dependency") | 2 (0 "changed
> dependency") |
> | step 1 under \{{-T 1C}} | n/a | 47 modules SUCCESS, 0 failures, checkstyle
> included |
> The reproduced cascade (30 modules / 28 "changed dependency") matches the CI
> logs exactly, and the
> flag eliminates it. The \{{~/.m2}} cache helps once it is populated (master
> pushes and subsequent PR
> runs); the first cold run still pays the download. Exact before/after
> wall-clock is visible on the
> PR's own CI run, since the change modifies \{{commit-check.yml}} itself.
> h2. PR
> [apache/ignite#13281|https://github.com/apache/ignite/pull/13281]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
