This is an automated email from the ASF dual-hosted git repository.
yuqi1129 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new a80f6fd2e7 [#12583] improvement(ci): Optimize build test execution
(#12584)
a80f6fd2e7 is described below
commit a80f6fd2e70c5d9e76ed98b3fbae706533687740
Author: Qi Yu <[email protected]>
AuthorDate: Fri Aug 28 11:44:42 2026 +0800
[#12583] improvement(ci): Optimize build test execution (#12584)
### What changes were proposed in this pull request?
- Write each Gradle test report to a task-specific directory so
independent test tasks can run in parallel.
- Serialize projects containing Docker-tagged tests through a shared
Gradle build service because they share containers and other test
resources.
- Limit the build job to two Gradle workers to avoid CPU and memory
contention between test JVMs.
- Raise the build job timeout from 90 to 120 minutes.
### Why are the changes needed?
The build job frequently approaches or exceeds its 90-minute timeout.
Gradle parallel execution is enabled, but the shared test report
directory forces all test tasks to run one at a time. Docker-based tests
cannot all be made parallel because several projects share servers,
databases, containers, and configuration files.
### Measured result
The serial baseline ([run
32871678601](https://github.com/apache/gravitino/actions/runs/32871678601/job/97880344037))
executed 62 test tasks with a maximum concurrency of one:
| Metric | Serial baseline |
|---|---:|
| Sum of test task durations | 68.43 min |
| First test start to last test end | 69.65 min |
| `Build with Gradle` | 70.03 min |
| Complete build job | 78.33 min |
Three successful revisions of this PR with `--max-workers=2` produced
consistent results ([run
32729975754](https://github.com/apache/gravitino/actions/runs/32729975754/job/97439835322),
[run
32823784437](https://github.com/apache/gravitino/actions/runs/32823784437/job/97727370085),
and [run
32833427151](https://github.com/apache/gravitino/actions/runs/32833427151/job/97769263031)):
| Metric | Two workers |
|---|---:|
| Maximum concurrent test tasks | 2 |
| Sum of test task durations | 70.19–70.96 min |
| First test start to last test end | 63.04–63.79 min |
| `Build with Gradle` | 63.43–64.13 min |
| Complete build job | 72.52–76.58 min |
This reduces the test and Gradle critical path by about six minutes, or
8–9%. Complete job time varies more because `publishToMavenLocal` alone
varied from about 5 to 12 minutes across these runs.
An experiment without the worker limit ([run
32943743979](https://github.com/apache/gravitino/actions/runs/32943743979/job/98100012872))
allowed four concurrent test tasks, but increased their summed duration
to 82.07 minutes and the test wall time to 68.32 minutes. The Gradle
step regressed to 68.83 minutes because resource contention offset most
of the additional parallelism. The two-worker limit is therefore
retained.
### Remaining bottleneck
Twenty-two test tasks must still hold the shared Docker lock and account
for roughly 61–62 minutes with two workers. `:core:test` is the largest
part of that chain. A material reduction beyond this PR requires
sharding the Docker-heavy tests across separate runners and aggregating
their coverage reports.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Verified from three CI runs that independent test tasks reach a
concurrency of two while Docker-locked tasks remain serialized.
- Verified the uncapped run reaches four concurrent test tasks and
regresses due to resource contention.
- Ran `gradle help --max-workers=2 -PskipWeb=true` successfully with
proxies disabled.
- Ran `git diff --check`.
---
.github/workflows/build.yml | 3 ++-
build.gradle.kts | 33 ++++++++++++++++++++++++++++++++-
2 files changed, 34 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index e7f9117c22..2c3f5a5426 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -152,7 +152,7 @@ jobs:
strategy:
matrix:
java-version: [ 17 ]
- timeout-minutes: 90
+ timeout-minutes: 120
needs: changes
if: needs.changes.outputs.source_changes == 'true'
# Steps represent a sequence of tasks that will be executed as part of the
job
@@ -192,6 +192,7 @@ jobs:
gradle_args=(
build
+ --max-workers=2
-PskipWeb=true
-PskipITs
-PskipDockerTests=false
diff --git a/build.gradle.kts b/build.gradle.kts
index 94ca8ac447..e1d288f139 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -25,6 +25,8 @@ import com.github.jk1.license.render.ReportRenderer
import com.github.vlsi.gradle.dsl.configureEach
import net.ltgt.gradle.errorprone.errorprone
import org.gradle.api.attributes.java.TargetJvmVersion
+import org.gradle.api.services.BuildService
+import org.gradle.api.services.BuildServiceParameters
import org.gradle.api.tasks.testing.logging.TestExceptionFormat
import org.gradle.internal.hash.ChecksumService
import org.gradle.internal.os.OperatingSystem
@@ -34,6 +36,8 @@ import java.util.Locale
Locale.setDefault(Locale.US)
+abstract class SharedTestEnvironmentLock :
BuildService<BuildServiceParameters.None>
+
plugins {
`maven-publish`
id("java")
@@ -61,6 +65,13 @@ plugins {
alias(libs.plugins.errorprone)
}
+val sharedTestEnvironmentLock = gradle.sharedServices.registerIfAbsent(
+ "sharedTestEnvironmentLock",
+ SharedTestEnvironmentLock::class
+) {
+ maxParallelUsages.set(1)
+}
+
val snappyJavaVersion: String = libs.versions.snappy.java.get()
val scalaVersion: String = project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
@@ -677,7 +688,11 @@ subprojects {
showCauses = true
showStackTraces = true
}
-
reports.html.outputLocation.set(file("${rootProject.projectDir}/build/reports/"))
+ // Distinct outputs let Gradle execute independent test tasks in parallel.
+ val testReportPath = path.removePrefix(":").replace(':', '/')
+ reports.html.outputLocation.set(
+ rootProject.layout.buildDirectory.dir("reports/tests/$testReportPath")
+ )
val skipTests = project.hasProperty("skipTests")
if (!skipTests) {
val extraArgs = project.property("extraJvmArgs") as List<String>
@@ -1253,6 +1268,22 @@ gradle.projectsEvaluated {
subprojectJarOutputDirs.map {
it.get().asFile.toPath().toAbsolutePath().normalize() }
allprojects {
+ val runsIntegrationTestsOnly = rootProject.hasProperty("skipTests")
+ val hasDockerTests =
+ rootProject.extra["dockerTest"] == true && fileTree("src/test") {
+ include("**/*.java", "**/*.kt")
+ }.any { it.readText().contains("gravitino-docker-test") }
+
+ // Integration tests in different projects share the same Gravitino
server, database,
+ // containers, and configuration files. Running them together lets one
test stop or reset
+ // resources while another test is still using them. Normal builds keep
independent unit tests
+ // parallel and serialize only projects that contain Docker-tagged tests.
+ if (runsIntegrationTestsOnly || hasDockerTests) {
+ tasks.withType<Test>().configureEach {
+ usesService(sharedTestEnvironmentLock)
+ }
+ }
+
tasks.withType<Jar>().configureEach {
mustRunAfter(cleanDistributionPackageTask)
}
