This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch release/v1.1.0-incubating
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/release/v1.1.0-incubating by
this push:
new b91416567a perf(ci): trim redundant scala compile and reorder for
early lint failure (#4638)
b91416567a is described below
commit b91416567a482984854af48a8506e5eba32f6e45
Author: Yicong Huang <[email protected]>
AuthorDate: Sat May 2 02:59:59 2026 +0000
perf(ci): trim redundant scala compile and reorder for early lint failure
(#4638)
### What changes were proposed in this PR?
Tighten the scala job in `build.yml`:
- Drop `Compile with sbt: sbt clean package` — its `package` output was
unused and it re-cleaned a tree the dist step had just compiled.
- Drop the leading `clean;` from the dist step so it can reuse the lint
compile.
- Merge `scalafmt`, `scalafix`, and all per-module `dist` commands into
a single `sbt` invocation with each as its own argument, so the whole
chain runs in one JVM and sbt exits at the first failing command.
- Move `Create Databases` ahead of any sbt step (the JOOQ source
generators connect to `texera_db` during compile).
- Move `Install dependencies` (pip) just before `Run backend tests`,
since only the test step needs the python deps.
New step order:
```
Create Databases
Setup sbt launcher / coursier cache
sbt scalafmtCheckAll "scalafixAll --check" <Service>/dist ... # one JVM,
fail-fast
Unzip / license check / audit
Install dependencies (pip)
Create texera_db_for_test_cases
Set docker-java API version
Run backend tests
```
### Any related issues, documentation, discussions?
Closes #4637.
### How was this PR tested?
Exercised by this PR's own scala matrix. Each individual command
(scalafmt, scalafix, dist, license check, audit, tests) is unchanged;
only ordering, the merged sbt invocation, and the removal of redundant
`sbt clean package` differ.
Timing comparison on the scala job, sbt-touching steps only (run
[25239784635](https://github.com/apache/texera/actions/runs/25239784635)
before, run
[25241165819](https://github.com/apache/texera/actions/runs/25241165819)
after):
| step | before | after |
|---|---|---|
| Lint with scalafmt | 45 s | (merged) |
| Build distributable bundles (`sbt 'clean; X/dist; ...'`) | 3 m 4 s |
(merged) |
| Compile with sbt (`sbt clean package`) | 1 m 26 s | removed |
| Lint with scalafix | 47 s | (merged) |
| **Combined `sbt scalafmtCheckAll "scalafixAll --check" X/dist ...`** |
— | **4 m 31 s** |
| sbt subtotal | **6 m 2 s** | **4 m 31 s** |
Net savings on the sbt portion ~1 m 30 s (matches the dropped redundant
compile plus one fewer sbt JVM cold-start). uv pip migration is
independent (#4636) and would shave another ~45 s off the python
`Install dependencies` step.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.7
Co-authored-by: github-actions[bot]
<github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
(backported from commit 8a4f2ddf5d086457a555b0898543e4abf0918026)
---
.github/workflows/build.yml | 85 +++++++++++++++++++++++++--------------------
1 file changed, 47 insertions(+), 38 deletions(-)
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 0e96a20281..0635d52cb2 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -157,31 +157,42 @@ jobs:
python-version: "3.11"
- name: Show Python
run: python --version || python3 --version
- - name: Install dependencies
- run: |
- python -m pip install --upgrade pip
- if [ -f amber/requirements.txt ]; then pip install -r
amber/requirements.txt; fi
- if [ -f amber/operator-requirements.txt ]; then pip install -r
amber/operator-requirements.txt; fi
- - name: Setup sbt launcher
- uses: sbt/setup-sbt@508b753e53cb6095967669e0911487d2b9bc9f41 # v1.1.22
- - uses: coursier/cache-action@90c37294538be80a558fd665531fcdc2b467b475 #
v8.1.0
- with:
- extraSbtFiles: '["*.sbt", "project/**.{scala,sbt}",
"project/build.properties" ]'
- - name: Lint with scalafmt
- run: sbt scalafmtCheckAll
- name: Create Databases
+ # Must run before any sbt compile step: the build's JOOQ source
+ # generators connect to texera_db while compiling.
run: |
psql -h localhost -U postgres -f sql/texera_ddl.sql
psql -h localhost -U postgres -f sql/iceberg_postgres_catalog.sql
psql -h localhost -U postgres -f sql/texera_lakefs.sql
env:
PGPASSWORD: postgres
- - name: Build distributable bundles for license check
- # Build every dist-producing module so the union of bundled jars can
- # be diffed against LICENSE-binary.
- run: sbt 'clean; ConfigService/dist; AccessControlService/dist;
FileService/dist; ComputingUnitManagingService/dist;
WorkflowCompilingService/dist; WorkflowExecutionService/dist'
- - name: Unzip JVM distributable bundles
+ - name: Setup sbt launcher
+ uses: sbt/setup-sbt@508b753e53cb6095967669e0911487d2b9bc9f41 # v1.1.22
+ - uses: coursier/cache-action@90c37294538be80a558fd665531fcdc2b467b475 #
v8.1.0
+ with:
+ extraSbtFiles: '["*.sbt", "project/**.{scala,sbt}",
"project/build.properties" ]'
+ - name: Lint and build distributable bundles
+ # Single sbt invocation runs scalafmt -> scalafix -> per-module dist
+ # in order; sbt exits at the first failing command (fail-fast). Each
+ # command is a separate sbt arg, not joined with ';', so a dist
+ # failure aborts the rest. scalafix triggers compile (and JOOQ
+ # codegen), which the dist commands then reuse incrementally.
run: |
+ sbt scalafmtCheckAll \
+ "scalafixAll --check" \
+ ConfigService/dist \
+ AccessControlService/dist \
+ FileService/dist \
+ ComputingUnitManagingService/dist \
+ WorkflowCompilingService/dist \
+ WorkflowExecutionService/dist
+ - name: Unzip dists and check binary licenses
+ # Unzips every service's dist bundle, runs the binding LICENSE-binary
+ # check, then runs the advisory per-dep audit. The audit always runs
+ # (mirroring the previous 'if: always()' on its own step) and never
+ # fails the step; the binding check's exit code drives the result.
+ run: |
+ set -euo pipefail
mkdir -p /tmp/dists
for zip in \
config-service/target/universal/config-service-*.zip \
@@ -192,33 +203,31 @@ jobs:
amber/target/universal/amber-*.zip; do
unzip -q "$zip" -d /tmp/dists/
done
- - name: Check bundled jars against LICENSE-binary
- run: |
- ./bin/licensing/check_binary_deps.py jar \
- /tmp/dists/config-service-*/lib \
- /tmp/dists/access-control-service-*/lib \
- /tmp/dists/file-service-*/lib \
- /tmp/dists/computing-unit-managing-service-*/lib \
- /tmp/dists/workflow-compiling-service-*/lib \
+
+ lib_paths=(
+ /tmp/dists/config-service-*/lib
+ /tmp/dists/access-control-service-*/lib
+ /tmp/dists/file-service-*/lib
+ /tmp/dists/computing-unit-managing-service-*/lib
+ /tmp/dists/workflow-compiling-service-*/lib
/tmp/dists/amber-*/lib
- - name: Audit per-dep license preservation (advisory)
- if: always()
+ )
+
+ check_exit=0
+ ./bin/licensing/check_binary_deps.py jar "${lib_paths[@]}" ||
check_exit=$?
+ ./bin/licensing/audit_jar_licenses.py "${lib_paths[@]}" || true
+ exit "$check_exit"
+ - name: Install dependencies
+ # Only the backend test step needs the python deps; install just
+ # before tests so a lint or dist failure does not pay for them.
run: |
- ./bin/licensing/audit_jar_licenses.py \
- /tmp/dists/config-service-*/lib \
- /tmp/dists/access-control-service-*/lib \
- /tmp/dists/file-service-*/lib \
- /tmp/dists/computing-unit-managing-service-*/lib \
- /tmp/dists/workflow-compiling-service-*/lib \
- /tmp/dists/amber-*/lib
+ python -m pip install --upgrade pip
+ if [ -f amber/requirements.txt ]; then pip install -r
amber/requirements.txt; fi
+ if [ -f amber/operator-requirements.txt ]; then pip install -r
amber/operator-requirements.txt; fi
- name: Create texera_db_for_test_cases
run: psql -h localhost -U postgres -v DB_NAME=texera_db_for_test_cases
-f sql/texera_ddl.sql
env:
PGPASSWORD: postgres
- - name: Compile with sbt
- run: sbt clean package
- - name: Lint with scalafix
- run: sbt "scalafixAll --check"
- name: Set docker-java API version
run: |
echo "api.version=1.52" >> ~/.docker-java.properties
