Yicong-Huang opened a new issue, #4637: URL: https://github.com/apache/texera/issues/4637
### Task Summary Trim redundant compile in the scala CI job and reorder steps so lint failures surface before the expensive dist + tests run. ### Pain point A successful scala CI run on `main` currently takes ~14 min 44 s. Step breakdown for run `25239784635`: | step | duration | |---|---| | Install dependencies (pip) | 1 m 21 s | | Lint with scalafmt | 45 s | | Build distributable bundles (`sbt 'clean; X/dist; ...'`) | 3 m 4 s | | **Compile with sbt (`sbt clean package`)** | **1 m 26 s** | | Lint with scalafix | 47 s | | Run backend tests | 6 m 36 s | Two concrete waste points and one ordering problem: - `Compile with sbt: sbt clean package` runs after dist already compiled every module; it re-cleans the working tree and re-packages, and the resulting jars are not consumed by any subsequent step. - `Build distributable bundles` prefixes its sbt invocation with `clean`, which is a no-op on the very first sbt build but discards earlier compile output if other sbt steps run first. - scalafix lives at the bottom of the job. A bad scalafix lint costs the developer all 3 m 4 s of dist + a redundant 1 m 26 s compile before they see the error. ### Proposal Re-order the scala job's steps to put both linters in front so the cheap checks fail fast, then run dist (without `clean`) which reuses scalafix's compile output, then license/audit/test. Specifically: 1. Drop `Compile with sbt: sbt clean package` entirely. 2. Drop the leading `clean;` from `Build distributable bundles`. 3. Move `Lint with scalafix` to run before `Build distributable bundles` so its compile is reused by dist. 4. Move `Install dependencies` (pip) to run just before `Run backend tests` since only the tests need the python deps; lint and dist failures no longer pay the install cost. Expected savings on a clean run: ~1 m 30 s (dropped redundant compile) plus a few seconds on the dropped clean and on the lint-failure path. ### Priority P2 – Medium ### Task Type - [x] DevOps / Deployment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
