mengw15 opened a new issue, #7975:
URL: https://github.com/apache/texera/issues/7975
### Task Summary
`build / frontend (macos-latest)` fails on a **V8 heap exhaustion inside a
Vitest worker**, not on any test. The
worker is killed mid-run, so the leg reports `200 passed (201)` — one spec
file never reports at all — with zero
failed assertions:
```
[22268:0xcbe80c000] 408527 ms: Mark-Compact 2031.6 (2049.3) -> 2026.0
(2052.3) MB, pooled: 0 MB,
7444.75 / 0.00 ms (average mu = 0.211, current mu =
0.027) allocation failure
FATAL ERROR: Ineffective mark-compacts near heap limit - Allocation failed -
JavaScript heap out of memory
Test Files 200 passed (201)
Error: [vitest-pool]: Worker forks emitted error.
Caused by: Error: Worker exited unexpectedly
```
It is currently red on **five open PRs** (#7917, #7961, #7966, #7967, #7968
— same signature in all five) and on
**5 of the last 10 pushes to `main`**. The spec that dies is different every
time (`search-bar.component`,
`codearea-custom-template.component`, ...), because it depends only on which
files a given worker was handed.
### Cause: the configured heap limit never reaches the worker
`frontend/package.json` asks for 8 GB:
```json
"test:ci": "node --max-old-space-size=8192 ./node_modules/nx/dist/bin/nx.js
test ..."
```
That flag applies to the parent process only. When Vitest (4.1.10) spawns
its fork-pool workers it rebuilds
`execArgv` from scratch and keeps **only profiling flags**:
```js
execArgv: [
...process.execArgv.filter(a => a.startsWith("--cpu-prof")
|| a.startsWith("--heap-prof")
|| a.startsWith("--diagnostic-dir")),
"--experimental-import-meta-resolve",
...
]
```
`--max-old-space-size=8192` is dropped, so every worker runs at V8's
**default** heap limit. Reproduced locally:
| process | heap limit |
|---|---|
| parent, `node --max-old-space-size=8192` | 8288 MB |
| child forked with `execArgv: []` | **4192 MB** (the default — parent's
flag has no effect) |
| child, with `NODE_OPTIONS=--max-old-space-size=6144` | **6240 MB** |
`NODE_OPTIONS` is an environment variable and *is* inherited by child
processes; the command-line flag is not.
The macOS image (`macos-26-arm64`) defaults to ~2.05 GB per worker, which is
exactly the ceiling in the crash
log; the ubuntu and windows images have more headroom and have never hit it.
That is why this is a macOS-only
failure even though the suite is identical everywhere.
### Why now, and its relation to #7713
The suite is 201 unit spec files / 5069 tests and still growing. Running the
56 `src/app/dashboard` specs on a
single worker with `logHeapUsage: true` shows this is **not a monotonic
leak** — the heap returns to a
130–275 MB baseline between files — but individual specs peak high:
| peak | spec |
|---|---|
| 712 MB | `search.component.spec.ts` |
| 586 MB | `workflow-execution-history.component.spec.ts` |
| 575 MB | `dataset-detail.component.spec.ts` (195 tests) |
| 379 MB | `filters.component.spec.ts` |
Those are the same specs #7713 listed as timing out on macOS a week ago.
#7713 read the symptom as runner
variance against thin timeout margins and raised
`testTimeout`/`hookTimeout`; what was actually happening is
that a worker near the 2 GB ceiling stalls in stop-the-world GC — single
mark-compact pauses of **7.4 s** and
**29.6 s** appear in these logs — long enough to blow a `beforeEach`.
#7966's one "real" failure this week is
that same effect: `filters.component.spec.ts` failed with `Hook timed out in
30000ms` inside
`TestBed.createComponent`, in the same run as the OOM.
Raising the timeouts bought about a week. The heap now exhausts outright, so
the worker dies instead of stalling.
### An amplifier: the frontend matrix has no `fail-fast: false`
`build.yml` sets `fail-fast: false` on four other matrices but not on
`frontend`, so when the macOS leg dies the
still-running legs are cancelled. On #7917 the windows leg was reported red
purely for this reason — its own run
was `Test Files 201 passed (201)`, `Tests 5068 passed | 1 skipped`, then
`The operation was canceled`.
### Proposed fix
1. Give the workers the memory the repo already intended, via an inherited
variable, on the `frontend` job:
```yaml
env:
NODE_OPTIONS: --max-old-space-size=3072
```
The macOS leg runs 2 concurrent forks (two worker pids appear in the
logs), so 3 GB each keeps the total
within the image's RAM. `poolOptions.forks.execArgv` in
`vitest.config.ts` is an equivalent, more targeted
alternative that would also apply to local runs.
2. Add `fail-fast: false` to the `frontend` matrix, matching the other four
matrices in the same workflow, so one
dead leg stops cancelling the others.
3. Follow-up, not a blocker: the four specs above account for a
disproportionate share of the peak. Worth a look
once the leg is green.
### Task Type
- [ ] Refactor / Cleanup
- [x] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
