Rangsh commented on issue #12266: URL: https://github.com/apache/seatunnel/issues/12266#issuecomment-5864623717
Thanks @SEZ9 — fully aligned, and thank you for confirming the Phase A → Phase B split and for holding both PRs until #12173 is healthy. On the heap numbers you asked for: * **Normal Benchmarks / Diagnostics fork budget** for this growth workload is fixed by `IMapJobStorageBenchmark`'s `@Fork` JVM args: `-Xms4g` / `-Xmx4g` (with G1 + `AlwaysPreTouch`). The Benchmarks / Diagnostics workflows do not override those JMH heap args, so **4g** is the acceptance heap budget for `initialStoredJobCount=1000`. * **Earlier full-WAL reload OOM**: I do not have a precise peak-heap capture from that failure (the fork died before a usable heap dump / constrained profile landed). The concrete incident was Diagnostics run https://github.com/Rangsh/seatunnel/actions/runs/34317023975 on `432bdb3d9`, which hit `Java heap space` inside `WALReader.loadAllData` / `FileMapStore.loadAll` while doing a first-iteration **full-batch** durability reload under `initialStoredJobCount=1000` stacked on resident pressure — i.e. it exhausted the normal **4g** fork budget. After narrowing to a single representative-key sample (`5bbc304b2`), a local `-Xmx2g` smoke completed without OOM. So for Phase A I will treat “must not OOM under the 4g Benchmarks/Diagnostics fork at `initialStoredJobCount=1000`” as the bar, and include a constrained-heap diagnostic / regression guard as **supporting evidence** alongside the deterministic selective-load tests (as Daniel suggested) — without relying on heap heuristics for the contract itself. For Phase B, agreed: I will add an explicit test/assertion that the durability sample runs only from the iteration / trial tear-down hooks (outside the measured `SingleShot` path), so measured growth scores cannot silently absorb reload cost. Still no implementation PR for this issue until #12173 is healthy. Thanks again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
