Rangsh commented on issue #12063: URL: https://github.com/apache/seatunnel/issues/12063#issuecomment-5585771019
Thanks @nzw921rx — followed your guidance and used **Benchmarks Diagnostics** (fork) + downloaded artifacts for local analysis, modeled after #12176. ### Causal claim The high CV is driven by the growth fixture calling `evict` + `FileMapStore.loadAll` on **every JMH iteration tear-down**. `FileMapStore.loadAll` replays the entire append-only WAL (`WALReader.loadAllData`), so reload cost grows across SingleShot samples and pollutes the next measurement. ### Diagnostics runs (same tool, same method/params) Method: `IMapJobStorageBenchmark.runningJobGrowth$` Params: `initialStoredJobCount=0` JDK: 11 JMH args: `-f 1 -wi 2 -i 3` | Rev | Ref | GC+JFR | Wall | | --- | --- | --- | --- | | Before | `dev` | https://github.com/Rangsh/seatunnel/actions/runs/34229761252 | https://github.com/Rangsh/seatunnel/actions/runs/34230694806 | | After | PR branch `improve/zeta-job-lifecycle-growth-stability-12063` (#12173) | https://github.com/Rangsh/seatunnel/actions/runs/34229094919 | https://github.com/Rangsh/seatunnel/actions/runs/34230337852 | Artifacts were downloaded locally (`profile-summary.md`, `summary-wall.txt`, flame HTML, JFR). ### Before: wall evidence — iteration tear-down full WAL reload Wall stacks on `dev` repeatedly show: ```text MapProxyImpl.loadAll IMapJobGrowthBenchmarkWorkload.reloadFromMapStore IMapJobGrowthBenchmarkWorkload.verifyLastRunningJobDurability # called from iteration tear-down path (verifyGrowthPhase) ... WALReader.loadAllData IMapFileStorage.loadAll FileMapStore.loadAll ``` Frame counts in `summary-wall.txt` (before): | Frame | Count | | --- | ---: | | `reloadFromMapStore` | 8 | | `verifyLastRunningJobDurability` | 8 | | `verifyLastGrowthPhaseDurability` | 0 | | `WALReader.loadAllData` | 5 | | `FileMapStore.loadAll` | 4 | ### After: wall evidence — reload only on trial tear-down After the fixture change, the same reload stack appears only under trial durability: ```text MapProxyImpl.loadAll reloadFromMapStore verifyLastRunningJobDurability verifyLastGrowthPhaseDurability # trial tear-down only ``` Frame counts (after): | Frame | Count | | --- | ---: | | `reloadFromMapStore` | 3 | | `verifyLastRunningJobDurability` | 3 | | `verifyLastGrowthPhaseDurability` | 3 | | `WALReader.loadAllData` | 1 | | `FileMapStore.loadAll` | 1 | So the expensive full-WAL `loadAll` is no longer on the iteration tear-down path; it runs once per trial, matching the intended coverage split. ### GC diagnostics (same method/params) | Rev | Alloc/op | GC count | GC time | | --- | ---: | ---: | ---: | | Before (`dev`) | 340,361 B/op | 1 | 5 ms | | After (PR) | 232,200 B/op | 0 | n/a | (Profiled Score is diagnostic-only and not compared to normal benchmark Score, per the workflow summary.) ### Causal chain (complete) 1. **Symptom**: high sample-to-sample CV on growth SingleShot methods (#12063 table). 2. **Mechanism**: iteration tear-down → `reloadFromMapStore` → `FileMapStore.loadAll` → `WALReader.loadAllData` (full WAL replay). 3. **Evidence**: before wall stacks + higher reload/WAL frame counts; after stacks move reload under `verifyLastGrowthPhaseDurability` with fewer `loadAll` frames; GC alloc/op drops. 4. **Fix**: keep resident checks per iteration; move durable `loadAll` sample to trial tear-down (+ unique keys). Production single-`put` remains separately documented in #12173; happy to split it if preferred once this fixture causal chain is accepted. Flame HTML / JFR are in the workflow artifacts above (download products). I can add cropped screenshots of the forward wall flames if you want them inlined like #12176. Next: I can run the same GC+wall pair for `completedJobHistoryGrowth` and/or `initialStoredJobCount=1000` if you want that matrix completed the same way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
