Rangsh commented on issue #12058: URL: https://github.com/apache/seatunnel/issues/12058#issuecomment-5578521367
@SEZ9 @nzw921rx following up with the first local reproduction you asked for on this issue. ### Scope - **One method only:** `CheckpointStorageBenchmark.checkpointOverviewIncrementalUpdate` - **Codebase:** `origin/dev` @ `8bea8c681` (baseline, **not** #12081) — so this is evidence about the original symptom, not about the PR’s sync-collapse claim - Existing JMH settings retained (no CLI override of forks/warmup/measurement) ### Environment | Item | Value | | --- | --- | | Machine | Apple M1, 16 GB, macOS 14.7.4 (`Darwin 23.6.0 arm64`) | | JDK | Corretto **11.0.26** (`JAVA_HOME` pointed at this JDK for build + run) | | Build | `./mvnw -Pbenchmark -pl seatunnel-benchmarks -am -DskipTests -Dskip.spotless=true package` | | Command | `java -jar seatunnel-benchmarks/target/benchmarks.jar 'CheckpointStorageBenchmark.checkpointOverviewIncrementalUpdate$' -rf json -rff results.json -v EXTRA` | ### JMH / state-store configuration (existing defaults) From annotations / templates (unchanged): - Mode: `SingleShotTime`, unit `us/op`, `@Threads(1)` - `@Fork(3)` with `-Xms4g -Xmx4g -XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+DisableExplicitGC -XX:ActiveProcessorCount=4 -Djava.net.preferIPv4Stack=true` - `@Warmup(iterations = 3)`, `@Measurement(iterations = 5)` (from `BenchmarkBase`) - `@OperationsPerInvocation(100)` — each iteration times 100 overview updates, then normalizes to us/op - State store: Hazelcast `engine*` MapStore, `write-delay-seconds: 0`, `FileMapStoreFactory`, `storage.type: hdfs`, **`fs.defaultFS: file:///`** (`hazelcast-storage.yaml.template`) ### Results (Score / Error / CV) **JMH summary** ```text Benchmark Mode Cnt Score Error Units CheckpointStorageBenchmark.checkpointOverviewIncrementalUpdate ss 15 190.087 ± 65.997 us/op ``` - Score: **190.087 us/op** - Error (99.9% CI half-width): **±65.997 us/op** (~**34.7%** of Score) - From raw iteration samples (n=15): mean **190.087**, stdev **61.734**, **CV ≈ 32.5%** - Range: **128.4 – 393.3 us/op** (max/min ≈ **3.06×**) So the high within-run dispersion from #12058 **reproduces locally** on a consistent machine (here CV is even higher than the Actions-reported ~20%). ### Per-iteration raw samples (measurement only) | Fork | Iterations (us/op) | | --- | --- | | 0 | 182.7, 149.7, 159.5, **240.3**, 188.2 | | 1 | 171.3, 154.7, 177.1, 170.7, 164.8 | | 2 | 198.3, 128.4, 175.1, **393.3**, 197.0 | Observations: 1. **Within-fork spikes exist** (not only between forks): Fork 0 has one ~240 us/op sample; Fork 2 has one ~393 us/op sample (~2.2× that fork’s median). 2. Fork 1 is comparatively tight (~155–177 us/op) — so dispersion is **bursty**, not a uniform “always noisy” distribution. 3. Using median **175.1** and “slower = >1.2× median”: the clear slower samples are **240.3** and **393.3**. Removing only the 393 outlier drops CV from ~32.5% to ~**15.1%** — i.e. a small number of heavy iterations dominate the reported Error/CV. 4. Warmup Iteration 1 on each fork was visually polluted in the log by `SeaTunnelConfig` INFO lines (no clean us/op printed for WI1). Measurement iterations look clean; still flagging as a **measurement-boundary** note per your methodology (warmup/setup logging vs timed phase). ### Hypotheses (ruled in / out so far) | Hypothesis | Status | Notes | | --- | --- | --- | | High CV is only an Actions runner artifact | **Ruled out (for this method)** | Reproduced on one local M1 machine with fixed JDK/settings | | Variance is only between forks (different JVMs) | **Partially ruled out** | Clear within-fork outliers in forks 0 and 2 | | `file:///` LocalFileSystem path is what the benchmark measures | **Ruled in** | Confirmed by template `fs.defaultFS: file:///` + MapStore write-through (`write-delay-seconds: 0`) | | Redundant HDFS-branch syncs (3× `hsync`/`hflush` on real DFS) fully explain **this** local CV | **Not supported yet / likely weak** | Local path uses `LocalFileSystem`, not `HdfsDataOutputStream`/`DFSOutputStream`. Method-call-count collapse on the HDFS branches does not automatically map to disk-sync-count on `file:///`. Treating #12081 sync-collapse as a **working hypothesis only**, as already agreed | | LocalFileSystem `hsync`/`hflush` are interchangeable for mid-stream visibility in tests | **Caution** | Separately, #12081’s durability test needed `setWriteChecksum(false)` to match production; checksum buffering can hide mid-stream visibility. That is a **test/config** lesson, not yet a CV root cause | | Fork-Actions before/after from #12081 closes this issue | **Ruled out as closing evidence** | Different CPUs + bundled changes; kept only as prior input | ### Open questions / where I’m unsure how to isolate next 1. What exactly are the slow iterations waiting on? I do **not** have `ASYNC_PROFILER_HOME` installed locally yet, so I have not captured wall/CPU/lock profiles for the slow samples. Next step options: install async-profiler and run `tools/benchmarks/profile_benchmarks.sh profile wall` on this same method, and/or use the `Benchmarks Diagnostics` workflow. 2. Are the spikes GC/safepoint related? Not measured yet (`-prof gc` / JFR next). 3. How much of the timed path is MapStore WAL append vs monitor/overview CPU work? Need wall profile attribution to `FileMapStore` / `HdfsWriter.flush` / Hazelcast vs `CheckpointMonitorService` itself. 4. Should we treat the WI1 logging noise as a benchmark hygiene issue (separate from production variance)? ### Ask Does this local reproduction look like a good baseline to you? If yes, I’ll proceed with **wall (+ optionally GC/JFR)** on the same machine/settings, focusing on contrasting a normal iteration (~160–180 us/op) vs a slow one (~390 us/op), before proposing any production or benchmark change. Happy to adjust the next experiment if you want a different isolation order (e.g. GC first, or also run `checkpointIdAtomicIncrement` the same way). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
