Rangsh commented on issue #12058: URL: https://github.com/apache/seatunnel/issues/12058#issuecomment-5611766762
<img width="2868" height="1424" alt="Image" src="https://github.com/user-attachments/assets/72e176b0-9af1-40c6-be69-509ba2170d63"; /> <img width="2870" height="1428" alt="Image" src="https://github.com/user-attachments/assets/8e09ebf3-a6f2-4503-9569-ae4f90efdbc2"; /> <img width="2878" height="1426" alt="Image" src="https://github.com/user-attachments/assets/90415df0-7508-4a8c-8d65-36a4835e2e84"; /> [flame-wall-forward.html](https://github.com/user-attachments/files/32033623/flame-wall-forward.html) [flame-wall-reverse.html](https://github.com/user-attachments/files/32033621/flame-wall-reverse.html) [flame-cpu-forward.html](https://github.com/user-attachments/files/32033624/flame-cpu-forward.html) [gc-profile-summary.md](https://github.com/user-attachments/files/32033622/gc-profile-summary.md) @SEZ9 @nzw921rx thanks — answering the three open diagnostics points. Still on unchanged `origin/dev` only; #12081 remains Related-only and is not used as CV evidence. ### 1. Screenshot + original HTML flame graphs Inline screenshots from the local Diagnostics run (`profile_benchmarks.sh`): 1. **Wall reverse overview** (`flame-wall-reverse.html`) 2. **Wall forward overview** (`flame-wall-forward.html`) 3. **CPU zoomed to the write/sync path** (`flame-cpu-forward.html`, search `hsync`, `Matched: 0.82%`): `WALWorkHandler` → `HdfsWriter.write/flush` → `FSDataOutputStream.hsync` → `RawLocalFileSystem$LocalFSFileOutputStream.write` Original HTML attached in this comment (download + open locally): - `flame-wall-forward.html` - `flame-wall-reverse.html` - `flame-cpu-forward.html` - `flame-cpu-reverse.html` - `gc-profile-summary.md` Note: `profile gc` produces the summary/metrics only (no flame HTML). Wall/CPU modes produce the flame graphs above. Environment for these flames: | Item | Value | | --- | --- | | Code | `origin/dev` baseline (**not** #12081) | | Method | `CheckpointStorageBenchmark.checkpointOverviewIncrementalUpdate$` | | JDK | Corretto **11.0.26** / Apple M1 | | Tooling | `tools/benchmarks/profile_benchmarks.sh` (`wall` / `cpu` / `gc`) | ### 2. GC vs Run B samples `205.64` and `115.29` Using the existing A/A gist artifacts ([e213029…](https://gist.github.com/Rangsh/e21302908815c2954edf4392eafc62c8)): | Sample | Where | Matching GC log | What the GC log shows | | --- | --- | --- | --- | | **205.64 us/op** | Run B, JMH Fork 2 / Iteration 2 | `run-b-gc-fork2.log` | Young pauses exist (including one ~35–40 ms early pause around ~4.3s uptime, consistent with warmup/init). Later young pauses are ~ms-scale. | | **115.29 us/op** | Run B, JMH Fork 1 / Iteration 4 | `run-b-gc-fork1.log` | No evidence of a pause explaining this sample: it is a **fast** iteration, not a spike. | Honest limit: JMH does not emit per-iteration timestamps aligned to `-Xlog:gc*`, so I **cannot** claim the 205.64 sample landed inside a specific pause. What we can say on unchanged code: - GC activity is **not flat** (young pauses are present in every fork). - The large early pause is **not** a credible explanation for a ~+40 µs/op measurement delta by itself without tighter correlation. - The **115.29** sample is inconsistent with “GC pause caused the anomaly” (GC would slow a sample, not speed it up). So: GC remains a **plausible** contributor in principle, but these logs do **not** establish a causal match for the Run B outliers/extremes. ### 3. Keep root-cause search on unchanged code Agreed with @nzw921rx / @SEZ9: - continue on original `origin/dev` artifacts only; - no candidate-fix A/B until a measurable mechanism is agreed; - if GC stays uncorrelated under tighter instrumentation, next single-variable experiment should isolate JIT/scheduling/`file:///` MapStore wait on the same commit — not an A/B against #12081. Happy to take the next agreed narrow step (JFR pause overlap, wall profile on a high-CV day, or another Diagnostics mode). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
