DanielLeens commented on PR #12165: URL: https://github.com/apache/seatunnel/pull/12165#issuecomment-6210071482
Thanks for the detailed benchmark work and for keeping this thread updated, @Vivek1106-04, @nzw921rx and @SEZ9. My review state for this PR is unchanged: head is still `c773cc091` (no push since 2026-09-22), and my APPROVED on that head stands. The cleanup in #12418 is a different matter, so I want to keep the two separate: 1. **What this PR changes versus what the benchmark measures.** The question @nzw921rx raised (CV shrinking as the pipeline count grows) is about the measurement harness in #12418, not about the code in this PR. Vivek's analysis in the comment on #12418 (the 200 ms interval floor giving the one-pipeline point about 10x fewer triggers and therefore much slower JIT warmup, the drift at `pipelineNum = 1`, and the heavy-tail sampling noise at 10 and 100) reads as a plausible explanation to me. I have not re-run the sweep myself, so I am treating the numbers as the author's measurements and not as something I verified. 2. **Why it still matters here.** I agree with the sequencing already proposed: the efficiency comparison of this PR against `dev` should be re-run only after the harness fix lands, because until then the one-pipeline point is dominated by warmup drift and would not be a fair comparison either way. At the time of writing I could not find a PR for the `fix-12418-benchmark-warmup` branch yet, so that is the next step on the author's side. 3. **When the comparison is re-run.** Please post the result on this PR. If the rerun needs any new commit on top of `c773cc091`, or if you sync with `dev` first (the branch is currently 176 commits behind, with no conflict reported by GitHub), please call that out so the CI attribution on the earlier fork run can be redone on the new head instead of carried over. I will re-review any new head promptly. A committer's final look is still needed for the merge itself, which I cannot trigger from a comment-only account. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
