goutamadwant commented on code in PR #12307: URL: https://github.com/apache/seatunnel/pull/12307#discussion_r4003078993
########## tools/benchmarks/suites/benchmarks_core.txt: ########## @@ -14,20 +14,11 @@ # limitations under the License. # Fundamental data-path operations. -SeaTunnelRowBenchmark -IntermediateQueueBenchmark -DebeziumJsonFormatBenchmark SeaTunnelPipelineBenchmark.sourceSink$ SeaTunnelPipelineBenchmark.sourceTransformSink$ +SeaTunnelPipelineBenchmark.sourceTransformSinkWithObservabilityAndTrace$ # Checkpoint coordination and storage. CheckpointingTimeBenchmark.checkpointSingleInput$ -CheckpointStorageBenchmark.checkpointPersistenceTransaction$ - -# High-frequency IMap state paths. -IMapJobStorageBenchmark.taskGroupStateTransition$ -IMapJobStorageBenchmark.runningMetricsReport$ - -# DAG persistence and reload. -IMapDagStorageBenchmark.finishedJobDagStore$ -IMapDagStorageBenchmark.finishedJobDagLoad$ +CheckpointStorageBenchmark.checkpointIdAtomicIncrement$ +CheckpointStorageBenchmark.checkpointOverviewIncrementalUpdate$ Review Comment: The two benchmarks added here are still being investigated in #12058 for high measurement variance. In the published run, their CVs were 12.7–20.9%, compared with 3.8–6.7% for `checkpointPersistenceTransaction`, which this change removes. That level of noise can hide a real regression or make a normal run appear regressed. Could we keep `checkpointPersistenceTransaction` in the daily suite until these two benchmarks are stabilized, or add controlled results showing that their variance is now acceptable? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
