goutamadwant commented on code in PR #12307: URL: https://github.com/apache/seatunnel/pull/12307#discussion_r4003078993
########## tools/benchmarks/suites/benchmarks_core.txt: ########## @@ -14,20 +14,11 @@ # limitations under the License. # Fundamental data-path operations. -SeaTunnelRowBenchmark -IntermediateQueueBenchmark -DebeziumJsonFormatBenchmark SeaTunnelPipelineBenchmark.sourceSink$ SeaTunnelPipelineBenchmark.sourceTransformSink$ +SeaTunnelPipelineBenchmark.sourceTransformSinkWithObservabilityAndTrace$ # Checkpoint coordination and storage. CheckpointingTimeBenchmark.checkpointSingleInput$ -CheckpointStorageBenchmark.checkpointPersistenceTransaction$ - -# High-frequency IMap state paths. -IMapJobStorageBenchmark.taskGroupStateTransition$ -IMapJobStorageBenchmark.runningMetricsReport$ - -# DAG persistence and reload. -IMapDagStorageBenchmark.finishedJobDagStore$ -IMapDagStorageBenchmark.finishedJobDagLoad$ +CheckpointStorageBenchmark.checkpointIdAtomicIncrement$ +CheckpointStorageBenchmark.checkpointOverviewIncrementalUpdate$ Review Comment: These two methods are still the subject of #12058 because their within-run variance is high. The official run cited there reports CVs of 14.40%/20.88% on Java 8 and 12.71%/19.72% on Java 11, while the `checkpointPersistenceTransaction` selector removed above was 3.80%/6.70%. Since the benchmark implementation is unchanged at this head, putting the noisy methods into the daily core suite makes ordinary checkpoint deltas inconclusive rather than improving the regression signal. Please keep the transaction benchmark in the core suite until these two methods are stabilized, or provide controlled evidence that their variance is now low enough for daily use. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
