Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 44d56510339c89635255d554d1a9754abba0b4d8
https://github.com/WebKit/WebKit/commit/44d56510339c89635255d554d1a9754abba0b4d8
Author: Dewei Zhu <[email protected]>
Date: 2026-09-10 (Thu, 10 Sep 2026)
Changed paths:
M Tools/Scripts/webkitpy/benchmark_runner/benchmark_results.py
M Tools/Scripts/webkitpy/benchmark_runner/benchmark_results_unittest.py
M Tools/Scripts/webkitpy/benchmark_runner/benchmark_runner.py
M Tools/Scripts/webkitpy/benchmark_runner/benchmark_runner_unittest.py
M Tools/Scripts/webkitpy/benchmark_runner/run_benchmark.py
M Tools/Scripts/webkitpy/benchmark_runner/webserver_benchmark_runner.py
Log Message:
-----------
`run-benchmark` should show per-iteration value after each iteration and in
the final output
https://bugs.webkit.org/show_bug.cgi?id=323177
rdar://186420926
Reviewed by Aakash Jain.
BenchmarkResults flattened the values of every iteration together before
aggregating them, so a run only ever reported one number per metric once all
of the iterations had finished. A long benchmark therefore gave no feedback
at all until the very end.
Keep the shape of the subtest values so that they can be aggregated per
iteration, and report the aggregate of each iteration as a "per-iteration"
suffix in the summary. BenchmarkResults.format() takes a new
show_iteration_aggregates argument so that a caller can ask for the summary
without that suffix. It defaults to None, which means "not
show_iteration_raw_values", so the output of every existing caller is
unchanged.
BenchmarkRunner uses that to report progress while the benchmark is running.
After each iteration it formats that iteration's own result with max_depth=1,
which reports the top level metric only and never the subtests, and logs it
right after the "End the iteration" line. The iteration aggregates are turned
off there because they are meaningless for a single iteration. A result that
cannot be formatted is logged as a warning instead, so that it cannot abort an
otherwise successful run.
Rename --show-iteration-values to --show-iteration-raw-values, keeping the old
spelling as an alias, and label its output "raw" to tell it apart from the
per-iteration suffix.
* Tools/Scripts/webkitpy/benchmark_runner/benchmark_results.py:
(BenchmarkResults.format):
(BenchmarkResults._format_tests):
(BenchmarkResults._format_values):
(BenchmarkResults._mean):
(BenchmarkResults._aggregate_results_for_test):
(BenchmarkResults._aggregate_values_by_iteration):
(BenchmarkResults._subtest_values_by_config_iteration):
(BenchmarkResults._value_buckets_like):
(BenchmarkResults._collect_values):
* Tools/Scripts/webkitpy/benchmark_runner/benchmark_results_unittest.py:
(BenchmarkResultsTest.test_format):
(BenchmarkResultsTest.test_format_with_depth_limit):
(BenchmarkResultsTest.test_format_without_iteration_aggregates):
(BenchmarkResultsTest.test_format_values_with_iteration_raw_values):
(BenchmarkResultsTest.test_format_values_with_no_unit_scaling_and_iteration_raw_values):
(BenchmarkResultsTest.test_format_values_with_iteration_aggregates):
(BenchmarkResultsTest.test_aggregate_results_with_gropus):
(BenchmarkResultsTest.test_format_results_with_groups):
* Tools/Scripts/webkitpy/benchmark_runner/benchmark_runner.py:
(BenchmarkRunner.__init__):
(BenchmarkRunner._run_benchmark):
(BenchmarkRunner.show_results):
(BenchmarkRunner._format_iteration_results):
(BenchmarkRunner._show_iteration_results):
* Tools/Scripts/webkitpy/benchmark_runner/benchmark_runner_unittest.py:
(iteration_result_with_score):
(FormatIterationResultsTest.test_only_top_level_metrics_are_formatted):
(FormatIterationResultsTest.test_scale_unit_is_honored):
(FormatIterationResultsTest.test_debug_output_is_ignored):
(FormatIterationResultsTest.test_grouped_values_within_one_iteration):
(MockBrowserDriver.prepare_initial_env):
(MockBrowserDriver.prepare_env):
(MockBrowserDriver.restore_env):
(MockBrowserDriver.restore_env_after_all_testing):
(FakeBenchmarkRunner.__init__):
(FakeBenchmarkRunner._run_one_test):
(ShowIterationResultsTest.test_logs_top_level_metrics):
(ShowIterationResultsTest.test_honors_runner_formatting_options):
(ShowIterationResultsTest.test_honors_show_iteration_raw_values):
(ShowIterationResultsTest.test_malformed_results_are_reported_but_not_raised):
(RunBenchmarkIterationLoggingTest.test_top_level_metrics_are_logged_after_each_iteration):
* Tools/Scripts/webkitpy/benchmark_runner/run_benchmark.py:
(config_argument_parser):
(run_benchmark_plan):
(start):
* Tools/Scripts/webkitpy/benchmark_runner/webserver_benchmark_runner.py:
(WebServerBenchmarkRunner.__init__):
Canonical link: https://commits.webkit.org/320873@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
