steveloughran commented on code in PR #15623: URL: https://github.com/apache/iceberg/pull/15623#discussion_r2941734156
########## site/docs/benchmarks.md: ########## @@ -20,111 +20,212 @@ title: "Benchmarks" ## Available Benchmarks and how to run them -Benchmarks are located under `<project-name>/jmh`. It is generally favorable to only run the tests of interest rather than running all available benchmarks. -Also note that JMH benchmarks run within the same JVM as the system-under-test, so results might vary between runs. +Benchmarks are located under `<module>/src/jmh`. It is generally better to run only the benchmarks you are investigating instead of the full suite. +Also note that JMH benchmarks run in the same JVM as the system under test, so results may vary between runs. Review Comment: If look for uses of `@Fork(1)` in the benchmarks you'll see this isn't true. The only @Fork(0) right now is in the PR I'm working on, as I want fast iterations. (#15629). But even with the forking, performance varies a lot on a local system because of what else you are doing on the same system, memory consumption etc. If you kick off a benchmark then have a conversation with an AI agent you will get bad numbers. Better to say --- Iceberg benchmarks execute in a new JVM for each test, for isolation and reproducibility. Other work taking place on the same computer may create demand for CPU, memory or other system resource, and so produce inconsistent results. Benchmarks results should be more reliable if executed on a system without other CPU, RAM or IO processes active at the same time. Even better: run them on a host dedicated exclusively to the benchmark. In-cloud testing with servers similar to production systems is ideal here, though the "noisy neighbour" problem is still a problem. --- -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
