[ https://issues.apache.org/jira/browse/AVRO-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymie Stata updated AVRO-2269: ------------------------------- Description: The class {{org.apache.avro.ipc.io.Perf}} is Avro's performance test suite. This JIRA aims to make it easier to use. Specifically: * Added a file {{performance-testing.html}} with guidance on how to use the suite * Added script {{run-script.sh}} that uses {{Perf}} to run structured experiments. * Added tests for performance of resolution of unchanged unions and enumerations, which will be subject to future optimizations. * Tweaks to {{Perf}} for better experimentation (e.g., support for minimum as well as average aggregation). was: In attempting to use Perf.java to show that proposed performance changes actually improved performance, different runs of Perf.java using the exact same code base resulted variances of 5% or greater – and often 10% or greater – for about half the test cases. With variance this high within a code base, it's impossible to tell if a proposed "improved" code base indeed improves performance. I will post to the wiki and elsewhere some documents and scripts I developed to reduce this variance. This JIRA is for changes to Perf.java that reduce the variance. Specifically: * Access the {{reader}} and {{writer}} instance variables directly in the inner-loop for {{SpecificTest}}, as well as switched to a "reuse" object for reading records, rather than constructing fresh objects for each read. Both helped to significantly reduce variance for {{FooBarSpecificRecordTestWrite}}, a major target of recent performance-improvement efforts. * Switched to {{DirectBinaryEncoder}} instead of {{BufferedBinaryEncoder}} for write tests. Although this slowed writer-tests a bit, it reduced variance a lot, especially for performance tests of primitives like booleans, making it a better choice for measuring the performance-impact of code changes. * Started the timer of a test after the encoder/decoder for the test is constructed, rather than before. Helps a little. * Added the ability to output the _minimum_ runtime of a test case across multiple cycles (vs the total runtime across all cycles). This was inspired by JVMSpec, which used to use a minimum. I was able to reduce the variance of total runtime enough to obviate the need for this metric, but since it's helpful diagnostically, I left it in. > Improve usability of Perf.java > ------------------------------ > > Key: AVRO-2269 > URL: https://issues.apache.org/jira/browse/AVRO-2269 > Project: Apache Avro > Issue Type: Test > Components: java > Reporter: Raymie Stata > Assignee: Raymie Stata > Priority: Major > > The class {{org.apache.avro.ipc.io.Perf}} is Avro's performance test suite. > This JIRA aims to make it easier to use. Specifically: > * Added a file {{performance-testing.html}} with guidance on how to use the > suite > * Added script {{run-script.sh}} that uses {{Perf}} to run structured > experiments. > * Added tests for performance of resolution of unchanged unions and > enumerations, which will be subject to future optimizations. > * Tweaks to {{Perf}} for better experimentation (e.g., support for minimum as > well as average aggregation). -- This message was sent by Atlassian JIRA (v7.6.3#76005)