[
https://issues.apache.org/jira/browse/AVRO-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raymie Stata updated AVRO-2269:
-------------------------------
Summary: Improve usability of Perf.java (was: Improve variances seen
across Perf.java runs)
> Improve usability of Perf.java
> ------------------------------
>
> Key: AVRO-2269
> URL: https://issues.apache.org/jira/browse/AVRO-2269
> Project: Apache Avro
> Issue Type: Test
> Components: java
> Reporter: Raymie Stata
> Assignee: Raymie Stata
> Priority: Major
>
> In attempting to use Perf.java to show that proposed performance changes
> actually improved performance, different runs of Perf.java using the exact
> same code base resulted variances of 5% or greater – and often 10% or greater
> – for about half the test cases. With variance this high within a code base,
> it's impossible to tell if a proposed "improved" code base indeed improves
> performance. I will post to the wiki and elsewhere some documents and scripts
> I developed to reduce this variance. This JIRA is for changes to Perf.java
> that reduce the variance. Specifically:
> * Access the {{reader}} and {{writer}} instance variables directly in the
> inner-loop for {{SpecificTest}}, as well as switched to a "reuse" object for
> reading records, rather than constructing fresh objects for each read. Both
> helped to significantly reduce variance for
> {{FooBarSpecificRecordTestWrite}}, a major target of recent
> performance-improvement efforts.
> * Switched to {{DirectBinaryEncoder}} instead of {{BufferedBinaryEncoder}}
> for write tests. Although this slowed writer-tests a bit, it reduced variance
> a lot, especially for performance tests of primitives like booleans, making
> it a better choice for measuring the performance-impact of code changes.
> * Started the timer of a test after the encoder/decoder for the test is
> constructed, rather than before. Helps a little.
> * Added the ability to output the _minimum_ runtime of a test case across
> multiple cycles (vs the total runtime across all cycles). This was inspired
> by JVMSpec, which used to use a minimum. I was able to reduce the variance
> of total runtime enough to obviate the need for this metric, but since it's
> helpful diagnostically, I left it in.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
