[
https://issues.apache.org/jira/browse/AVRO-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scott Carey updated AVRO-743:
-----------------------------
Status: Patch Available (was: Open)
Patch implements a simple solution that worked best:
Keep track of the thread that created the GenericDatumReader, and allow only
this thread to access the local resolver reference. Other threads use the
global resolver cache.
Performance results:
{noformat}
GenericRead: 1824 ms, 3.654 million entries/sec.
141.829 million bytes/sec
GenericNested_Read: 2940 ms, 2.267 million entries/sec.
87.989 million bytes/sec
GenericWithDefault_Read: 3364 ms, 1.981 million entries/sec.
76.898 million bytes/sec
GenericWithOutOfOrder_Read: 1799 ms, 3.705 million entries/sec.
143.792 million bytes/sec
GenericWithPromotion_Read: 2020 ms, 3.300 million entries/sec.
128.059 million bytes/sec
GenericOneTimeDecoderUse_Read: 1813 ms, 3.676 million entries/sec.
142.682 million bytes/sec
GenericOneTimeReaderUse_Read: 2290 ms, 2.910 million entries/sec.
112.944 million bytes/sec
GenericOneTimeUse_Read: 2253 ms, 2.959 million entries/sec.
114.831 million bytes/sec
{noformat}
It is just as thread-safe as the prior version. If multiple threads are
concurrently doing setSchema() or setExcpeted() other threads may not see the
changes at the same time. It should be stable, and eventually consistent. If
AVRO-650 was good enough, this should be too.
> Java: Performance Regression and memory pressure with GenericDatumReader
> ------------------------------------------------------------------------
>
> Key: AVRO-743
> URL: https://issues.apache.org/jira/browse/AVRO-743
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.0
> Reporter: Scott Carey
> Assignee: Scott Carey
> Priority: Critical
> Fix For: 1.5.0
>
> Attachments: AVRO-743.patch
>
>
> AVRO-650 introduced a large performance regression and memory bloat issue
> with GenericDatumReader.
> Performance plummets for some Perf.java tests (One test took 1 hour to finish
> on my laptop).
> Some minor changes I tried result in it passing in shorter time, but with
> still an 80% performance degredation.
> This is associated with memory bloat related to ThreadLocals.
> More details provided in comments.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
