[
https://issues.apache.org/jira/browse/AVRO-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985550#action_12985550
]
Scott Carey commented on AVRO-743:
----------------------------------
On trunk, Perf.java does not complete in reasonable time for me. I can make
some changes that help, but these get the following performance:
{noformat}
NestedRecordTest: 1525 ms, 32.77809426848799 million entries/sec.
82.48148529345245 million bytes/sec
ResolverTest: 746 ms, 11.162490388711324 million entries/sec.
433.1971901548255 million bytes/sec
MigrationWithDefaultTest: 2986 ms, 2.7902796608268243 million entries/sec.
108.28598876454534 million bytes/sec
GenericReaderTest: 1510 ms, 2.758188340605312 million entries/sec.
107.0486888261762 million bytes/sec
GenericReaderOneTimeUseReaderTest: 7258 ms, 0.5739884392267267 million
entries/sec. 22.27995399275671 million bytes/sec
GenericReaderOneTimeUseDecoderTest: 1569 ms, 2.6550866272593567 million
entries/sec. 103.03571383053837 million bytes/sec
GenericReaderOneTimeUseTest: 7662 ms, 0.5437721933184865 million entries/sec.
21.10381644161428 million bytes/sec
GenericReaderTestWithDefaultTest: 2615 ms, 1.593089132314149 million
entries/sec. 61.826693773206664 million bytes/sec
GenericReaderTestWithOutOfOrderTest: 1542 ms, 2.7016336892067576 million
entries/sec. 104.87577931741035 million bytes/sec
GenericReaderTestWithPromotionTest: 1558 ms, 2.6727042262917378 million
entries/sec. 103.74088785226581 million bytes/sec
{noformat}
Performance when I revert the AVRO-650 changes is much better for GenericReader
tests:
{noformat}
NestedRecordTest: 1324 ms, 37.760699871915705 million entries/sec.
95.01951472969381 million bytes/sec
ResolverTest: 715 ms, 11.648566400183402 million entries/sec.
452.0609701034574 million bytes/sec
MigrationWithDefaultTest: 2919 ms, 2.8541473504518238 million entries/sec.
110.76458473406217 million bytes/sec
GenericReaderTest: 1109 ms, 3.754252987904214 million entries/sec.
145.70943802892032 million bytes/sec
GenericReaderOneTimeUseReaderTest: 1454 ms, 2.864026615891172 million
entries/sec. 111.15055988231816 million bytes/sec
GenericReaderOneTimeUseDecoderTest: 1132 ms, 3.6795452959841604 million
entries/sec. 142.80018121749544 million bytes/sec
GenericReaderOneTimeUseTest: 1471 ms, 2.830739253347936 million entries/sec.
109.8712732110088 million bytes/sec
GenericReaderTestWithDefaultTest: 2020 ms, 2.06191332095476 million
entries/sec. 80.03430501173852 million bytes/sec
GenericReaderTestWithOutOfOrderTest: 1124 ms, 3.7038683266586956 million
entries/sec. 143.73524875433927 million bytes/sec
GenericReaderTestWithPromotionTest: 1440 ms, 2.8925109202440358 million
entries/sec. 112.26017811174887 million bytes/sec
{noformat}
My conclusion after several variations and experiments is that:
A new ThreadLocal per GenericDatumReader simply won't work, its very expensive.
You can do better by keeping a static
ThreadLocal<WeakIdentityHashMap<GenericDatumReader,ResolvingDecoder> but this
is still too slow and creates a lot of mess on the heap. In the
"GenericReaderOneTimeUseReaderTest it bloats the map size tremendously because
the map keeps an entry for every object not yet GC'd. 'jmap' output looks like
this during the test:
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 371287 23762368
org.apache.avro.util.WeakIdentityHashMap$IdentityWeakReference
2: 359761 17268528 java.util.HashMap$Entry
3: 171171 6846840 org.apache.avro.generic.GenericDatumReader
4: 258 4241968 [Ljava.util.HashMap$Entry;
{noformat}
When I revert AVRO-650 it is much faster and the most common objects on the
heap are much more sane:
{noformat}
num #instances #bytes class name
----------------------------------------------
1: 299380 7185120 java.lang.Long
2: 299125 7179000 java.lang.Double
3: 1470 6658728 [B
4: 12485 1707464 <constMethodKlass>
{noformat}
What it comes down to, is that either our map, or the one on the thread, has to
keep track of every instance created between garbage collections if we want to
track per thread state for an object. The global ThreadLocal for resolvers
works because its key is (Schema actual, Schema expected) we aren't creating
and destroying a large number of Schema objects and attempting to resolve them
and that is a use case that makes little sense.
The best long term options I have thought of so far are then:
* Leave GenericDatumReader in the state before AVRO-650 -- not thread safe --
and fix the places elsewhere that assume it is thread safe.
* Make GenericDatumReader threadsafe via immutability. This would require
some other changes (builder pattern?) to allow for the cases where we now call
setSchema and setExpected after construction.
I think the second option is the better one, but it would be an API break in a
significant way. The more I think about it, the more that I feel that all of
the complicated behavioral objects should be immutable -- Schema/Protocol,
DatumReader/DatumWriter, ResolvingDecoders, etc. That or not thread safe --
mutability + threadsafe is either hard, bug prone, or performs poorly.
The first option is easier and might be OK for 1.5.0 -- fix the stuff in ipc
that is re-using decoders across threads calling setSchema or setExpected.
> Java: Performance Regression and memory pressure with GenericDatumReader
> ------------------------------------------------------------------------
>
> Key: AVRO-743
> URL: https://issues.apache.org/jira/browse/AVRO-743
> Project: Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.5.0
> Reporter: Scott Carey
> Priority: Critical
> Fix For: 1.5.0
>
>
> AVRO-650 introduced a large performance regression and memory bloat issue
> with GenericDatumReader.
> Performance plummets for some Perf.java tests (One test took 1 hour to finish
> on my laptop).
> Some minor changes I tried result in it passing in shorter time, but with
> still an 80% performance degredation.
> This is associated with memory bloat related to ThreadLocals.
> More details provided in comments.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
