[
https://issues.apache.org/jira/browse/AVRO-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872656#comment-13872656
]
Tie Liu commented on AVRO-1428:
-------------------------------
Our code doesn't compute hashcode directly, but our schema are pretty big, it
contains hundreds of types, and quite some types contain dozens of fields.
Different types do have fields of same type. With the way current computeHash
works, every time when it need to generate the hashcode for the schema of a
type, it will generate the hashcode for every fields even though the hashcode
for that field type has been computed before. Below is a stack trace of typical
computeHash:
TRACE 313672: (thread=200023)
java.util.LinkedHashMap$LinkedHashIterator.<init>(LinkedHashMap.java:366)
java.util.LinkedHashMap$LinkedHashIterator.<init>(LinkedHashMap.java:366)
java.util.LinkedHashMap$EntryIterator.<init>(LinkedHashMap.java:412)
java.util.LinkedHashMap$EntryIterator.<init>(LinkedHashMap.java:412)
java.util.LinkedHashMap.newEntryIterator(LinkedHashMap.java:419)
java.util.HashMap$EntrySet.iterator(HashMap.java:1050)
java.util.AbstractMap.hashCode(AbstractMap.java:492)
org.apache.avro.Schema.computeHash(Schema.java:327)
org.apache.avro.Schema$UnionSchema.computeHash(Schema.java:787)
org.apache.avro.Schema$Field.hashCode(Schema.java:403)
java.util.AbstractList.hashCode(AbstractList.java:541)
org.apache.avro.Schema$RecordSchema.computeHash(Schema.java:605)
org.apache.avro.Schema$UnionSchema.computeHash(Schema.java:787)
org.apache.avro.Schema$Field.hashCode(Schema.java:403)
java.util.AbstractList.hashCode(AbstractList.java:541)
org.apache.avro.Schema$RecordSchema.computeHash(Schema.java:605)
org.apache.avro.Schema$UnionSchema.computeHash(Schema.java:787)
org.apache.avro.Schema$Field.hashCode(Schema.java:403)
java.util.AbstractList.hashCode(AbstractList.java:541)
org.apache.avro.Schema$RecordSchema.computeHash(Schema.java:605)
As you can see, the hashcode for the schema will be computed many many times
unnecessarily. The code my colleague provide is just to demonstrate that
computeHash is expensive operation.
> Schema.computeHash() to add if check to avoid unnecessary hashcode computation
> ------------------------------------------------------------------------------
>
> Key: AVRO-1428
> URL: https://issues.apache.org/jira/browse/AVRO-1428
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Tie Liu
> Attachments: AVRO-1428.patch, AVRO-1428.patch-V2, TestAvro.java
>
>
> In current Schma.java we have following implementation:
> public final int hashCode() {
> if (hashCode == NO_HASHCODE)
> hashCode = computeHash();
> return hashCode;
> }
> int computeHash() { return getType().hashCode() + props.hashCode(); }
> While hashCode is doing the checking of "if (hashCode == NO_HASHCODE)", the
> computeHash method is not. But the computeHash method is being called from
> Schema$Field.hashCode and the subclasses hashCode implementation like
> following:
> public int hashCode() { return name.hashCode() + schema.computeHash(); }
> //this is from Schema$Field class
> This is causing the the calculation of hashCode getting called
> unnecessarily extensively. The proposed changed is to add the "if" check
> inside the computeHash method instead:
> int computeHash()
> {
> if (hashCode == NO_HASHCODE)
> {
> hashCode = getType().hashCode() + props.hashCode();
> }
> return hashCode;
> }
> We did a simple test to compare the performance difference, below is a
> summary of the heap snapshot of comparing the difference:
> As a test I wrote a small program that creates a HashMap<Schema.Field,
> Integer>() and enters a loop simply identifying whether various Schema.Field
> instances are keys in the map. Obviously this is a pathological test case,
> but when running with the current implementation of Schema.Field it has (in
> about 30 seconds) used up nearly 8 GBytes of heap in instantiating
> intermediate objects associated with calling Schema.computeHash():
> Heap
> PSYoungGen total 17432576K, used 8666481K [0x0000000340000000,
> 0x0000000800000000, 0x0000000800000000)
> eden space 14942208K, 58% used
> [0x0000000340000000,0x0000000550f5c650,0x00000006d0000000)
> from space 2490368K, 0% used
> [0x0000000768000000,0x0000000768000000,0x0000000800000000)
> to space 2490368K, 0% used
> [0x00000006d0000000,0x00000006d0000000,0x0000000768000000)
> ParOldGen total 1048576K, used 0K [0x0000000300000000,
> 0x0000000340000000, 0x0000000340000000)
> object space 1048576K, 0% used
> [0x0000000300000000,0x0000000300000000,0x0000000340000000)
> PSPermGen total 21504K, used 5782K [0x00000002fae00000,
> 0x00000002fc300000, 0x0000000300000000)
> object space 21504K, 26% used
> [0x00000002fae00000,0x00000002fb3a5818,0x00000002fc300000)
> When running with the modified implementation (and no other change) all the
> object allocation vanishes:
> Heap
> PSYoungGen total 17432576K, used 896532K [0x0000000340000000,
> 0x0000000800000000, 0x0000000800000000)
> eden space 14942208K, 6% used
> [0x0000000340000000,0x0000000376b852d0,0x00000006d0000000)
> from space 2490368K, 0% used
> [0x0000000768000000,0x0000000768000000,0x0000000800000000)
> to space 2490368K, 0% used
> [0x00000006d0000000,0x00000006d0000000,0x0000000768000000)
> ParOldGen total 1048576K, used 0K [0x0000000300000000,
> 0x0000000340000000, 0x0000000340000000)
> object space 1048576K, 0% used
> [0x0000000300000000,0x0000000300000000,0x0000000340000000)
> PSPermGen total 21504K, used 5768K [0x00000002fae00000,
> 0x00000002fc300000, 0x0000000300000000)
> object space 21504K, 26% used
> [0x00000002fae00000,0x00000002fb3a2240,0x00000002fc300000)
> As a side-effect the test runs x3 faster with the modified hashCode()
> implementation.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
