[ https://issues.apache.org/jira/browse/AVRO-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421809#comment-17421809 ]
Erik Krogen commented on AVRO-3156: ----------------------------------- [~steven.aerts] this should be fixed by [#1206|https://github.com/apache/avro/pull/1206] from AVRO-3048, can you take a look? > Performance degradation in SpecificRecordBuilder introduced in 1.9.0 > -------------------------------------------------------------------- > > Key: AVRO-3156 > URL: https://issues.apache.org/jira/browse/AVRO-3156 > Project: Apache Avro > Issue Type: Improvement > Components: java > Affects Versions: 1.9.0, 1.10.0 > Environment: Using SpecificData in environments with multiple > classloaders. > Reporter: Steven Aerts > Priority: Major > Attachments: image-2021-06-11-14-27-16-689.png > > > The change introduced in spark 1.9.0 which changed: > {code:java} > SpecificData.get(){code} > into: > {code:java} > SpecificData.getForSchema(schema){code} > introduced a significant performance degradation in environments where the > class of {{schema}} is provided by a different classloader then the > classloader containing {{SpecificData}}. > > A possible solution is to use the {{classCache}} of the default > {{SpecificData}} so the sometimes expensive classloader codepath is cached. > (PR coming up) > We noticed this in after trying out a spark upstep from spark 3.1.0 (avro > 1.8.2) to 3.2.0 (spark 1.10.2) where 74% of the time was spend in millions of > times resolving the same class. > With this patch this resolving time was brought back from 74% to 0.70%. > JMC flamegraph showing this issue: > !image-2021-06-11-14-27-16-689.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)