[
https://issues.apache.org/jira/browse/PARQUET-1261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425885#comment-16425885
]
ASF GitHub Bot commented on PARQUET-1261:
-----------------------------------------
julienledem commented on issue #92: PARQUET-1261 - Remove string interning
URL: https://github.com/apache/parquet-format/pull/92#issuecomment-378676662
thanks a lot for the details Scott.
I'd like to add that the number of distinct strings here is not that big,
since it is the name of the fields in schemas (individual field names not fully
qualified paths). They are referred many times though especially when
inspecting footers from many files. If that's a problem we can switch to a
different deduping mechanism. It seems the overhead of a separate map would
still be reasonable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Parquet-format interns strings when reading filemetadata
> --------------------------------------------------------
>
> Key: PARQUET-1261
> URL: https://issues.apache.org/jira/browse/PARQUET-1261
> Project: Parquet
> Issue Type: Bug
> Affects Versions: 1.9.0
> Reporter: Robert Kruszewski
> Assignee: Robert Kruszewski
> Priority: Major
>
> Parquet-format when deserializing metadata will intern strings. References I
> could find suggested that it had been done to reduce memory pressure early
> on. Java (and jvm in particular) went a long way since then and interning is
> generally discouraged, see
> [https://shipilev.net/jvm-anatomy-park/10-string-intern/] for a good
> explanation. What is more since java 8 there's string deduplication
> implemented at GC level per [http://openjdk.java.net/jeps/192.] During our
> usage and testing we found the interning to cause significant gc pressure for
> long running applications due to bigger GC root set.
> This issue proposes removing interning given it's questionable whether it
> should be used in modern jvms.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
