[
https://issues.apache.org/jira/browse/SPARK-58175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58175:
-----------------------------------
Labels: pull-request-available (was: )
> Cache descriptor trees built from a binary FileDescriptorSet in
> from_protobuf/to_protobuf
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-58175
> URL: https://issues.apache.org/jira/browse/SPARK-58175
> Project: Spark
> Issue Type: Improvement
> Components: Protobuf
> Affects Versions: 4.2.0
> Reporter: Ben Hollis
> Priority: Minor
> Labels: pull-request-available
>
> from_protobuf / to_protobuf used with a binary FileDescriptorSet hold their
> descriptor as a "@transient lazy val", so each task instance in an executor
> JVM independently rebuilds the entire descriptor object graph from the
> descriptor bytes, with no caching. Under high fan-out (many partitions ->
> many concurrent tasks in one JVM) with large descriptor sets (hundreds of KB,
> thousands of transitive fields), this produces many redundant,
> concurrently-retained copies of the same descriptor tree (tens of millions of
> Descriptors$FieldDescriptor instances), driving sustained executor GC
> pressure.
> Proposal: add a process-wide bounded cache so concurrent task instances share
> one descriptor tree per FileDescriptorSet instead of each rebuilding it.
> Cache at the parseFileDescriptorSet layer (which materializes the
> FieldDescriptor graph) so a given set is parsed at most once per JVM
> regardless of entry point (both the descriptor-tree build and the
> convertAnyFieldsToJson TypeRegistry build share one parsed graph), with
> descriptor trees additionally cached on top of the parse.
> Key the caches on the message name, a content hash of the descriptor bytes,
> and whether extension support is enabled; collapse concurrent misses into a
> single build; do not cache failed loads. Bound the size by a new internal
> config spark.sql.protobuf.descriptorCacheSize (default 8; 0 disables caching
> as a kill-switch).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
