[
https://issues.apache.org/jira/browse/SPARK-60829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-60829.
---------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 60044
[https://github.com/apache/spark/pull/60044]
> Avoid copying the whole SQL config registry on every SQLConf entry
> registration
> -------------------------------------------------------------------------------
>
> Key: SPARK-60829
> URL: https://issues.apache.org/jira/browse/SPARK-60829
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Aaron Kwan
> Assignee: Aaron Kwan
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> SPARK-35701 changed {{SQLConf.sqlConfEntries}} to a copy-on-write {{HashMap}}
> so that lookups don't take a lock. A side effect is that every {{register}}
> call copies the entire map, so registering n entries during {{SQLConf}} class
> initialization costs O(n^2) entry copies and allocations. With the ~900
> entries defined in {{SQLConf}}, plus entries from other modules and
> extensions, this adds measurable time to the first access of {{SQLConf}},
> which is on the startup path of every driver and executor.
> A {{ConcurrentHashMap}} keeps lookups lock-free (as SPARK-34573 originally
> proposed) while making registration O(1). Writers still serialize on the
> existing lock, so the duplicate-key check stays atomic. Iteration over
> {{getConfigEntries()}} becomes weakly consistent instead of a snapshot; no
> caller relies on snapshot semantics, and it never throws
> {{ConcurrentModificationException}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
