[
https://issues.apache.org/jira/browse/SPARK-60829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-60829:
--------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Avoid copying the whole SQL config registry on every SQLConf entry
> registration
> -------------------------------------------------------------------------------
>
> Key: SPARK-60829
> URL: https://issues.apache.org/jira/browse/SPARK-60829
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Aaron Kwan
> Assignee: Aaron Kwan
> Priority: Major
> Labels: pull-request-available
>
> SPARK-35701 changed {{SQLConf.sqlConfEntries}} to a copy-on-write {{HashMap}}
> so that lookups don't take a lock. A side effect is that every {{register}}
> call copies the entire map, so registering n entries during {{SQLConf}} class
> initialization costs O(n^2) entry copies and allocations. With the ~900
> entries defined in {{SQLConf}}, plus entries from other modules and
> extensions, this adds measurable time to the first access of {{SQLConf}},
> which is on the startup path of every driver and executor.
> A {{ConcurrentHashMap}} keeps lookups lock-free (as SPARK-34573 originally
> proposed) while making registration O(1). Writers still serialize on the
> existing lock, so the duplicate-key check stays atomic. Iteration over
> {{getConfigEntries()}} becomes weakly consistent instead of a snapshot; no
> caller relies on snapshot semantics, and it never throws
> {{ConcurrentModificationException}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
