Aaron Kwan created SPARK-60829:
----------------------------------
Summary: Avoid copying the whole SQL config registry on every
SQLConf entry registration
Key: SPARK-60829
URL: https://issues.apache.org/jira/browse/SPARK-60829
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Aaron Kwan
SPARK-35701 changed {{SQLConf.sqlConfEntries}} to a copy-on-write {{HashMap}}
so that lookups don't take a lock. A side effect is that every {{register}}
call copies the entire map, so registering n entries during {{SQLConf}} class
initialization costs O(n^2) entry copies and allocations. With the ~900 entries
defined in {{SQLConf}}, plus entries from other modules and extensions, this
adds measurable time to the first access of {{SQLConf}}, which is on the
startup path of every driver and executor.
A {{ConcurrentHashMap}} keeps lookups lock-free (as SPARK-34573 originally
proposed) while making registration O(1). Writers still serialize on the
existing lock, so the duplicate-key check stays atomic. Iteration over
{{getConfigEntries()}} becomes weakly consistent instead of a snapshot; no
caller relies on snapshot semantics, and it never throws
{{ConcurrentModificationException}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
