aaronkwan-db opened a new pull request, #60044:
URL: https://github.com/apache/spark/pull/60044
### What changes were proposed in this pull request?
`SQLConf.sqlConfEntries` changes from a `@volatile` copy-on-write
`java.util.HashMap` to a `ConcurrentHashMap`:
- `register` and `unregister` still hold `sqlConfEntriesUpdateLock`, keep
the duplicate-key check, and still advance `sqlConfEntriesVersion` after the
update. They now `put` or `remove` in place instead of copying the map.
- `ConcurrentHashMap` rejects `null` keys, so a small `registeredEntry`
helper keeps `getConfigEntry(null)` and `containsConfigKey(null)` returning
`null` and `false`, as before.
### Why are the changes needed?
Since SPARK-35701, every registration copies the whole registry, so
registering n entries at class initialization does O(n^2) work. `SQLConf` alone
registers ~900 entries, so its first access performs hundreds of thousands of
entry copies plus repeated large `HashMap` allocations, all on the startup
path. `ConcurrentHashMap` keeps the lock-free reads that motivated SPARK-35701
and makes each registration O(1).
Initialization of `object SQLConf` in a fresh JVM drops from 926 ms to 889
ms (median of 15 interleaved runs, JDK 17).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- New tests in `SQLConfEntrySuite`:
- Iterating `getConfigEntries()` while another thread registers entries.
This guards against a non-concurrent map throwing
`ConcurrentModificationException`; every entry registered before the iteration
began must be seen.
- Null-key lookups.
- `build/mvn -pl sql/core -Dtest=none
-DwildcardSuites=org.apache.spark.sql.internal.SQLConfEntrySuite,org.apache.spark.sql.internal.SQLConfSuite
test` passes.
- Micro-benchmark: timed
`Class.forName("org.apache.spark.sql.internal.SQLConf$")` in fresh JVMs,
alternating the patched and unpatched `spark-catalyst` jar for 15 runs each.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
