John Zhuge created SPARK-61843:
----------------------------------
Summary: Follow-up tests for spark.sql.extensions.defaultList
Key: SPARK-61843
URL: https://issues.apache.org/jira/browse/SPARK-61843
Project: Spark
Issue Type: Task
Components: Spark Core
Affects Versions: 4.3.0
Reporter: John Zhuge
Follow-up to SPARK-60483 (PR #59743), which adds
spark.sql.extensions.defaultList. That PR pins the default list ahead of user
extensions, the default-only case, a shared extension applied once, and
last-win ordering with user-list repeats. These cases are not covered yet.
1. Repeats inside the default list, and the warning count when the two lists
overlap. Every default-list test in #59743 configures a single entry, so these
mutations pass the whole suite:
- defaultClassNames.distinct.filterNot(userClassNames.contains(_)) drops
repeats inside the default list. Caught by default = MyExtensions,MyExtensions
with user = MyExtensions2, asserting MyRule runs twice.
- defaultClassNames.diff(userClassNames) keeps default-list occurrences
beyond the user count. Caught by default = MyExtensions,MyExtensions with user
= MyExtensions, asserting MyRule runs once.
- Dropping .distinct from the warning loop logs one warning per default-list
occurrence instead of per class. Caught by the same case, counting the matching
log messages (1, not 2); an exists() check does not see it.
Also assert zero matching warnings when the two lists are disjoint, so
warning once per default entry cannot pass.
2. spark.conf.getOption("spark.sql.extensions") is empty when only the default
list is set, and returns the raw user value when spark.sql.extensions is set;
SET spark.sql.extensions shows <undefined> before it is set and the raw user
value after. Assert through getOption so the test does not depend on
null-versus-throw for an unset optional entry.
3. A runtime SET spark.sql.extensions.defaultList, or spark.conf.set on it,
throws CANNOT_MODIFY_STATIC_CONFIG. This pins the buildStaticConf registration;
the guard itself is requireNonStaticConf, which is key-independent apart from
its documented spark.default.parallelism carve-out.
The follow-up PR should also list these cases under "How was this patch
tested?" (sarutak, Nov-13 inline review).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]