jzhuge commented on PR #59743: URL: https://github.com/apache/spark/pull/59743#issuecomment-6452883617
Thanks @sarutak @pan3793 @knkarthik01 for the reviews and the verification. @knkarthik01 also shared feedback offline: he reproduced the bug on Spark 4.0.2 with Iceberg 1.10.0 (Iceberg set in spark-defaults.conf, Delta added via --conf, Iceberg silently dropped, WRITE ORDERED BY fails to parse), and this branch passes SparkSessionExtensionSuite locally. **Direction.** I'd land this scoped fix first and treat a general `<key>.defaultList` convention as a follow-up. A general version must define semantics per config (plugins de-dup, extension lists allow repeats), which is where SPARK-37602 / #34856 stalled. Happy to rework if reviewers prefer to settle it now. **De-dup.** Agreed. A defaultList entry that also appears in spark.sql.extensions is dropped; the remaining defaults apply first, then the user list (`defaults.filterNot(user.contains) ++ user`). Why: - Jobs that still list an extension after the operator moves it to defaultList behave exactly as today. - User config overriding platform defaults matches Spark convention. - Post-Hoc Resolution, Pre CBO Rules, and Early Filter and Projection Push-Down are Once batches, so running a non-idempotent rule twice can change results. Repeats within spark.sql.extensions stay allowed. I'll log a warning when a default is dropped, replace test 3 with an applied-once assertion, and document it. **Static conf.** Agreed: register defaultList via buildStaticConf (docs, SET -v, version, fail-fast on runtime writes). This also stops SET spark.sql.extensions from reporting classes that were never applied. **Docs.** The spark.sql.extensions doc will say defaults are prepended and that get() / SET shows only the user part. **SPI / SPARK-35380.** Separate follow-up. I'll push an updated patch and update the PR description to match. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
