gengliangwang commented on code in PR #58298: URL: https://github.com/apache/spark/pull/58298#discussion_r4021446600
########## docs/sql-migration-guide.md: ########## @@ -24,6 +24,8 @@ license: | ## Upgrading from Spark SQL 4.3 to 4.4 +- Since Spark 4.4, when an already-analyzed Data Source V2 plan is refreshed after a compatible schema change, connectors can safely return more data columns from the current table schema in `Scan.readSchema()` than requested by `SupportsPushDownRequiredColumns.pruneColumns`. Previously, this partial pruning could make planning fail because the scan reported data columns absent from the analyzed relation output. +- Since Spark 4.4, if expanding a struct used as a map key causes distinct keys to collide under a previously analyzed schema, the query fails with `DUPLICATED_MAP_KEY` by default instead of silently returning duplicate keys. With `spark.sql.mapKeyDedupPolicy=LAST_WIN`, the last value is kept. Review Comment: Shall we combine the two changes into one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
