johanl-db opened a new pull request, #58632: URL: https://github.com/apache/spark/pull/58632
### What changes were proposed in this pull request? This changes extends the DSv2 connector interface to allow connectors to configure schema alignment behavior and opt-out of the default behavior when required: - spark.sql.storeAssignmentPolicy=LEGACY being rejected for DSv2 writes. - Strict implicit casts check at analysis time for DSv2 writes under sparksql.storeAssignmentPolicy=ANSI ### Why are the changes needed? Delta is migrating to DSv2, and historically has always allowed: - writes under spark.sql.storeAssignmentPolicy=LEGACY - Many implicit casts at analysis, relying instead on execution time checks to catch malformed values or overflows. Without a way to configure this. behavior, this effectively blocks migrating Delta to DSv2. ### How was this patch tested? Add Spark suite SchemaAlignmentConfigSuite covering the configuration mechanism Add Delta DSv2 variant of existing DeltaInsertIntoImplicitCastSuite, showing that casting behavior is preserved when running on DSv2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
