[
https://issues.apache.org/jira/browse/SPARK-59342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johan Lasperas updated SPARK-59342:
-----------------------------------
Description:
Allow connectors configuring following schema alignment behavior:
* allow writes under spark.sql.storeAssignmentPolicy=LEGACY
* accept implicit casts at analysis, and rely on execution time checks to
catch malformed values or overflows
The proposal is to extend the DSv2 interface to allow configuring this behavior
per table:
{code:java}
public interface Table {
...
default SchemaAlignmentConfig schemaAlignmentConfig() {
return SchemaAlignmentConfig.DEFAULT; }
}
public interface SchemaAlignmentConfig {
default boolean allowLegacyStoreAssignmentPolicy() {
return false;
}
default boolean deferCastValidationToRuntime() {
return false;
}
}
{code}
was:
Delta is migrating to DSv2, and historically has always allowed:
* writes under spark.sql.storeAssignmentPolicy=LEGACY
* Many implicit casts at analysis, relying instead on execution time checks to
catch malformed values or overflows.
Without a way to configure this. behavior, this effectively blocks migrating
Delta to DSv2.
The proposal is to extend the DSv2 interface to allow configuring this behavior
per table:
{code:java}
public interface Table {
...
default SchemaAlignmentConfig schemaAlignmentConfig() {
return SchemaAlignmentConfig.DEFAULT; }
}
public interface SchemaAlignmentConfig {
default boolean allowLegacyStoreAssignmentPolicy() {
return false;
}
default boolean deferCastValidationToRuntime() {
return false;
}
}
{code}
> Allow configuring schema alignment behavior for DSv2 writes
> -----------------------------------------------------------
>
> Key: SPARK-59342
> URL: https://issues.apache.org/jira/browse/SPARK-59342
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Johan Lasperas
> Priority: Major
> Labels: pull-request-available
>
> Allow connectors configuring following schema alignment behavior:
> * allow writes under spark.sql.storeAssignmentPolicy=LEGACY
> * accept implicit casts at analysis, and rely on execution time checks to
> catch malformed values or overflows
> The proposal is to extend the DSv2 interface to allow configuring this
> behavior per table:
>
> {code:java}
> public interface Table {
> ...
> default SchemaAlignmentConfig schemaAlignmentConfig() {
> return SchemaAlignmentConfig.DEFAULT; }
> }
> public interface SchemaAlignmentConfig {
> default boolean allowLegacyStoreAssignmentPolicy() {
> return false;
> }
> default boolean deferCastValidationToRuntime() {
> return false;
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
