Szehon Ho created SPARK-60413:
---------------------------------
Summary: Warn when evolving a DSv2 table that accepts any schema
Key: SPARK-60413
URL: https://issues.apache.org/jira/browse/SPARK-60413
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Szehon Ho
DSv2 tables can advertise both AUTOMATIC_SCHEMA_EVOLUTION and
ACCEPT_ANY_SCHEMA. Automatic schema evolution applies requested schema changes
during write analysis, while ACCEPT_ANY_SCHEMA skips Spark's native write
schema resolution and alignment. A write can consequently fail after the table
schema has been updated; a nontransactional catalog retains those changes.
Add a warning immediately before Spark applies pending schema changes to a
table reporting both capabilities. Identify the data source and table, explain
the risk, and suggest disabling ACCEPT_ANY_SCHEMA. Emit the warning once for
the batch of changes. Writes without pending evolution, or tables reporting
only one capability, should remain quiet.
Cover INSERT and MERGE through public DataFrame write APIs, including exact
warning counts and persistent schema changes after a failed MERGE.
Related connector discussion: https://github.com/apache/iceberg/pull/17957
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
