Szehon Ho created SPARK-54172:
---------------------------------
Summary: Merge Into schema evolution should only add referenced
fields
Key: SPARK-54172
URL: https://issues.apache.org/jira/browse/SPARK-54172
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: Szehon Ho
SPARK-52991 added schema evolution support for MERGE INTO statements. However,
it is a bit too broad. In some instances, source table may have many more
fields than target tables. But only a few new ones are needed to be added to
the target. We should limit the scope of schema evolution to only add
columns/nested fields that are referenced in the MERGE INTO query via UPDATE or
INSERT statements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
