Szehon Ho created SPARK-56550:
---------------------------------
Summary: Support filling missing nested struct fields with null in
INSERT INTO WITH SCHEMA EVOLUTION
Key: SPARK-56550
URL: https://issues.apache.org/jira/browse/SPARK-56550
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: Szehon Ho
When using INSERT INTO ... WITH SCHEMA EVOLUTION, if the source has fewer
nested struct fields than the target table, the insert fails with
INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA or STRUCT_MISSING_FIELDS errors.
MERGE INTO already supports this via the
spark.sql.mergeNestedTypeCoercion.enabled config flag, which enables RECURSE
mode in TableOutputResolver to fill missing nested fields with null.
This ticket adds the equivalent support for INSERT INTO, gated by a new
spark.sql.insertNestedTypeCoercion.enabled config flag (default false). When
enabled together with WITH SCHEMA EVOLUTION, missing nested struct fields,
fields in structs nested within arrays/maps, and trailing by-position columns
are filled with their default values or null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
