Sudhanva Huruli created SPARK-62029:
---------------------------------------
Summary: Full refresh of Iceberg tables fails with 'Cannot process
delete snapshot' and column type changes are rejected
Key: SPARK-62029
URL: https://issues.apache.org/jira/browse/SPARK-62029
Project: Spark
Issue Type: New Feature
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Sudhanva Huruli
With --full-refresh-all, Declarative Pipelines empties each Iceberg table with
TRUNCATE, which adds a delete snapshot to the table history. Downstream
streaming flows then replay the old appends and fail with 'Cannot process
delete snapshot'. Changing the column type of a materialized view fails with
'Cannot change column type' and leaves the table empty. Proposed: when the
catalog supports atomic table replacement, a table rebuilt from scratch is
replaced in one step with a new version that has the new schema, instead of
being truncated and altered in place.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]