Sudhanva Huruli created SPARK-62029:
---------------------------------------

             Summary: Full refresh of Iceberg tables fails with 'Cannot process 
delete snapshot' and column type changes are rejected
                 Key: SPARK-62029
                 URL: https://issues.apache.org/jira/browse/SPARK-62029
             Project: Spark
          Issue Type: New Feature
          Components: Declarative Pipelines
    Affects Versions: 4.3.0
            Reporter: Sudhanva Huruli


With --full-refresh-all, Declarative Pipelines empties each Iceberg table with 
TRUNCATE, which adds a delete snapshot to the table history. Downstream 
streaming flows then replay the old appends and fail with 'Cannot process 
delete snapshot'. Changing the column type of a materialized view fails with 
'Cannot change column type' and leaves the table empty. Proposed: when the 
catalog supports atomic table replacement, a table rebuilt from scratch is 
replaced in one step with a new version that has the new schema, instead of 
being truncated and altered in place.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to