Sudhanva Huruli created SPARK-62028:
---------------------------------------

             Summary: Re-running a pipeline fails with 
DELTA_CANNOT_CHANGE_PROVIDER for an existing streaming table declared with 
format delta
                 Key: SPARK-62028
                 URL: https://issues.apache.org/jira/browse/SPARK-62028
             Project: Spark
          Issue Type: Bug
          Components: Declarative Pipelines
    Affects Versions: 4.3.0
            Reporter: Sudhanva Huruli


On a second run against a persistent catalog, a pipeline that declares a table 
with format="delta" fails during table setup with 
"[DELTA_CANNOT_CHANGE_PROVIDER] 'provider' is a reserved table property, and 
cannot be altered". The pipeline re-applies the declared format to the existing 
table as a change to the provider property, even when the format has not 
changed. Reproduction: with the Delta Lake jars on the classpath and a 
persistent catalog, declare a streaming table with format="delta" and run 
spark-pipelines run twice. Expected: the second run completes, and a format 
that really differs from the existing table stops the run before any data is 
written, with an error that names the table and both formats.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to