Sudhanva Huruli created SPARK-62028:
---------------------------------------
Summary: Re-running a pipeline fails with
DELTA_CANNOT_CHANGE_PROVIDER for an existing streaming table declared with
format delta
Key: SPARK-62028
URL: https://issues.apache.org/jira/browse/SPARK-62028
Project: Spark
Issue Type: Bug
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Sudhanva Huruli
On a second run against a persistent catalog, a pipeline that declares a table
with format="delta" fails during table setup with
"[DELTA_CANNOT_CHANGE_PROVIDER] 'provider' is a reserved table property, and
cannot be altered". The pipeline re-applies the declared format to the existing
table as a change to the provider property, even when the format has not
changed. Reproduction: with the Delta Lake jars on the classpath and a
persistent catalog, declare a streaming table with format="delta" and run
spark-pipelines run twice. Expected: the second run completes, and a format
that really differs from the existing table stops the run before any data is
written, with an error that names the table and both formats.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]