Sudhanva Huruli created SPARK-62026:
---------------------------------------
Summary: A pipeline silently replaces a materialized view created
by a different pipeline with the same table name
Key: SPARK-62026
URL: https://issues.apache.org/jira/browse/SPARK-62026
Project: Spark
Issue Type: Bug
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Sudhanva Huruli
Two pipeline projects define a materialized view with the same name in the same
catalog and schema. Repro: spark-pipelines init --name hello; run it
(example_python_materialized_view has 10 rows); spark-pipelines init --name
hello2 and define @dp.materialized_view example_python_materialized_view
returning spark.range(3) in its transformations; run hello2 against the same
metastore. The second run exits 0, reports the flow as completed, and the table
now has 3 rows. Nothing warned that the table already existed and was
maintained by a different project. The same happens with --full-refresh of a
streaming table another pipeline maintains. Expected: the run fails before
writing anything with an error that says the table belongs to a different
pipeline, and the first pipeline's rows are left in place. Proposed: record the
owning pipeline's storage root in a table property when a table is created or
recomputed from scratch, and reject a materialized view refresh or a streaming
table full refresh from a pipeline with a different storage root. Incremental
streaming runs and tables without a recorded owner keep working.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]