Sudhanva Huruli created SPARK-62026:
---------------------------------------

             Summary: A pipeline silently replaces a materialized view created 
by a different pipeline with the same table name
                 Key: SPARK-62026
                 URL: https://issues.apache.org/jira/browse/SPARK-62026
             Project: Spark
          Issue Type: Bug
          Components: Declarative Pipelines
    Affects Versions: 4.3.0
            Reporter: Sudhanva Huruli


Two pipeline projects define a materialized view with the same name in the same 
catalog and schema. Repro: spark-pipelines init --name hello; run it 
(example_python_materialized_view has 10 rows); spark-pipelines init --name 
hello2 and define @dp.materialized_view example_python_materialized_view 
returning spark.range(3) in its transformations; run hello2 against the same 
metastore. The second run exits 0, reports the flow as completed, and the table 
now has 3 rows. Nothing warned that the table already existed and was 
maintained by a different project. The same happens with --full-refresh of a 
streaming table another pipeline maintains. Expected: the run fails before 
writing anything with an error that says the table belongs to a different 
pipeline, and the first pipeline's rows are left in place. Proposed: record the 
owning pipeline's storage root in a table property when a table is created or 
recomputed from scratch, and reject a materialized view refresh or a streaming 
table full refresh from a pipeline with a different storage root. Incremental 
streaming runs and tables without a recorded owner keep working.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to