[ 
https://issues.apache.org/jira/browse/SPARK-62026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudhanva Huruli updated SPARK-62026:
------------------------------------
    Description: Two pipeline projects define a materialized view with the same 
name in the same catalog and schema. Repro: start one Spark Connect server with 
a persistent catalog (for example sbin/start-connect-server.sh --conf 
spark.sql.catalogImplementation=hive) so both pipelines see the same tables. 
spark-pipelines init --name hello; run it with spark-pipelines run --remote 
sc://localhost:15002 (example_python_materialized_view has 10 rows); 
spark-pipelines init --name hello2 and define @dp.materialized_view 
example_python_materialized_view returning spark.range(3) in its 
transformations; run hello2 against the same server with --remote 
sc://localhost:15002. The second run exits 0, reports the flow as completed, 
and the table now has 3 rows. Nothing warned that the table already existed and 
was maintained by a different project. The same happens with --full-refresh of 
a streaming table another pipeline maintains. Expected: the run fails before 
writing anything with an error that says the table belongs to a different 
pipeline, and the first pipeline's rows are left in place. Proposed: record the 
owning pipeline's storage root in a table property when a table is created or 
recomputed from scratch, and reject a materialized view refresh or a streaming 
table full refresh from a pipeline with a different storage root. Incremental 
streaming runs and tables without a recorded owner keep working.  (was: Two 
pipeline projects define a materialized view with the same name in the same 
catalog and schema. Repro: spark-pipelines init --name hello; run it 
(example_python_materialized_view has 10 rows); spark-pipelines init --name 
hello2 and define @dp.materialized_view example_python_materialized_view 
returning spark.range(3) in its transformations; run hello2 against the same 
metastore. The second run exits 0, reports the flow as completed, and the table 
now has 3 rows. Nothing warned that the table already existed and was 
maintained by a different project. The same happens with --full-refresh of a 
streaming table another pipeline maintains. Expected: the run fails before 
writing anything with an error that says the table belongs to a different 
pipeline, and the first pipeline's rows are left in place. Proposed: record the 
owning pipeline's storage root in a table property when a table is created or 
recomputed from scratch, and reject a materialized view refresh or a streaming 
table full refresh from a pipeline with a different storage root. Incremental 
streaming runs and tables without a recorded owner keep working.)

> A pipeline silently replaces a materialized view created by a different 
> pipeline with the same table name
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62026
>                 URL: https://issues.apache.org/jira/browse/SPARK-62026
>             Project: Spark
>          Issue Type: Bug
>          Components: Declarative Pipelines
>    Affects Versions: 4.3.0
>            Reporter: Sudhanva Huruli
>            Priority: Minor
>
> Two pipeline projects define a materialized view with the same name in the 
> same catalog and schema. Repro: start one Spark Connect server with a 
> persistent catalog (for example sbin/start-connect-server.sh --conf 
> spark.sql.catalogImplementation=hive) so both pipelines see the same tables. 
> spark-pipelines init --name hello; run it with spark-pipelines run --remote 
> sc://localhost:15002 (example_python_materialized_view has 10 rows); 
> spark-pipelines init --name hello2 and define @dp.materialized_view 
> example_python_materialized_view returning spark.range(3) in its 
> transformations; run hello2 against the same server with --remote 
> sc://localhost:15002. The second run exits 0, reports the flow as completed, 
> and the table now has 3 rows. Nothing warned that the table already existed 
> and was maintained by a different project. The same happens with 
> --full-refresh of a streaming table another pipeline maintains. Expected: the 
> run fails before writing anything with an error that says the table belongs 
> to a different pipeline, and the first pipeline's rows are left in place. 
> Proposed: record the owning pipeline's storage root in a table property when 
> a table is created or recomputed from scratch, and reject a materialized view 
> refresh or a streaming table full refresh from a pipeline with a different 
> storage root. Incremental streaming runs and tables without a recorded owner 
> keep working.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to