[
https://issues.apache.org/jira/browse/SPARK-62026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudhanva Huruli updated SPARK-62026:
------------------------------------
Description: Two pipeline projects define a materialized view with the same
name in the same catalog and schema. Repro: start one Spark Connect server with
a persistent catalog (for example sbin/start-connect-server.sh --conf
spark.sql.catalogImplementation=hive) so both pipelines see the same tables.
spark-pipelines init --name hello; run it with spark-pipelines run --remote
sc://localhost:15002 (example_python_materialized_view has 10 rows);
spark-pipelines init --name hello2 and define @dp.materialized_view
example_python_materialized_view returning spark.range(3) in its
transformations; run hello2 against the same server with --remote
sc://localhost:15002. The second run exits 0, reports the flow as completed,
and the table now has 3 rows. Nothing warned that the table already existed and
was maintained by a different project. The same happens with --full-refresh of
a streaming table another pipeline maintains. Expected: the run fails before
writing anything with an error that says the table belongs to a different
pipeline, and the first pipeline's rows are left in place. Proposed: record the
owning pipeline's storage root in a table property when a table is created or
recomputed from scratch, and reject a materialized view refresh or a streaming
table full refresh from a pipeline with a different storage root. Incremental
streaming runs and tables without a recorded owner keep working. (was: Two
pipeline projects define a materialized view with the same name in the same
catalog and schema. Repro: spark-pipelines init --name hello; run it
(example_python_materialized_view has 10 rows); spark-pipelines init --name
hello2 and define @dp.materialized_view example_python_materialized_view
returning spark.range(3) in its transformations; run hello2 against the same
metastore. The second run exits 0, reports the flow as completed, and the table
now has 3 rows. Nothing warned that the table already existed and was
maintained by a different project. The same happens with --full-refresh of a
streaming table another pipeline maintains. Expected: the run fails before
writing anything with an error that says the table belongs to a different
pipeline, and the first pipeline's rows are left in place. Proposed: record the
owning pipeline's storage root in a table property when a table is created or
recomputed from scratch, and reject a materialized view refresh or a streaming
table full refresh from a pipeline with a different storage root. Incremental
streaming runs and tables without a recorded owner keep working.)
> A pipeline silently replaces a materialized view created by a different
> pipeline with the same table name
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62026
> URL: https://issues.apache.org/jira/browse/SPARK-62026
> Project: Spark
> Issue Type: Bug
> Components: Declarative Pipelines
> Affects Versions: 4.3.0
> Reporter: Sudhanva Huruli
> Priority: Minor
>
> Two pipeline projects define a materialized view with the same name in the
> same catalog and schema. Repro: start one Spark Connect server with a
> persistent catalog (for example sbin/start-connect-server.sh --conf
> spark.sql.catalogImplementation=hive) so both pipelines see the same tables.
> spark-pipelines init --name hello; run it with spark-pipelines run --remote
> sc://localhost:15002 (example_python_materialized_view has 10 rows);
> spark-pipelines init --name hello2 and define @dp.materialized_view
> example_python_materialized_view returning spark.range(3) in its
> transformations; run hello2 against the same server with --remote
> sc://localhost:15002. The second run exits 0, reports the flow as completed,
> and the table now has 3 rows. Nothing warned that the table already existed
> and was maintained by a different project. The same happens with
> --full-refresh of a streaming table another pipeline maintains. Expected: the
> run fails before writing anything with an error that says the table belongs
> to a different pipeline, and the first pipeline's rows are left in place.
> Proposed: record the owning pipeline's storage root in a table property when
> a table is created or recomputed from scratch, and reject a materialized view
> refresh or a streaming table full refresh from a pipeline with a different
> storage root. Incremental streaming runs and tables without a recorded owner
> keep working.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]