[
https://issues.apache.org/jira/browse/SPARK-57670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101330#comment-18101330
]
Power John commented on SPARK-57670:
------------------------------------
I traced this issue to DatasetManager.evolveTable. We cannot safely determine
which properties to remove by diffing the current catalog properties against
the current table definition, because Table.properties() may contain
catalog-managed, non-reserved keys such as clusteringColumns. The fix therefore
needs a persisted record of which properties originated from the Spark
Pipelines table definition.
Before implementing it, I would like to confirm the intended ownership model.
AutoCdcScd1MultiPipelineSuite covers separate DataflowGraph and
PipelineUpdateContext instances materializing the same target table.
If Pipeline A declares property `a` and Pipeline B declares property `b`,
should B's table definition be authoritative for the physical table and remove
`a`, or should B preserve properties owned by A?
My current recommendation is table-scoped ownership, with the latest table
definition being authoritative. Table properties describe the physical table.
Under this model, schema changes, property removals, property updates, and the
ownership-metadata update can be submitted in a single alterTable call. The
TableCatalog contract requires those changes to be applied in order and rejects
the entire change set if any change is rejected.
The implementation would persist a versioned record of canonicalized entries
originating from the graph Table.properties map. Catalog-injected entries and
properties synthesized solely from the separate table.format and table.comment
fields would not be recorded. Missing state would bootstrap without removals
for backward compatibility.
If per-pipeline ownership is intended instead, we would need a stable pipeline
identity and per-pipeline state, likely under storageRoot. That is a materially
larger change and cannot be implemented safely with a single table-level marker.
An ordinary marker stored in the target table properties would also be visible
through SHOW CREATE TABLE and SHOW TBLPROPERTIES unless it is explicitly
treated as internal metadata.
Does the table-scoped, latest-definition-wins model match the intended
semantics? If so, I will proceed with the implementation and tests.
> Spark Pipelines table properties are not dropped on subsequent run
> ------------------------------------------------------------------
>
> Key: SPARK-57670
> URL: https://issues.apache.org/jira/browse/SPARK-57670
> Project: Spark
> Issue Type: Bug
> Components: Declarative Pipelines
> Affects Versions: 4.1.0, 4.1.1, 4.1.2
> Reporter: Anish Mahto
> Priority: Major
>
> Consider a using creating a pipeline with one table, with the following
> properties:
> ```
> from pyspark import pipelines as dp
>
> @dp.materialized_view(
> table_properties={
> "myproperty":"value",
> "myotherproperty":"othervalue",
> },
> )
> def my_table():
> return spark.range(10)
> ```
> Before the next pipeline run, they might change their pipeline definition as
> such:
> ```
> from pyspark import pipelines as dp
>
> @dp.materialized_view(
> table_properties={
> "myproperty":"value",
> },
> )
> def my_table():
> return spark.range(10)
> ```
>
> The expectation would be on the second pipeline run, `myotherproperty` is
> dropped from the table in the configured catalog during table materialization.
>
> In reality however, pipelines table property evolution is additive only
> today; we do not detect and produce a remove property table change for
> dropped properties across runs. Code pointer:
> https://github.com/apache/spark/blob/master/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala#L331
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
