[
https://issues.apache.org/jira/browse/SPARK-58330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated SPARK-58330:
-----------------------------------------
Affects Version/s: 4.2.0
(was: 5.0.0)
> Dynamic table options are dropped when the same table is referenced twice in
> a DML statement
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-58330
> URL: https://issues.apache.org/jira/browse/SPARK-58330
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Anurag Mantripragada
> Priority: Major
>
> Spark supports a per-statement `WITH (key = value)` dynamic table options
> clause on `SELECT` (SPARK-36680), `INSERT` (SPARK-49098), `UPDATE`
> (SPARK-57681), and `MERGE` (SPARK-58007). When a single statement references
> the same table more than once with different options, only the first-resolved
> reference's options are honored; the other reference's options are silently
> dropped.
> Examples (each drops one side's options):
> ```sql
> -- INSERT self-select: the target's write options are dropped
> INSERT INTO t WITH (`write.split-size` = 10) SELECT * FROM t
> -- MERGE self-merge: the source's read options are dropped
> MERGE INTO t WITH (`write.split-size` = 10)
> USING t WITH (`split-size` = 5) s
> ON t.id = s.id
> WHEN MATCHED THEN UPDATE SET ...
> -- UPDATE self-reference in a subquery/CTE: the subquery's options are dropped
> UPDATE t WITH (`write.split-size` = 10) SET ...
> WHERE id IN (SELECT id FROM t WITH (`split-size` = 5))
> ```
> No error is raised; an option the user explicitly wrote is silently ignored.
> Root cause: the per-query analyzer relation cache (`RelationResolution`) is
> keyed by catalog + namespace + name + time-travel spec only -- options are
> not part of the key. On a cache hit the cached relation is returned as-is,
> without applying the current reference's options, so the second reference to
> the same table inherits the first reference's options. Which reference "wins"
> is a traversal-order accident. The shared (cross-query) relation cache
> already re-applies the caller's options on a hit, so this is an inconsistency
> between the two cache paths.
> This affects INSERT, UPDATE, and MERGE and should be fixed once in the shared
> resolution path so each reference honors exactly its own options. This
> subsumes the separately-noted INSERT self-select option-drop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
