[
https://issues.apache.org/jira/browse/SPARK-58330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anurag Mantripragada updated SPARK-58330:
-----------------------------------------
Description:
Spark supports a per-statement `WITH (key = value)` dynamic table options
clause on DMLs. When a single statement references the same table more than
once with different options, only the first-resolved reference's options are
honored; the other reference's options are silently dropped.
Examples (each drops one side's options):
```sql
– INSERT self-select: the target's write options are dropped
INSERT INTO t WITH (`write.split-size` = 10) SELECT * FROM t
– MERGE self-merge: the source's read options are dropped
MERGE INTO t WITH (`write.split-size` = 10)
USING t WITH (`split-size` = 5) s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET ...
– UPDATE self-reference in a subquery/CTE: the subquery's options are dropped
UPDATE t WITH (`write.split-size` = 10) SET ...
WHERE id IN (SELECT id FROM t WITH (`split-size` = 5))
```
was:
Spark supports a per-statement `WITH (key = value)` dynamic table options
clause on `SELECT` (SPARK-36680), `INSERT` (SPARK-49098), `UPDATE`
(SPARK-57681), and `MERGE` (SPARK-58007). When a single statement references
the same table more than once with different options, only the first-resolved
reference's options are honored; the other reference's options are silently
dropped.
Examples (each drops one side's options):
```sql
-- INSERT self-select: the target's write options are dropped
INSERT INTO t WITH (`write.split-size` = 10) SELECT * FROM t
-- MERGE self-merge: the source's read options are dropped
MERGE INTO t WITH (`write.split-size` = 10)
USING t WITH (`split-size` = 5) s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET ...
-- UPDATE self-reference in a subquery/CTE: the subquery's options are dropped
UPDATE t WITH (`write.split-size` = 10) SET ...
WHERE id IN (SELECT id FROM t WITH (`split-size` = 5))
```
No error is raised; an option the user explicitly wrote is silently ignored.
Root cause: the per-query analyzer relation cache (`RelationResolution`) is
keyed by catalog + namespace + name + time-travel spec only -- options are not
part of the key. On a cache hit the cached relation is returned as-is, without
applying the current reference's options, so the second reference to the same
table inherits the first reference's options. Which reference "wins" is a
traversal-order accident. The shared (cross-query) relation cache already
re-applies the caller's options on a hit, so this is an inconsistency between
the two cache paths.
This affects INSERT, UPDATE, and MERGE and should be fixed once in the shared
resolution path so each reference honors exactly its own options. This subsumes
the separately-noted INSERT self-select option-drop.
> Prevent silent dropping of dynamic options on same table references in DML
> --------------------------------------------------------------------------
>
> Key: SPARK-58330
> URL: https://issues.apache.org/jira/browse/SPARK-58330
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Anurag Mantripragada
> Priority: Major
>
> Spark supports a per-statement `WITH (key = value)` dynamic table options
> clause on DMLs. When a single statement references the same table more than
> once with different options, only the first-resolved reference's options are
> honored; the other reference's options are silently dropped.
> Examples (each drops one side's options):
> ```sql
> – INSERT self-select: the target's write options are dropped
> INSERT INTO t WITH (`write.split-size` = 10) SELECT * FROM t
> – MERGE self-merge: the source's read options are dropped
> MERGE INTO t WITH (`write.split-size` = 10)
> USING t WITH (`split-size` = 5) s
> ON t.id = s.id
> WHEN MATCHED THEN UPDATE SET ...
> – UPDATE self-reference in a subquery/CTE: the subquery's options are dropped
> UPDATE t WITH (`write.split-size` = 10) SET ...
> WHERE id IN (SELECT id FROM t WITH (`split-size` = 5))
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
