szehon-ho commented on code in PR #59895:
URL: https://github.com/apache/spark/pull/59895#discussion_r4407460132
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -1139,9 +1139,12 @@ class Analyzer(
if (metaCols.isEmpty) {
node
} else {
- val newNode = node.mapChildren(
- addMetadataCol(_, metaCols.map(_.exprId).toSet,
onlyUniqueAndNecessaryMetadataColumns)
- )
+ val requiredAttrIds = metaCols.map(_.exprId).toSet
+ val newNode = node.mapChildren {
+ case child if isRowLevelCommandTarget(node, child) => child
Review Comment:
Could you check whether this breaks a previously working delta DELETE when
its predicate references metadata that the operation doesn't request?
For example, `DeltaBasedNoMetadataDeleteFromTableSuite` exposes
`_partition`, requests no write metadata, and uses `pk` as the row ID. Given
columns `pk INT NOT NULL, id INT, dep STRING`, partitioned by `dep`, consider:
```sql
DELETE FROM t
WHERE _partition = 'hr' AND id < 1.5;
```
From tracing the code, before this patch type coercion causes another
analyzer iteration, which adds `_partition` to the scan. The delta writer
projects only the row ID, so this can execute correctly. With this patch, the
rewrite appears to omit `_partition` from the scan, leaving the filter with a
missing input.
I haven't run this example. Could you verify it before and after the patch
and add coverage? Ideally, both this form and `id < 2` should succeed without
including predicate-only metadata in the write schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
