dongjoon-hyun opened a new pull request, #59918: URL: https://github.com/apache/spark/pull/59918
### What changes were proposed in this pull request? This PR fixes `ResolveRowLevelCommandAssignments` so that DSv2 `UPDATE` and `MERGE` can reference metadata columns (e.g. `_partition`) of a target table that has `CHAR` or `VARCHAR` columns. When the rule aligns the assignments of a rewritable `UPDATE` or `MERGE`, `cleanAttrMetadata` replaces the target `DataSourceV2Relation` with a copy whose output has no `__CHAR_VARCHAR_TYPE_STRING` metadata. The copy is a new instance, so its lazy `metadataOutput` creates the metadata attributes again with new expression IDs. The condition and the assignments of the command still refer to the metadata attributes of the original relation. However, `RewriteUpdateTable` and `RewriteMergeIntoTable` decide which metadata columns to read based on the copy. They resolve the metadata columns that the operation requires by name on the copy, and they read the other metadata columns that the command references (SPARK-60673) only if they are in the metadata output of the copy. For an insert-only `MERGE`, `AddMetadataColumns` also looks them up in the copy. The analysis fails because the plan no longer produces the attributes that the command refers to. Without `CHAR`/`VARCHAR` columns, the co py is equal to the original relation, so `transform` keeps the original instance and the references stay valid. This PR makes `cleanAttrMetadata` also return a map from the metadata output of the original relation to that of the copy. The rule then rewrites the references of the command with `QueryPlan.rewriteAttrs`, which also updates outer references in subqueries. `ResolveSchemaEvolution.replaceWriteTarget` uses the same approach for the output when it replaces the target relation of a write. When there is nothing to clean, the relation is kept and the map is empty. Alternatives that were not taken: - Keep the original relation: the cleaned output is what makes `AssignmentUtils.aligned` accept the aligned assignments, because it compares the assignment value type with the raw `CHAR`/`VARCHAR` type while the metadata is present. - Keep the original metadata attributes in the copy: each `DataSourceV2Relation` instance derives its metadata output from the table, and other copies such as `newInstance()` rely on getting new expression IDs. - Resolve the referenced metadata columns by name in the rewrite rules: the original relation is gone at that point, and a name alone cannot tell a target metadata column from a metadata column of a DSv2 source. ### Why are the changes needed? To fix a bug. For example, take a DSv2 table that supports row-level operations, exposes the metadata column `_partition`, and has the schema `pk INT NOT NULL, id INT, dep STRING, name VARCHAR(10)` (partitioned by `dep`): ```sql UPDATE t SET id = -1 WHERE id IN (1, 2) AND _partition = 'hr' ``` ``` [MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION] Resolved attribute(s) "_partition" missing from "pk", "id", "dep", "name", "_partition", "index" in operator !Filter (id#80 IN (1,2) AND (_partition#84 = hr)). Attribute(s) with the same name appear in the operation: "_partition". ``` The same statements succeed when `name` is a `STRING` column. The failure happens for group-based and delta-based tables (including split updates) in these cases: - `UPDATE` whose condition or assignment values reference a metadata column - `MERGE` with `UPDATE` or `INSERT` actions whose clause conditions or assignment values reference a target metadata column - insert-only `MERGE` whose `ON` condition references a target metadata column (`MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT`) If the operation requests none of the referenced metadata columns, the first two cases fail with `MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT` instead. `DELETE` and a `MERGE` with only `DELETE` actions are not affected because Spark does not align them and keeps the original relation. By code reading, the same code path exists since Apache Spark 3.5.0, where `ResolveRowLevelCommandAssignments` started to align the assignments of `UPDATE` (SPARK-42151) and `MERGE` (SPARK-43204) for tables that support row-level operations. ### Does this PR introduce _any_ user-facing change? Yes. On a DSv2 table with `CHAR`/`VARCHAR` columns, the `UPDATE` and `MERGE` statements above no longer fail with `MISSING_ATTRIBUTES` and behave as they do on a table without such columns. Statements that worked before are not affected. ### How was this patch tested? Pass the CIs with the newly added test cases. - `UpdateTableSuiteBase`: `SPARK-60708: update char/varchar table with metadata columns` - `MergeIntoTableSuiteBase`: `SPARK-60708: merge into char/varchar table with metadata columns` and `SPARK-60708: insert-only merge into char/varchar table with metadata column in ON` They run in the group-based, delta-based, split-update and no-metadata suites (and the NoCodegen MERGE suites), and they fail without this fix. I also checked manually that a `CHAR` column, outer references to metadata columns in subqueries, and a `MERGE` with multiple `NOT MATCHED` clauses behave as they do on a table without `CHAR`/`VARCHAR` columns. I also ran these suites locally, and all of them passed: the UPDATE, MERGE (including schema evolution) and DELETE suites, `MetadataColumnSuite`, `PlanResolutionSuite`, `AlignUpdateAssignmentsSuite`, `AlignMergeAssignmentsSuite`, the row-level operation runtime filter suites, `MergeIntoDataFrameSuite`, and the CHAR/VARCHAR test suites. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
