sunchao commented on code in PR #58412:
URL: https://github.com/apache/spark/pull/58412#discussion_r3887066268
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -765,15 +765,60 @@ abstract class InMemoryBaseTable(
/** Predicates recorded by [[filter]], for test assertions only. */
def pushedCatalystPredicates: Seq[CatalystExpression] =
catalystPredicates.toSeq
- /** AttributeReferences matching the partition-key InternalRow field
order. */
- private def partitionAttributes: Seq[AttributeReference] = {
+ /**
+ * The `AttributeReference`s standing for the partition key InternalRow
fields, in its field
+ * order, each paired with the name-part sequence of its partition column.
The parts are kept
+ * unflattened so a quoted top-level column `a.b` (parts `Seq("a.b")`)
stays distinct from a
+ * nested column `a`.`b` (parts `Seq("a", "b")`). Example:
+ * - `PARTITIONED BY (part, s.nested)` -> `(Seq("part"),
AttributeReference(part))`, then
+ * `(Seq("s", "nested"), AttributeReference(s.nested))`
+ */
+ private def partitionAttributes: Seq[(Seq[String], AttributeReference)] = {
partitioning.flatMap(_.references()).flatMap { ref =>
- val name = ref.fieldNames.mkString(".")
- readSchema.find(_.name == name).orElse(tableSchema.find(_.name ==
name)).map { f =>
- AttributeReference(f.name, f.dataType, f.nullable)()
+ val path = ref.fieldNames.toImmutableArraySeq
+ val resolver = SQLConf.get.resolver
+ readSchema.findNestedField(path, resolver = resolver)
+ .orElse(tableSchema.findNestedField(path, resolver = resolver)).map {
+ case (_, f) =>
+ path -> AttributeReference(ref.fieldNames.mkString("."),
f.dataType, f.nullable)()
Review Comment:
[P2] Preserve transform semantics for nested partition filters
`partitioning.flatMap(_.references())` includes non-identity transforms, but
this nested lookup binds each source field as if its original value occupied
the partition-key slot. For the fixture's supported `PARTITIONED BY
(truncate(s.part, 1))`, a row with `s.part = 'AB'` is stored under key `'A'` by
`getKey`. A scalar-subquery runtime predicate such as `s.part = (SELECT
max(val) FROM dim)`, where `max(val)` is `'AB'`, is then remapped to `'A' =
'AB'`, and `filter()` drops the matching partition before residual evaluation
can recover it. Before this change the fixture did not advertise `s.part`, so
the row survived for the residual filter.
Please restrict direct key-slot binding to identity transforms while
preserving actual partition-key ordinals, or project predicates through the
transform. This finding is limited to the in-memory test connector and was
independently checked by two reviewers from source; it was not reproduced at
runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
