ulysses-you commented on code in PR #58501:
URL: https://github.com/apache/spark/pull/58501#discussion_r3946259121
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/InternalRowComparableWrapper.scala:
##########
@@ -105,21 +105,59 @@ object InternalRowComparableWrapper {
def apply(
partition: InputPartition with HasPartitionKey,
partitionExpression: Seq[Expression]): InternalRowComparableWrapper = {
- new InternalRowComparableWrapper(
- partition.asInstanceOf[HasPartitionKey].partitionKey(),
partitionExpression.map(_.dataType))
+ apply(partition.partitionKey(), partitionExpression)
}
def apply(
partitionRow: InternalRow,
partitionExpression: Seq[Expression]): InternalRowComparableWrapper = {
- new InternalRowComparableWrapper(partitionRow,
partitionExpression.map(_.dataType))
+
getInternalRowComparableWrapperFactory(partitionExpression.map(_.dataType))(partitionRow)
+ }
+
+ /**
+ * The types a row is compared at, which is the given types with their
naming erased: struct field
+ * names and every nullability go, and nothing else does.
+ *
+ * Two rows of the same value belong together whatever the columns they came
from were called. A
+ * storage-partitioned join relies on that: an equi-join across two structs
whose fields are named
+ * differently is legal, `identity` carries that name into the key type, and
+ * `KeyedShuffleSpec.createPartitioning` puts one side's expressions over
the other side's keys.
+ * So the naming is erased once, here, and every wrapper compares at these
types.
+ *
+ * Everything else is kept exactly, since it decides where a value belongs:
a collation and a
+ * decimal precision still tell two rows apart.
+ */
+ def comparableTypes(dataTypes: Seq[DataType]): Seq[DataType] = {
Review Comment:
I'm a bit concerned about we may miss call comparableTypes in future since
the related code exists in many key place. Can we narrow the comparableTypes in
the InternalRowComparableWrapper file ? It seems for now, the two key code path
depends on it:
1. InternalRowComparableWrapper.hashCode/equals
2. InternalRowComparableWrapper.getInternalRowComparableWrapperFactory
We can normalize the data types inside these method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
