cloud-fan commented on code in PR #52153:
URL: https://github.com/apache/spark/pull/52153#discussion_r2317698107
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:
##########
@@ -946,3 +946,43 @@ case class ShuffleSpecCollection(specs: Seq[ShuffleSpec])
extends ShuffleSpec {
specs.head.numPartitions
}
}
+
+/**
+ * Represents a partitioning where partition IDs are passed through directly
from the
+ * DirectShufflePartitionID expression. This partitioning scheme is used when
users
+ * want to directly control partition placement rather than using hash-based
partitioning.
+ *
+ * This partitioning maps directly to the PartitionIdPassthrough RDD
partitioner.
+ */
+case class ShufflePartitionIdPassThrough(
+ expr: DirectShufflePartitionID,
+ numPartitions: Int) extends Expression with Partitioning with Unevaluable {
+
+ // TODO(SPARK-53401): Support Shuffle Spec in Direct Partition ID Pass
Through
+ def partitionIdExpression: Expression = Pmod(expr.child,
Literal(numPartitions))
Review Comment:
Here we strip `DirectShufflePartitionID`, which means we can make
`DirectShufflePartitionID` extend `Unevaluable`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
