[GitHub] [spark] ulysses-you commented on a change in pull request #33099: [SPARK-35904][SQL] Collapse above RebalancePartitions
ulysses-you commented on a change in pull request #33099:
URL: https://github.com/apache/spark/pull/33099#discussion_r659322585
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -1373,16 +1373,18 @@ object RepartitionByExpression {
*/
case class RebalancePartitions(
partitionExpressions: Seq[Expression],
-child: LogicalPlan) extends UnaryNode {
- override def maxRows: Option[Long] = child.maxRows
- override def output: Seq[Attribute] = child.output
+child: LogicalPlan) extends RepartitionOperation {
Review comment:
sorry for the late, I prefer not to do this. Since
`RepartitionOperation` can affect the behavior of AQE, the semantics of
`RepartitionOperation` have a little different from previous.
Currently, the `RepartitionOperation` behavior with AQE framwork follows:
* `Repartition` does not use AQE stage optimizer rules
* `RepartitionByExpression` depends on `optNumPartitions` and
`partitionExpressions`
* `RebalancePartitions`(if extend) depends on `partitionExpressions`
And the rule `CollapseRepartition` may not effective in all use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #33099: [SPARK-35904][SQL] Collapse above RebalancePartitions
ulysses-you commented on a change in pull request #33099:
URL: https://github.com/apache/spark/pull/33099#discussion_r659322585
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -1373,16 +1373,18 @@ object RepartitionByExpression {
*/
case class RebalancePartitions(
partitionExpressions: Seq[Expression],
-child: LogicalPlan) extends UnaryNode {
- override def maxRows: Option[Long] = child.maxRows
- override def output: Seq[Attribute] = child.output
+child: LogicalPlan) extends RepartitionOperation {
Review comment:
sorry for the late, I prefer not to do this. Since
`RepartitionOperation` can affect the behavior of AQE, the semantics of
`RepartitionOperation` have a little different from previous.
Currently, the `RepartitionOperation` behavior with AQE framwork follows:
* `Repartition` does not use AQE
* `RepartitionByExpression` use AQE depends on `optNumPartitions`
* `RebalancePartitions`(if extend) always use AQE
And the rule `CollapseRepartition` may not effective in all use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
