Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1506968570 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -250,11 +281,15 @@ case class BroadcastQueryStageExec( reu

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1506968570 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -250,11 +281,15 @@ case class BroadcastQueryStageExec( reu

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on PR #45234: URL: https://github.com/apache/spark/pull/45234#issuecomment-1971998442 Should we also backport this patch to `v3.4.x` and `v3.5.x`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

ulysses-you commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1508373311 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -198,10 +221,18 @@ case class ShuffleQueryStageExec( reuse }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1509646119 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -198,10 +221,18 @@ case class ShuffleQueryStageExec( reuse

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

ulysses-you commented on PR #45234: URL: https://github.com/apache/spark/pull/45234#issuecomment-1978266178 cc @cloud-fan @maryannxue as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1512850096 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -148,6 +148,18 @@ abstract class QueryStageExec extends LeafExecNode {

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1513862684 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -148,6 +148,18 @@ abstract class QueryStageExec extends LeafExe

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1514403933 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec( currentP

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1514404676 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -148,6 +148,17 @@ abstract class QueryStageExec extends LeafExecNode {

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1515240053 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1515240053 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1515240053 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1515240053 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1515240053 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1515240053 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1515243849 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -148,6 +148,17 @@ abstract class QueryStageExec extends LeafExe

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1517390394 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec( currentP

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1518243547 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1518243547 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1518243547 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1518243547 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala: ## @@ -790,7 +790,8 @@ case class AdaptiveSparkPlanExec(

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1519578742 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite } } +

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1519582719 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite } } +

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1520503365 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1525339983 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1525339983 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1525339983 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1544936688 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1544936688 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1544936688 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1548387200 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +897,138 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1551401862 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +900,85 @@ class AdaptiveQueryExecSuite } } +

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1551407392 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +900,85 @@ class AdaptiveQueryExecSuite } } +

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1573095590 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +900,85 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1573096494 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +900,85 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1573096494 ## sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala: ## @@ -897,6 +900,85 @@ class AdaptiveQueryExecSuite }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on PR #45234: URL: https://github.com/apache/spark/pull/45234#issuecomment-2068474192 Thanks @cloud-fan and @ulysses-you for the reviews and approval. I have just rebased to get green build. -- This is an automated message from the Apache Git Service. To resp

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on PR #45234: URL: https://github.com/apache/spark/pull/45234#issuecomment-2070062199 Build is green now so PR is ready to be merged. Thanks in advance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1575603485 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -198,18 +215,23 @@ case class ShuffleQueryStageExec( reuse }

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1575677485 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -51,13 +51,30 @@ abstract class QueryStageExec extends LeafExecNode {

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1582461066 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -198,18 +215,23 @@ case class ShuffleQueryStageExec( reuse

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1582463921 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -51,13 +51,30 @@ abstract class QueryStageExec extends LeafExec

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

erenavsarogullari commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1582463921 ## sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/QueryStageExec.scala: ## @@ -51,13 +51,30 @@ abstract class QueryStageExec extends LeafExec

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan closed pull request #45234: [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation URL: https://github.com/apache/spark/pull/45234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on PR #45234: URL: https://github.com/apache/spark/pull/45234#issuecomment-2082863674 thanks, merging to master! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1684094611 ## sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala: ## @@ -47,6 +47,15 @@ import org.apache.spark.util.random.XORShiftRandom

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1684091860 ## sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala: ## @@ -47,6 +47,15 @@ import org.apache.spark.util.random.XORShiftRandom

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1684091860 ## sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala: ## @@ -47,6 +47,15 @@ import org.apache.spark.util.random.XORShiftRandom

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1684101953 ## sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala: ## @@ -47,6 +47,15 @@ import org.apache.spark.util.random.XORShiftRandom

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

cloud-fan commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1684103901 ## sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala: ## @@ -47,6 +47,15 @@ import org.apache.spark.util.random.XORShiftRandom

Re: [PR] [SPARK-47148][SQL] Avoid to materialize AQE ExchangeQueryStageExec on the cancellation [spark]

ulysses-you commented on code in PR #45234: URL: https://github.com/apache/spark/pull/45234#discussion_r1687324526 ## sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala: ## @@ -47,6 +47,15 @@ import org.apache.spark.util.random.XORShiftRand