[GitHub] [spark] AmplabJenkins commented on pull request #30177: [SPARK-33277][PYSPARK][SQL] Use ContextAwareIterator to stop consuming after the task ends.

AmplabJenkins commented on pull request #30177: URL: https://github.com/apache/spark/pull/30177#issuecomment-718370669 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-718379863 **[Test build #130395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130395/testReport)** for PR 30062 at commit [`b3fc7d0`](https://github.com

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r514001854 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,905 @@ +/* + * Licensed to

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r514003255 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,893 @@ +/* + * Licensed to

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r514003770 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,905 @@ +/* + * Licensed to

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r514003670 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,883 @@ +/* + * Licensed to

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r514003367 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,905 @@ +/* + * Licensed to

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r514007203 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,893 @@ +/* + * Licensed to

[GitHub] [spark] luluorta commented on a change in pull request #29882: [SPARK-33008][SQL] Division by zero on divide-like operations returns incorrect result

luluorta commented on a change in pull request #29882: URL: https://github.com/apache/spark/pull/29882#discussion_r514008045 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala ## @@ -500,4 +524,53 @@ class Arithm

[GitHub] [spark] SparkQA removed a comment on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

SparkQA removed a comment on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-718348533 **[Test build #130394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130394/testReport)** for PR 29247 at commit [`1854e74`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

SparkQA commented on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-718384689 **[Test build #130394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130394/testReport)** for PR 29247 at commit [`1854e74`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

AmplabJenkins commented on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-718385194 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

AmplabJenkins removed a comment on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-718385194 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29247: [SPARK-32446][SHS] Add new executor metrics summary REST APIs

AmplabJenkins removed a comment on pull request #29247: URL: https://github.com/apache/spark/pull/29247#issuecomment-718385203 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130

[GitHub] [spark] SparkQA commented on pull request #30162: [SPARK-33263][SS] Configurable StateStore compression codec

SparkQA commented on pull request #30162: URL: https://github.com/apache/spark/pull/30162#issuecomment-718386994 **[Test build #130390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130390/testReport)** for PR 30162 at commit [`8e435a1`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30162: [SPARK-33263][SS] Configurable StateStore compression codec

SparkQA removed a comment on pull request #30162: URL: https://github.com/apache/spark/pull/30162#issuecomment-718308057 **[Test build #130390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130390/testReport)** for PR 30162 at commit [`8e435a1`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #30162: [SPARK-33263][SS] Configurable StateStore compression codec

AmplabJenkins commented on pull request #30162: URL: https://github.com/apache/spark/pull/30162#issuecomment-718387807 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30162: [SPARK-33263][SS] Configurable StateStore compression codec

AmplabJenkins removed a comment on pull request #30162: URL: https://github.com/apache/spark/pull/30162#issuecomment-718387807 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AngersZhuuuu commented on pull request #30139: [SPARK-31069][CORE] high cpu caused by chunksBeingTransferred in external shuffle service

AngersZh commented on pull request #30139: URL: https://github.com/apache/spark/pull/30139#issuecomment-718389398 @otterc Know what you are concern, I haven't thought of such a good solution for a while ping @jiangxb1987 @HeartSaVioR @dongjoon-hyun Any more advise? -

[GitHub] [spark] SparkQA commented on pull request #30156: [SPARK-33248][SQL] Add a configuration to control the legacy behavior of whether need to pad null value when value size less then schema size

SparkQA commented on pull request #30156: URL: https://github.com/apache/spark/pull/30156#issuecomment-718394716 **[Test build #130391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130391/testReport)** for PR 30156 at commit [`3148608`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30156: [SPARK-33248][SQL] Add a configuration to control the legacy behavior of whether need to pad null value when value size less then sch

SparkQA removed a comment on pull request #30156: URL: https://github.com/apache/spark/pull/30156#issuecomment-718314992 **[Test build #130391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130391/testReport)** for PR 30156 at commit [`3148608`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #30156: [SPARK-33248][SQL] Add a configuration to control the legacy behavior of whether need to pad null value when value size less then schem

AmplabJenkins commented on pull request #30156: URL: https://github.com/apache/spark/pull/30156#issuecomment-718395498 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] mridulm commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

mridulm commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r514024472 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,893 @@ +/* + * Licensed to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30156: [SPARK-33248][SQL] Add a configuration to control the legacy behavior of whether need to pad null value when value size less th

AmplabJenkins removed a comment on pull request #30156: URL: https://github.com/apache/spark/pull/30156#issuecomment-718395498 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] Karl-WangSK closed pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates

Karl-WangSK closed pull request #29360: URL: https://github.com/apache/spark/pull/29360 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] mridulm commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

mridulm commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r514026014 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,905 @@ +/* + * Licensed to

[GitHub] [spark] mridulm commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

mridulm commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r514027473 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,883 @@ +/* + * Licensed to

[GitHub] [spark] cloud-fan commented on a change in pull request #29882: [SPARK-33008][SQL] Division by zero on divide-like operations returns incorrect result

cloud-fan commented on a change in pull request #29882: URL: https://github.com/apache/spark/pull/29882#discussion_r514027971 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala ## @@ -500,4 +524,53 @@ class Arith

[GitHub] [spark] SparkQA commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-718399594 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34998/ -

[GitHub] [spark] LuciferYang opened a new pull request #30179: [SPARK-12667][CORE][FOLLOWUP] Remove useless externalBlockStoreSize from RDDInfo

LuciferYang opened a new pull request #30179: URL: https://github.com/apache/spark/pull/30179 ### What changes were proposed in this pull request? "external block store" API was removed after SPARK-12667, `externalBlockStoreSize` in `RDDInfo` looks like always 0 and useless. So this pr

[GitHub] [spark] cloud-fan commented on a change in pull request #30154: [SPARK-32405][SQL] Apply table options while creating tables in JDBC Table Catalog

cloud-fan commented on a change in pull request #30154: URL: https://github.com/apache/spark/pull/30154#discussion_r514035984 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalog.scala ## @@ -123,8 +123,14 @@ class JDBCTable

<    2   3   4   5   6   7