[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r517544531 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,959 @@ +/* + * Licensed to

[GitHub] [spark] SparkQA commented on pull request #30243: [SPARK-33335][SQL] Support `array_contains_array` func

SparkQA commented on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-721898084 **[Test build #130603 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130603/testReport)** for PR 30243 at commit [`613f1c3`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30243: [SPARK-33335][SQL] Support `array_contains_array` func

SparkQA removed a comment on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-721743429 **[Test build #130603 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130603/testReport)** for PR 30243 at commit [`613f1c3`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #30243: [SPARK-33335][SQL] Support `array_contains_array` func

AmplabJenkins commented on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-721899503 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30243: [SPARK-33335][SQL] Support `array_contains_array` func

AmplabJenkins removed a comment on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-721899503 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] huaxingao commented on pull request #29695: [SPARK-22390][SPARK-32833][SQL] [WIP]JDBC V2 Datasource aggregate push down

huaxingao commented on pull request #29695: URL: https://github.com/apache/spark/pull/29695#issuecomment-721901391 @rf972 Thank you very much for taking time evaluating this patch! Also thanks for the reproduce. I know there are some issues. I am busy with something else recently and proba

[GitHub] [spark] SparkQA commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721905314 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35214/ -

[GitHub] [spark] SparkQA commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721905606 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35213/ ---

[GitHub] [spark] AmplabJenkins commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721905623 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721905623 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721905631 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8

[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

SparkQA commented on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721913702 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35215/ -

[GitHub] [spark] sarveshdave1 opened a new pull request #30252: [MINOR][SS][DOCS] Update join type in stream static joins code examples

sarveshdave1 opened a new pull request #30252: URL: https://github.com/apache/spark/pull/30252 ### What changes were proposed in this pull request? Update join type in stream static joins code examples in structured streaming programming guide. 1) Scala, Java and Python examples have

[GitHub] [spark] AmplabJenkins commented on pull request #30252: [MINOR][SS][DOCS] Update join type in stream static joins code examples

AmplabJenkins commented on pull request #30252: URL: https://github.com/apache/spark/pull/30252#issuecomment-721916621 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] xkrogen commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

xkrogen commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721916626 @mridulm FYI this should be ready for final review now. I added unit tests for the parsing logic. I will do another test of the updated logic on a real cluster to be sure. ---

[GitHub] [spark] AmplabJenkins commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721916361 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721916335 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35214/ ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721916361 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30252: [MINOR][SS][DOCS] Update join type in stream static joins code examples

AmplabJenkins commented on pull request #30252: URL: https://github.com/apache/spark/pull/30252#issuecomment-721917138 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30252: [MINOR][SS][DOCS] Update join type in stream static joins code examples

AmplabJenkins removed a comment on pull request #30252: URL: https://github.com/apache/spark/pull/30252#issuecomment-721916621 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] venkata91 commented on a change in pull request #30164: [SPARK-32919][SHUFFLE] Driver side changes for coordinating push based shuffle by selecting external shuffle services for mergi

venkata91 commented on a change in pull request #30164: URL: https://github.com/apache/spark/pull/30164#discussion_r517568438 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -657,6 +681,13 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] SparkQA commented on pull request #30251: [SPARK-33347]Cleanup useless variables in MutableApplicationInfo

SparkQA commented on pull request #30251: URL: https://github.com/apache/spark/pull/30251#issuecomment-721918796 **[Test build #130608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130608/testReport)** for PR 30251 at commit [`2284905`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #30251: [SPARK-33347]Cleanup useless variables in MutableApplicationInfo

AmplabJenkins commented on pull request #30251: URL: https://github.com/apache/spark/pull/30251#issuecomment-721919700 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] mridulm commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

mridulm commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721921611 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] mridulm commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

mridulm commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721921508 Jenkins, add to whitelist This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [spark] RogerDunn commented on pull request #30106: [WIP]Introducing a separation of concerns between data sources that are gi…

RogerDunn commented on pull request #30106: URL: https://github.com/apache/spark/pull/30106#issuecomment-721923042 @cloud-fan Your idea sounds just right. Are you proposing to make that change in the work you're already doing (in which case I'll remove this PR)?

[GitHub] [spark] SparkQA commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

SparkQA commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721924617 **[Test build #130615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130615/testReport)** for PR 30096 at commit [`b6fd413`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30251: [SPARK-33347]Cleanup useless variables in MutableApplicationInfo

AmplabJenkins removed a comment on pull request #30251: URL: https://github.com/apache/spark/pull/30251#issuecomment-721919700 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

AmplabJenkins removed a comment on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-712502179 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] SparkQA removed a comment on pull request #30251: [SPARK-33347]Cleanup useless variables in MutableApplicationInfo

SparkQA removed a comment on pull request #30251: URL: https://github.com/apache/spark/pull/30251#issuecomment-721842771 **[Test build #130608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130608/testReport)** for PR 30251 at commit [`2284905`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30251: [SPARK-33347]Cleanup useless variables in MutableApplicationInfo

AmplabJenkins removed a comment on pull request #30251: URL: https://github.com/apache/spark/pull/30251#issuecomment-721919713 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130

[GitHub] [spark] AmplabJenkins commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

AmplabJenkins commented on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721925740 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

SparkQA commented on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721925715 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35215/ ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

AmplabJenkins removed a comment on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721925740 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

AmplabJenkins removed a comment on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721925747 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8

[GitHub] [spark] SparkQA commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721928842 **[Test build #130610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130610/testReport)** for PR 30062 at commit [`7f8303a`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721929469 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721859821 **[Test build #130610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130610/testReport)** for PR 30062 at commit [`7f8303a`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721929469 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721929481 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130

[GitHub] [spark] bart-samwel commented on a change in pull request #28027: [SPARK-31255][SQL] Add SupportsMetadataColumns to DSv2

bart-samwel commented on a change in pull request #28027: URL: https://github.com/apache/spark/pull/28027#discussion_r517582852 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala ## @@ -48,6 +48,15 @@ case class D

[GitHub] [spark] SparkQA commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

SparkQA commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721933850 **[Test build #130615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130615/testReport)** for PR 30096 at commit [`b6fd413`](https://github.co

[GitHub] [spark] attilapiros commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

attilapiros commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r517585393 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,959 @@ +/* + * License

[GitHub] [spark] AmplabJenkins commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

AmplabJenkins commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721933991 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

SparkQA removed a comment on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721924617 **[Test build #130615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130615/testReport)** for PR 30096 at commit [`b6fd413`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

AmplabJenkins removed a comment on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721933991 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] sunchao commented on a change in pull request #30187: [SPARK-33290][SQL] REFRESH TABLE should invalidate cache even though the table itself may not be cached

sunchao commented on a change in pull request #30187: URL: https://github.com/apache/spark/pull/30187#discussion_r517590557 ## File path: sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala ## @@ -524,14 +527,17 @@ class CatalogImpl(sparkSession: SparkSessi

[GitHub] [spark] SparkQA commented on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

SparkQA commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721939478 **[Test build #130616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130616/testReport)** for PR 30242 at commit [`aec13c2`](https://github.com

[GitHub] [spark] tgravescs commented on a change in pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

tgravescs commented on a change in pull request #30096: URL: https://github.com/apache/spark/pull/30096#discussion_r517591670 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -1172,6 +1189,26 @@ private[spark] class Client(

[GitHub] [spark] tgravescs commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.

tgravescs commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-721946217 test this please This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [spark] zero323 commented on a change in pull request #30181: [SPARK-33250][PYTHON][DOCS] Migration to NumPy documentation style in SQL (pyspark.sql.*)

zero323 commented on a change in pull request #30181: URL: https://github.com/apache/spark/pull/30181#discussion_r517599907 ## File path: python/pyspark/sql/avro/functions.py ## @@ -25,7 +25,6 @@ from pyspark.util import _print_missing_jar -@since(3.0) Review comment:

[GitHub] [spark] SparkQA commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

SparkQA commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721947736 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35216/ -

[GitHub] [spark] gaborgsomogyi commented on pull request #30151: [SPARK-33223][SS][UI]Structured Streaming Web UI state information

gaborgsomogyi commented on pull request #30151: URL: https://github.com/apache/spark/pull/30151#issuecomment-721948315 Sure, started to have a look what tests are possible to add. Doing some other experiments so will take some time.

[GitHub] [spark] SparkQA commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.

SparkQA commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-721949728 **[Test build #130617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130617/testReport)** for PR 28528 at commit [`c8ce5a3`](https://github.com

[GitHub] [spark] zero323 commented on a change in pull request #30181: [SPARK-33250][PYTHON][DOCS] Migration to NumPy documentation style in SQL (pyspark.sql.*)

zero323 commented on a change in pull request #30181: URL: https://github.com/apache/spark/pull/30181#discussion_r517599907 ## File path: python/pyspark/sql/avro/functions.py ## @@ -25,7 +25,6 @@ from pyspark.util import _print_missing_jar -@since(3.0) Review comment:

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r517604518 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,959 @@ +/* + * Licensed to

[GitHub] [spark] viirya commented on a change in pull request #30246: [SPARK-33338][SQL] GROUP BY using literal map should not fail

viirya commented on a change in pull request #30246: URL: https://github.com/apache/spark/pull/30246#discussion_r517605777 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala ## @@ -316,6 +316,8 @@ case class Literal (value: Any,

[GitHub] [spark] tgravescs commented on a change in pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

tgravescs commented on a change in pull request #30096: URL: https://github.com/apache/spark/pull/30096#discussion_r517607144 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -1172,6 +1189,26 @@ private[spark] class Client(

[GitHub] [spark] sunchao commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

sunchao commented on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721956706 So this has been discussed sometime back: https://github.com/sbt/sbt-assembly/issues/120, but I don't know whether there is already a fix or not. -

[GitHub] [spark] SparkQA commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

SparkQA commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721958422 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35216/ ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

AmplabJenkins removed a comment on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721958443 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

AmplabJenkins commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721958443 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

SparkQA commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721961871 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35217/ -

[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

SparkQA commented on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721962965 **[Test build #130609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130609/testReport)** for PR 30250 at commit [`babab2e`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

SparkQA removed a comment on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721847050 **[Test build #130609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130609/testReport)** for PR 30250 at commit [`babab2e`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

AmplabJenkins commented on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721963823 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

AmplabJenkins removed a comment on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721963823 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

AmplabJenkins removed a comment on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721963831 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130

[GitHub] [spark] SparkQA commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.

SparkQA commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-721965973 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35218/ -

[GitHub] [spark] SparkQA commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721967395 **[Test build #130613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130613/testReport)** for PR 30062 at commit [`ef9af3a`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721875852 **[Test build #130613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130613/testReport)** for PR 30062 at commit [`ef9af3a`](https://gi

[GitHub] [spark] SparkQA commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721967649 **[Test build #130612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130612/testReport)** for PR 30062 at commit [`26ee13e`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

SparkQA removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721872023 **[Test build #130612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130612/testReport)** for PR 30062 at commit [`26ee13e`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721968179 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins commented on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721968386 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721968391 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

AmplabJenkins removed a comment on pull request #30062: URL: https://github.com/apache/spark/pull/30062#issuecomment-721968179 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] xkrogen commented on a change in pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

xkrogen commented on a change in pull request #30096: URL: https://github.com/apache/spark/pull/30096#discussion_r517624169 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -1172,6 +1189,26 @@ private[spark] class Client(

[GitHub] [spark] SparkQA commented on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

SparkQA commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721970110 **[Test build #130616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130616/testReport)** for PR 30242 at commit [`aec13c2`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

SparkQA removed a comment on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721939478 **[Test build #130616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130616/testReport)** for PR 30242 at commit [`aec13c2`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

AmplabJenkins removed a comment on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721970426 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] AmplabJenkins commented on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

AmplabJenkins commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721970426 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] xkrogen commented on a change in pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

xkrogen commented on a change in pull request #30096: URL: https://github.com/apache/spark/pull/30096#discussion_r517627254 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ## @@ -1080,9 +1085,9 @@ private[spark] class Client(

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

AmplabJenkins removed a comment on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721970434 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130

[GitHub] [spark] AmplabJenkins commented on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

AmplabJenkins commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721971770 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] tgravescs commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

tgravescs commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r517620874 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,959 @@ +/* + * Licensed

[GitHub] [spark] SparkQA commented on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

SparkQA commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721971757 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35217/ ---

[GitHub] [spark] SparkQA commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

SparkQA commented on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721972171 **[Test build #130614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130614/testReport)** for PR 30250 at commit [`babab2e`](https://github.co

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

AmplabJenkins removed a comment on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721971770 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

SparkQA removed a comment on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721891101 **[Test build #130614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130614/testReport)** for PR 30250 at commit [`babab2e`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

AmplabJenkins commented on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721973187 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] xkrogen commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

xkrogen commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721973407 New push fixes a bug when the driver container hasn't yet started (thanks Thomas) and hides the feature behind a disabled-by-default config. I would be curious if anyone has tho

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30250: [SPARK-33343][BUILD] Fix the build with sbt to copy hadoop-client-runtime.jar

AmplabJenkins removed a comment on pull request #30250: URL: https://github.com/apache/spark/pull/30250#issuecomment-721973187 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30096: [SPARK-33185][YARN] Set up yarn.Client to print direct links to driver stdout/stderr

SparkQA commented on pull request #30096: URL: https://github.com/apache/spark/pull/30096#issuecomment-721973715 **[Test build #130618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130618/testReport)** for PR 30096 at commit [`745c8cf`](https://github.com

[GitHub] [spark] ueshin commented on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

ueshin commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721974986 Jenkins, retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] otterc commented on a change in pull request #30062: [SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode

otterc commented on a change in pull request #30062: URL: https://github.com/apache/spark/pull/30062#discussion_r517632529 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java ## @@ -0,0 +1,959 @@ +/* + * Licensed to

[GitHub] [spark] SparkQA commented on pull request #30242: [SPARK-33277][PYSPARK][SQL][FOLLOW-UP] Block TaskCompletion event until the thread ends.

SparkQA commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-721977085 **[Test build #130619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130619/testReport)** for PR 30242 at commit [`aec13c2`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.

SparkQA commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-721978507 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35218/ ---

[GitHub] [spark] AmplabJenkins commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.

AmplabJenkins commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-721978538 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.

AmplabJenkins removed a comment on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-721978538 This is an automated message from the Apache Git Service. To respond to the message, please log on

  1   2   3   4   5   6   7   8   >