[GitHub] [spark] SparkQA commented on pull request #28463: [SPARK-31399][CORE] Support indylambda Scala closure in ClosureCleaner

2020-05-08 Thread GitBox
SparkQA commented on pull request #28463: URL: https://github.com/apache/spark/pull/28463#issuecomment-625688089 **[Test build #122430 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122430/testReport)** for PR 28463 at commit

[GitHub] [spark] Ngone51 commented on pull request #26440: [SPARK-20628][CORE][K8S] Start to improve Spark decommissioning & preemption support

2020-05-08 Thread GitBox
Ngone51 commented on pull request #26440: URL: https://github.com/apache/spark/pull/26440#issuecomment-625681613 > Right so just send SIGPWR to the worker. Are you saying in standalone mode you have one worker with multiple executors and you want to decommission a specific executor?

[GitHub] [spark] jiangxb1987 commented on a change in pull request #28444: [SPARK-31632][CORE][WEBUI] Make the ApplicationInfo always available when accessed

2020-05-08 Thread GitBox
jiangxb1987 commented on a change in pull request #28444: URL: https://github.com/apache/spark/pull/28444#discussion_r42198 ## File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala ## @@ -35,8 +36,24 @@ private[spark] class AppStatusStore( val

[GitHub] [spark] cloud-fan commented on a change in pull request #28477: [SPARK-31405][SQL] Fail by default when reading/writing ancient datetime values from/to Parquet/Avro files

2020-05-08 Thread GitBox
cloud-fan commented on a change in pull request #28477: URL: https://github.com/apache/spark/pull/28477#discussion_r421984688 ## File path: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java ## @@ -102,14 +103,14 @@ // The

[GitHub] [spark] cloud-fan commented on a change in pull request #28477: [SPARK-31405][SQL] Fail by default when reading/writing ancient datetime values from/to Parquet/Avro files

2020-05-08 Thread GitBox
cloud-fan commented on a change in pull request #28477: URL: https://github.com/apache/spark/pull/28477#discussion_r421982556 ## File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ## @@ -46,17 +47,40 @@ class AvroSerializer(

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28474: [SPARK-31658] Fix SQL UI not showing write commands of AQE plan

2020-05-08 Thread GitBox
AmplabJenkins removed a comment on pull request #28474: URL: https://github.com/apache/spark/pull/28474#issuecomment-625671206 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] yaooqinn edited a comment on pull request #28106: [SPARK-31335][SQL] Add try function support

2020-05-08 Thread GitBox
yaooqinn edited a comment on pull request #28106: URL: https://github.com/apache/spark/pull/28106#issuecomment-625671160 One more thing, the scope that the `try` can cover is not that large you have imagined. The cases you listed is actually are all invalid, e.g. sum(a, b)??? ```sql

[GitHub] [spark] yaooqinn commented on pull request #28106: [SPARK-31335][SQL] Add try function support

2020-05-08 Thread GitBox
yaooqinn commented on pull request #28106: URL: https://github.com/apache/spark/pull/28106#issuecomment-625671160 One more things, the scope that the `try` can cover is not that large you have imagined. The cases you listed is actually are all invalid, e.g. sum(a, b)??? ```sql --

[GitHub] [spark] SparkQA removed a comment on pull request #28474: [SPARK-31658] Fix SQL UI not showing write commands of AQE plan

2020-05-08 Thread GitBox
SparkQA removed a comment on pull request #28474: URL: https://github.com/apache/spark/pull/28474#issuecomment-625627264 **[Test build #122427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122427/testReport)** for PR 28474 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28474: [SPARK-31658] Fix SQL UI not showing write commands of AQE plan

2020-05-08 Thread GitBox
AmplabJenkins commented on pull request #28474: URL: https://github.com/apache/spark/pull/28474#issuecomment-625671199 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28474: [SPARK-31658] Fix SQL UI not showing write commands of AQE plan

2020-05-08 Thread GitBox
AmplabJenkins removed a comment on pull request #28474: URL: https://github.com/apache/spark/pull/28474#issuecomment-625671199 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #28474: [SPARK-31658] Fix SQL UI not showing write commands of AQE plan

2020-05-08 Thread GitBox
SparkQA commented on pull request #28474: URL: https://github.com/apache/spark/pull/28474#issuecomment-625670916 **[Test build #122427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122427/testReport)** for PR 28474 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
AmplabJenkins removed a comment on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625666500 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
AmplabJenkins removed a comment on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625666493 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
SparkQA removed a comment on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625663877 **[Test build #122429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122429/testReport)** for PR 27944 at commit

[GitHub] [spark] SparkQA commented on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
SparkQA commented on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625666481 **[Test build #122429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122429/testReport)** for PR 27944 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
AmplabJenkins commented on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625666493 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] Ngone51 commented on a change in pull request #28476: [SPARK-31485][CORE][FOLLOW-UP] Also refer blacklisting in error message if barrier scheduling fail

2020-05-08 Thread GitBox
Ngone51 commented on a change in pull request #28476: URL: https://github.com/apache/spark/pull/28476#discussion_r421973198 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ## @@ -688,10 +688,11 @@ private[spark] class TaskSchedulerImpl(

[GitHub] [spark] holdenk commented on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned

2020-05-08 Thread GitBox
holdenk commented on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-625665354 Jenkins retest this please. I feel like I’ve seen these particular tests fail before; let’s dig into what’s causing them to fail.

[GitHub] [spark] jiangxb1987 commented on a change in pull request #28476: [SPARK-31485][CORE][FOLLOW-UP] Also refer blacklisting in error message if barrier scheduling fail

2020-05-08 Thread GitBox
jiangxb1987 commented on a change in pull request #28476: URL: https://github.com/apache/spark/pull/28476#discussion_r421973578 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ## @@ -688,10 +688,11 @@ private[spark] class

[GitHub] [spark] Ngone51 commented on a change in pull request #28476: [SPARK-31485][CORE][FOLLOW-UP] Also refer blacklisting in error message if barrier scheduling fail

2020-05-08 Thread GitBox
Ngone51 commented on a change in pull request #28476: URL: https://github.com/apache/spark/pull/28476#discussion_r421973198 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ## @@ -688,10 +688,11 @@ private[spark] class TaskSchedulerImpl(

[GitHub] [spark] rednaxelafx commented on a change in pull request #28463: [SPARK-31399][CORE] Support indylambda Scala closure in ClosureCleaner

2020-05-08 Thread GitBox
rednaxelafx commented on a change in pull request #28463: URL: https://github.com/apache/spark/pull/28463#discussion_r421973199 ## File path: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala ## @@ -538,43 +604,56 @@ private[spark] object IndylambdaScalaClosures

[GitHub] [spark] holdenk commented on pull request #28370: [SPARK-20732][CORE] Decommission cache blocks to other executors when an executor is decommissioned

2020-05-08 Thread GitBox
holdenk commented on pull request #28370: URL: https://github.com/apache/spark/pull/28370#issuecomment-625664892 Right so just send SIGPWR to the worker. Are you saying in standalone mode you have one worker with multiple executors and you want to decommission a specific executor?

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
AmplabJenkins removed a comment on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625664443 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
SparkQA removed a comment on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625660830 **[Test build #122428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122428/testReport)** for PR 27944 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
AmplabJenkins removed a comment on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625664438 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
SparkQA commented on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625664424 **[Test build #122428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122428/testReport)** for PR 27944 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
AmplabJenkins commented on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625664438 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] turboFei commented on a change in pull request #26339: [SPARK-27194][SPARK-29302][SQL] For dynamic partition overwrite operation, fix speculation task conflict issue and FileAlreadyEx

2020-05-08 Thread GitBox
turboFei commented on a change in pull request #26339: URL: https://github.com/apache/spark/pull/26339#discussion_r421966564 ## File path: sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala ## @@ -157,3 +161,49 @@ class PartitionedWriteSuite

[GitHub] [spark] SparkQA commented on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
SparkQA commented on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625663877 **[Test build #122429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122429/testReport)** for PR 27944 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
AmplabJenkins removed a comment on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625661311 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
AmplabJenkins commented on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625661311 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #27944: [SPARK-31180][ML] Implement PowerTransform

2020-05-08 Thread GitBox
SparkQA commented on pull request #27944: URL: https://github.com/apache/spark/pull/27944#issuecomment-625660830 **[Test build #122428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122428/testReport)** for PR 27944 at commit

[GitHub] [spark] turboFei commented on a change in pull request #26339: [SPARK-27194][SPARK-29302][SQL] For dynamic partition overwrite operation, fix speculation task conflict issue and FileAlreadyEx

2020-05-08 Thread GitBox
turboFei commented on a change in pull request #26339: URL: https://github.com/apache/spark/pull/26339#discussion_r421966564 ## File path: sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala ## @@ -157,3 +161,49 @@ class PartitionedWriteSuite

[GitHub] [spark] stijndehaes commented on pull request #28423: [SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

2020-05-08 Thread GitBox
stijndehaes commented on pull request #28423: URL: https://github.com/apache/spark/pull/28423#issuecomment-625656337 @holdenk @dongjoon-hyun I have tested this code in production and it works. I have a couple of jobs that take roughly 4 hours to finish, these all failed without the fix

[GitHub] [spark] zhengruifeng commented on pull request #28473: [SPARK-31656][ML][PYSPARK] AFT blockify input vectors

2020-05-08 Thread GitBox
zhengruifeng commented on pull request #28473: URL: https://github.com/apache/spark/pull/28473#issuecomment-625650251 Merged to master This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] yaooqinn edited a comment on pull request #28106: [SPARK-31335][SQL] Add try function support

2020-05-08 Thread GitBox
yaooqinn edited a comment on pull request #28106: URL: https://github.com/apache/spark/pull/28106#issuecomment-625648533 I see the concerns. The behavior of `try` here is simply that it wraps a `try-catch` for its root child expression. And the meanings of `try` will be spread based

[GitHub] [spark] yaooqinn commented on pull request #28106: [SPARK-31335][SQL] Add try function support

2020-05-08 Thread GitBox
yaooqinn commented on pull request #28106: URL: https://github.com/apache/spark/pull/28106#issuecomment-625648533 I see the concerns. The behavior of `try` here is simply that it wraps a `try-catch` for its root child expression. And the meanings of `try` will be spread based on how

<    1   2   3   4