[GitHub] [spark] SparkQA removed a comment on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread GitBox
SparkQA removed a comment on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-632390624 **[Test build #122949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122949/testReport)** for PR 28596 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input

2020-05-21 Thread GitBox
dongjoon-hyun commented on a change in pull request #28604: URL: https://github.com/apache/spark/pull/28604#discussion_r429009076 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -49,6 +49,24 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] SparkQA commented on pull request #28596: [SPARK-31784][CORE][TEST] Fix test BarrierTaskContextSuite."share messages with allGather() call"

2020-05-21 Thread GitBox
SparkQA commented on pull request #28596: URL: https://github.com/apache/spark/pull/28596#issuecomment-632441024 **[Test build #122949 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122949/testReport)** for PR 28596 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input

2020-05-21 Thread GitBox
dongjoon-hyun commented on a change in pull request #28604: URL: https://github.com/apache/spark/pull/28604#discussion_r429008842 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala ## @@ -49,6 +49,24 @@ class SQLQuerySuite extends QueryTest with

[GitHub] [spark] maropu commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
maropu commented on a change in pull request #28560: URL: https://github.com/apache/spark/pull/28560#discussion_r429007625 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala ## @@ -338,6 +340,75 @@ abstract class

[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632439127 cc @dbtsai and @holdenk This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632438386 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632438386 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632438465 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632438452 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27597/

[GitHub] [spark] SparkQA removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632386056 **[Test build #122947 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122947/testReport)** for PR 28601 at commit

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632437537 **[Test build #122947 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122947/testReport)** for PR 28601 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632437332 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632437332 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28592: [SPARK-31771][SQL] Disable Narrow TextStyle for datetime pattern 'G/M/L/E/u/Q/q'

2020-05-21 Thread GitBox
SparkQA commented on pull request #28592: URL: https://github.com/apache/spark/pull/28592#issuecomment-632437045 **[Test build #122958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122958/testReport)** for PR 28592 at commit

[GitHub] [spark] HyukjinKwon commented on pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs

2020-05-21 Thread GitBox
HyukjinKwon commented on pull request #28603: URL: https://github.com/apache/spark/pull/28603#issuecomment-632437124 Shell as also add a unit test? Also, please describe before/after this fix in "Does this PR introduce any user-facing change?". Technically I think this IS a user-facing

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs

2020-05-21 Thread GitBox
HyukjinKwon commented on a change in pull request #28603: URL: https://github.com/apache/spark/pull/28603#discussion_r429005432 ## File path: python/pyspark/context.py ## @@ -864,8 +865,20 @@ def union(self, rdds): first_jrdd_deserializer = rdds[0]._jrdd_deserializer

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632431676 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27597/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632430611 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632430605 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632430605 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632429899 **[Test build #122946 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122946/testReport)** for PR 28601 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632381162 **[Test build #122946 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122946/testReport)** for PR 28601 at commit

[GitHub] [spark] viirya commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
viirya commented on a change in pull request #28560: URL: https://github.com/apache/spark/pull/28560#discussion_r428998943 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala ## @@ -338,6 +340,75 @@ abstract class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632427679 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632427677 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] maropu commented on a change in pull request #28540: [SPARK-31719][SQL] Refactor JoinSelection

2020-05-21 Thread GitBox
maropu commented on a change in pull request #28540: URL: https://github.com/apache/spark/pull/28540#discussion_r428996754 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala ## @@ -208,3 +208,161 @@ object

[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632427677 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632375746 **[Test build #122944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122944/testReport)** for PR 28601 at commit

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632426852 **[Test build #122944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122944/testReport)** for PR 28601 at commit

[GitHub] [spark] jalpan-randeri commented on pull request #26585: [WIP][SPARK-25351][SQL][Python] Handle Pandas category type when converting from Python with Arrow

2020-05-21 Thread GitBox
jalpan-randeri commented on pull request #26585: URL: https://github.com/apache/spark/pull/26585#issuecomment-632423189 gentle ping. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28600: [SPARK-31761][SQL] cast integer to Long to avoid IntegerOverflow for IntegralDivide operator

2020-05-21 Thread GitBox
HyukjinKwon commented on a change in pull request #28600: URL: https://github.com/apache/spark/pull/28600#discussion_r428991924 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala ## @@ -337,11 +340,19 @@ trait DivModLike

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #27019: URL: https://github.com/apache/spark/pull/27019#issuecomment-632421353 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] maropu commented on pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
maropu commented on pull request #28560: URL: https://github.com/apache/spark/pull/28560#issuecomment-632421429 Looks okay cc: @dongjoon-hyun @dbtsai This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] maropu commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
maropu commented on a change in pull request #28560: URL: https://github.com/apache/spark/pull/28560#discussion_r428991332 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala ## @@ -338,6 +340,75 @@ abstract class

[GitHub] [spark] AmplabJenkins commented on pull request #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #27019: URL: https://github.com/apache/spark/pull/27019#issuecomment-632421353 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632421071 **[Test build #122956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122956/testReport)** for PR 28601 at commit

[GitHub] [spark] SparkQA commented on pull request #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec

2020-05-21 Thread GitBox
SparkQA commented on pull request #27019: URL: https://github.com/apache/spark/pull/27019#issuecomment-632421118 **[Test build #122957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122957/testReport)** for PR 27019 at commit

[GitHub] [spark] maropu commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
maropu commented on a change in pull request #28560: URL: https://github.com/apache/spark/pull/28560#discussion_r428991085 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala ## @@ -341,6 +341,100 @@ class

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28600: [SPARK-31761][SQL] cast integer to Long to avoid IntegerOverflow for IntegralDivide operator

2020-05-21 Thread GitBox
HyukjinKwon commented on a change in pull request #28600: URL: https://github.com/apache/spark/pull/28600#discussion_r428991160 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala ## @@ -505,4 +505,9 @@ class

[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632420623 The one failure in the last K8s IT is a known flaky one. ``` KubernetesSuite: - Run SparkPi with no resources *** FAILED *** The code passed to eventually

[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632420252 Retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] skambha commented on pull request #27627: [WIP][SPARK-28067][SQL] Fix incorrect results for decimal aggregate sum by returning null on decimal overflow

2020-05-21 Thread GitBox
skambha commented on pull request #27627: URL: https://github.com/apache/spark/pull/27627#issuecomment-632420083 sure @cloud-fan. I will take a look and add comments there. This is an automated message from the Apache Git

[GitHub] [spark] HyukjinKwon closed pull request #28597: fix typo

2020-05-21 Thread GitBox
HyukjinKwon closed pull request #28597: URL: https://github.com/apache/spark/pull/28597 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28597: fix typo

2020-05-21 Thread GitBox
HyukjinKwon commented on a change in pull request #28597: URL: https://github.com/apache/spark/pull/28597#discussion_r428989788 ## File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ## @@ -1797,7 +1797,7 @@ object functions { /** * Returns the

[GitHub] [spark] HyukjinKwon commented on pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers

2020-05-21 Thread GitBox
HyukjinKwon commented on pull request #28598: URL: https://github.com/apache/spark/pull/28598#issuecomment-632419544 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] maropu commented on pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))

2020-05-21 Thread GitBox
maropu commented on pull request #27803: URL: https://github.com/apache/spark/pull/27803#issuecomment-632419597 Yea, that sounds reasonable. Anyway, thanks for your reviews and attentions, @dongjoon-hyun and @dilipbiswal.

[GitHub] [spark] HyukjinKwon closed pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers

2020-05-21 Thread GitBox
HyukjinKwon closed pull request #28598: URL: https://github.com/apache/spark/pull/28598 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632418445 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632418438 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632418419 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27593/

[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632418438 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28598: [SPARK-31785][SQL][TESTS] Add a helper function to test all parquet readers

2020-05-21 Thread GitBox
HyukjinKwon commented on a change in pull request #28598: URL: https://github.com/apache/spark/pull/28598#discussion_r428987921 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTest.scala ## @@ -162,4 +162,36 @@ private[sql]

[GitHub] [spark] AmplabJenkins commented on pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28560: URL: https://github.com/apache/spark/pull/28560#issuecomment-632417641 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28363: [SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28363: URL: https://github.com/apache/spark/pull/28363#issuecomment-632417665 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28363: [SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28363: URL: https://github.com/apache/spark/pull/28363#issuecomment-632417665 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28560: URL: https://github.com/apache/spark/pull/28560#issuecomment-632417641 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] maropu commented on a change in pull request #28600: [SPARK-31761][SQL] cast integer to Long to avoid IntegerOverflow for IntegralDivide operator

2020-05-21 Thread GitBox
maropu commented on a change in pull request #28600: URL: https://github.com/apache/spark/pull/28600#discussion_r428987485 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala ## @@ -337,11 +340,19 @@ trait DivModLike extends

[GitHub] [spark] SparkQA commented on pull request #28363: [SPARK-27188][SS] FileStreamSink: provide a new option to have retention on output files

2020-05-21 Thread GitBox
SparkQA commented on pull request #28363: URL: https://github.com/apache/spark/pull/28363#issuecomment-632417393 **[Test build #122955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122955/testReport)** for PR 28363 at commit

[GitHub] [spark] SparkQA commented on pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
SparkQA commented on pull request #28560: URL: https://github.com/apache/spark/pull/28560#issuecomment-632417352 **[Test build #122954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122954/testReport)** for PR 28560 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))

2020-05-21 Thread GitBox
dongjoon-hyun commented on pull request #27803: URL: https://github.com/apache/spark/pull/27803#issuecomment-632416179 Sorry for being late, @maropu . +1 for revisit this later. Thanks. This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28485: [SPARK-31642] Add Pagination Support for Structured Streaming Page

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28485: URL: https://github.com/apache/spark/pull/28485#issuecomment-632415396 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28485: [SPARK-31642] Add Pagination Support for Structured Streaming Page

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28485: URL: https://github.com/apache/spark/pull/28485#issuecomment-632415396 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28485: [SPARK-31642] Add Pagination Support for Structured Streaming Page

2020-05-21 Thread GitBox
SparkQA removed a comment on pull request #28485: URL: https://github.com/apache/spark/pull/28485#issuecomment-632293771 **[Test build #122941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122941/testReport)** for PR 28485 at commit

[GitHub] [spark] SparkQA commented on pull request #28485: [SPARK-31642] Add Pagination Support for Structured Streaming Page

2020-05-21 Thread GitBox
SparkQA commented on pull request #28485: URL: https://github.com/apache/spark/pull/28485#issuecomment-632414682 **[Test build #122941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122941/testReport)** for PR 28485 at commit

[GitHub] [spark] TJX2014 commented on a change in pull request #28534: [SPARK-31710][SQL] Adds TIMESTAMP_SECONDS, TIMESTAMP_MILLIS and TIMESTAMP_MICROS functions

2020-05-21 Thread GitBox
TJX2014 commented on a change in pull request #28534: URL: https://github.com/apache/spark/pull/28534#discussion_r428984616 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala ## @@ -401,6 +401,83 @@ case class

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632412704 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27593/

[GitHub] [spark] maropu closed pull request #27803: [SPARK-31049][SQL] Support nested adjacent generators, e.g., explode(explode(v))

2020-05-21 Thread GitBox
maropu closed pull request #27803: URL: https://github.com/apache/spark/pull/27803 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] github-actions[bot] commented on pull request #26503: [SPARK-29880][CORE][YARN] Handle submit exception when submit to federation cluster

2020-05-21 Thread GitBox
github-actions[bot] commented on pull request #26503: URL: https://github.com/apache/spark/pull/26503#issuecomment-632409128 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] github-actions[bot] closed pull request #27231: [SPARK-28478][SQL] Remove redundant null checks

2020-05-21 Thread GitBox
github-actions[bot] closed pull request #27231: URL: https://github.com/apache/spark/pull/27231 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] github-actions[bot] commented on pull request #27213: [SPARK-30516][SQL] involve partition filters in the statistic estimation of FileScan

2020-05-21 Thread GitBox
github-actions[bot] commented on pull request #27213: URL: https://github.com/apache/spark/pull/27213#issuecomment-632409120 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue

[GitHub] [spark] github-actions[bot] closed pull request #27436: [SPARK-30705][SQL] Improve CaseWhen sub-expression equality

2020-05-21 Thread GitBox
github-actions[bot] closed pull request #27436: URL: https://github.com/apache/spark/pull/27436 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28604: URL: https://github.com/apache/spark/pull/28604#issuecomment-632409285 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28604: URL: https://github.com/apache/spark/pull/28604#issuecomment-632409285 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input

2020-05-21 Thread GitBox
SparkQA commented on pull request #28604: URL: https://github.com/apache/spark/pull/28604#issuecomment-632408912 **[Test build #122953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122953/testReport)** for PR 28604 at commit

[GitHub] [spark] maropu commented on pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input

2020-05-21 Thread GitBox
maropu commented on pull request #28604: URL: https://github.com/apache/spark/pull/28604#issuecomment-632408836 @cloud-fan @gengliangwang I've checked the proposal in https://github.com/apache/spark/pull/27024 again and I think it's been already fixed in the master. So, we could close

[GitHub] [spark] maropu opened a new pull request #28604: [SPARK-29854][SQL][TESTS] Add tests to check lpad/rpad throw an exception for invalid length input

2020-05-21 Thread GitBox
maropu opened a new pull request #28604: URL: https://github.com/apache/spark/pull/28604 ### What changes were proposed in this pull request? This PR intends to add trivial tests to check https://github.com/apache/spark/pull/27024 has already been fixed in the master.

[GitHub] [spark] viirya commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
viirya commented on a change in pull request #28560: URL: https://github.com/apache/spark/pull/28560#discussion_r428975902 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala ## @@ -48,7 +53,10 @@ object

[GitHub] [spark] AmplabJenkins commented on pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28603: URL: https://github.com/apache/spark/pull/28603#issuecomment-632402291 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28603: URL: https://github.com/apache/spark/pull/28603#issuecomment-632401913 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] AmplabJenkins commented on pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28603: URL: https://github.com/apache/spark/pull/28603#issuecomment-632401913 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] redsanket opened a new pull request #28603: [SPARK-31788][CORE] Fix UnionRDD of PairRDDs

2020-05-21 Thread GitBox
redsanket opened a new pull request #28603: URL: https://github.com/apache/spark/pull/28603 ### What changes were proposed in this pull request? UnionRDD of PairRDDs causing a bug. The fix is to check for instance type before proceeding ### Why are the changes needed? We can

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399668 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399864 **[Test build #122952 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122952/testReport)** for PR 28601 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399661 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632399475 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632399475 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399661 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632399647 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27589/

[GitHub] [spark] SparkQA commented on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator

2020-05-21 Thread GitBox
SparkQA commented on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632398947 **[Test build #122945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122945/testReport)** for PR 28553 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28553: [SPARK-31734][ML][PySpark] Add weight support in ClusteringEvaluator

2020-05-21 Thread GitBox
SparkQA removed a comment on pull request #28553: URL: https://github.com/apache/spark/pull/28553#issuecomment-632375761 **[Test build #122945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122945/testReport)** for PR 28553 at commit

[GitHub] [spark] maropu commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
maropu commented on a change in pull request #28560: URL: https://github.com/apache/spark/pull/28560#discussion_r428969502 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala ## @@ -48,7 +53,10 @@ object

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632398364 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27589/

[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632397839 Since there is a revert on master branch (https://github.com/apache/spark/commit/db5e5fce68f7e3a724eb4516908c4ba9786b0db3), I'll rebase once more to make it sure.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632395037 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
SparkQA commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632395021 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/27588/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins removed a comment on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632395035 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2

2020-05-21 Thread GitBox
AmplabJenkins commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632395035 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] viirya commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
viirya commented on a change in pull request #28560: URL: https://github.com/apache/spark/pull/28560#discussion_r428965331 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala ## @@ -341,6 +341,78 @@ class

[GitHub] [spark] viirya commented on a change in pull request #28560: [SPARK-27217][SQL] Nested column aliasing for more operators which can prune nested column

2020-05-21 Thread GitBox
viirya commented on a change in pull request #28560: URL: https://github.com/apache/spark/pull/28560#discussion_r428965232 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala ## @@ -48,7 +53,10 @@ object

<    1   2   3   4   5   6   >