[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662426192 **[Test build #126313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126313/testReport)** for PR 29085 at commit [`670f21b`](https://github.co

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662421917 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662422087 **[Test build #126333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126333/testReport)** for PR 29085 at commit [`7916d72`](https://github.com

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29145: [SPARK-32346][SQL] Support filters pushdown in Avro datasource

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #29145: URL: https://github.com/apache/spark/pull/29145#discussion_r458750581 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/OrderedFilters.scala ## @@ -94,7 +91,7 @@ class CSVFilters(filters: Seq[sourc

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662421911 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662342533 **[Test build #126324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126324/testReport)** for PR 29160 at commit [`16eaf45`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662421911 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458750565 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -20,78 +20,44 @@ package org.apache.sp

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662421393 **[Test build #126324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126324/testReport)** for PR 29160 at commit [`16eaf45`](https://github.co

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28977: [SPARK-32389][TESTS] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
dongjoon-hyun edited a comment on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662397644 Ya. I understand what you mean by `org.apache.spark.sql.hive.execution suites are now whitelisted`. However, it will enforce all new test suites under that namespac

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458745259 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala ## @@ -87,17 +181,72 @@ trait BaseScriptTransforma

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458742052 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -745,7 +745,7 @@ class AstBuilder(conf: SQLConf) ext

[GitHub] [spark] constzhou commented on pull request #29094: [SPARK-24983][SQL] limit number of leaf expressions in a single project when collapse project to prevent driver oom

2020-07-22 Thread GitBox
constzhou commented on pull request #29094: URL: https://github.com/apache/spark/pull/29094#issuecomment-662414958 code changed. Please feel free to comment. Thanks! @maropu This is an automated message from the Apache Git S

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458741224 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ## @@ -679,64 +680,78 @@ class SparkSqlAstBuilder(conf: SQLConf)

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458735510 ## File path: sql/core/src/test/resources/sql-tests/inputs/transform.sql ## @@ -0,0 +1,89 @@ +-- Test data. +CREATE OR REPLACE TEMPORARY VIEW t AS SELECT *

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #28977: URL: https://github.com/apache/spark/pull/28977#discussion_r458734446 ## File path: project/SparkBuild.scala ## @@ -486,9 +485,11 @@ object SparkParallelTestGrouping { ) private val DEFAULT_TEST_GROUP = "default_

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458731441 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class HiveScriptTra

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #28977: URL: https://github.com/apache/spark/pull/28977#discussion_r458730964 ## File path: project/SparkBuild.scala ## @@ -486,9 +485,11 @@ object SparkParallelTestGrouping { ) private val DEFAULT_TEST_GROUP = "default_

[GitHub] [spark] AmplabJenkins commented on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662405396 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662405522 **[Test build #126332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126332/testReport)** for PR 29160 at commit [`7ca5da6`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662405396 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662324250 **[Test build #126319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126319/testReport)** for PR 28977 at commit [`21b9292`](https://gi

[GitHub] [spark] SparkQA commented on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
SparkQA commented on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662404795 **[Test build #126319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126319/testReport)** for PR 28977 at commit [`21b9292`](https://github.co

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
dongjoon-hyun edited a comment on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662397644 Ya. I understand what you mean by `org.apache.spark.sql.hive.execution suites are now whitelisted`. However, it will enforce all new test suites under that namespac

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
dongjoon-hyun edited a comment on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662397644 Ya. I understand what you mean by `org.apache.spark.sql.hive.execution suites are now whitelisted`. However, it will enforce all new test suites under that namespac

[GitHub] [spark] dongjoon-hyun commented on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
dongjoon-hyun commented on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662397644 Ya. I understand what you mean by `org.apache.spark.sql.hive.execution suites are now whitelisted`. So, it will enforce all new test suites under that namespace and there

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
dongjoon-hyun edited a comment on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662397644 Ya. I understand what you mean by `org.apache.spark.sql.hive.execution suites are now whitelisted`. However, it will enforce all new test suites under that namespac

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662397408 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662396395 Thank you, @HyukjinKwon . This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [spark] Fokko commented on a change in pull request #29110: [SPARK-32313][PYSPARK] Remove Python 2 artifacts

2020-07-22 Thread GitBox
Fokko commented on a change in pull request #29110: URL: https://github.com/apache/spark/pull/29110#discussion_r458719481 ## File path: python/pyspark/cloudpickle.py ## @@ -1010,11 +1006,7 @@ def save_ellipsis(self, obj): def save_not_implemented(self, obj): self.

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662397408 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] Fokko closed pull request #29110: [SPARK-32313][PYSPARK] Remove Python 2 artifacts

2020-07-22 Thread GitBox
Fokko closed pull request #29110: URL: https://github.com/apache/spark/pull/29110 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [spark] Fokko commented on a change in pull request #29110: [SPARK-32313][PYSPARK] Remove Python 2 artifacts

2020-07-22 Thread GitBox
Fokko commented on a change in pull request #29110: URL: https://github.com/apache/spark/pull/29110#discussion_r458719410 ## File path: python/pyspark/_globals.py ## @@ -59,10 +59,6 @@ def __new__(cls): cls.__instance = super(_NoValueType, cls).__new__(cls)

[GitHub] [spark] dongjoon-hyun commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
dongjoon-hyun commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662394998 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662394309 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662394393 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662394393 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662394309 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662393017 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662393010 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA removed a comment on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662392331 **[Test build #126330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126330/testReport)** for PR 29188 at commit [`c448b23`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
SparkQA commented on pull request #29166: URL: https://github.com/apache/spark/pull/29166#issuecomment-662393165 **[Test build #126331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126331/testReport)** for PR 29166 at commit [`cc4ee59`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
SparkQA commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662392996 **[Test build #126330 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126330/testReport)** for PR 29188 at commit [`c448b23`](https://github.co

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662367871 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662393010 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
SparkQA commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662392331 **[Test build #126330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126330/testReport)** for PR 29188 at commit [`c448b23`](https://github.com

[GitHub] [spark] HyukjinKwon commented on pull request #28977: [SPARK-32389][Tests] Add all hive.execution suite in the parallel test group

2020-07-22 Thread GitBox
HyukjinKwon commented on pull request #28977: URL: https://github.com/apache/spark/pull/28977#issuecomment-662392321 @dongjoon-hyun, do you mean we should update: https://github.com/apache/spark/blob/026b0b926dfd40038f2cee932f38b917eb25b77e/project/SparkBuild.scala#L445-L461

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662392114 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662392114 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662286321 **[Test build #126312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126312/testReport)** for PR 29104 at commit [`e090286`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

2020-07-22 Thread GitBox
SparkQA commented on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-662391482 **[Test build #126312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126312/testReport)** for PR 29104 at commit [`e090286`](https://github.co

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458712933 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -754,7 +755,80 @@ class AstBuilder(conf: SQLCo

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662387970 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662387962 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662388154 **[Test build #126329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126329/testReport)** for PR 29085 at commit [`08d97c8`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662387962 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662384929 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662384929 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662383656 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662383645 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
SparkQA commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662383880 **[Test build #126328 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126328/testReport)** for PR 29187 at commit [`31c73b9`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662383645 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] Fokko commented on pull request #29180: [SPARK-17333][PYSPARK] Enable mypy on the repository

2020-07-22 Thread GitBox
Fokko commented on pull request #29180: URL: https://github.com/apache/spark/pull/29180#issuecomment-662383147 @zero323 The annotations aren't random at all. This PR fixes all the outstanding violations on the current master: ``` fokkodriesprong@Fan python % mypy --ignore-missing-impo

[GitHub] [spark] gaborgsomogyi commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
gaborgsomogyi commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662383093 cc @HeartSaVioR @zsxwing @xuanyuanking This is an automated message from the Apache Git Service. To respo

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458697351 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,343 @@ +/* + * Licensed to th

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458697351 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,343 @@ +/* + * Licensed to th

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458696650 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,343 @@ +/* + * Licensed to th

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29166: URL: https://github.com/apache/spark/pull/29166#issuecomment-662374731 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29166: URL: https://github.com/apache/spark/pull/29166#issuecomment-662374731 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458692902 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -754,7 +755,80 @@ class AstBuilder(conf: SQLConf) ex

[GitHub] [spark] Ngone51 commented on a change in pull request #29166: [SPARK-32372][SQL] ResolveReferences.dedupRight should only rewrite attributes for ancestor nodes of the conflict plan

2020-07-22 Thread GitBox
Ngone51 commented on a change in pull request #29166: URL: https://github.com/apache/spark/pull/29166#discussion_r458691890 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1237,20 +1250,79 @@ class Analyzer( if (con

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458687735 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala ## @@ -0,0 +1,343 @@ +/* + * Licensed to the Apac

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458691259 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite +

[GitHub] [spark] HyukjinKwon commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662372392 @BryanCutler, @huaxingao, @ueshin, @viirya, @srowen, @dongjoon-hyun, @WeichenXu123, @zhengruifeng, @holdenk, @zero323, can you guys take a look when you are available? ---

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458689199 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite +

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #29188: URL: https://github.com/apache/spark/pull/29188#discussion_r458688852 ## File path: python/docs/source/conf.py ## @@ -14,12 +14,23 @@ import sys import os +import shutil # If extensions (or modules to document wit

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458688383 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationSuite.scala ## @@ -206,75 +169,83 @@ class HiveScr

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #29188: URL: https://github.com/apache/spark/pull/29188#discussion_r458688468 ## File path: python/docs/source/_templates/class_with_docs.rst ## @@ -0,0 +1,79 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #29188: URL: https://github.com/apache/spark/pull/29188#discussion_r458688468 ## File path: python/docs/source/_templates/class_with_docs.rst ## @@ -0,0 +1,79 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662368327 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon commented on a change in pull request #29188: URL: https://github.com/apache/spark/pull/29188#discussion_r458686040 ## File path: .gitignore ## @@ -64,6 +64,7 @@ python/lib/pyspark.zip python/.eggs/ python/deps python/docs/_site/ +python/docs/source/reference/api

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662368318 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] SparkQA commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662368200 **[Test build #126317 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126317/testReport)** for PR 29160 at commit [`4019789`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662316877 **[Test build #126317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126317/testReport)** for PR 29160 at commit [`4019789`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662357223 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29160: [SPARK-32364][SQL] Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29160: URL: https://github.com/apache/spark/pull/29160#issuecomment-662368318 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
SparkQA commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662368464 **[Test build #126327 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126327/testReport)** for PR 29187 at commit [`9d214c9`](https://github.com

[GitHub] [spark] AmplabJenkins commented on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662367871 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] HyukjinKwon opened a new pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base

2020-07-22 Thread GitBox
HyukjinKwon opened a new pull request #29188: URL: https://github.com/apache/spark/pull/29188 ### What changes were proposed in this pull request? This PR proposes to redesign the PySpark documentation. I made a demo site to make it easier to review: https://hyukjin-spark.read

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AngersZh commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458681551 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala ## @@ -87,17 +178,69 @@ trait BaseScriptTran

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458681628 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite +-- Num

[GitHub] [spark] AmplabJenkins commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662364371 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662364371 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-662364401 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
AmplabJenkins removed a comment on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662362036 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
AmplabJenkins commented on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662362036 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
SparkQA commented on pull request #29085: URL: https://github.com/apache/spark/pull/29085#issuecomment-662362247 **[Test build #126326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126326/testReport)** for PR 29085 at commit [`4615733`](https://github.com

[GitHub] [spark] SparkQA removed a comment on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
SparkQA removed a comment on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662289378 **[Test build #126314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126314/testReport)** for PR 29185 at commit [`cb1148c`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29185: [SPARK-32384][CORE] repartitionAndSortWithinPartitions avoid shuffle with same partitioner

2020-07-22 Thread GitBox
SparkQA commented on pull request #29185: URL: https://github.com/apache/spark/pull/29185#issuecomment-662361283 **[Test build #126314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126314/testReport)** for PR 29185 at commit [`cb1148c`](https://github.co

[GitHub] [spark] maropu commented on a change in pull request #29085: [SPARK-32106][SQL]Implement SparkScriptTransformationExec in sql/core

2020-07-22 Thread GitBox
maropu commented on a change in pull request #29085: URL: https://github.com/apache/spark/pull/29085#discussion_r458676081 ## File path: sql/core/src/test/resources/sql-tests/results/transform.sql.out ## @@ -0,0 +1,160 @@ +-- Automatically generated by SQLQueryTestSuite +-- Num

<    3   4   5   6   7   8   9   10   11   >