[GitHub] [spark] AmplabJenkins commented on pull request #30883: [SPARK-33878][SQL][TESTS] Fix resolving of `spark_catalog` in v1 Hive catalog tests

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30883: URL: https://github.com/apache/spark/pull/30883#issuecomment-749397252 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37796/

[GitHub] [spark] SparkQA commented on pull request #30883: [SPARK-33878][SQL][TESTS] Fix resolving of `spark_catalog` in v1 Hive catalog tests

2020-12-21 Thread GitBox
SparkQA commented on pull request #30883: URL: https://github.com/apache/spark/pull/30883#issuecomment-749396300 **[Test build #133199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133199/testReport)** for PR 30883 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30212: [SPARK-33308][SQL] Refactor current grouping analytics

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30212: URL: https://github.com/apache/spark/pull/30212#issuecomment-748607367 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] maropu commented on a change in pull request #30212: [SPARK-33308][SQL] Refactor current grouping analytics

2020-12-21 Thread GitBox
maropu commented on a change in pull request #30212: URL: https://github.com/apache/spark/pull/30212#discussion_r547116200 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -850,29 +850,62 @@ class AstBuilder extends

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #29966: URL: https://github.com/apache/spark/pull/29966#issuecomment-745429482 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132831/

[GitHub] [spark] SparkQA commented on pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path

2020-12-21 Thread GitBox
SparkQA commented on pull request #29966: URL: https://github.com/apache/spark/pull/29966#issuecomment-749395209 **[Test build #133198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133198/testReport)** for PR 29966 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30863: [SPARK-33858][SQL][TESTS] Unify v1 and v2 ALTER TABLE .. RENAME PARTITION tests

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30863: URL: https://github.com/apache/spark/pull/30863#issuecomment-749099159 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133152/

[GitHub] [spark] MaxGekk commented on a change in pull request #30863: [SPARK-33858][SQL][TESTS] Unify v1 and v2 ALTER TABLE .. RENAME PARTITION tests

2020-12-21 Thread GitBox
MaxGekk commented on a change in pull request #30863: URL: https://github.com/apache/spark/pull/30863#discussion_r547114868 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableRenamePartitionSuite.scala ## @@ -0,0 +1,158 @@ +/* + *

[GitHub] [spark] MaxGekk opened a new pull request #30883: [SPARK-33878][SQL][TESTS] Fix resolving of `spark_catalog` in v1 Hive catalog tests

2020-12-21 Thread GitBox
MaxGekk opened a new pull request #30883: URL: https://github.com/apache/spark/pull/30883 ### What changes were proposed in this pull request? 1. Recognize `spark_catalog` as the default session catalog in the checks of `TestHiveQueryExecution`. 2. Move v2 and v1 in-memory catalog

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30877: URL: https://github.com/apache/spark/pull/30877#issuecomment-749393606 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37794/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30865: [WIP][SPARK-33861][SQL] Simplify conditional in predicate

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30865: URL: https://github.com/apache/spark/pull/30865#issuecomment-748820434 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133125/

[GitHub] [spark] AmplabJenkins commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30877: URL: https://github.com/apache/spark/pull/30877#issuecomment-749393606 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37794/

[GitHub] [spark] wangyum commented on pull request #30865: [WIP][SPARK-33861][SQL] Simplify conditional in predicate

2020-12-21 Thread GitBox
wangyum commented on pull request #30865: URL: https://github.com/apache/spark/pull/30865#issuecomment-749393504 It seems we need to add a new rule, this is because we can not add it to `ReplaceNullWithFalseInPredicate ` or `SimplifyConditionals`, example: `select if(null, true, false)`

[GitHub] [spark] SparkQA commented on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
SparkQA commented on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749393372 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37795/

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #29966: [SPARK-33084][CORE][SQL] Add jar support ivy path

2020-12-21 Thread GitBox
AngersZh commented on a change in pull request #29966: URL: https://github.com/apache/spark/pull/29966#discussion_r547110204 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala ## @@ -1219,6 +1219,22 @@ class HiveQuerySuite

[GitHub] [spark] maropu commented on pull request #29893: [SPARK-32976][SQL]Support column list in INSERT statement

2020-12-21 Thread GitBox
maropu commented on pull request #29893: URL: https://github.com/apache/spark/pull/29893#issuecomment-749381388 @yaooqinn kindly ping: I've filed jira so that we don't forget to do it. https://issues.apache.org/jira/browse/SPARK-33877 `branch-3.1` includes this commit, so I think its

[GitHub] [spark] LuciferYang commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method

2020-12-21 Thread GitBox
LuciferYang commented on pull request #30484: URL: https://github.com/apache/spark/pull/30484#issuecomment-749380105 > @LuciferYang I am very sorry but do you mind pointing out which commit added that codes and removed the usages? It would be much easier to review with that.

[GitHub] [spark] SparkQA removed a comment on pull request #30243: [SPARK-33335][SQL] Support `has_all` func

2020-12-21 Thread GitBox
SparkQA removed a comment on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-749303793 **[Test build #133190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133190/testReport)** for PR 30243 at commit

[GitHub] [spark] LuciferYang commented on pull request #30663: [SPARK-33700][SQL] Avoid file meta reading when enableFilterPushDown is true and filters is empty for Orc

2020-12-21 Thread GitBox
LuciferYang commented on pull request #30663: URL: https://github.com/apache/spark/pull/30663#issuecomment-749378837 thx @HyukjinKwon @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] LuciferYang commented on pull request #30663: [SPARK-33700][SQL] Avoid file meta reading when enableFilterPushDown is true and filters is empty for Orc

2020-12-21 Thread GitBox
LuciferYang commented on pull request #30663: URL: https://github.com/apache/spark/pull/30663#issuecomment-749378207 @HyukjinKwon @dongjoon-hyun It seems that it is not easy to prove this optimization through UT. I did the following test, taking DataSourceV1 as an example:

[GitHub] [spark] AmplabJenkins commented on pull request #30243: [SPARK-33335][SQL] Support `has_all` func

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-749377852 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133190/

[GitHub] [spark] weixiuli commented on a change in pull request #30716: [SPARK-33747][CORE] Avoid calling unregisterMapOutput when the map stage is being rerunning.

2020-12-21 Thread GitBox
weixiuli commented on a change in pull request #30716: URL: https://github.com/apache/spark/pull/30716#discussion_r546631296 ## File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ## @@ -2035,6 +2040,107 @@ class DAGSchedulerSuite extends

[GitHub] [spark] SparkQA commented on pull request #30243: [SPARK-33335][SQL] Support `has_all` func

2020-12-21 Thread GitBox
SparkQA commented on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-749377070 **[Test build #133190 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133190/testReport)** for PR 30243 at commit

[GitHub] [spark] imback82 commented on a change in pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
imback82 commented on a change in pull request #30881: URL: https://github.com/apache/spark/pull/30881#discussion_r547097927 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveAttribute.scala ## @@ -0,0 +1,35 @@ +/* + * Licensed to the

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #30243: [SPARK-33335][SQL] Support `has_all` func

2020-12-21 Thread GitBox
AngersZh commented on a change in pull request #30243: URL: https://github.com/apache/spark/pull/30243#discussion_r547096480 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ## @@ -3999,3 +3999,203 @@ case

[GitHub] [spark] SparkQA commented on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
SparkQA commented on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749374255 **[Test build #133197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133197/testReport)** for PR 30881 at commit

[GitHub] [spark] HyukjinKwon commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method

2020-12-21 Thread GitBox
HyukjinKwon commented on pull request #30484: URL: https://github.com/apache/spark/pull/30484#issuecomment-749372549 @LuciferYang I am very sorry but do you mind pointing out which commit added that codes and removed the usages? It would be much easier to review with that.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30882: [SPARK-33876][SQL] Add length-check for reading char/varchar from tables w/ a external location

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30882: URL: https://github.com/apache/spark/pull/30882#issuecomment-749370232 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37792/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #27019: URL: https://github.com/apache/spark/pull/27019#issuecomment-749370234 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133188/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30880: [MINOR][CORE] Remove unused variable CompressionCodec.DEFAULT_COMPRESSION_CODEC

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30880: URL: https://github.com/apache/spark/pull/30880#issuecomment-749320394 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749370231 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133191/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30443: [SPARK-33497][SQL] Override maxRows in some LogicalPlan

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30443: URL: https://github.com/apache/spark/pull/30443#issuecomment-749370233 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37793/

[GitHub] [spark] AmplabJenkins commented on pull request #30880: [MINOR][CORE] Remove unused variable CompressionCodec.DEFAULT_COMPRESSION_CODEC

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30880: URL: https://github.com/apache/spark/pull/30880#issuecomment-749370229 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133192/

[GitHub] [spark] AmplabJenkins commented on pull request #30443: [SPARK-33497][SQL] Override maxRows in some LogicalPlan

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30443: URL: https://github.com/apache/spark/pull/30443#issuecomment-749370233 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37793/

[GitHub] [spark] AmplabJenkins commented on pull request #30882: [SPARK-33876][SQL] Add length-check for reading char/varchar from tables w/ a external location

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30882: URL: https://github.com/apache/spark/pull/30882#issuecomment-749370232 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37792/

[GitHub] [spark] AmplabJenkins commented on pull request #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #27019: URL: https://github.com/apache/spark/pull/27019#issuecomment-749370234 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133188/

[GitHub] [spark] AmplabJenkins commented on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749370231 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133191/

[GitHub] [spark] viirya edited a comment on pull request #30812: [SPARK-33814][SS] Provide preferred locations for stateful operations without reported state store locations

2020-12-21 Thread GitBox
viirya edited a comment on pull request #30812: URL: https://github.com/apache/spark/pull/30812#issuecomment-749356702 > I see. This makes sense. But why do we need to avoid this? > What's the cost did you mean? The execution memory used by states? > It would be great if you can

[GitHub] [spark] ulysses-you commented on pull request #30864: [SPARK-33857][SQL] Unify random functions and make Uuid Shuffle support seed in SQL

2020-12-21 Thread GitBox
ulysses-you commented on pull request #30864: URL: https://github.com/apache/spark/pull/30864#issuecomment-749364673 thanks @maropu @dongjoon-hyun , will to narrow the goal and make this PR focus on one thing. This is an

[GitHub] [spark] viirya edited a comment on pull request #30812: [SPARK-33814][SS] Provide preferred locations for stateful operations without reported state store locations

2020-12-21 Thread GitBox
viirya edited a comment on pull request #30812: URL: https://github.com/apache/spark/pull/30812#issuecomment-749356702 > I see. This makes sense. But why do we need to avoid this? > What's the cost did you mean? The execution memory used by states? > It would be great if you can

[GitHub] [spark] ulysses-you commented on pull request #30868: [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

2020-12-21 Thread GitBox
ulysses-you commented on pull request #30868: URL: https://github.com/apache/spark/pull/30868#issuecomment-749363672 thanks for merging ! This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] HyukjinKwon commented on pull request #30868: [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

2020-12-21 Thread GitBox
HyukjinKwon commented on pull request #30868: URL: https://github.com/apache/spark/pull/30868#issuecomment-749362580 It has a conflict with branch-2.4 but I think we don't have to bother. This is an automated message from

[GitHub] [spark] HyukjinKwon edited a comment on pull request #30868: [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

2020-12-21 Thread GitBox
HyukjinKwon edited a comment on pull request #30868: URL: https://github.com/apache/spark/pull/30868#issuecomment-749361814 Merged to master, branch-3.1 and branch-3.0. This is an automated message from the Apache Git

[GitHub] [spark] HyukjinKwon commented on pull request #30868: [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

2020-12-21 Thread GitBox
HyukjinKwon commented on pull request #30868: URL: https://github.com/apache/spark/pull/30868#issuecomment-749361814 Merged to master, branch-3.1, branch-3.0 and branch-2.4. This is an automated message from the Apache Git

[GitHub] [spark] HyukjinKwon closed pull request #30868: [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

2020-12-21 Thread GitBox
HyukjinKwon closed pull request #30868: URL: https://github.com/apache/spark/pull/30868 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] HyukjinKwon commented on a change in pull request #30868: [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

2020-12-21 Thread GitBox
HyukjinKwon commented on a change in pull request #30868: URL: https://github.com/apache/spark/pull/30868#discussion_r547087403 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala ## @@ -457,7 +457,9 @@ object

[GitHub] [spark] ulysses-you commented on a change in pull request #30868: [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

2020-12-21 Thread GitBox
ulysses-you commented on a change in pull request #30868: URL: https://github.com/apache/spark/pull/30868#discussion_r547086735 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala ## @@ -457,7 +457,9 @@ object

[GitHub] [spark] viirya edited a comment on pull request #30812: [SPARK-33814][SS] Provide preferred locations for stateful operations without reported state store locations

2020-12-21 Thread GitBox
viirya edited a comment on pull request #30812: URL: https://github.com/apache/spark/pull/30812#issuecomment-749356702 > I see. This makes sense. But why do we need to avoid this? > What's the cost did you mean? The execution memory used by states? > It would be great if you can

[GitHub] [spark] SparkQA removed a comment on pull request #30880: [MINOR][CORE] Remove unused variable CompressionCodec.DEFAULT_COMPRESSION_CODEC

2020-12-21 Thread GitBox
SparkQA removed a comment on pull request #30880: URL: https://github.com/apache/spark/pull/30880#issuecomment-749318866 **[Test build #133192 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133192/testReport)** for PR 30880 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
SparkQA removed a comment on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749319402 **[Test build #133191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133191/testReport)** for PR 30881 at commit

[GitHub] [spark] SparkQA commented on pull request #30880: [MINOR][CORE] Remove unused variable CompressionCodec.DEFAULT_COMPRESSION_CODEC

2020-12-21 Thread GitBox
SparkQA commented on pull request #30880: URL: https://github.com/apache/spark/pull/30880#issuecomment-749358204 **[Test build #133192 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133192/testReport)** for PR 30880 at commit

[GitHub] [spark] HyukjinKwon commented on a change in pull request #30868: [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

2020-12-21 Thread GitBox
HyukjinKwon commented on a change in pull request #30868: URL: https://github.com/apache/spark/pull/30868#discussion_r547084623 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala ## @@ -457,7 +457,9 @@ object

[GitHub] [spark] SparkQA commented on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
SparkQA commented on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749358016 **[Test build #133191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133191/testReport)** for PR 30881 at commit

[GitHub] [spark] SparkQA commented on pull request #30882: [SPARK-33876][SQL] Add length-check for reading char/varchar from tables w/ a external location

2020-12-21 Thread GitBox
SparkQA commented on pull request #30882: URL: https://github.com/apache/spark/pull/30882#issuecomment-749357902 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37792/

[GitHub] [spark] HyukjinKwon commented on a change in pull request #30868: [SPARK-33860][SQL] Make CatalystTypeConverters.convertToCatalyst match special Array value

2020-12-21 Thread GitBox
HyukjinKwon commented on a change in pull request #30868: URL: https://github.com/apache/spark/pull/30868#discussion_r547084623 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala ## @@ -457,7 +457,9 @@ object

[GitHub] [spark] SparkQA removed a comment on pull request #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec

2020-12-21 Thread GitBox
SparkQA removed a comment on pull request #27019: URL: https://github.com/apache/spark/pull/27019#issuecomment-749287847 **[Test build #133188 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133188/testReport)** for PR 27019 at commit

[GitHub] [spark] SparkQA commented on pull request #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec

2020-12-21 Thread GitBox
SparkQA commented on pull request #27019: URL: https://github.com/apache/spark/pull/27019#issuecomment-749357196 **[Test build #133188 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133188/testReport)** for PR 27019 at commit

[GitHub] [spark] viirya commented on pull request #30812: [SPARK-33814][SS] Provide preferred locations for stateful operations without reported state store locations

2020-12-21 Thread GitBox
viirya commented on pull request #30812: URL: https://github.com/apache/spark/pull/30812#issuecomment-749356702 > I see. This makes sense. But why do we need to avoid this? > What's the cost did you mean? The execution memory used by states? > It would be great if you can explain your

[GitHub] [spark] SparkQA commented on pull request #30443: [SPARK-33497][SQL] Override maxRows in some LogicalPlan

2020-12-21 Thread GitBox
SparkQA commented on pull request #30443: URL: https://github.com/apache/spark/pull/30443#issuecomment-749356083 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37793/

[GitHub] [spark] SparkQA commented on pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API

2020-12-21 Thread GitBox
SparkQA commented on pull request #30877: URL: https://github.com/apache/spark/pull/30877#issuecomment-749352528 **[Test build #133196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133196/testReport)** for PR 30877 at commit

[GitHub] [spark] xkrogen commented on a change in pull request #30877: [SPARK-23862][SQL] Support Java enums from Scala Dataset API

2020-12-21 Thread GitBox
xkrogen commented on a change in pull request #30877: URL: https://github.com/apache/spark/pull/30877#discussion_r547079407 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala ## @@ -232,6 +232,11 @@ object ScalaReflection extends

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30851: [SPARK-33846][SQL] Include Comments for a nested schema in StructType.toDDL

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30851: URL: https://github.com/apache/spark/pull/30851#issuecomment-749347995 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749347996 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37789/

[GitHub] [spark] AmplabJenkins commented on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749347996 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37789/

[GitHub] [spark] AmplabJenkins commented on pull request #30851: [SPARK-33846][SQL] Include Comments for a nested schema in StructType.toDDL

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30851: URL: https://github.com/apache/spark/pull/30851#issuecomment-749347995 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30443: [SPARK-33497][SQL] Override maxRows in some LogicalPlan

2020-12-21 Thread GitBox
SparkQA commented on pull request #30443: URL: https://github.com/apache/spark/pull/30443#issuecomment-749346069 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37793/

[GitHub] [spark] SparkQA commented on pull request #30882: [SPARK-33876][SQL] Add length-check for reading char/varchar from tables w/ a external location

2020-12-21 Thread GitBox
SparkQA commented on pull request #30882: URL: https://github.com/apache/spark/pull/30882#issuecomment-749345887 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37792/

[GitHub] [spark] dongjoon-hyun commented on pull request #30472: [SPARK-32221][k8s] Avoid possible errors due to incorrect file size or type supplied in spark conf.

2020-12-21 Thread GitBox
dongjoon-hyun commented on pull request #30472: URL: https://github.com/apache/spark/pull/30472#issuecomment-749345172 Please let me know if this is ready back, @ScrapCodes . This is an automated message from the Apache Git

[GitHub] [spark] weixiuli commented on pull request #30716: [SPARK-33747][CORE] Avoid calling unregisterMapOutput when the map stage is being rerunning.

2020-12-21 Thread GitBox
weixiuli commented on pull request #30716: URL: https://github.com/apache/spark/pull/30716#issuecomment-749343263 @Ngone51 @mridulm @jiangxb1987 @dongjoon-hyun PTAL. This is an automated message from the Apache Git Service.

[GitHub] [spark] zsxwing commented on pull request #30812: [SPARK-33814][SS] Provide preferred locations for stateful operations without reported state store locations

2020-12-21 Thread GitBox
zsxwing commented on pull request #30812: URL: https://github.com/apache/spark/pull/30812#issuecomment-749343031 > When the first batch takes payload from latest offsets, this batch possibly finishes very quick. An executor might be assigned more than one task because the executor

[GitHub] [spark] dungdm93 commented on pull request #30738: [SPARK-33759][K8S] docker entrypoint should using `spark-class` for spark executor

2020-12-21 Thread GitBox
dungdm93 commented on pull request #30738: URL: https://github.com/apache/spark/pull/30738#issuecomment-749341877 @dongjoon-hyun Yes, It's OK. So feel free to close this MR if it is not suitable for you. This is an

[GitHub] [spark] SparkQA commented on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
SparkQA commented on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749341419 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37789/

[GitHub] [spark] SparkQA commented on pull request #30851: [SPARK-33846][SQL] Include Comments for a nested schema in StructType.toDDL

2020-12-21 Thread GitBox
SparkQA commented on pull request #30851: URL: https://github.com/apache/spark/pull/30851#issuecomment-749340702 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37791/

[GitHub] [spark] wangyum commented on a change in pull request #30852: [SPARK-33847][SQL] Replace None of elseValue inside CaseWhen if all branches are FalseLiteral

2020-12-21 Thread GitBox
wangyum commented on a change in pull request #30852: URL: https://github.com/apache/spark/pull/30852#discussion_r547069637 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicate.scala ## @@ -94,6 +94,7 @@ object

[GitHub] [spark] SparkQA removed a comment on pull request #30851: [SPARK-33846][SQL] Include Comments for a nested schema in StructType.toDDL

2020-12-21 Thread GitBox
SparkQA removed a comment on pull request #30851: URL: https://github.com/apache/spark/pull/30851#issuecomment-749321404 **[Test build #133193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133193/testReport)** for PR 30851 at commit

[GitHub] [spark] SparkQA commented on pull request #30851: [SPARK-33846][SQL] Include Comments for a nested schema in StructType.toDDL

2020-12-21 Thread GitBox
SparkQA commented on pull request #30851: URL: https://github.com/apache/spark/pull/30851#issuecomment-749338745 **[Test build #133193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133193/testReport)** for PR 30851 at commit

[GitHub] [spark] wangyum commented on a change in pull request #30852: [SPARK-33847][SQL] Replace None of elseValue inside CaseWhen if all branches are FalseLiteral

2020-12-21 Thread GitBox
wangyum commented on a change in pull request #30852: URL: https://github.com/apache/spark/pull/30852#discussion_r547000365 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicate.scala ## @@ -94,6 +94,7 @@ object

[GitHub] [spark] wangyum commented on a change in pull request #30852: [SPARK-33847][SQL] Replace None of elseValue inside CaseWhen if all branches are FalseLiteral

2020-12-21 Thread GitBox
wangyum commented on a change in pull request #30852: URL: https://github.com/apache/spark/pull/30852#discussion_r546997382 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicate.scala ## @@ -94,6 +94,7 @@ object

[GitHub] [spark] dongjoon-hyun commented on pull request #30738: [SPARK-33759][K8S] docker entrypoint should using `spark-class` for spark executor

2020-12-21 Thread GitBox
dongjoon-hyun commented on pull request #30738: URL: https://github.com/apache/spark/pull/30738#issuecomment-749337595 Hi, @dungdm93 . Apache Spark distributions provide docker files and build scripts instead of docker image. It seems that you can do the following to achieve your use

[GitHub] [spark] SparkQA commented on pull request #30881: [SPARK-33875][SQL] Implement DESCRIBE COLUMN for v2 tables

2020-12-21 Thread GitBox
SparkQA commented on pull request #30881: URL: https://github.com/apache/spark/pull/30881#issuecomment-749333100 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37789/

[GitHub] [spark] SparkQA commented on pull request #30443: [SPARK-33497][SQL] Override maxRows in some LogicalPlan

2020-12-21 Thread GitBox
SparkQA commented on pull request #30443: URL: https://github.com/apache/spark/pull/30443#issuecomment-749332723 **[Test build #133195 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133195/testReport)** for PR 30443 at commit

[GitHub] [spark] SparkQA commented on pull request #30851: [SPARK-33846][SQL] Include Comments for a nested schema in StructType.toDDL

2020-12-21 Thread GitBox
SparkQA commented on pull request #30851: URL: https://github.com/apache/spark/pull/30851#issuecomment-749332132 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37791/

[GitHub] [spark] SparkQA commented on pull request #30882: [SPARK-33876][SQL] Add length-check for reading char/varchar from tables w/ a external location

2020-12-21 Thread GitBox
SparkQA commented on pull request #30882: URL: https://github.com/apache/spark/pull/30882#issuecomment-749332139 **[Test build #133194 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133194/testReport)** for PR 30882 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30243: [SPARK-33335][SQL] Support `has_all` func

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-749306881 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30443: [SPARK-33497][SQL] Override maxRows in some LogicalPlan

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30443: URL: https://github.com/apache/spark/pull/30443#issuecomment-749331841 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133185/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30812: [SPARK-33814][SS] Provide preferred locations for stateful operations without reported state store locations

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30812: URL: https://github.com/apache/spark/pull/30812#issuecomment-749331843 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133184/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30663: [SPARK-33700][SQL] Avoid file meta reading when enableFilterPushDown is true and filters is empty for Orc

2020-12-21 Thread GitBox
AmplabJenkins removed a comment on pull request #30663: URL: https://github.com/apache/spark/pull/30663#issuecomment-749331838 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30812: [SPARK-33814][SS] Provide preferred locations for stateful operations without reported state store locations

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30812: URL: https://github.com/apache/spark/pull/30812#issuecomment-749331843 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133184/

[GitHub] [spark] AmplabJenkins commented on pull request #30243: [SPARK-33335][SQL] Support `has_all` func

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-749331840 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37788/

[GitHub] [spark] AmplabJenkins commented on pull request #30663: [SPARK-33700][SQL] Avoid file meta reading when enableFilterPushDown is true and filters is empty for Orc

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30663: URL: https://github.com/apache/spark/pull/30663#issuecomment-749331838 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #30443: [SPARK-33497][SQL] Override maxRows in some LogicalPlan

2020-12-21 Thread GitBox
AmplabJenkins commented on pull request #30443: URL: https://github.com/apache/spark/pull/30443#issuecomment-749331841 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133185/

[GitHub] [spark] dongjoon-hyun commented on pull request #30864: [SPARK-33857][SQL] Unify random functions and make Uuid Shuffle support seed in SQL

2020-12-21 Thread GitBox
dongjoon-hyun commented on pull request #30864: URL: https://github.com/apache/spark/pull/30864#issuecomment-749331667 Oh, I commented before reading @maropu 's last comment. Ya. This looks like two orthogonal purposes.

[GitHub] [spark] yaooqinn commented on pull request #30882: [SPARK-33876][SQL] Add length-check for reading char/varchar from tables w/ a external location

2020-12-21 Thread GitBox
yaooqinn commented on pull request #30882: URL: https://github.com/apache/spark/pull/30882#issuecomment-749330997 cc @cloud-fan @maropu @HyukjinKwon thanks for checking this This is an automated message from the Apache Git

[GitHub] [spark] dongjoon-hyun closed pull request #30663: [SPARK-33700][SQL] Avoid file meta reading when enableFilterPushDown is true and filters is empty for Orc

2020-12-21 Thread GitBox
dongjoon-hyun closed pull request #30663: URL: https://github.com/apache/spark/pull/30663 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] yaooqinn opened a new pull request #30882: [SPARK-33876][SQL] Add length-check for reading char/varchar from tables w/ a external location

2020-12-21 Thread GitBox
yaooqinn opened a new pull request #30882: URL: https://github.com/apache/spark/pull/30882 ### What changes were proposed in this pull request? ### Why are the changes needed? ```sql spark-sql> INSERT INTO t2 VALUES ('1', 'b12345'); Time taken: 0.141 seconds

[GitHub] [spark] ulysses-you commented on pull request #30443: [SPARK-33497][SQL] Override maxRows in some LogicalPlan

2020-12-21 Thread GitBox
ulysses-you commented on pull request #30443: URL: https://github.com/apache/spark/pull/30443#issuecomment-749327611 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun commented on pull request #30876: [SPARK-33870][CORE] Enable spark.storage.replication.proactive by default

2020-12-21 Thread GitBox
dongjoon-hyun commented on pull request #30876: URL: https://github.com/apache/spark/pull/30876#issuecomment-749327332 Also, cc @mridulm . This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun commented on pull request #29414: [SPARK-32106][SQL] Implement script transform in sql/core

2020-12-21 Thread GitBox
dongjoon-hyun commented on pull request #29414: URL: https://github.com/apache/spark/pull/29414#issuecomment-749327113 Thank you for informing, @maropu ! This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] SparkQA removed a comment on pull request #30663: [SPARK-33700][SQL] Avoid file meta reading when enableFilterPushDown is true and filters is empty for Orc

2020-12-21 Thread GitBox
SparkQA removed a comment on pull request #30663: URL: https://github.com/apache/spark/pull/30663#issuecomment-749304279 **[Test build #133189 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133189/testReport)** for PR 30663 at commit

[GitHub] [spark] SparkQA commented on pull request #30663: [SPARK-33700][SQL] Avoid file meta reading when enableFilterPushDown is true and filters is empty for Orc

2020-12-21 Thread GitBox
SparkQA commented on pull request #30663: URL: https://github.com/apache/spark/pull/30663#issuecomment-749325825 **[Test build #133189 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133189/testReport)** for PR 30663 at commit

  1   2   3   4   5   6   7   8   >