[GitHub] [spark] cloud-fan commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

2021-02-02 Thread GitBox
cloud-fan commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r568391286 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Found

[GitHub] [spark] SparkQA commented on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

2021-02-02 Thread GitBox
SparkQA commented on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-771447464 **[Test build #134752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134752/testReport)** for PR 31316 at commit [`a6ba6d4`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

2021-02-02 Thread GitBox
SparkQA removed a comment on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-771321877 **[Test build #134752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134752/testReport)** for PR 31316 at commit [`a6ba6d4`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

2021-02-02 Thread GitBox
SparkQA commented on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771448222 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39351/ -

[GitHub] [spark] SparkQA commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

2021-02-02 Thread GitBox
SparkQA commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771450192 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39347/ -

[GitHub] [spark] SparkQA commented on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

2021-02-02 Thread GitBox
SparkQA commented on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771451301 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39350/ -

[GitHub] [spark] HeartSaVioR opened a new pull request #31435: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

2021-02-02 Thread GitBox
HeartSaVioR opened a new pull request #31435: URL: https://github.com/apache/spark/pull/31435 ### What changes were proposed in this pull request? This PR proposes to fix the UTs being added in SPARK-31793, so that all things contributing the length limit are properly accounted.

[GitHub] [spark] SparkQA commented on pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

2021-02-02 Thread GitBox
SparkQA commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771453019 **[Test build #134768 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134768/testReport)** for PR 31434 at commit [`43f7c21`](https://github.com

[GitHub] [spark] HeartSaVioR commented on a change in pull request #31435: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

2021-02-02 Thread GitBox
HeartSaVioR commented on a change in pull request #31435: URL: https://github.com/apache/spark/pull/31435#discussion_r568399650 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/DataSourceScanExecRedactionSuite.scala ## @@ -137,9 +137,24 @@ class DataSourceS

[GitHub] [spark] HeartSaVioR commented on pull request #31435: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

2021-02-02 Thread GitBox
HeartSaVioR commented on pull request #31435: URL: https://github.com/apache/spark/pull/31435#issuecomment-771453884 cc.ing @gengliangwang @cloud-fan @HyukjinKwon @maropu who are author/reviewers of #28610 This is an automat

[GitHub] [spark] HeartSaVioR commented on a change in pull request #31435: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

2021-02-02 Thread GitBox
HeartSaVioR commented on a change in pull request #31435: URL: https://github.com/apache/spark/pull/31435#discussion_r568399650 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/DataSourceScanExecRedactionSuite.scala ## @@ -137,9 +137,24 @@ class DataSourceS

[GitHub] [spark] SparkQA commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread GitBox
SparkQA commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771454099 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39348/ -

[GitHub] [spark] SparkQA commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

2021-02-02 Thread GitBox
SparkQA commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771458360 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39349/ ---

[GitHub] [spark] cloud-fan commented on pull request #31424: [SPARK-34317][SQL] Introduce relationTypeMismatchHint to UnresolvedTable for a better error message

2021-02-02 Thread GitBox
cloud-fan commented on pull request #31424: URL: https://github.com/apache/spark/pull/31424#issuecomment-771458661 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] gengliangwang commented on pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

2021-02-02 Thread GitBox
gengliangwang commented on pull request #31421: URL: https://github.com/apache/spark/pull/31421#issuecomment-771458714 @cloud-fan Thanks for the comments. I have addressed them in https://github.com/apache/spark/pull/31434 T

[GitHub] [spark] cloud-fan closed pull request #31424: [SPARK-34317][SQL] Introduce relationTypeMismatchHint to UnresolvedTable for a better error message

2021-02-02 Thread GitBox
cloud-fan closed pull request #31424: URL: https://github.com/apache/spark/pull/31424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] gengliangwang commented on pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

2021-02-02 Thread GitBox
gengliangwang commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771458928 @cloud-fan yes, done. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] cloud-fan commented on pull request #31420: [SPARK-34312][SQL] Support partition(s) truncation by `Supports(Atomic)PartitionManagement`

2021-02-02 Thread GitBox
cloud-fan commented on pull request #31420: URL: https://github.com/apache/spark/pull/31420#issuecomment-771459608 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] cloud-fan closed pull request #31420: [SPARK-34312][SQL] Support partition(s) truncation by `Supports(Atomic)PartitionManagement`

2021-02-02 Thread GitBox
cloud-fan closed pull request #31420: URL: https://github.com/apache/spark/pull/31420 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] SparkQA commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

2021-02-02 Thread GitBox
SparkQA commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771459889 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39346/ ---

[GitHub] [spark] SparkQA commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

2021-02-02 Thread GitBox
SparkQA commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771460537 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39352/ ---

[GitHub] [spark] SparkQA commented on pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

2021-02-02 Thread GitBox
SparkQA commented on pull request #31421: URL: https://github.com/apache/spark/pull/31421#issuecomment-771463158 **[Test build #134756 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134756/testReport)** for PR 31421 at commit [`971ccbd`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

2021-02-02 Thread GitBox
SparkQA commented on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771462980 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39351/ ---

[GitHub] [spark] SparkQA removed a comment on pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

2021-02-02 Thread GitBox
SparkQA removed a comment on pull request #31421: URL: https://github.com/apache/spark/pull/31421#issuecomment-771341045 **[Test build #134756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134756/testReport)** for PR 31421 at commit [`971ccbd`](https://gi

[GitHub] [spark] cloud-fan commented on pull request #31398: [SPARK-34297][SQL][SS] Add metrics for data loss and offset out range for KafkaMicroBatchStream

2021-02-02 Thread GitBox
cloud-fan commented on pull request #31398: URL: https://github.com/apache/spark/pull/31398#issuecomment-771463490 The feature makes a lot of sense, but to me building the infra to report data source metrics is the most important thing here. @viirya can you add more details about the DS v2

[GitHub] [spark] ScrapCodes opened a new pull request #31436: [SPARK-34327][BUILD] Strip passwords from inlining into build information while releasing.

2021-02-02 Thread GitBox
ScrapCodes opened a new pull request #31436: URL: https://github.com/apache/spark/pull/31436 ### What changes were proposed in this pull request? Strip passwords from getting inlined into build information, inadvertently. ### Why are the changes needed? This can be a

[GitHub] [spark] SparkQA commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread GitBox
SparkQA commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771464376 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39348/ ---

[GitHub] [spark] wangyum commented on pull request #31393: [SPARK-34289][SQL] Parquet vectorized reader support column index

2021-02-02 Thread GitBox
wangyum commented on pull request #31393: URL: https://github.com/apache/spark/pull/31393#issuecomment-771464400 Benchmark with production data: ```sql CREATE TABLE test11.benchmark_column_index_2 using parquet CLUSTERED BY (FDBK_RCVR_USER_ID) SORTED BY (FDBK_GIVER_USER_ID) into 200

[GitHub] [spark] SparkQA commented on pull request #31422: [SPARK-34313][SQL] Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to use UnresolvedTable to resolve the identifier

2021-02-02 Thread GitBox
SparkQA commented on pull request #31422: URL: https://github.com/apache/spark/pull/31422#issuecomment-771464438 **[Test build #134755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134755/testReport)** for PR 31422 at commit [`e0ccd1a`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31422: [SPARK-34313][SQL] Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to use UnresolvedTable to resolve the identifier

2021-02-02 Thread GitBox
SparkQA removed a comment on pull request #31422: URL: https://github.com/apache/spark/pull/31422#issuecomment-771341008 **[Test build #134755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134755/testReport)** for PR 31422 at commit [`e0ccd1a`](https://gi

[GitHub] [spark] ScrapCodes commented on pull request #31436: [SPARK-34327][BUILD] Strip passwords from inlining into build information while releasing.

2021-02-02 Thread GitBox
ScrapCodes commented on pull request #31436: URL: https://github.com/apache/spark/pull/31436#issuecomment-771464936 Hi @srowen, Please take a look. Our build process uses the script `build/spark-build-info` to inline the build information into jars. This should fix it. I have

[GitHub] [spark] cloud-fan commented on a change in pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

2021-02-02 Thread GitBox
cloud-fan commented on a change in pull request #31434: URL: https://github.com/apache/spark/pull/31434#discussion_r568413503 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowPartitionsSuiteBase.scala ## @@ -161,7 +161,7 @@ trait ShowPartitionsS

[GitHub] [spark] cloud-fan commented on a change in pull request #31422: [SPARK-34313][SQL] Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to use UnresolvedTable to resolve the identifier

2021-02-02 Thread GitBox
cloud-fan commented on a change in pull request #31422: URL: https://github.com/apache/spark/pull/31422#discussion_r568414692 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -3493,14 +3493,16 @@ class AstBuilder extends Sql

[GitHub] [spark] dongjoon-hyun closed pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

2021-02-02 Thread GitBox
dongjoon-hyun closed pull request #31430: URL: https://github.com/apache/spark/pull/31430 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

2021-02-02 Thread GitBox
SparkQA commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771467460 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39347/ ---

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

2021-02-02 Thread GitBox
AngersZh commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r568415995 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ## @@ -129,7 +129,7 @@ class HiveCommandSuite exten

[GitHub] [spark] cloud-fan commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

2021-02-02 Thread GitBox
cloud-fan commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771469218 > Does this PR introduce any user-facing change? > No Can we have more description there? This definitely has user-facing change, right? More queries are allowed than

[GitHub] [spark] attilapiros commented on pull request #31133: [SPARK-26836][SQL] Supporting Avro schema evolution for partitioned Hive tables

2021-02-02 Thread GitBox
attilapiros commented on pull request #31133: URL: https://github.com/apache/spark/pull/31133#issuecomment-771469227 @dongjoon-hyun may I ask for another review from you? This is an automated message from the Apache Git Servi

[GitHub] [spark] dongjoon-hyun commented on pull request #31133: [SPARK-26836][SQL] Supporting Avro schema evolution for partitioned Hive tables

2021-02-02 Thread GitBox
dongjoon-hyun commented on pull request #31133: URL: https://github.com/apache/spark/pull/31133#issuecomment-771470866 Sure, I'll review right now, @attilapiros . This is an automated message from the Apache Git Service. To r

[GitHub] [spark] zhengruifeng commented on a change in pull request #31394: [SPARK-34291][ML] LSH hashDistance optimization

2021-02-02 Thread GitBox
zhengruifeng commented on a change in pull request #31394: URL: https://github.com/apache/spark/pull/31394#discussion_r568419326 ## File path: mllib/src/main/scala/org/apache/spark/ml/feature/BucketedRandomProjectionLSH.scala ## @@ -97,7 +97,19 @@ class BucketedRandomProjectio

[GitHub] [spark] d80tb7 commented on pull request #31399: [SPARK-34259][SQL] Don't attempt to parse file-based partitions as special timestamps

2021-02-02 Thread GitBox
d80tb7 commented on pull request #31399: URL: https://github.com/apache/spark/pull/31399#issuecomment-771471526 Hi @MaxGekk The other PR looks good to me- but just wondering where we are with this one. It looks like @cloud-fan agrees we should disallow special values being eligibl

[GitHub] [spark] SparkQA commented on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

2021-02-02 Thread GitBox
SparkQA commented on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771471941 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39350/ ---

[GitHub] [spark] zhengruifeng commented on a change in pull request #31394: [SPARK-34291][ML] LSH hashDistance optimization

2021-02-02 Thread GitBox
zhengruifeng commented on a change in pull request #31394: URL: https://github.com/apache/spark/pull/31394#discussion_r568421067 ## File path: mllib/src/main/scala/org/apache/spark/ml/feature/MinHashLSH.scala ## @@ -109,9 +109,16 @@ class MinHashLSHModel private[ml]( overrid

[GitHub] [spark] AmplabJenkins commented on pull request #31422: [SPARK-34313][SQL] Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to use UnresolvedTable to resolve the identifier

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31422: URL: https://github.com/apache/spark/pull/31422#issuecomment-771475123 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134755/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-771475134 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134752/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771475125 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771475127 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39347/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771475131 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39346/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771475135 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39351/ -

[GitHub] [spark] gengliangwang commented on a change in pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

2021-02-02 Thread GitBox
gengliangwang commented on a change in pull request #31434: URL: https://github.com/apache/spark/pull/31434#discussion_r568423494 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowPartitionsSuiteBase.scala ## @@ -161,7 +161,7 @@ trait ShowPartiti

[GitHub] [spark] AmplabJenkins commented on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771475126 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39350/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31421: URL: https://github.com/apache/spark/pull/31421#issuecomment-771475121 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134756/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771475130 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39348/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771475136 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39349/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771475136 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39349/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771475126 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39350/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-771475134 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134752/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771475130 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39348/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771475127 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39347/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771475135 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39351/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771475131 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39346/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31421: URL: https://github.com/apache/spark/pull/31421#issuecomment-771475121 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134756/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31422: [SPARK-34313][SQL] Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to use UnresolvedTable to resolve the identifier

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31422: URL: https://github.com/apache/spark/pull/31422#issuecomment-771475123 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134755/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771475125 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31133: [SPARK-26836][SQL] Supporting Avro schema evolution for partitioned Hive tables

2021-02-02 Thread GitBox
dongjoon-hyun commented on a change in pull request #31133: URL: https://github.com/apache/spark/pull/31133#discussion_r568424773 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ## @@ -388,6 +394,9 @@ private[hive] object HiveTableUtil { priva

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31133: [SPARK-26836][SQL] Supporting Avro schema evolution for partitioned Hive tables

2021-02-02 Thread GitBox
dongjoon-hyun commented on a change in pull request #31133: URL: https://github.com/apache/spark/pull/31133#discussion_r568428103 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala ## @@ -388,6 +394,9 @@ private[hive] object HiveTableUtil { priva

[GitHub] [spark] SparkQA commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

2021-02-02 Thread GitBox
SparkQA commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771481043 **[Test build #134767 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134767/testReport)** for PR 31433 at commit [`12be68c`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

2021-02-02 Thread GitBox
SparkQA removed a comment on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771423559 **[Test build #134767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134767/testReport)** for PR 31433 at commit [`12be68c`](https://gi

[GitHub] [spark] viirya commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread GitBox
viirya commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771481739 cc @cloud-fan @sunchao @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to th

[GitHub] [spark] cloud-fan commented on a change in pull request #31423: [SPARK-34314][SQL] Create new file index after partition schema inferring w/ the schema

2021-02-02 Thread GitBox
cloud-fan commented on a change in pull request #31423: URL: https://github.com/apache/spark/pull/31423#discussion_r568432026 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ## @@ -413,9 +413,13 @@ case class DataSource(

[GitHub] [spark] MaxGekk commented on pull request #31399: [SPARK-34259][SQL] Don't attempt to parse file-based partitions as special timestamps

2021-02-02 Thread GitBox
MaxGekk commented on pull request #31399: URL: https://github.com/apache/spark/pull/31399#issuecomment-771482897 I agree with disabling special dates/timestamps as partition values but I would prefer to create dedicated partition formatter like we have the fraction formatter now: https://

[GitHub] [spark] cloud-fan commented on pull request #31423: [SPARK-34314][SQL] Create new file index after partition schema inferring w/ the schema

2021-02-02 Thread GitBox
cloud-fan commented on pull request #31423: URL: https://github.com/apache/spark/pull/31423#issuecomment-771483498 @MaxGekk can you provide more code-wise details about how this bug is triggered? This is an automated message

[GitHub] [spark] cloud-fan commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread GitBox
cloud-fan commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771484471 Looks reasonable. Does it cause any user-facing problems? This is an automated message from the Apache Git Ser

[GitHub] [spark] SparkQA commented on pull request #31436: [SPARK-34327][BUILD] Strip passwords from inlining into build information while releasing.

2021-02-02 Thread GitBox
SparkQA commented on pull request #31436: URL: https://github.com/apache/spark/pull/31436#issuecomment-771486521 **[Test build #134769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134769/testReport)** for PR 31436 at commit [`c1558b0`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

2021-02-02 Thread GitBox
SparkQA commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771486575 **[Test build #134771 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134771/testReport)** for PR 31434 at commit [`15e9785`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771486627 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134767/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771486627 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134767/ -

[GitHub] [spark] AngersZhuuuu commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

2021-02-02 Thread GitBox
AngersZh commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771487154 > > Does this PR introduce any user-facing change? > > No > > Can we have more description there? This definitely has user-facing change, right? More queries are a

[GitHub] [spark] cloud-fan commented on pull request #30829: [SPARK-33832][SQL] Add an option in AQE to mitigate skew even if it c…

2021-02-02 Thread GitBox
cloud-fan commented on pull request #30829: URL: https://github.com/apache/spark/pull/30829#issuecomment-771487468 I think it's just a very conservative check. We can skip this check if the config to force-apply skew join optimization is turned on.

[GitHub] [spark] viirya commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread GitBox
viirya commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771487515 > Looks reasonable. Does it cause any user-facing problems? IIUC, `FileTable` is not a public API.

[GitHub] [spark] SparkQA commented on pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

2021-02-02 Thread GitBox
SparkQA commented on pull request #31378: URL: https://github.com/apache/spark/pull/31378#issuecomment-771487869 **[Test build #134772 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134772/testReport)** for PR 31378 at commit [`90c07f4`](https://github.com

[GitHub] [spark] cloud-fan commented on pull request #31405: [SPARK-34304][SQL] Remove view checks in v1 alter table commands

2021-02-02 Thread GitBox
cloud-fan commented on pull request #31405: URL: https://github.com/apache/spark/pull/31405#issuecomment-771487936 Hi @MaxGekk , can you resolve the conflicts? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

2021-02-02 Thread GitBox
cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r568438630 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala ## @@ -159,11 +159,19 @@ class CacheManager extends Logging wit

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

2021-02-02 Thread GitBox
cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r568439758 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala ## @@ -159,11 +159,19 @@ class CacheManager extends Logging wit

[GitHub] [spark] cloud-fan commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread GitBox
cloud-fan commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771491124 do users hit any query failure or wrong result when file source v2 is enabled? This is an automated message f

[GitHub] [spark] cloud-fan commented on a change in pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

2021-02-02 Thread GitBox
cloud-fan commented on a change in pull request #31402: URL: https://github.com/apache/spark/pull/31402#discussion_r568442858 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -3170,7 +3170,9 @@ class Analyzer(override val ca

[GitHub] [spark] cloud-fan commented on a change in pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

2021-02-02 Thread GitBox
cloud-fan commented on a change in pull request #31402: URL: https://github.com/apache/spark/pull/31402#discussion_r568443085 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala ## @@ -633,7 +633,9 @@ case class CumeDist(

[GitHub] [spark] cloud-fan commented on pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

2021-02-02 Thread GitBox
cloud-fan commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771493411 LGTM, can we update the PR title? This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #31435: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

2021-02-02 Thread GitBox
SparkQA commented on pull request #31435: URL: https://github.com/apache/spark/pull/31435#issuecomment-771494956 **[Test build #134770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134770/testReport)** for PR 31435 at commit [`36943c6`](https://github.com

[GitHub] [spark] gengliangwang commented on pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Revise the version and doc of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

2021-02-02 Thread GitBox
gengliangwang commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771495056 > LGTM, can we update the PR title? Done This is an automated message from the Apache Git Service.

[GitHub] [spark] viirya commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

2021-02-02 Thread GitBox
viirya commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771503216 > do users hit any query failure or wrong result when file source v2 is enabled? Currently, I don't see any place checking table truncation capability by `TableHelper.supp

[GitHub] [spark] SparkQA commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

2021-02-02 Thread GitBox
SparkQA commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771505153 **[Test build #134759 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134759/testReport)** for PR 31429 at commit [`11a6e24`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

2021-02-02 Thread GitBox
SparkQA removed a comment on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771369355 **[Test build #134759 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134759/testReport)** for PR 31429 at commit [`11a6e24`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31436: [SPARK-34327][BUILD] Strip passwords from inlining into build information while releasing.

2021-02-02 Thread GitBox
SparkQA commented on pull request #31436: URL: https://github.com/apache/spark/pull/31436#issuecomment-771507713 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39353/ -

[GitHub] [spark] SparkQA commented on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

2021-02-02 Thread GitBox
SparkQA commented on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771510645 **[Test build #134757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134757/testReport)** for PR 31413 at commit [`3d348a6`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

2021-02-02 Thread GitBox
SparkQA removed a comment on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771366278 **[Test build #134757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134757/testReport)** for PR 31413 at commit [`3d348a6`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

2021-02-02 Thread GitBox
SparkQA commented on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771511446 **[Test build #134764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134764/testReport)** for PR 31430 at commit [`8d43399`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

2021-02-02 Thread GitBox
SparkQA removed a comment on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771417687 **[Test build #134764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134764/testReport)** for PR 31430 at commit [`8d43399`](https://gi

[GitHub] [spark] AmplabJenkins commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

2021-02-02 Thread GitBox
AmplabJenkins commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771513400 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134759/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

2021-02-02 Thread GitBox
AmplabJenkins removed a comment on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771513400 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134759/ -

  1   2   3   4   5   6   7   8   9   10   >