[GitHub] [spark] cloud-fan commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

cloud-fan commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r568391121 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Found

[GitHub] [spark] turboFei commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

turboFei commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771444379 The UT passed in our spark-2.3 branch, will try to find out why it failed in community master branch. This is

[GitHub] [spark] SparkQA commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771444120 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39352/ -

[GitHub] [spark] SparkQA commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

SparkQA commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771443108 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39349/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

AmplabJenkins removed a comment on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771442106 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134753/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

AmplabJenkins removed a comment on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771442108 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134760/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31286: [SPARK-34199][SQL] Block `count(table.*)` to follow ANSI standard and other SQL engines

AmplabJenkins removed a comment on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771442107 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134754/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

AmplabJenkins commented on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771442108 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134760/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

AmplabJenkins commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771442106 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134753/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31286: [SPARK-34199][SQL] Block `count(table.*)` to follow ANSI standard and other SQL engines

AmplabJenkins commented on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771442107 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134754/ -

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

HyukjinKwon commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568386466 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan closed pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

cloud-fan closed pull request #31286: URL: https://github.com/apache/spark/pull/31286 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] cloud-fan commented on pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

cloud-fan commented on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771440146 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] cloud-fan commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

cloud-fan commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r568385314 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ## @@ -129,7 +129,7 @@ class HiveCommandSuite extends

[GitHub] [spark] cloud-fan commented on pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

cloud-fan commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771439102 @gengliangwang I've left more comments in the original PR, can you address them as well? thanks! This is an a

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568384363 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] HyukjinKwon commented on pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

HyukjinKwon commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771437952 cc @cloud-fan too This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

AngersZh commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r568382649 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ## @@ -129,7 +129,7 @@ class HiveCommandSuite exten

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

HyukjinKwon commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568382451 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568382197 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowPartitionsSuiteBase.scala ## @@ -154,4 +154,15 @@ trait ShowPartitions

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568381329 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568381165 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568381010 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -472,9 +472,11 @@ class AstBuilder extends SqlBas

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568380805 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -500,9 +502,11 @@ class AstBuilder extends SqlBas

[GitHub] [spark] maropu commented on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

maropu commented on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-771434590 okay, could you copy the statement above into the PR description? This is an automated message from the Apache Gi

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568379292 ## File path: docs/sql-migration-guide.md ## @@ -41,6 +41,8 @@ license: | - In Spark 3.2, the auto-generated `Cast` (such as those added by type c

[GitHub] [spark] SparkQA removed a comment on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA removed a comment on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771320748 **[Test build #134751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134751/testReport)** for PR 31402 at commit [`96037a8`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771432746 **[Test build #134751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134751/testReport)** for PR 31402 at commit [`96037a8`](https://github.co

[GitHub] [spark] gengliangwang commented on pull request #31434: [SPARK-33591][SQL][FollowUp] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

gengliangwang commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771432824 cc @HyukjinKwon I will create two backport PRs to branch 3.1 and 3.0 once this is merged. This is an a

[GitHub] [spark] sarutak edited a comment on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

sarutak edited a comment on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-771432295 > Does this PR supports all types in PostgreSQL? I think so except for serial types as I mentioned [above](https://github.com/apache/spark/pull/31419#issuecomment-

[GitHub] [spark] gengliangwang opened a new pull request #31434: [SPARK-33591][SQL][FollowUp] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

gengliangwang opened a new pull request #31434: URL: https://github.com/apache/spark/pull/31434 ### What changes were proposed in this pull request? Correct the version of SQL configuration `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral` from 3.2.0 to 3.0.2.

[GitHub] [spark] sarutak commented on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

sarutak commented on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-771432295 > Does this PR supports all types in PostgreSQL? are there unsupported types? I think so except for serial types as I mentioned [above](https://github.com/apache/spark/pul

[GitHub] [spark] maropu commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

maropu commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r568376839 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ## @@ -129,7 +129,7 @@ class HiveCommandSuite extends Que

[GitHub] [spark] SparkQA removed a comment on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

SparkQA removed a comment on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771365500 **[Test build #134760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134760/testReport)** for PR 31204 at commit [`9009b9a`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

SparkQA commented on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771429364 **[Test build #134760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134760/testReport)** for PR 31204 at commit [`9009b9a`](https://github.co

[GitHub] [spark] Ngone51 commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

Ngone51 commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771428466 thanks all! This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [spark] SparkQA removed a comment on pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

SparkQA removed a comment on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771318932 **[Test build #134754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134754/testReport)** for PR 31286 at commit [`19340d4`](https://gi

[GitHub] [spark] HyukjinKwon closed pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

HyukjinKwon closed pull request #31429: URL: https://github.com/apache/spark/pull/31429 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

HyukjinKwon commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771427885 Merged to master, branch-3.1 and branch-3.0. This is an automated message from the Apache Git Service. To re

[GitHub] [spark] SparkQA commented on pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

SparkQA commented on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771427572 **[Test build #134754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134754/testReport)** for PR 31286 at commit [`19340d4`](https://github.co

[GitHub] [spark] SparkQA removed a comment on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

SparkQA removed a comment on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771318802 **[Test build #134753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134753/testReport)** for PR 31429 at commit [`0ab2b7a`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

SparkQA commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771427109 **[Test build #134753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134753/testReport)** for PR 31429 at commit [`0ab2b7a`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

SparkQA commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771425039 **[Test build #134762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134762/testReport)** for PR 31432 at commit [`584dcbb`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771424391 **[Test build #134766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134766/testReport)** for PR 31402 at commit [`bfc1af1`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

AmplabJenkins removed a comment on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771420996 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] SparkQA commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

SparkQA commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771423559 **[Test build #134767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134767/testReport)** for PR 31433 at commit [`12be68c`](https://github.com

[GitHub] [spark] HyukjinKwon commented on pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

HyukjinKwon commented on pull request #31421: URL: https://github.com/apache/spark/pull/31421#issuecomment-771423568 Merged to master. @gengliangwang would you mind changing the version and making backporting PRs please? -

[GitHub] [spark] offthewall123 commented on a change in pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

offthewall123 commented on a change in pull request #31433: URL: https://github.com/apache/spark/pull/31433#discussion_r568370590 ## File path: core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala ## @@ -20,12 +20,11 @@ package org.apache.spark.shuffle.sor

[GitHub] [spark] sunchao commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

sunchao commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568370098 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

AmplabJenkins removed a comment on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771416669 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] SparkQA commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

SparkQA commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771422618 **[Test build #134763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134763/testReport)** for PR 31431 at commit [`62e4027`](https://github.com

[GitHub] [spark] HyukjinKwon closed pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

HyukjinKwon closed pull request #31421: URL: https://github.com/apache/spark/pull/31421 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] AmplabJenkins commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

AmplabJenkins commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771420996 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

AmplabJenkins removed a comment on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771411424 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] dongjoon-hyun commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

dongjoon-hyun commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771419982 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

dongjoon-hyun commented on a change in pull request #31433: URL: https://github.com/apache/spark/pull/31433#discussion_r568367864 ## File path: core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala ## @@ -20,12 +20,11 @@ package org.apache.spark.shuffle.sor

[GitHub] [spark] SparkQA commented on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

SparkQA commented on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771418173 **[Test build #134765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134765/testReport)** for PR 31413 at commit [`9a6999d`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

SparkQA commented on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771417687 **[Test build #134764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134764/testReport)** for PR 31430 at commit [`8d43399`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

AmplabJenkins removed a comment on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-770997062 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134735/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

AmplabJenkins removed a comment on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771410963 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

AmplabJenkins removed a comment on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771410968 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39343/

[GitHub] [spark] AmplabJenkins commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

AmplabJenkins commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771416669 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

AmplabJenkins removed a comment on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771410967 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39345/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

AmplabJenkins removed a comment on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771411511 Can one of the admins verify this patch? This is an automated message from the Apache Git Service.

[GitHub] [spark] maropu commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

maropu commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r568364686 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [spark] maropu commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

maropu commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771414229 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [spark] MaxGekk commented on a change in pull request #31423: [SPARK-34314][SQL] Create new file index after partition schema inferring w/ the schema

MaxGekk commented on a change in pull request #31423: URL: https://github.com/apache/spark/pull/31423#discussion_r568362848 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ## @@ -413,9 +413,13 @@ case class DataSource(

[GitHub] [spark] turboFei removed a comment on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

turboFei removed a comment on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771405635 For temporary view, its logical plan has been stored in catalog and the logical plan has been analyzed. So, even we refresh its underlying table, it logical plan

[GitHub] [spark] MaxGekk commented on a change in pull request #31423: [SPARK-34314][SQL] Create new file index after partition schema inferring w/ the schema

MaxGekk commented on a change in pull request #31423: URL: https://github.com/apache/spark/pull/31423#discussion_r568362026 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionedTablePerfStatsSuite.scala ## @@ -370,7 +370,7 @@ class PartitionedTablePerfSta

[GitHub] [spark] SparkQA commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

SparkQA commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771412722 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39346/ -

[GitHub] [spark] maropu commented on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

maropu commented on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-771412232 Does this PR supports all types in PostgreSQL? are there unsupported types? This is an automated message from the

[GitHub] [spark] AmplabJenkins commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

AmplabJenkins commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771411511 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

AmplabJenkins commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771411424 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

AmplabJenkins commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771410963 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

AmplabJenkins commented on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771410967 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39345/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

AmplabJenkins commented on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771410968 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39343/ -

[GitHub] [spark] maropu commented on a change in pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

maropu commented on a change in pull request #31419: URL: https://github.com/apache/spark/pull/31419#discussion_r568360410 ## File path: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala ## @@ -74,7 +74,9 @@ class Postgre

[GitHub] [spark] Ngone51 commented on pull request #31102: [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

Ngone51 commented on pull request #31102: URL: https://github.com/apache/spark/pull/31102#issuecomment-771410418 kindly ping @holdenk This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [spark] Ngone51 commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

Ngone51 commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-771409987 I left some minor comments. Overall, looks good to me. This is an automated message from the Apache Git Service.

[GitHub] [spark] c21 commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

c21 commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568359233 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec( lo

[GitHub] [spark] offthewall123 commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

offthewall123 commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771408356 @srowen @HeartSaVioR Please take a review, thanks. This is an automated message from the Apache Git Servic

[GitHub] [spark] Ngone51 commented on a change in pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

Ngone51 commented on a change in pull request #31249: URL: https://github.com/apache/spark/pull/31249#discussion_r568358423 ## File path: core/src/test/scala/org/apache/spark/scheduler/HealthTrackerSuite.scala ## @@ -554,6 +554,50 @@ class HealthTrackerSuite extends SparkFunSu

[GitHub] [spark] offthewall123 opened a new pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

offthewall123 opened a new pull request #31433: URL: https://github.com/apache/spark/pull/31433 ### What changes were proposed in this pull request? Remove shuffleBlockResolver in SortShuffleWriter. ### Why are the changes needed? For better code understanding.

[GitHub] [spark] c21 commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

c21 commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568357088 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec( lo

[GitHub] [spark] turboFei commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

turboFei commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771405635 For temporary view, its logical plan has been stored in catalog and the logical plan has been analyzed. So, even we refresh its underlying table, it logical plan would no

[GitHub] [spark] linhongliu-db commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

linhongliu-db commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r568356352 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software F

[GitHub] [spark] Ngone51 commented on a change in pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

Ngone51 commented on a change in pull request #31249: URL: https://github.com/apache/spark/pull/31249#discussion_r568356238 ## File path: core/src/test/scala/org/apache/spark/scheduler/HealthTrackerSuite.scala ## @@ -554,6 +554,50 @@ class HealthTrackerSuite extends SparkFunSu

[GitHub] [spark] igreenfield commented on pull request #26624: [SPARK-8981][CORE][test-hadoop3.2][test-java11] Add MDC support in Executor

igreenfield commented on pull request #26624: URL: https://github.com/apache/spark/pull/26624#issuecomment-771405161 @alefischer13 Hi please look at this PR that was merged later and changed the way you need to configure the log4j.properties: #28801 ---

[GitHub] [spark] maropu commented on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

maropu commented on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771403087 LGTM nit: could you add a link to the release note of `1.4.8-3` in the PR description? This is an automated mes

[GitHub] [spark] sunchao commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

sunchao commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568353916 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] SparkQA removed a comment on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA removed a comment on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771366254 **[Test build #134758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134758/testReport)** for PR 31402 at commit [`5c4c256`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771401976 **[Test build #134758 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134758/testReport)** for PR 31402 at commit [`5c4c256`](https://github.co

[GitHub] [spark] dongjoon-hyun commented on pull request #31428: [SPARK-34316][K8S] Support spark.kubernetes.executor.disableConfigMap

dongjoon-hyun commented on pull request #31428: URL: https://github.com/apache/spark/pull/31428#issuecomment-771397570 Thank you, @ScrapCodes . Yes, for security, some namespaces do not allow the driver pod to create new ConfigMaps. As a result, Spark jobs are unable to be executed in thos

[GitHub] [spark] c21 commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

c21 commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568353327 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec( lo

[GitHub] [spark] viirya opened a new pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

viirya opened a new pull request #31432: URL: https://github.com/apache/spark/pull/31432 ### What changes were proposed in this pull request? This patch proposes to remove `TRUNCATE` from the default `capabilities` list from `FileTable`. ### Why are the changes nee

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

AngersZh commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r568352532 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ## @@ -129,7 +129,7 @@ class HiveCommandSuite exten

[GitHub] [spark] maropu commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

maropu commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568352335 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] ScrapCodes commented on pull request #31428: [SPARK-34316][K8S] Support spark.kubernetes.executor.disableConfigMap

ScrapCodes commented on pull request #31428: URL: https://github.com/apache/spark/pull/31428#issuecomment-771399674 Ahh, makes sense then. Wish, we could autodetect and fallback too. This is an automated message from the Apac

[GitHub] [spark] c21 commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

c21 commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568351355 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec( lo

[GitHub] [spark] maropu commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

maropu commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r568350914 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundati

  1   2   3   4   5   6   7   8   >