[GitHub] [spark] cloud-fan commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

cloud-fan commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r568391121 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] turboFei commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

turboFei commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771444379 The UT passed in our spark-2.3 branch, will try to find out why it failed in community master branch. This is

[GitHub] [spark] SparkQA commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771444120 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39352/

[GitHub] [spark] SparkQA commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

SparkQA commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771443108 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39349/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

AmplabJenkins removed a comment on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771442106 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134753/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

AmplabJenkins removed a comment on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771442108 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134760/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31286: [SPARK-34199][SQL] Block `count(table.*)` to follow ANSI standard and other SQL engines

AmplabJenkins removed a comment on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771442107 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134754/

[GitHub] [spark] AmplabJenkins commented on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

AmplabJenkins commented on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771442108 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134760/

[GitHub] [spark] AmplabJenkins commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

AmplabJenkins commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771442106 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134753/

[GitHub] [spark] AmplabJenkins commented on pull request #31286: [SPARK-34199][SQL] Block `count(table.*)` to follow ANSI standard and other SQL engines

AmplabJenkins commented on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771442107 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134754/

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

HyukjinKwon commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568386466 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan closed pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

cloud-fan closed pull request #31286: URL: https://github.com/apache/spark/pull/31286 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] cloud-fan commented on pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

cloud-fan commented on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771440146 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] cloud-fan commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

cloud-fan commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r568385314 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ## @@ -129,7 +129,7 @@ class HiveCommandSuite extends

[GitHub] [spark] cloud-fan commented on pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

cloud-fan commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771439102 @gengliangwang I've left more comments in the original PR, can you address them as well? thanks! This is an

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568384363 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] HyukjinKwon commented on pull request #31434: [SPARK-33591][SQL][FOLLOW-UP] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

HyukjinKwon commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771437952 cc @cloud-fan too This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

AngersZh commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r568382649 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ## @@ -129,7 +129,7 @@ class HiveCommandSuite

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

HyukjinKwon commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568382451 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568382197 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowPartitionsSuiteBase.scala ## @@ -154,4 +154,15 @@ trait

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568381329 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568381165 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2472,6 +2472,16 @@ object SQLConf { .booleanConf

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568381010 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -472,9 +472,11 @@ class AstBuilder extends

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568380805 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -500,9 +502,11 @@ class AstBuilder extends

[GitHub] [spark] maropu commented on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

maropu commented on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-771434590 okay, could you copy the statement above into the PR description? This is an automated message from the Apache

[GitHub] [spark] cloud-fan commented on a change in pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

cloud-fan commented on a change in pull request #31421: URL: https://github.com/apache/spark/pull/31421#discussion_r568379292 ## File path: docs/sql-migration-guide.md ## @@ -41,6 +41,8 @@ license: | - In Spark 3.2, the auto-generated `Cast` (such as those added by type

[GitHub] [spark] SparkQA removed a comment on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA removed a comment on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771320748 **[Test build #134751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134751/testReport)** for PR 31402 at commit

[GitHub] [spark] SparkQA commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771432746 **[Test build #134751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134751/testReport)** for PR 31402 at commit

[GitHub] [spark] gengliangwang commented on pull request #31434: [SPARK-33591][SQL][FollowUp] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

gengliangwang commented on pull request #31434: URL: https://github.com/apache/spark/pull/31434#issuecomment-771432824 cc @HyukjinKwon I will create two backport PRs to branch 3.1 and 3.0 once this is merged. This is an

[GitHub] [spark] sarutak edited a comment on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

sarutak edited a comment on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-771432295 > Does this PR supports all types in PostgreSQL? I think so except for serial types as I mentioned

[GitHub] [spark] gengliangwang opened a new pull request #31434: [SPARK-33591][SQL][FollowUp] Correct the version of `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral`

gengliangwang opened a new pull request #31434: URL: https://github.com/apache/spark/pull/31434 ### What changes were proposed in this pull request? Correct the version of SQL configuration `spark.sql.legacy.parseNullPartitionSpecAsStringLiteral` from 3.2.0 to 3.0.2.

[GitHub] [spark] sarutak commented on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

sarutak commented on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-771432295 > Does this PR supports all types in PostgreSQL? are there unsupported types? I think so except for serial types as I mentioned

[GitHub] [spark] maropu commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

maropu commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r568376839 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ## @@ -129,7 +129,7 @@ class HiveCommandSuite extends

[GitHub] [spark] SparkQA removed a comment on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

SparkQA removed a comment on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771365500 **[Test build #134760 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134760/testReport)** for PR 31204 at commit

[GitHub] [spark] SparkQA commented on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

SparkQA commented on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771429364 **[Test build #134760 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134760/testReport)** for PR 31204 at commit

[GitHub] [spark] Ngone51 commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

Ngone51 commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771428466 thanks all! This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

SparkQA removed a comment on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771318932 **[Test build #134754 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134754/testReport)** for PR 31286 at commit

[GitHub] [spark] HyukjinKwon closed pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

HyukjinKwon closed pull request #31429: URL: https://github.com/apache/spark/pull/31429 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] HyukjinKwon commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

HyukjinKwon commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771427885 Merged to master, branch-3.1 and branch-3.0. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

SparkQA commented on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-771427572 **[Test build #134754 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134754/testReport)** for PR 31286 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

SparkQA removed a comment on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771318802 **[Test build #134753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134753/testReport)** for PR 31429 at commit

[GitHub] [spark] SparkQA commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

SparkQA commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771427109 **[Test build #134753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134753/testReport)** for PR 31429 at commit

[GitHub] [spark] SparkQA commented on pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

SparkQA commented on pull request #31432: URL: https://github.com/apache/spark/pull/31432#issuecomment-771425039 **[Test build #134762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134762/testReport)** for PR 31432 at commit

[GitHub] [spark] SparkQA commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771424391 **[Test build #134766 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134766/testReport)** for PR 31402 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

AmplabJenkins removed a comment on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771420996 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] SparkQA commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

SparkQA commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771423559 **[Test build #134767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134767/testReport)** for PR 31433 at commit

[GitHub] [spark] HyukjinKwon commented on pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

HyukjinKwon commented on pull request #31421: URL: https://github.com/apache/spark/pull/31421#issuecomment-771423568 Merged to master. @gengliangwang would you mind changing the version and making backporting PRs please?

[GitHub] [spark] offthewall123 commented on a change in pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

offthewall123 commented on a change in pull request #31433: URL: https://github.com/apache/spark/pull/31433#discussion_r568370590 ## File path: core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala ## @@ -20,12 +20,11 @@ package

[GitHub] [spark] sunchao commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

sunchao commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568370098 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

AmplabJenkins removed a comment on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771416669 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] SparkQA commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

SparkQA commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771422618 **[Test build #134763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134763/testReport)** for PR 31431 at commit

[GitHub] [spark] HyukjinKwon closed pull request #31421: [SPARK-33591][SQL][FOLLOWUP] Add legacy config for recognizing null partition spec values

HyukjinKwon closed pull request #31421: URL: https://github.com/apache/spark/pull/31421 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

AmplabJenkins commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771420996 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

AmplabJenkins removed a comment on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771411424 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] dongjoon-hyun commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

dongjoon-hyun commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771419982 ok to test This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

dongjoon-hyun commented on a change in pull request #31433: URL: https://github.com/apache/spark/pull/31433#discussion_r568367864 ## File path: core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala ## @@ -20,12 +20,11 @@ package

[GitHub] [spark] SparkQA commented on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

SparkQA commented on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771418173 **[Test build #134765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134765/testReport)** for PR 31413 at commit

[GitHub] [spark] SparkQA commented on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

SparkQA commented on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771417687 **[Test build #134764 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134764/testReport)** for PR 31430 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

AmplabJenkins removed a comment on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-770997062 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134735/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

AmplabJenkins removed a comment on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771410963 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

AmplabJenkins removed a comment on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771410968 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39343/

[GitHub] [spark] AmplabJenkins commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

AmplabJenkins commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771416669 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

AmplabJenkins removed a comment on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771410967 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39345/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

AmplabJenkins removed a comment on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771411511 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] maropu commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

maropu commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r568364686 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] maropu commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

maropu commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771414229 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] MaxGekk commented on a change in pull request #31423: [SPARK-34314][SQL] Create new file index after partition schema inferring w/ the schema

MaxGekk commented on a change in pull request #31423: URL: https://github.com/apache/spark/pull/31423#discussion_r568362848 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala ## @@ -413,9 +413,13 @@ case class DataSource(

[GitHub] [spark] turboFei removed a comment on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

turboFei removed a comment on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771405635 For temporary view, its logical plan has been stored in catalog and the logical plan has been analyzed. So, even we refresh its underlying table, it logical plan

[GitHub] [spark] MaxGekk commented on a change in pull request #31423: [SPARK-34314][SQL] Create new file index after partition schema inferring w/ the schema

MaxGekk commented on a change in pull request #31423: URL: https://github.com/apache/spark/pull/31423#discussion_r568362026 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionedTablePerfStatsSuite.scala ## @@ -370,7 +370,7 @@ class

[GitHub] [spark] SparkQA commented on pull request #31429: [SPARK-34319][SQL] Resolve duplicate attributes for FlatMapCoGroupsInPandas/MapInPandas

SparkQA commented on pull request #31429: URL: https://github.com/apache/spark/pull/31429#issuecomment-771412722 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39346/

[GitHub] [spark] maropu commented on pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

maropu commented on pull request #31419: URL: https://github.com/apache/spark/pull/31419#issuecomment-771412232 Does this PR supports all types in PostgreSQL? are there unsupported types? This is an automated message from

[GitHub] [spark] AmplabJenkins commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

AmplabJenkins commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771411511 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

AmplabJenkins commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771411424 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

AmplabJenkins commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771410963 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #31204: [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

AmplabJenkins commented on pull request #31204: URL: https://github.com/apache/spark/pull/31204#issuecomment-771410967 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39345/

[GitHub] [spark] AmplabJenkins commented on pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

AmplabJenkins commented on pull request #31413: URL: https://github.com/apache/spark/pull/31413#issuecomment-771410968 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39343/

[GitHub] [spark] maropu commented on a change in pull request #31419: [SPARK-34311][SQL] PostgresDialect can't treat arrays of some types

maropu commented on a change in pull request #31419: URL: https://github.com/apache/spark/pull/31419#discussion_r568360410 ## File path: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala ## @@ -74,7 +74,9 @@ class

[GitHub] [spark] Ngone51 commented on pull request #31102: [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

Ngone51 commented on pull request #31102: URL: https://github.com/apache/spark/pull/31102#issuecomment-771410418 kindly ping @holdenk This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] Ngone51 commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

Ngone51 commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-771409987 I left some minor comments. Overall, looks good to me. This is an automated message from the Apache Git

[GitHub] [spark] c21 commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

c21 commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568359233 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] offthewall123 commented on pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

offthewall123 commented on pull request #31433: URL: https://github.com/apache/spark/pull/31433#issuecomment-771408356 @srowen @HeartSaVioR Please take a review, thanks. This is an automated message from the Apache Git

[GitHub] [spark] Ngone51 commented on a change in pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

Ngone51 commented on a change in pull request #31249: URL: https://github.com/apache/spark/pull/31249#discussion_r568358423 ## File path: core/src/test/scala/org/apache/spark/scheduler/HealthTrackerSuite.scala ## @@ -554,6 +554,50 @@ class HealthTrackerSuite extends

[GitHub] [spark] offthewall123 opened a new pull request #31433: [SPARK-34325][CORE] remove_shuffleBlockResolver_in_SortShuffleWriter

offthewall123 opened a new pull request #31433: URL: https://github.com/apache/spark/pull/31433 ### What changes were proposed in this pull request? Remove shuffleBlockResolver in SortShuffleWriter. ### Why are the changes needed? For better code understanding.

[GitHub] [spark] c21 commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

c21 commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568357088 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] turboFei commented on pull request #31431: [SPARK-34322][SQL] When refreshing a non-temporary view, also refresh its underlying tables

turboFei commented on pull request #31431: URL: https://github.com/apache/spark/pull/31431#issuecomment-771405635 For temporary view, its logical plan has been stored in catalog and the logical plan has been analyzed. So, even we refresh its underlying table, it logical plan would

[GitHub] [spark] linhongliu-db commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

linhongliu-db commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r568356352 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] Ngone51 commented on a change in pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

Ngone51 commented on a change in pull request #31249: URL: https://github.com/apache/spark/pull/31249#discussion_r568356238 ## File path: core/src/test/scala/org/apache/spark/scheduler/HealthTrackerSuite.scala ## @@ -554,6 +554,50 @@ class HealthTrackerSuite extends

[GitHub] [spark] igreenfield commented on pull request #26624: [SPARK-8981][CORE][test-hadoop3.2][test-java11] Add MDC support in Executor

igreenfield commented on pull request #26624: URL: https://github.com/apache/spark/pull/26624#issuecomment-771405161 @alefischer13 Hi please look at this PR that was merged later and changed the way you need to configure the log4j.properties: #28801

[GitHub] [spark] maropu commented on pull request #31430: [SPARK-34323][BUILD] Upgrade zstd-jni to 1.4.8-3

maropu commented on pull request #31430: URL: https://github.com/apache/spark/pull/31430#issuecomment-771403087 LGTM nit: could you add a link to the release note of `1.4.8-3` in the PR description? This is an automated

[GitHub] [spark] sunchao commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

sunchao commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568353916 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] SparkQA removed a comment on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA removed a comment on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771366254 **[Test build #134758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134758/testReport)** for PR 31402 at commit

[GitHub] [spark] SparkQA commented on pull request #31402: [SPARK-34296][SQL] AggregateWindowFunction frame should not always use UnboundedPreceding

SparkQA commented on pull request #31402: URL: https://github.com/apache/spark/pull/31402#issuecomment-771401976 **[Test build #134758 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134758/testReport)** for PR 31402 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #31428: [SPARK-34316][K8S] Support spark.kubernetes.executor.disableConfigMap

dongjoon-hyun commented on pull request #31428: URL: https://github.com/apache/spark/pull/31428#issuecomment-771397570 Thank you, @ScrapCodes . Yes, for security, some namespaces do not allow the driver pod to create new ConfigMaps. As a result, Spark jobs are unable to be executed in

[GitHub] [spark] c21 commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

c21 commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568353327 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] viirya opened a new pull request #31432: [SPARK-34324][SQL] FileTable should not list TRUNCATE in capabilities by default

viirya opened a new pull request #31432: URL: https://github.com/apache/spark/pull/31432 ### What changes were proposed in this pull request? This patch proposes to remove `TRUNCATE` from the default `capabilities` list from `FileTable`. ### Why are the changes

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

AngersZh commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r568352532 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala ## @@ -129,7 +129,7 @@ class HiveCommandSuite

[GitHub] [spark] maropu commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

maropu commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568352335 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] ScrapCodes commented on pull request #31428: [SPARK-34316][K8S] Support spark.kubernetes.executor.disableConfigMap

ScrapCodes commented on pull request #31428: URL: https://github.com/apache/spark/pull/31428#issuecomment-771399674 Ahh, makes sense then. Wish, we could autodetect and fallback too. This is an automated message from the

[GitHub] [spark] c21 commented on a change in pull request #31413: [SPARK-32985][SQL] Decouple bucket scan and bucket filter pruning for data source v1

c21 commented on a change in pull request #31413: URL: https://github.com/apache/spark/pull/31413#discussion_r568351355 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala ## @@ -591,20 +590,48 @@ case class FileSourceScanExec(

[GitHub] [spark] maropu commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

maropu commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r568350914 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software

  1   2   3   4   5   6   7   8   >