[GitHub] [spark] viirya commented on a change in pull request #31476: [SPARK-34366][SQL] Add interface for DS v2 metrics

viirya commented on a change in pull request #31476: URL: https://github.com/apache/spark/pull/31476#discussion_r580807887 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/CustomMetric.java ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] cloud-fan closed pull request #31606: [SPARK-34490][SQL] Analysis should fail if the view refers a dropped table

cloud-fan closed pull request #31606: URL: https://github.com/apache/spark/pull/31606 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] cloud-fan commented on pull request #31606: [SPARK-34490][SQL] Analysis should fail if the view refers a dropped table

cloud-fan commented on pull request #31606: URL: https://github.com/apache/spark/pull/31606#issuecomment-783982949 thanks, merging to master/3.1! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] c21 commented on a change in pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

c21 commented on a change in pull request #31595: URL: https://github.com/apache/spark/pull/31595#discussion_r580825530 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala ## @@ -808,6 +813,53 @@ class DataFrameSetOperationsSuite

[GitHub] [spark] SparkQA removed a comment on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

SparkQA removed a comment on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-783848308 **[Test build #135361 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135361/testReport)** for PR 31595 at commit

[GitHub] [spark] SparkQA commented on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

SparkQA commented on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-783981752 **[Test build #135361 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135361/testReport)** for PR 31595 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580824030 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ## @@ -541,40 +552,31 @@ object ViewHelper { } /** -

[GitHub] [spark] SparkQA removed a comment on pull request #31567: [SPARK-26138][SQL] Pushdown limit through InnerLike when condition is empty

SparkQA removed a comment on pull request #31567: URL: https://github.com/apache/spark/pull/31567#issuecomment-783849430 **[Test build #135362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135362/testReport)** for PR 31567 at commit

[GitHub] [spark] SparkQA commented on pull request #31567: [SPARK-26138][SQL] Pushdown limit through InnerLike when condition is empty

SparkQA commented on pull request #31567: URL: https://github.com/apache/spark/pull/31567#issuecomment-783980512 **[Test build #135362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135362/testReport)** for PR 31567 at commit

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580815988 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala ## @@ -301,16 +307,16 @@ abstract class

[GitHub] [spark] dongjoon-hyun commented on pull request #31618: [SPARK-34503][CORE] Use zstd for spark.eventLog.compression.codec by default

dongjoon-hyun commented on pull request #31618: URL: https://github.com/apache/spark/pull/31618#issuecomment-783979247 I updated the indirect benchmark result by using `lzbench`. This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

SparkQA commented on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-783979315 **[Test build #135371 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135371/testReport)** for PR 31595 at commit

[GitHub] [spark] SparkQA commented on pull request #31618: [SPARK-34503][CORE] Use zstd for spark.eventLog.compression.codec by default

SparkQA commented on pull request #31618: URL: https://github.com/apache/spark/pull/31618#issuecomment-783975896 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39946/

[GitHub] [spark] SparkQA commented on pull request #31606: [SPARK-34490][SQL] Analysis should fail if the view refers a dropped table

SparkQA commented on pull request #31606: URL: https://github.com/apache/spark/pull/31606#issuecomment-783975697 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39947/

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580818323 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ## @@ -541,40 +552,31 @@ object ViewHelper { } /** -

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580817665 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ## @@ -541,40 +552,31 @@ object ViewHelper { } /** -

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580817158 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ## @@ -489,8 +500,8 @@ object ViewHelper { path:

[GitHub] [spark] dongjoon-hyun commented on pull request #31618: [SPARK-34503][CORE] Use zstd for spark.eventLog.compression.codec by default

dongjoon-hyun commented on pull request #31618: URL: https://github.com/apache/spark/pull/31618#issuecomment-783973623 No~ This only decides write codec for new logs. This is an automated message from the Apache Git Service.

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580816822 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ## @@ -192,6 +192,17 @@ case class CreateViewCommand(

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580815988 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala ## @@ -301,16 +307,16 @@ abstract class

[GitHub] [spark] SparkQA removed a comment on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

SparkQA removed a comment on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783968196 **[Test build #135370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135370/testReport)** for PR 31619 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

AmplabJenkins removed a comment on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783971632 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135370/

[GitHub] [spark] SparkQA commented on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

SparkQA commented on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783971612 **[Test build #135370 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135370/testReport)** for PR 31619 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

AmplabJenkins commented on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783971632 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135370/

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580815261 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveHintsSuite.scala ## @@ -30,37 +30,48 @@ import

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580813778 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala ## @@ -443,21 +444,23 @@ case class

[GitHub] [spark] cloud-fan commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580813358 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ## @@ -838,9 +836,11 @@ class SessionCatalog(

[GitHub] [spark] shardulm94 commented on pull request #31591: [SPARK-34472][CORE] Ship ivySettings file to driver in cluster mode

shardulm94 commented on pull request #31591: URL: https://github.com/apache/spark/pull/31591#issuecomment-783969152 I thought about this a bit more. I think the main concern here is where/how to find the ivySettings file. I believe this becomes tricky because the same code and properties

[GitHub] [spark] SparkQA commented on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

SparkQA commented on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783968196 **[Test build #135370 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135370/testReport)** for PR 31619 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

AmplabJenkins removed a comment on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783945983 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] cloud-fan commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-783967099 Let me take a closer look first. It's better to record the create time for df temp view correctly, if the effort is small.

[GitHub] [spark] AmplabJenkins commented on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

AmplabJenkins commented on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783967501 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135368/

[GitHub] [spark] viirya commented on a change in pull request #31476: [SPARK-34366][SQL] Add interface for DS v2 metrics

viirya commented on a change in pull request #31476: URL: https://github.com/apache/spark/pull/31476#discussion_r580807887 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/CustomMetric.java ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] SparkQA commented on pull request #31606: [SPARK-34490][SQL] Analysis should fail if the view refers a dropped table

SparkQA commented on pull request #31606: URL: https://github.com/apache/spark/pull/31606#issuecomment-783960757 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39947/

[GitHub] [spark] SparkQA commented on pull request #31618: [SPARK-34503][CORE] Use zstd for spark.eventLog.compression.codec by default

SparkQA commented on pull request #31618: URL: https://github.com/apache/spark/pull/31618#issuecomment-783960565 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39946/

[GitHub] [spark] SparkQA removed a comment on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

SparkQA removed a comment on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783949815 **[Test build #135368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135368/testReport)** for PR 31619 at commit

[GitHub] [spark] SparkQA commented on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

SparkQA commented on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783950446 **[Test build #135368 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135368/testReport)** for PR 31619 at commit

[GitHub] [spark] SparkQA commented on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

SparkQA commented on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-783950201 **[Test build #135369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135369/testReport)** for PR 31316 at commit

[GitHub] [spark] SparkQA commented on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

SparkQA commented on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783949815 **[Test build #135368 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135368/testReport)** for PR 31619 at commit

[GitHub] [spark] viirya commented on a change in pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

viirya commented on a change in pull request #31595: URL: https://github.com/apache/spark/pull/31595#discussion_r580799487 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala ## @@ -808,6 +813,27 @@ class DataFrameSetOperationsSuite

[GitHub] [spark] SparkQA commented on pull request #31606: [SPARK-34490][SQL] Analysis should fail if the view refers a dropped table

SparkQA commented on pull request #31606: URL: https://github.com/apache/spark/pull/31606#issuecomment-783949730 **[Test build #135367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135367/testReport)** for PR 31606 at commit

[GitHub] [spark] SparkQA commented on pull request #31618: [SPARK-34503][CORE] Use zstd for spark.eventLog.compression.codec by default

SparkQA commented on pull request #31618: URL: https://github.com/apache/spark/pull/31618#issuecomment-783949409 **[Test build #135366 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135366/testReport)** for PR 31618 at commit

[GitHub] [spark] SparkQA commented on pull request #31618: [SPARK-34503][CORE] Use zstd for spark.eventLog.compression.codec by default

SparkQA commented on pull request #31618: URL: https://github.com/apache/spark/pull/31618#issuecomment-783946203 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39945/

[GitHub] [spark] AmplabJenkins commented on pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

AmplabJenkins commented on pull request #31619: URL: https://github.com/apache/spark/pull/31619#issuecomment-783945983 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39948/

[GitHub] [spark] viirya commented on a change in pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

viirya commented on a change in pull request #31595: URL: https://github.com/apache/spark/pull/31595#discussion_r580796187 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -501,6 +502,42 @@ object RemoveNoopOperators

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31616: [MINOR][DOCS] Replace http to https when possible in PySpark documentation

AmplabJenkins removed a comment on pull request #31616: URL: https://github.com/apache/spark/pull/31616#issuecomment-783943750 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135358/

[GitHub] [spark] AmplabJenkins commented on pull request #31616: [MINOR][DOCS] Replace http to https when possible in PySpark documentation

AmplabJenkins commented on pull request #31616: URL: https://github.com/apache/spark/pull/31616#issuecomment-783943750 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135358/

[GitHub] [spark] SparkQA removed a comment on pull request #31616: [MINOR][DOCS] Replace http to https when possible in PySpark documentation

SparkQA removed a comment on pull request #31616: URL: https://github.com/apache/spark/pull/31616#issuecomment-783837428 **[Test build #135358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135358/testReport)** for PR 31616 at commit

[GitHub] [spark] SparkQA commented on pull request #31616: [MINOR][DOCS] Replace http to https when possible in PySpark documentation

SparkQA commented on pull request #31616: URL: https://github.com/apache/spark/pull/31616#issuecomment-783942974 **[Test build #135358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135358/testReport)** for PR 31616 at commit

[GitHub] [spark] karenfeng opened a new pull request #31619: [WIP][SPARK-33600] Group exception messages in execution/datasources/v2

karenfeng opened a new pull request #31619: URL: https://github.com/apache/spark/pull/31619 ### What changes were proposed in this pull request? This PR groups exception messages in `execution/datasources/v2`. ### Why are the changes needed? It will largely help with

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

AmplabJenkins removed a comment on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-783939742 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135360/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31617: [SPARK-34502][SQL] Remove unused parameters

AmplabJenkins removed a comment on pull request #31617: URL: https://github.com/apache/spark/pull/31617#issuecomment-783939744 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39943/

[GitHub] [spark] AmplabJenkins commented on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

AmplabJenkins commented on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-783939742 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135360/

[GitHub] [spark] AmplabJenkins commented on pull request #31617: [SPARK-34502][SQL] Remove unused parameters

AmplabJenkins commented on pull request #31617: URL: https://github.com/apache/spark/pull/31617#issuecomment-783939744 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39943/

[GitHub] [spark] SparkQA removed a comment on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

SparkQA removed a comment on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-783829107 **[Test build #135360 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135360/testReport)** for PR 31316 at commit

[GitHub] [spark] SparkQA commented on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

SparkQA commented on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-783935785 **[Test build #135360 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135360/testReport)** for PR 31316 at commit

[GitHub] [spark] imback82 commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

imback82 commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-783930159 So, do you prefer generating `CatalogTable` on the fly in `SessionCatalog` instead of changing `TemporaryViewRelation`?

[GitHub] [spark] SparkQA commented on pull request #31618: [SPARK-34503][CORE] Use zstd for spark.eventLog.compression.codec by default

SparkQA commented on pull request #31618: URL: https://github.com/apache/spark/pull/31618#issuecomment-783927059 **[Test build #135365 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135365/testReport)** for PR 31618 at commit

[GitHub] [spark] SparkQA commented on pull request #31617: [SPARK-34502][SQL] Remove unused parameters

SparkQA commented on pull request #31617: URL: https://github.com/apache/spark/pull/31617#issuecomment-783926232 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39943/

[GitHub] [spark] cloud-fan commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-783924839 I see. Since previously we don't have the `View` wrapper for df temp view, I think no one really cares about the table metadata of a temp view. We can probably just set the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31617: [SPARK-34502][SQL] Remove unused parameters

AmplabJenkins removed a comment on pull request #31617: URL: https://github.com/apache/spark/pull/31617#issuecomment-783901733 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135363/

[GitHub] [spark] SparkQA commented on pull request #31617: [SPARK-34502][SQL] Remove unused parameters

SparkQA commented on pull request #31617: URL: https://github.com/apache/spark/pull/31617#issuecomment-783918512 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39943/

[GitHub] [spark] dongjoon-hyun opened a new pull request #31618: [SPARK-34503][CORE] Use zstd for spark.eventLog.compression.codec by default

dongjoon-hyun opened a new pull request #31618: URL: https://github.com/apache/spark/pull/31618 ### What changes were proposed in this pull request? This PR aims to set `zstd` as the default value for `spark.eventLog.compression.codec` configuration. ### Why are the changes

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

AmplabJenkins removed a comment on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-783901731 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39941/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31567: [SPARK-26138][SQL] Pushdown limit through InnerLike when condition is empty

AmplabJenkins removed a comment on pull request #31567: URL: https://github.com/apache/spark/pull/31567#issuecomment-783901732 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39942/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

AmplabJenkins removed a comment on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-783901727 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

AmplabJenkins commented on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-783901731 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39941/

[GitHub] [spark] AmplabJenkins commented on pull request #31617: [SPARK-34502][SQL] Remove unused parameters

AmplabJenkins commented on pull request #31617: URL: https://github.com/apache/spark/pull/31617#issuecomment-783901733 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135363/

[GitHub] [spark] AmplabJenkins commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

AmplabJenkins commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-783901727 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #31567: [SPARK-26138][SQL] Pushdown limit through InnerLike when condition is empty

AmplabJenkins commented on pull request #31567: URL: https://github.com/apache/spark/pull/31567#issuecomment-783901732 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39942/

[GitHub] [spark] ijuma commented on pull request #30939: [DO-NOT-MERGE][SPARK-33913][SPARK-33921][BUILD] Upgrade kafka to 2.7.0 and Upgrade Scala version to 2.12.12

ijuma commented on pull request #30939: URL: https://github.com/apache/spark/pull/30939#issuecomment-783898758 FYI, we adjusted the optimizer configs for Kafka to avoid the exact Scala patch version dependency issue: https://github.com/apache/kafka/pull/10174 It will be part of

[GitHub] [spark] huaxingao commented on pull request #31617: [SPARK-34502][SQL] Remove unused parameters

huaxingao commented on pull request #31617: URL: https://github.com/apache/spark/pull/31617#issuecomment-783891003 jira opened @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] imback82 commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

imback82 commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-783882301 > > However, there were tests that compare analyzed plans which were failing due to CatalogTable.createTime difference > > Can you elaborate on it? analyzed plan does

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

dongjoon-hyun commented on a change in pull request #31595: URL: https://github.com/apache/spark/pull/31595#discussion_r580766926 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala ## @@ -808,6 +813,27 @@ class

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

dongjoon-hyun commented on a change in pull request #31595: URL: https://github.com/apache/spark/pull/31595#discussion_r580766586 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -501,6 +502,42 @@ object

[GitHub] [spark] c21 commented on pull request #31617: [MINOR][SQL] Remove unused parameters

c21 commented on pull request #31617: URL: https://github.com/apache/spark/pull/31617#issuecomment-783873360 LGTM if tests are passed. My bad. I probably introduced those in files `CoalesceBucketsInJoin` and `ShuffledHashJoinExec`. Also checked `CartesianProductExec` and the change makes

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31499: [SPARK-31891][SQL] Support `MSCK REPAIR TABLE .. [{ADD|DROP|SYNC} PARTITIONS]`

dongjoon-hyun commented on a change in pull request #31499: URL: https://github.com/apache/spark/pull/31499#discussion_r580765104 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ## @@ -791,8 +799,26 @@ case class

[GitHub] [spark] dongjoon-hyun commented on pull request #31499: [SPARK-31891][SQL] Support `MSCK REPAIR TABLE .. [{ADD|DROP|SYNC} PARTITIONS]`

dongjoon-hyun commented on pull request #31499: URL: https://github.com/apache/spark/pull/31499#issuecomment-783870399 Oops. Sorry, @MaxGekk . Could you rebase to the master once more please? This is an automated message

[GitHub] [spark] SparkQA commented on pull request #31567: [SPARK-26138][SQL] Pushdown limit through InnerLike when condition is empty

SparkQA commented on pull request #31567: URL: https://github.com/apache/spark/pull/31567#issuecomment-783870391 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39942/

[GitHub] [spark] c21 commented on a change in pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

c21 commented on a change in pull request #31595: URL: https://github.com/apache/spark/pull/31595#discussion_r580761460 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -501,6 +502,42 @@ object RemoveNoopOperators extends

[GitHub] [spark] cloud-fan commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

cloud-fan commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-783867340 > However, there were tests that compare analyzed plans which were failing due to CatalogTable.createTime difference Can you elaborate on it? analyzed plan does not

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31611: [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified

AmplabJenkins removed a comment on pull request #31611: URL: https://github.com/apache/spark/pull/31611#issuecomment-783865396 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

AmplabJenkins removed a comment on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-783865395 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39940/

[GitHub] [spark] AmplabJenkins commented on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

AmplabJenkins commented on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-783865395 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39940/

[GitHub] [spark] AmplabJenkins commented on pull request #31611: [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

AmplabJenkins commented on pull request #31611: URL: https://github.com/apache/spark/pull/31611#issuecomment-783865397 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] imback82 commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

imback82 commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580758902 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveHintsSuite.scala ## @@ -30,37 +30,48 @@ import

[GitHub] [spark] huaxingao opened a new pull request #31617: [MINOR][SQL] Remove unused parameters

huaxingao opened a new pull request #31617: URL: https://github.com/apache/spark/pull/31617 ### What changes were proposed in this pull request? Remove unused parameters ### Why are the changes needed? Clean up ### Does this PR introduce _any_ user-facing

[GitHub] [spark] imback82 commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

imback82 commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580758321 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ## @@ -683,7 +681,7 @@ class SessionCatalog(

[GitHub] [spark] imback82 commented on a change in pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

imback82 commented on a change in pull request #31273: URL: https://github.com/apache/spark/pull/31273#discussion_r580757641 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ## @@ -838,9 +836,11 @@ class SessionCatalog(

[GitHub] [spark] imback82 commented on pull request #31273: [SPARK-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

imback82 commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-783861974 Thanks @cloud-fan for the suggestion. I pushed new changes that do not require `View` signature changes. Basically, the following changes are needed: 1.

[GitHub] [spark] zsxwing commented on a change in pull request #31476: [SPARK-34366][SQL] Add interface for DS v2 metrics

zsxwing commented on a change in pull request #31476: URL: https://github.com/apache/spark/pull/31476#discussion_r580756025 ## File path: sql/catalyst/src/main/java/org/apache/spark/sql/connector/CustomMetric.java ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] SparkQA commented on pull request #31567: [SPARK-26138][SQL] Pushdown limit through InnerLike when condition is empty

SparkQA commented on pull request #31567: URL: https://github.com/apache/spark/pull/31567#issuecomment-783861624 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39942/

[GitHub] [spark] SparkQA commented on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

SparkQA commented on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-783857159 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39940/

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh edited a comment on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-783854398 > The file commit is a driver side thing, why do we need to update `BasicWriteJobStatsTracker`? I think we can follow `BroadcastExchangeExec` and simply call

[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

AngersZh commented on pull request #31522: URL: https://github.com/apache/spark/pull/31522#issuecomment-783854398 > The file commit is a driver side thing, why do we need to update `BasicWriteJobStatsTracker`? I think we can follow `BroadcastExchangeExec` and simply call

[GitHub] [spark] SparkQA commented on pull request #31316: [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

SparkQA commented on pull request #31316: URL: https://github.com/apache/spark/pull/31316#issuecomment-783851813 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39940/

[GitHub] [spark] SparkQA commented on pull request #31611: [SPARK-34488][CORE] Support task Metrics Distributions and executor Metrics Distributions in the REST API call for a specified stage

SparkQA commented on pull request #31611: URL: https://github.com/apache/spark/pull/31611#issuecomment-783851621 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39939/

[GitHub] [spark] SparkQA commented on pull request #31567: [SPARK-26138][SQL] Pushdown limit through InnerLike when condition is empty

SparkQA commented on pull request #31567: URL: https://github.com/apache/spark/pull/31567#issuecomment-783849430 **[Test build #135362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135362/testReport)** for PR 31567 at commit

[GitHub] [spark] SparkQA commented on pull request #31595: [SPARK-34474][SQL] Remove unnecessary Union under Distinct/Deduplicate

SparkQA commented on pull request #31595: URL: https://github.com/apache/spark/pull/31595#issuecomment-783848308 **[Test build #135361 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135361/testReport)** for PR 31595 at commit

[GitHub] [spark] cloud-fan commented on pull request #31601: [SPARK-34484][SQL] Introduce a new syntax to represent attributes with the Catalyst DSL

cloud-fan commented on pull request #31601: URL: https://github.com/apache/spark/pull/31601#issuecomment-783843413 I agree that the `$"abc"` is not great as it conflicts with the well-known syntax defined in sql/core. But I do like the existing `"abc".attr` syntax instead of the new one.

  1   2   3   4   5   >