[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31680: [SPARK-34568][SQL] We should respect enableHiveSupport when initialize SparkSession

AngersZh commented on a change in pull request #31680: URL: https://github.com/apache/spark/pull/31680#discussion_r584497705 ## File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ## @@ -946,7 +946,11 @@ object SparkSession extends Logging {

[GitHub] [spark] HyukjinKwon edited a comment on pull request #31692: [SPARK-34578][SQL][TESTS][test-maven] Ignore ORC encryption tests when ORC shim is loaded by old Hadoop library

HyukjinKwon edited a comment on pull request #31692: URL: https://github.com/apache/spark/pull/31692#issuecomment-787735738 Hm, GA tests are skipped too (https://github.com/apache/spark/pull/31692/checks?check_run_id=2002074971#step:9:12751) ``` [info] - Write and read an encrypte

[GitHub] [spark] HyukjinKwon commented on pull request #31692: [SPARK-34578][SQL][TESTS][test-maven] Ignore ORC encryption tests when ORC shim is loaded by old Hadoop library

HyukjinKwon commented on pull request #31692: URL: https://github.com/apache/spark/pull/31692#issuecomment-787735738 Hm, GA tests are skipped too (https://github.com/apache/spark/pull/31692/checks?check_run_id=2002074971#step:9:12751) ``` [info] - Write and read an encrypted file

[GitHub] [spark] Ngone51 commented on pull request #31600: [SPARK-34482][SS] Correct the active SparkSession for StreamExecution.logicalPlan

Ngone51 commented on pull request #31600: URL: https://github.com/apache/spark/pull/31600#issuecomment-787735717 retest this please This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [spark] maropu commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

maropu commented on a change in pull request #28715: URL: https://github.com/apache/spark/pull/28715#discussion_r584503039 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala ## @@ -51,6 +51,84 @@ class WholeStageCodegenSuite exten

[GitHub] [spark] cloud-fan commented on pull request #31600: [SPARK-34482][SS] Correct the active SparkSession for StreamExecution.logicalPlan

cloud-fan commented on pull request #31600: URL: https://github.com/apache/spark/pull/31600#issuecomment-787735324 Looks reasonable to me. I'll merge it within a few days if there is no objection. This is an automated messag

[GitHub] [spark] viirya commented on pull request #31656: [SPARK-34548][SQL] Remove unnecessary children from Union under Distince and Deduplicate

viirya commented on pull request #31656: URL: https://github.com/apache/spark/pull/31656#issuecomment-78770 cc @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #31675: [SPARK-34560][SQL] Generate unique output attributes in the `SHOW TABLES` logical node

SparkQA commented on pull request #31675: URL: https://github.com/apache/spark/pull/31675#issuecomment-787732936 **[Test build #135584 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135584/testReport)** for PR 31675 at commit [`b9d4ce7`](https://github.com

[GitHub] [spark] MaxGekk commented on a change in pull request #31675: [SPARK-34560][SQL] Generate unique output attributes in the `SHOW TABLES` logical node

MaxGekk commented on a change in pull request #31675: URL: https://github.com/apache/spark/pull/31675#discussion_r584500748 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -491,13 +491,10 @@ case class RenameTable(

[GitHub] [spark] MaxGekk commented on a change in pull request #31675: [SPARK-34560][SQL] Generate unique output attributes in the `SHOW TABLES` logical node

MaxGekk commented on a change in pull request #31675: URL: https://github.com/apache/spark/pull/31675#discussion_r584499837 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala ## @@ -334,8 +334,8 @@ class DataSourceV2S

[GitHub] [spark] cloud-fan commented on pull request #31102: [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

cloud-fan commented on pull request #31102: URL: https://github.com/apache/spark/pull/31102#issuecomment-787730436 retest this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31691: [SPARK-34575][SQL] Push down limit through window when partitionSpec is empty

AmplabJenkins removed a comment on pull request #31691: URL: https://github.com/apache/spark/pull/31691#issuecomment-787729730 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40160/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31680: [SPARK-34568][SQL] We should respect enableHiveSupport when initialize SparkSession

AmplabJenkins removed a comment on pull request #31680: URL: https://github.com/apache/spark/pull/31680#issuecomment-787729728 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135572/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

AmplabJenkins removed a comment on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-787729731 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40158/

[GitHub] [spark] AmplabJenkins commented on pull request #31680: [SPARK-34568][SQL] We should respect enableHiveSupport when initialize SparkSession

AmplabJenkins commented on pull request #31680: URL: https://github.com/apache/spark/pull/31680#issuecomment-787729728 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135572/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

AmplabJenkins commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-787729731 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40158/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31691: [SPARK-34575][SQL] Push down limit through window when partitionSpec is empty

AmplabJenkins commented on pull request #31691: URL: https://github.com/apache/spark/pull/31691#issuecomment-787729730 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40160/ -

[GitHub] [spark] maropu commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

maropu commented on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-787728721 @karuppayya Are you still here? Could you fix the build failure? This is an automated message from the Apache Git

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31680: [SPARK-34568][SQL] We should respect enableHiveSupport when initialize SparkSession

AngersZh commented on a change in pull request #31680: URL: https://github.com/apache/spark/pull/31680#discussion_r584497705 ## File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ## @@ -946,7 +946,11 @@ object SparkSession extends Logging {

[GitHub] [spark] sririshindra edited a comment on pull request #31477: [SPARK-34369][SQL] Track number of pairs processed out of Join

sririshindra edited a comment on pull request #31477: URL: https://github.com/apache/spark/pull/31477#issuecomment-787552318 > @sririshindra could you check if no performance (e.g., TPCDS queries) changes, too? Sure, I am working that right now. cc: @maropu Update: I ran the

[GitHub] [spark] viirya commented on a change in pull request #31656: [SPARK-34548][SQL] Remove unnecessary children from Union under Distince and Deduplicate

viirya commented on a change in pull request #31656: URL: https://github.com/apache/spark/pull/31656#discussion_r584496283 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala ## @@ -526,21 +527,29 @@ object RemoveNoopUnion extends

[GitHub] [spark] maropu commented on a change in pull request #31680: [SPARK-34568][SQL] We should respect enableHiveSupport when initialize SparkSession

maropu commented on a change in pull request #31680: URL: https://github.com/apache/spark/pull/31680#discussion_r584495945 ## File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ## @@ -946,7 +946,11 @@ object SparkSession extends Logging { Spar

[GitHub] [spark] dongjoon-hyun commented on pull request #31690: [SPARK-34574][DOCS] Jekyll fails to generate Scala API docs for Scala 2.13

dongjoon-hyun commented on pull request #31690: URL: https://github.com/apache/spark/pull/31690#issuecomment-787724282 Thank you, @sarutak and @HyukjinKwon This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] SparkQA commented on pull request #31691: [SPARK-34575][SQL] Push down limit through window when partitionSpec is empty

SparkQA commented on pull request #31691: URL: https://github.com/apache/spark/pull/31691#issuecomment-787722123 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40160/ ---

[GitHub] [spark] viirya commented on a change in pull request #31656: [SPARK-34548][SQL] Remove unnecessary children from Union under Distince and Deduplicate

viirya commented on a change in pull request #31656: URL: https://github.com/apache/spark/pull/31656#discussion_r584493655 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala ## @@ -860,6 +860,22 @@ class DataFrameSetOperationsSuite ext

[GitHub] [spark] SparkQA removed a comment on pull request #31680: [SPARK-34568][SQL] We should respect enableHiveSupport when initialize SparkSession

SparkQA removed a comment on pull request #31680: URL: https://github.com/apache/spark/pull/31680#issuecomment-787603295 **[Test build #135572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135572/testReport)** for PR 31680 at commit [`f73421a`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31680: [SPARK-34568][SQL] We should respect enableHiveSupport when initialize SparkSession

SparkQA commented on pull request #31680: URL: https://github.com/apache/spark/pull/31680#issuecomment-787721078 **[Test build #135572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135572/testReport)** for PR 31680 at commit [`f73421a`](https://github.co

[GitHub] [spark] dongjoon-hyun commented on pull request #31692: [SPARK-34578][SQL][TESTS][test-maven] Ignore ORC encryption tests when ORC shim is loaded by old Hadoop library

dongjoon-hyun commented on pull request #31692: URL: https://github.com/apache/spark/pull/31692#issuecomment-787720705 Could you review this, @HyukjinKwon ? This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] peter-toth commented on a change in pull request #31682: [WIP][SPARK-34545][SQL] Fix row comparisons

peter-toth commented on a change in pull request #31682: URL: https://github.com/apache/spark/pull/31682#discussion_r584490309 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameStatSuite.scala ## @@ -436,7 +436,7 @@ class DataFrameStatSuite extends QueryTest w

[GitHub] [spark] peter-toth commented on a change in pull request #31682: [WIP][SPARK-34545][SQL] Fix row comparisons

peter-toth commented on a change in pull request #31682: URL: https://github.com/apache/spark/pull/31682#discussion_r584489967 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala ## @@ -447,7 +447,7 @@ trait Row extends Serializable { if (d1.co

[GitHub] [spark] sarutak edited a comment on pull request #31601: [SPARK-34484][SQL] Rename `map` to `mapAttr` in Catalyst DSL

sarutak edited a comment on pull request #31601: URL: https://github.com/apache/spark/pull/31601#issuecomment-787714489 > OK seems the ship is already sailed. Let's wait a bit more but I'm not sure we can change that decision... Yeah, I think so too. At least, symbol literals will be

[GitHub] [spark] SparkQA commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

SparkQA commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-787714868 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40158/ ---

[GitHub] [spark] sarutak edited a comment on pull request #31601: [SPARK-34484][SQL] Rename `map` to `mapAttr` in Catalyst DSL

sarutak edited a comment on pull request #31601: URL: https://github.com/apache/spark/pull/31601#issuecomment-787714489 > OK seems the ship is already sailed. Let's wait a bit more but I'm not sure we can change that decision... Yeah, I think so too. At least, symbol literals will be

[GitHub] [spark] sarutak commented on pull request #31601: [SPARK-34484][SQL] Rename `map` to `mapAttr` in Catalyst DSL

sarutak commented on pull request #31601: URL: https://github.com/apache/spark/pull/31601#issuecomment-787714489 > OK seems the ship is already sailed. Let's wait a bit more but I'm not sure we can change that decision... Yeah, I think so too. At least, symbol literal will be no long

[GitHub] [spark] cloud-fan commented on a change in pull request #31682: [WIP][SPARK-34545][SQL] Fix row comparisons

cloud-fan commented on a change in pull request #31682: URL: https://github.com/apache/spark/pull/31682#discussion_r584488733 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameStatSuite.scala ## @@ -436,7 +436,7 @@ class DataFrameStatSuite extends QueryTest wi

[GitHub] [spark] dongjoon-hyun opened a new pull request #31692: [SPARK-34578][SQL][TESTS][test-maven] Ignore ORC encryption tests when ORC shim is loaded by old Hadoop library

dongjoon-hyun opened a new pull request #31692: URL: https://github.com/apache/spark/pull/31692 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this

[GitHub] [spark] peter-toth commented on a change in pull request #31682: [WIP][SPARK-34545][SQL] Fix row comparisons

peter-toth commented on a change in pull request #31682: URL: https://github.com/apache/spark/pull/31682#discussion_r584488570 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/RowTest.scala ## @@ -108,6 +108,31 @@ class RowTest extends AnyFunSpec with Matchers {

[GitHub] [spark] yaooqinn commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

yaooqinn commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584488254 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionBuilderSuite.scala ## @@ -412,4 +415,32 @@ class SparkSessionBuilderSuite extends

[GitHub] [spark] cloud-fan commented on a change in pull request #31682: [WIP][SPARK-34545][SQL] Fix row comparisons

cloud-fan commented on a change in pull request #31682: URL: https://github.com/apache/spark/pull/31682#discussion_r584488221 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/Row.scala ## @@ -447,7 +447,7 @@ trait Row extends Serializable { if (d1.com

[GitHub] [spark] cloud-fan commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

cloud-fan commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584486820 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionBuilderSuite.scala ## @@ -412,4 +415,32 @@ class SparkSessionBuilderSuite extend

[GitHub] [spark] cloud-fan commented on pull request #31601: [SPARK-34484][SQL] Rename `map` to `mapAttr` in Catalyst DSL

cloud-fan commented on pull request #31601: URL: https://github.com/apache/spark/pull/31601#issuecomment-787709295 OK seems the ship is already sailed. Let's wait a bit more but I'm not sure we can change that decision... Th

[GitHub] [spark] yaooqinn commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

yaooqinn commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584485480 ## File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ## @@ -253,9 +253,8 @@ object SharedState extends Logging { v

[GitHub] [spark] viirya commented on a change in pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

viirya commented on a change in pull request #28715: URL: https://github.com/apache/spark/pull/28715#discussion_r584484987 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/GenerateExec.scala ## @@ -124,7 +124,7 @@ case class GenerateExec( } } - o

[GitHub] [spark] MaxGekk commented on a change in pull request #31676: [SPARK-34561][SQL] Fix drop/add columns from/to a dataset of v2 `DESCRIBE TABLE`

MaxGekk commented on a change in pull request #31676: URL: https://github.com/apache/spark/pull/31676#discussion_r584484615 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -347,7 +347,7 @@ case class DescribeRelation

[GitHub] [spark] cloud-fan commented on a change in pull request #31621: [SPARK-34498][SQL][TESTS] fix the remaining problems in #31560

cloud-fan commented on a change in pull request #31621: URL: https://github.com/apache/spark/pull/31621#discussion_r584484677 ## File path: sql/core/src/test/scala/org/apache/spark/sql/connector/SimpleWritableDataSource.scala ## @@ -41,11 +41,9 @@ import org.apache.spark.util.

[GitHub] [spark] cloud-fan commented on a change in pull request #31545: [SPARK-34417] [SQL] org.apache.spark.sql.DataFrameNaFunctions.fillMap(values: Seq[(String, Any)]) fails for column name having

cloud-fan commented on a change in pull request #31545: URL: https://github.com/apache/spark/pull/31545#discussion_r584484106 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala ## @@ -395,9 +395,9 @@ final class DataFrameNaFunctions private[s

[GitHub] [spark] yaooqinn commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

yaooqinn commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584484398 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionBuilderSuite.scala ## @@ -412,4 +415,32 @@ class SparkSessionBuilderSuite extends

[GitHub] [spark] sarutak edited a comment on pull request #31601: [SPARK-34484][SQL] Rename `map` to `mapAttr` in Catalyst DSL

sarutak edited a comment on pull request #31601: URL: https://github.com/apache/spark/pull/31601#issuecomment-787703342 No, but we get warning with Scala 2.13. EDIT: dotty seems not to support symbol literals. https://scalacenter.github.io/scala-3-migration-guide/docs/incompatibil

[GitHub] [spark] yaooqinn commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

yaooqinn commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584483797 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala ## @@ -424,7 +426,7 @@ object SetWarehouseLocationTest extend

[GitHub] [spark] viirya commented on a change in pull request #31682: [WIP][SPARK-34545][SQL] Fix row comparisons

viirya commented on a change in pull request #31682: URL: https://github.com/apache/spark/pull/31682#discussion_r584483705 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/RowTest.scala ## @@ -108,6 +108,31 @@ class RowTest extends AnyFunSpec with Matchers {

[GitHub] [spark] sarutak commented on pull request #31601: [SPARK-34484][SQL] Rename `map` to `mapAttr` in Catalyst DSL

sarutak commented on pull request #31601: URL: https://github.com/apache/spark/pull/31601#issuecomment-787703342 No, but we get warning with Scala 2.13. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] cloud-fan commented on pull request #31601: [SPARK-34484][SQL] Rename `map` to `mapAttr` in Catalyst DSL

cloud-fan commented on pull request #31601: URL: https://github.com/apache/spark/pull/31601#issuecomment-787702782 Is there a released Scala version that forbids the symbol's literal syntax? This is an automated message from

[GitHub] [spark] cloud-fan commented on pull request #31643: [SPARK-34534] Fix blockIds order when use FetchShuffleBlocks to fetch blocks

cloud-fan commented on pull request #31643: URL: https://github.com/apache/spark/pull/31643#issuecomment-787701759 Seems like there are two proposals: 1. regenerate the block ids when building the `FetchShuffleBlocks` request. 2. leverage the fact the block ids are ordered already.

[GitHub] [spark] SparkQA commented on pull request #31691: [SPARK-34575][SQL] Push down limit through window when partitionSpec is empty

SparkQA commented on pull request #31691: URL: https://github.com/apache/spark/pull/31691#issuecomment-787701049 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40160/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

AmplabJenkins removed a comment on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-787698737 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135575/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

AmplabJenkins commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-787698737 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135575/ -

[GitHub] [spark] SparkQA commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

SparkQA commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-787698217 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40158/ -

[GitHub] [spark] SparkQA removed a comment on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

SparkQA removed a comment on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-787622956 **[Test build #135575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135575/testReport)** for PR 31598 at commit [`3f795f5`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31598: [SPARK-34478][SQL] When build SparkSession, we should check config keys

SparkQA commented on pull request #31598: URL: https://github.com/apache/spark/pull/31598#issuecomment-787697464 **[Test build #135575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135575/testReport)** for PR 31598 at commit [`3f795f5`](https://github.co

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31318: [SPARK-34222][SQL] Enhance boolean simplification rule

AmplabJenkins removed a comment on pull request #31318: URL: https://github.com/apache/spark/pull/31318#issuecomment-787694455 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40161/

[GitHub] [spark] AmplabJenkins commented on pull request #31318: [SPARK-34222][SQL] Enhance boolean simplification rule

AmplabJenkins commented on pull request #31318: URL: https://github.com/apache/spark/pull/31318#issuecomment-787694455 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40161/ -

[GitHub] [spark] SparkQA commented on pull request #31318: [SPARK-34222][SQL] Enhance boolean simplification rule

SparkQA commented on pull request #31318: URL: https://github.com/apache/spark/pull/31318#issuecomment-787694433 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40161/ ---

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31690: [SPARK-34574][DOCS] Jekyll fails to generate Scala API docs for Scala 2.13

AmplabJenkins removed a comment on pull request #31690: URL: https://github.com/apache/spark/pull/31690#issuecomment-787693105 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40157/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

AmplabJenkins removed a comment on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-787693106 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

AmplabJenkins commented on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-787693106 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #31690: [SPARK-34574][DOCS] Jekyll fails to generate Scala API docs for Scala 2.13

AmplabJenkins commented on pull request #31690: URL: https://github.com/apache/spark/pull/31690#issuecomment-787693105 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40157/ -

[GitHub] [spark] cloud-fan commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

cloud-fan commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584475849 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala ## @@ -424,7 +426,7 @@ object SetWarehouseLocationTest exten

[GitHub] [spark] cloud-fan commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

cloud-fan commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584475526 ## File path: sql/core/src/test/scala/org/apache/spark/sql/SparkSessionBuilderSuite.scala ## @@ -412,4 +415,32 @@ class SparkSessionBuilderSuite extend

[GitHub] [spark] cloud-fan commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

cloud-fan commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584475139 ## File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ## @@ -264,12 +263,15 @@ object SharedState extends Logging {

[GitHub] [spark] cloud-fan commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

cloud-fan commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584475089 ## File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ## @@ -253,9 +253,8 @@ object SharedState extends Logging {

[GitHub] [spark] SparkQA commented on pull request #31690: [SPARK-34574][DOCS] Jekyll fails to generate Scala API docs for Scala 2.13

SparkQA commented on pull request #31690: URL: https://github.com/apache/spark/pull/31690#issuecomment-787690003 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40157/ ---

[GitHub] [spark] HyukjinKwon closed pull request #31690: [SPARK-34574][DOCS] Jekyll fails to generate Scala API docs for Scala 2.13

HyukjinKwon closed pull request #31690: URL: https://github.com/apache/spark/pull/31690 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on pull request #31690: [SPARK-34574][DOCS] Jekyll fails to generate Scala API docs for Scala 2.13

HyukjinKwon commented on pull request #31690: URL: https://github.com/apache/spark/pull/31690#issuecomment-787689397 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [spark] SparkQA commented on pull request #31318: [SPARK-34222][SQL] Enhance boolean simplification rule

SparkQA commented on pull request #31318: URL: https://github.com/apache/spark/pull/31318#issuecomment-787688874 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40161/ -

[GitHub] [spark] SparkQA commented on pull request #31690: [SPARK-34574][DOCS] Jekyll fails to generate Scala API docs for Scala 2.13

SparkQA commented on pull request #31690: URL: https://github.com/apache/spark/pull/31690#issuecomment-787686886 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40157/ -

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31675: [SPARK-34560][SQL] Generate unique output attributes in the `SHOW TABLES` logical node

AngersZh commented on a change in pull request #31675: URL: https://github.com/apache/spark/pull/31675#discussion_r584470863 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala ## @@ -334,8 +334,8 @@ class DataSour

[GitHub] [spark] MaxGekk commented on a change in pull request #31675: [SPARK-34560][SQL] Generate unique output attributes in the `SHOW TABLES` logical node

MaxGekk commented on a change in pull request #31675: URL: https://github.com/apache/spark/pull/31675#discussion_r584470889 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -491,13 +491,10 @@ case class RenameTable(

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #31676: [SPARK-34561][SQL] Fix drop/add columns from/to a dataset of v2 `DESCRIBE TABLE`

AngersZh edited a comment on pull request #31676: URL: https://github.com/apache/spark/pull/31676#issuecomment-787670462 > Looks like we should fix all other instances too? But could be done separately. Yea, have fix some of this. But maybe there are still incorrect instance.

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #31676: [SPARK-34561][SQL] Fix drop/add columns from/to a dataset of v2 `DESCRIBE TABLE`

AngersZh edited a comment on pull request #31676: URL: https://github.com/apache/spark/pull/31676#issuecomment-787670462 > Looks like we should fix all other instances too? But could be done separately. Yea, have fix some of this. But maybe there are still incorrect instance.

[GitHub] [spark] MaxGekk commented on a change in pull request #31675: [SPARK-34560][SQL] Generate unique output attributes in the `SHOW TABLES` logical node

MaxGekk commented on a change in pull request #31675: URL: https://github.com/apache/spark/pull/31675#discussion_r584469951 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala ## @@ -334,8 +334,8 @@ class DataSourceV2S

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31675: [SPARK-34560][SQL] Generate unique output attributes in the `SHOW TABLES` logical node

AngersZh commented on a change in pull request #31675: URL: https://github.com/apache/spark/pull/31675#discussion_r584469418 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala ## @@ -334,8 +334,8 @@ class DataSour

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31680: [SPARK-34568][SQL] We should respect enableHiveSupport when initialize SparkSession

AngersZh commented on a change in pull request #31680: URL: https://github.com/apache/spark/pull/31680#discussion_r584469068 ## File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ## @@ -946,7 +946,11 @@ object SparkSession extends Logging {

[GitHub] [spark] SparkQA removed a comment on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

SparkQA removed a comment on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-787673020 **[Test build #135578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135578/testReport)** for PR 28715 at commit [`d6f8459`](https://gi

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

SparkQA commented on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-787676740 **[Test build #135578 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135578/testReport)** for PR 28715 at commit [`d6f8459`](https://github.co

[GitHub] [spark] yaooqinn commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

yaooqinn commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584465609 ## File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ## @@ -253,9 +253,8 @@ object SharedState extends Logging { v

[GitHub] [spark] SparkQA commented on pull request #28715: [SPARK-31897][SQL] Enable codegen for GenerateExec

SparkQA commented on pull request #28715: URL: https://github.com/apache/spark/pull/28715#issuecomment-787673020 **[Test build #135578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135578/testReport)** for PR 28715 at commit [`d6f8459`](https://github.com

[GitHub] [spark] yaooqinn commented on a change in pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

yaooqinn commented on a change in pull request #31671: URL: https://github.com/apache/spark/pull/31671#discussion_r584465609 ## File path: sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala ## @@ -253,9 +253,8 @@ object SharedState extends Logging { v

[GitHub] [spark] AngersZhuuuu commented on pull request #31676: [SPARK-34561][SQL] Fix drop/add columns from/to a dataset of v2 `DESCRIBE TABLE`

AngersZh commented on pull request #31676: URL: https://github.com/apache/spark/pull/31676#issuecomment-787670462 > Looks like we should fix all other instances too? But could be done separately. Yea, have fix some of this. But maybe there are still incorrect instance. ---

[GitHub] [spark] yaooqinn commented on pull request #31671: [SPARK-34558][SQL] warehouse path should be qualified ahead of populating and use

yaooqinn commented on pull request #31671: URL: https://github.com/apache/spark/pull/31671#issuecomment-787670358 > What's concrete steps to reproduce this error by users? > > ``` > Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolu

[GitHub] [spark] SparkQA commented on pull request #31690: [SPARK-34574][DOCS] Jekyll fails to generate Scala API docs for Scala 2.13

SparkQA commented on pull request #31690: URL: https://github.com/apache/spark/pull/31690#issuecomment-787669760 **[Test build #135576 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135576/testReport)** for PR 31690 at commit [`f2d47d1`](https://github.com

[GitHub] [spark] cloud-fan commented on a change in pull request #31675: [SPARK-34560][SQL] Generate unique output attributes in the `SHOW TABLES` logical node

cloud-fan commented on a change in pull request #31675: URL: https://github.com/apache/spark/pull/31675#discussion_r584463268 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -491,13 +491,10 @@ case class RenameTable(

[GitHub] [spark] cloud-fan commented on a change in pull request #31654: [SPARK-34547][SQL] Only use metadata columns for resolution as last resort

cloud-fan commented on a change in pull request #31654: URL: https://github.com/apache/spark/pull/31654#discussion_r584462469 ## File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala ## @@ -2492,6 +2492,35 @@ class DataSourceV2SQLSuite

[GitHub] [spark] Swinky commented on a change in pull request #31318: [SPARK-34222][SQL] Enhance boolean simplification rule

Swinky commented on a change in pull request #31318: URL: https://github.com/apache/spark/pull/31318#discussion_r584462177 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala ## @@ -366,22 +367,34 @@ object BooleanSimplification

[GitHub] [spark] SparkQA commented on pull request #31318: [SPARK-34222][SQL] Enhance boolean simplification rule

SparkQA commented on pull request #31318: URL: https://github.com/apache/spark/pull/31318#issuecomment-787666371 **[Test build #135580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135580/testReport)** for PR 31318 at commit [`7ce6731`](https://github.com

[GitHub] [spark] MaxGekk commented on a change in pull request #31675: [SPARK-34560][SQL] Generate unique output attributes in the `SHOW TABLES` logical node

MaxGekk commented on a change in pull request #31675: URL: https://github.com/apache/spark/pull/31675#discussion_r584461988 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala ## @@ -491,13 +491,10 @@ case class RenameTable(

[GitHub] [spark] SparkQA commented on pull request #31691: [SPARK-34575][SQL] Push down limit through window when partitionSpec is empty

SparkQA commented on pull request #31691: URL: https://github.com/apache/spark/pull/31691#issuecomment-787666103 **[Test build #135579 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135579/testReport)** for PR 31691 at commit [`d04b678`](https://github.com

[GitHub] [spark] cloud-fan commented on a change in pull request #31654: [SPARK-34547][SQL] Only use metadata columns for resolution as last resort

cloud-fan commented on a change in pull request #31654: URL: https://github.com/apache/spark/pull/31654#discussion_r584461515 ## File path: sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala ## @@ -2492,6 +2492,35 @@ class DataSourceV2SQLSuite

[GitHub] [spark] maropu commented on a change in pull request #31680: [SPARK-34568][SQL] We should respect enableHiveSupport when initialize SparkSession

maropu commented on a change in pull request #31680: URL: https://github.com/apache/spark/pull/31680#discussion_r584461453 ## File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ## @@ -946,7 +946,11 @@ object SparkSession extends Logging { Spar

[GitHub] [spark] wangyum opened a new pull request #31691: [SPARK-34575][SQL] Push down limit through window when partitionSpec is empty

wangyum opened a new pull request #31691: URL: https://github.com/apache/spark/pull/31691 ### What changes were proposed in this pull request? Push down limit through `Window` when partitionSpec is empty and window function is range based. This is a real case from production:

[GitHub] [spark] cloud-fan commented on pull request #31684: [SPARK-34571][SQL][CORE] Provide a more convenient way to deprecate/remove/alternate configs

cloud-fan commented on pull request #31684: URL: https://github.com/apache/spark/pull/31684#issuecomment-787664414 Agree with @HyukjinKwon to do deprecation first. For the removed ones, we don't really need to keep the config entries for them anymore. We just need the names of removed conf

  1   2   3   4   5   >