[GitHub] [spark] cloud-fan commented on a change in pull request #33200: [SPARK-36006][SQL] Migrate ALTER TABLE ... ADD/REPLACE COLUMNS commands to use UnresolvedTable to resolve the identifier

cloud-fan commented on a change in pull request #33200: URL: https://github.com/apache/spark/pull/33200#discussion_r672827401 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -3574,15 +3568,64 @@ class Analyzer(override val

[GitHub] [spark] SparkQA commented on pull request #33430: [SPARK-36046][SQL][FOLLOWUP] Implement prettyName for MakeTimestampNTZ and MakeTimestampLTZ

SparkQA commented on pull request #33430: URL: https://github.com/apache/spark/pull/33430#issuecomment-883090499 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45810/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33431: [SPARK-36221][SQL] Make sure CustomShuffleReaderExec has at least one partition

SparkQA commented on pull request #33431: URL: https://github.com/apache/spark/pull/33431#issuecomment-883090213 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45809/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA removed a comment on pull request #33424: [SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec

SparkQA removed a comment on pull request #33424: URL: https://github.com/apache/spark/pull/33424#issuecomment-882968356 **[Test build #141288 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141288/testReport)** for PR 33424 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33350: [SPARK-36136][SQL][TESTS] Refactor PruneFileSourcePartitionsSuite etc to a different package

AmplabJenkins removed a comment on pull request #33350: URL: https://github.com/apache/spark/pull/33350#issuecomment-883087045 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45807/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

AmplabJenkins removed a comment on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-883087051 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45808/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33428: [SPARK-36220][PYTHON] Fix pyspark.sql.types.Row type annotation

AmplabJenkins removed a comment on pull request #33428: URL: https://github.com/apache/spark/pull/33428#issuecomment-883011701 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33424: [SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec

AmplabJenkins removed a comment on pull request #33424: URL: https://github.com/apache/spark/pull/33424#issuecomment-883087048 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141288/

[GitHub] [spark] cloud-fan commented on a change in pull request #33200: [SPARK-36006][SQL] Migrate ALTER TABLE ... ADD/REPLACE COLUMNS commands to use UnresolvedTable to resolve the identifier

cloud-fan commented on a change in pull request #33200: URL: https://github.com/apache/spark/pull/33200#discussion_r672821019 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -3574,15 +3568,64 @@ class Analyzer(override val

[GitHub] [spark] SparkQA commented on pull request #33410: [WIP][SPARK-36204][INFRA][BUILD] Deduplicate Scala 2.13 daily build

SparkQA commented on pull request #33410: URL: https://github.com/apache/spark/pull/33410#issuecomment-883088778 **[Test build #141303 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141303/testReport)** for PR 33410 at commit

[GitHub] [spark] SparkQA commented on pull request #33416: [SPARK-36207][PYTHON] Expose databaseExists in pyspark.sql.catalog

SparkQA commented on pull request #33416: URL: https://github.com/apache/spark/pull/33416#issuecomment-883088693 **[Test build #141302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141302/testReport)** for PR 33416 at commit

[GitHub] [spark] SparkQA commented on pull request #33429: [SPARK-36217][SQL] Rename CustomShuffleReader and OptimizeLocalShuffleReader in AQE

SparkQA commented on pull request #33429: URL: https://github.com/apache/spark/pull/33429#issuecomment-883088620 **[Test build #141300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141300/testReport)** for PR 33429 at commit

[GitHub] [spark] SparkQA commented on pull request #33428: [SPARK-36220][PYTHON] Fix pyspark.sql.types.Row type annotation

SparkQA commented on pull request #33428: URL: https://github.com/apache/spark/pull/33428#issuecomment-883088661 **[Test build #141301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141301/testReport)** for PR 33428 at commit

[GitHub] [spark] SparkQA commented on pull request #33432: [SPARK-32709][SQL] Support writing Hive bucketed table (Parquet/ORC format with Hive hash)

SparkQA commented on pull request #33432: URL: https://github.com/apache/spark/pull/33432#issuecomment-883088569 **[Test build #141299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141299/testReport)** for PR 33432 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

AmplabJenkins commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-883087051 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45808/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33350: [SPARK-36136][SQL][TESTS] Refactor PruneFileSourcePartitionsSuite etc to a different package

AmplabJenkins commented on pull request #33350: URL: https://github.com/apache/spark/pull/33350#issuecomment-883087045 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45807/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33424: [SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec

AmplabJenkins commented on pull request #33424: URL: https://github.com/apache/spark/pull/33424#issuecomment-883087048 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141288/ -- This

[GitHub] [spark] SparkQA commented on pull request #33429: [SPARK-36217][SQL] Rename CustomShuffleReader and OptimizeLocalShuffleReader in AQE

SparkQA commented on pull request #33429: URL: https://github.com/apache/spark/pull/33429#issuecomment-883084988 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45811/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33424: [SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec

SparkQA commented on pull request #33424: URL: https://github.com/apache/spark/pull/33424#issuecomment-883082141 **[Test build #141288 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141288/testReport)** for PR 33424 at commit

[GitHub] [spark] otterc commented on a change in pull request #33425: [SPARK-32919][FOLLOW-UP] Filter out driver in the merger locations and fix the return type of RemoveShufflePushMergerLocations

otterc commented on a change in pull request #33425: URL: https://github.com/apache/spark/pull/33425#discussion_r672821698 ## File path: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ## @@ -2093,6 +2093,9 @@ class BlockManagerSuite extends SparkFunSuite

[GitHub] [spark] cloud-fan commented on a change in pull request #33200: [SPARK-36006][SQL] Migrate ALTER TABLE ... ADD/REPLACE COLUMNS commands to use UnresolvedTable to resolve the identifier

cloud-fan commented on a change in pull request #33200: URL: https://github.com/apache/spark/pull/33200#discussion_r672821019 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -3574,15 +3568,64 @@ class Analyzer(override val

[GitHub] [spark] HyukjinKwon commented on pull request #33429: [SPARK-36217][SQL] Rename CustomShuffleReader and OptimizeLocalShuffleReader in AQE

HyukjinKwon commented on pull request #33429: URL: https://github.com/apache/spark/pull/33429#issuecomment-883075964 let me rebase. seems like it couldn't detect my GitHub actions job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] c21 commented on pull request #33432: [SPARK-32709][SQL] Support writing Hive bucketed table (Parquet/ORC format with Hive hash)

c21 commented on pull request #33432: URL: https://github.com/apache/spark/pull/33432#issuecomment-883074400 cc @cloud-fan could you help take a look when you have time? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] c21 opened a new pull request #33432: [SPARK-32709][SQL] Support writing Hive bucketed table (Parquet/ORC format with Hive hash)

c21 opened a new pull request #33432: URL: https://github.com/apache/spark/pull/33432 ### What changes were proposed in this pull request? This is a re-work of https://github.com/apache/spark/pull/30003, here we add support for writing Hive bucketed table with Parquet/ORC

[GitHub] [spark] SparkQA commented on pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-883072761 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45808/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33350: [SPARK-36136][SQL][TESTS] Refactor PruneFileSourcePartitionsSuite etc to a different package

SparkQA commented on pull request #33350: URL: https://github.com/apache/spark/pull/33350#issuecomment-883071887 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45807/ -- This is an automated message from the

[GitHub] [spark] dongjoon-hyun commented on pull request #33409: [SPARK-36201][SQL][FOLLOWUP] Schema check should check inner field too

dongjoon-hyun commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-883069750 It seems that the master branch's Java 17 job is suffering with the same reason. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #33409: [SPARK-36201][SQL][FOLLOWUP] Schema check should check inner field too

dongjoon-hyun edited a comment on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-883069206 The error code is the following. It looks like OOM happening again. - https://github.com/AngersZh/spark/runs/3110053861?check_suite_focus=true ```

[GitHub] [spark] dongjoon-hyun commented on pull request #33409: [SPARK-36201][SQL][FOLLOWUP] Schema check should check inner field too

dongjoon-hyun commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-883069206 The error code is the following. It looks like OOM happening again. ``` ./build/mvn: line 178: 1699 Killed "${MVN_BIN}" "$@"

[GitHub] [spark] dongjoon-hyun commented on pull request #33409: [SPARK-36201][SQL][FOLLOWUP] Schema check should check inner field too

dongjoon-hyun commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-883067722 Wow, the GitHub Action failures look really weird. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] jhu-chang commented on a change in pull request #33263: [SPARK-35027][CORE] Close the inputStream in FileAppender when writin…

jhu-chang commented on a change in pull request #33263: URL: https://github.com/apache/spark/pull/33263#discussion_r672813847 ## File path: core/src/main/scala/org/apache/spark/util/logging/FileAppender.scala ## @@ -76,7 +80,13 @@ private[spark] class FileAppender(inputStream:

[GitHub] [spark] viirya commented on a change in pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

viirya commented on a change in pull request #33239: URL: https://github.com/apache/spark/pull/33239#discussion_r672812727 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/metric/CustomMetrics.scala ## @@ -51,7 +51,7 @@ object CustomMetrics {

[GitHub] [spark] viirya commented on a change in pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

viirya commented on a change in pull request #33239: URL: https://github.com/apache/spark/pull/33239#discussion_r672812642 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriterMetricSuite.scala ## @@ -0,0 +1,96 @@ +/* + *

[GitHub] [spark] HyukjinKwon commented on pull request #33428: [SPARK-36220][PYTHON] Fix pyspark.sql.types.Row type annotation

HyukjinKwon commented on pull request #33428: URL: https://github.com/apache/spark/pull/33428#issuecomment-883060656 Jenkins, ok to test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

dongjoon-hyun commented on a change in pull request #33239: URL: https://github.com/apache/spark/pull/33239#discussion_r672811174 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriterMetricSuite.scala ## @@ -0,0 +1,96 @@ +/* + *

[GitHub] [spark] AmplabJenkins commented on pull request #33422: [SPARK-34806][SQL] Add Observation helper for Dataset.observe

AmplabJenkins commented on pull request #33422: URL: https://github.com/apache/spark/pull/33422#issuecomment-883059530 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141285/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #33422: [SPARK-34806][SQL] Add Observation helper for Dataset.observe

SparkQA removed a comment on pull request #33422: URL: https://github.com/apache/spark/pull/33422#issuecomment-882962473 **[Test build #141285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141285/testReport)** for PR 33422 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

dongjoon-hyun commented on a change in pull request #33239: URL: https://github.com/apache/spark/pull/33239#discussion_r672810180 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/metric/CustomMetrics.scala ## @@ -51,7 +51,7 @@ object CustomMetrics {

[GitHub] [spark] SparkQA commented on pull request #33422: [SPARK-34806][SQL] Add Observation helper for Dataset.observe

SparkQA commented on pull request #33422: URL: https://github.com/apache/spark/pull/33422#issuecomment-883058475 **[Test build #141285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141285/testReport)** for PR 33422 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

dongjoon-hyun commented on a change in pull request #33239: URL: https://github.com/apache/spark/pull/33239#discussion_r672809311 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/metric/CustomMetrics.scala ## @@ -51,7 +51,7 @@ object CustomMetrics {

[GitHub] [spark] mridulm commented on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

mridulm commented on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-883057074 Thanks for the clarifications ! This sounds good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] mridulm commented on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better

mridulm commented on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-883056728 +CC @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] mridulm commented on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better

mridulm commented on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-883056445 Merged to master and branch-3.2 Thanks for working on this @zhouyejoe ! Thanks for all the reviews @Ngone51, @otterc, @venkata91 :-) -- This is an automated message from

[GitHub] [spark] asfgit closed pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way

asfgit closed pull request #33078: URL: https://github.com/apache/spark/pull/33078 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] SparkQA commented on pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-883053190 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45808/ -- This is an automated message from the Apache

[GitHub] [spark] tobiasedwards commented on pull request #33428: [SPARK-36220][PYTHON] Fix pyspark.sql.types.Row type annotation

tobiasedwards commented on pull request #33428: URL: https://github.com/apache/spark/pull/33428#issuecomment-883053001 There we go, that should be better. When I botched the rebase the bot added some incorrect labels though, are you able to remove them, @HyukjinKwon? Thanks

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #33429: [SPARK-36217][SQL] Rename CustomShuffleReader and OptimizeLocalShuffleReader in AQE

dongjoon-hyun commented on a change in pull request #33429: URL: https://github.com/apache/spark/pull/33429#discussion_r672805734 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CoalesceShufflePartitions.scala ## @@ -88,23 +88,23 @@ case class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33409: [SPARK-36201][SQL] Schema check should check inner field too

AmplabJenkins removed a comment on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-883051150 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45805/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

AmplabJenkins removed a comment on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883051152 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45804/

[GitHub] [spark] SparkQA commented on pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-883052445 **[Test build #141298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141298/testReport)** for PR 33239 at commit

[GitHub] [spark] SparkQA commented on pull request #33429: [SPARK-36217][SQL] Rename CustomShuffleReader and OptimizeLocalShuffleReader in AQE

SparkQA commented on pull request #33429: URL: https://github.com/apache/spark/pull/33429#issuecomment-883052316 **[Test build #141297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141297/testReport)** for PR 33429 at commit

[GitHub] [spark] SparkQA commented on pull request #33430: [SPARK-36046][SQL][FOLLOWUP] Implement prettyName for MakeTimestampNTZ and MakeTimestampLTZ

SparkQA commented on pull request #33430: URL: https://github.com/apache/spark/pull/33430#issuecomment-883052246 **[Test build #141296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141296/testReport)** for PR 33430 at commit

[GitHub] [spark] SparkQA commented on pull request #33431: [SPARK-36221][SQL] Make sure CustomShuffleReaderExec has at least one partition

SparkQA commented on pull request #33431: URL: https://github.com/apache/spark/pull/33431#issuecomment-883052250 **[Test build #141295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141295/testReport)** for PR 33431 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

AmplabJenkins commented on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883051152 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45804/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33409: [SPARK-36201][SQL] Schema check should check inner field too

AmplabJenkins commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-883051150 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45805/ --

[GitHub] [spark] SparkQA commented on pull request #33350: [SPARK-36136][SQL][TESTS] Refactor PruneFileSourcePartitionsSuite etc to a different package

SparkQA commented on pull request #33350: URL: https://github.com/apache/spark/pull/33350#issuecomment-883050392 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45807/ -- This is an automated message from the Apache

[GitHub] [spark] tobiasedwards commented on pull request #33428: [SPARK-36220][PYTHON] Fix pyspark.sql.types.Row type annotation

tobiasedwards commented on pull request #33428: URL: https://github.com/apache/spark/pull/33428#issuecomment-883049975 Whoops I think I've messed up my rebase, give me a minute -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] HyukjinKwon commented on pull request #33429: [SPARK-36217][SQL] Rename CustomShuffleReader and OptimizeLocalShuffleReader in AQE

HyukjinKwon commented on pull request #33429: URL: https://github.com/apache/spark/pull/33429#issuecomment-883048266 cc @ulysses-you too FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33422: [SPARK-34806][SQL] Add Observation helper for Dataset.observe

HyukjinKwon commented on a change in pull request #33422: URL: https://github.com/apache/spark/pull/33422#discussion_r672800804 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33422: [SPARK-34806][SQL] Add Observation helper for Dataset.observe

HyukjinKwon commented on a change in pull request #33422: URL: https://github.com/apache/spark/pull/33422#discussion_r672800470 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [spark] HyukjinKwon commented on a change in pull request #33422: [SPARK-34806][SQL] Add Observation helper for Dataset.observe

HyukjinKwon commented on a change in pull request #33422: URL: https://github.com/apache/spark/pull/33422#discussion_r672800470 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [spark] HyukjinKwon commented on pull request #33429: [SPARK-36217][SQL] Rename CustomShuffleReader and OptimizeLocalShuffleReader in AQE

HyukjinKwon commented on pull request #33429: URL: https://github.com/apache/spark/pull/33429#issuecomment-883046619 cc @cloud-fan and @maryannxue can you take a look please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] viirya commented on pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

viirya commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-883045610 @dongjoon-hyun and @gengliangwang Thanks for reviewing. Please take another look on the suggested change/new tests. Thanks! -- This is an automated message from the Apache Git

[GitHub] [spark] viirya commented on pull request #30565: [WIP][SPARK-33625][SQL] Subexpression elimination for whole-stage codegen in Filter

viirya commented on pull request #30565: URL: https://github.com/apache/spark/pull/30565#issuecomment-883045221 I think it is much easier to solve it at query optimization (i.e. by the optimizer), instead of at codegen. It also looks like query optimization problem instead of codegen.

[GitHub] [spark] beliefer commented on pull request #33430: [SPARK-36046][SQL][FOLLOWUP] Implement prettyName for MakeTimestampNTZ and MakeTimestampLTZ

beliefer commented on pull request #33430: URL: https://github.com/apache/spark/pull/33430#issuecomment-883043550 > > This PR fix the incorrect alias usecase. > > @beliefer I wouldn't say that is incorrect..implementing `prettyName` is more reliable. OK. I updated the

[GitHub] [spark] gengliangwang commented on pull request #33430: [SPARK-36046][SQL][FOLLOWUP] Implement prettyName for MakeTimestampNTZ and MakeTimestampLTZ

gengliangwang commented on pull request #33430: URL: https://github.com/apache/spark/pull/33430#issuecomment-883042925 > This PR fix the incorrect alias usecase. @beliefer I wouldn't say that is incorrect..implementing `prettyName` is more reliable. -- This is an automated

[GitHub] [spark] ulysses-you opened a new pull request #33431: [SPARK-36221][SQL] Make sure CustomShuffleReaderExec has at least one partition

ulysses-you opened a new pull request #33431: URL: https://github.com/apache/spark/pull/33431 ### What changes were proposed in this pull request? * Add non-empty partition check in `CustomShuffleReaderExec` * Make sure `OptimizeLocalShuffleReader` doesn't return empty

[GitHub] [spark] HyukjinKwon edited a comment on pull request #33428: [SPARK-36220][PYTHON] Fix pyspark.sql.types.Row type annotation

HyukjinKwon edited a comment on pull request #33428: URL: https://github.com/apache/spark/pull/33428#issuecomment-883039820 ah actually this is the limitation .. I can;t retrigger the test because it belongs to your repo :-) .. can you rebase and push it again? e.g.) `git checkout

[GitHub] [spark] HyukjinKwon commented on pull request #33428: [SPARK-36220][PYTHON] Fix pyspark.sql.types.Row type annotation

HyukjinKwon commented on pull request #33428: URL: https://github.com/apache/spark/pull/33428#issuecomment-883039820 ah actually this is the limitation .. I can;t retrigger the test because it belongs to your repo :-) .. can you rebase and push it again? e.g.) `git checkout

[GitHub] [spark] HyukjinKwon closed pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

HyukjinKwon closed pull request #33427: URL: https://github.com/apache/spark/pull/33427 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] HyukjinKwon commented on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

HyukjinKwon commented on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883038497 Merged to master, branch-3.2, branch-3.1 and branch-3.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] viirya commented on a change in pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

viirya commented on a change in pull request #33239: URL: https://github.com/apache/spark/pull/33239#discussion_r672793126 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala ## @@ -41,7 +42,8 @@ import

[GitHub] [spark] beliefer opened a new pull request #33430: [SPARK-36046][SQL][FOLLOWUP] Implement prettyName for MakeTimestampNTZ and MakeTimestampLTZ

beliefer opened a new pull request #33430: URL: https://github.com/apache/spark/pull/33430 ### What changes were proposed in this pull request? This PR fix the incorrect use alias for `MakeTimestampNTZ` and `MakeTimestampLTZ` based on the discussion show below

[GitHub] [spark] HyukjinKwon commented on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

HyukjinKwon commented on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883038320 Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] SparkQA commented on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

SparkQA commented on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883038157 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45804/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33409: [SPARK-36201][SQL] Schema check should check inner field too

SparkQA commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-883037465 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45805/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path

SparkQA commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-883036620 **[Test build #141294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141294/testReport)** for PR 33239 at commit

[GitHub] [spark] HyukjinKwon opened a new pull request #33429: [SPARK-36217][SQL] Rename CustomShuffleReader and OptimizeLocalShuffleReader

HyukjinKwon opened a new pull request #33429: URL: https://github.com/apache/spark/pull/33429 ### What changes were proposed in this pull request? This PR proposes to rename: - Rename `*Reader`/`*reader` to `*Read`/`*read` for rules and execution plan (user-facing doc/config

[GitHub] [spark] SparkQA commented on pull request #33350: [SPARK-36136][SQL][TESTS] Refactor PruneFileSourcePartitionsSuite etc to a different package

SparkQA commented on pull request #33350: URL: https://github.com/apache/spark/pull/33350#issuecomment-883031312 **[Test build #141293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141293/testReport)** for PR 33350 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in

SparkQA removed a comment on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-882962666 **[Test build #141286 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141286/testReport)** for PR 33078 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the sta

AmplabJenkins removed a comment on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-883030102 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141286/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

AmplabJenkins removed a comment on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-883030099 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45806/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

AmplabJenkins removed a comment on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883030098 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141290/

[GitHub] [spark] cfmcgrady commented on a change in pull request #33212: [SPARK-35912][SQL] Fix nullability of `spark.read.json`

cfmcgrady commented on a change in pull request #33212: URL: https://github.com/apache/spark/pull/33212#discussion_r672787073 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala ## @@ -405,10 +405,18 @@ class JacksonParser(

[GitHub] [spark] AmplabJenkins commented on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

AmplabJenkins commented on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-883030099 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45806/ --

[GitHub] [spark] AmplabJenkins commented on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

AmplabJenkins commented on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883030098 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141290/ -- This

[GitHub] [spark] AmplabJenkins commented on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a

AmplabJenkins commented on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-883030102 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141286/ -- This

[GitHub] [spark] gengliangwang commented on pull request #32401: [SPARK-35276][CORE] Calculate checksum for shuffle data and write as checksum file

gengliangwang commented on pull request #32401: URL: https://github.com/apache/spark/pull/32401#issuecomment-883029244 @Ngone51 Yes let's see if we can make it before 3.2. Thanks for the work! -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] HyukjinKwon commented on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

HyukjinKwon commented on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883028518 test failures in GA should be unrelated. @dongjoon-hyun, mind taking a quick look please? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] SparkQA commented on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

SparkQA commented on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883027144 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45804/ -- This is an automated message from the Apache

[GitHub] [spark] yaooqinn commented on a change in pull request #33424: [SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec

yaooqinn commented on a change in pull request #33424: URL: https://github.com/apache/spark/pull/33424#discussion_r672783259 ## File path: sql/core/src/test/resources/sql-tests/results/describe.sql.out ## @@ -324,6 +324,37 @@ Location [not included in

[GitHub] [spark] SparkQA commented on pull request #33409: [SPARK-36201][SQL] Schema check should check inner field too

SparkQA commented on pull request #33409: URL: https://github.com/apache/spark/pull/33409#issuecomment-883027098 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45805/ -- This is an automated message from the Apache

[GitHub] [spark] SparkQA commented on pull request #33078: [SPARK-35546][Shuffle] Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better

SparkQA commented on pull request #33078: URL: https://github.com/apache/spark/pull/33078#issuecomment-883026671 **[Test build #141286 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141286/testReport)** for PR 33078 at commit

[GitHub] [spark] SparkQA commented on pull request #33352: [SPARK-34952][SQL] DSv2 Aggregate push down APIs

SparkQA commented on pull request #33352: URL: https://github.com/apache/spark/pull/33352#issuecomment-883026630 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45806/ --

[GitHub] [spark] tobiasedwards commented on pull request #33428: [SPARK-36220][PYTHON] Fix pyspark.sql.types.Row type annotation

tobiasedwards commented on pull request #33428: URL: https://github.com/apache/spark/pull/33428#issuecomment-883023465 Hey @HyukjinKwon, I've added a Jira ticket here: [SPARK-36220](https://issues.apache.org/jira/browse/SPARK-36220) and enabled GitHub Actions on my forked repo. Is there

[GitHub] [spark] ulysses-you commented on a change in pull request #33188: [SPARK-35989][SQL] Only remove redundant shuffle if shuffle origin is REPARTITION_BY_COL in AQE

ulysses-you commented on a change in pull request #33188: URL: https://github.com/apache/spark/pull/33188#discussion_r672779057 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala ## @@ -250,7 +250,12 @@ object

[GitHub] [spark] cloud-fan commented on a change in pull request #33424: [SPARK-36213][SQL] Normalize PartitionSpec for Describe Table Command with PartitionSpec

cloud-fan commented on a change in pull request #33424: URL: https://github.com/apache/spark/pull/33424#discussion_r672778569 ## File path: sql/core/src/test/resources/sql-tests/results/describe.sql.out ## @@ -324,6 +324,37 @@ Location [not included in

[GitHub] [spark] cloud-fan commented on a change in pull request #33422: [SPARK-34806][SQL] Add Observation helper for Dataset.observe

cloud-fan commented on a change in pull request #33422: URL: https://github.com/apache/spark/pull/33422#discussion_r672777257 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Observation.scala ## @@ -0,0 +1,150 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [spark] SparkQA removed a comment on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

SparkQA removed a comment on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883011237 **[Test build #141290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141290/testReport)** for PR 33427 at commit

[GitHub] [spark] SparkQA commented on pull request #33427: [SPARK-36216][PYTHON][TESTS] Increase timeout for StreamingLinearRegressionWithTests. test_parameter_convergence

SparkQA commented on pull request #33427: URL: https://github.com/apache/spark/pull/33427#issuecomment-883019062 **[Test build #141290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141290/testReport)** for PR 33427 at commit

  1   2   3   4   5   6   7   8   9   10   >