[GitHub] [spark] SparkQA removed a comment on pull request #31245: [SPARK-34157][SQL] Unify output of SHOW TABLES and pass output attributes properly

SparkQA removed a comment on pull request #31245: URL: https://github.com/apache/spark/pull/31245#issuecomment-772186623 **[Test build #134805 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134805/testReport)** for PR 31245 at commit [`350bf94`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31245: [SPARK-34157][SQL] Unify output of SHOW TABLES and pass output attributes properly

SparkQA commented on pull request #31245: URL: https://github.com/apache/spark/pull/31245#issuecomment-772312717 **[Test build #134805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134805/testReport)** for PR 31245 at commit [`350bf94`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #31454: [WIP] Skip zinc start and shutdown in aarch64 os.

AmplabJenkins commented on pull request #31454: URL: https://github.com/apache/spark/pull/31454#issuecomment-772312572 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To resp

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

AmplabJenkins removed a comment on pull request #31378: URL: https://github.com/apache/spark/pull/31378#issuecomment-772311963 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134804/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31447: [SPARK-33726][SQL][FOLLOW-UP] Add assertion to FixedLengthRowBasedKeyValueBatch

AmplabJenkins removed a comment on pull request #31447: URL: https://github.com/apache/spark/pull/31447#issuecomment-772311973 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134803/ -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31451: [WIP][SPARK-34338][SQL] Report metrics from Datasource v2 scan

AmplabJenkins removed a comment on pull request #31451: URL: https://github.com/apache/spark/pull/31451#issuecomment-772311968 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39399/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

AmplabJenkins removed a comment on pull request #31450: URL: https://github.com/apache/spark/pull/31450#issuecomment-772311962 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39400/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31449: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

AmplabJenkins removed a comment on pull request #31449: URL: https://github.com/apache/spark/pull/31449#issuecomment-772311967 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134807/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31451: [WIP][SPARK-34338][SQL] Report metrics from Datasource v2 scan

AmplabJenkins commented on pull request #31451: URL: https://github.com/apache/spark/pull/31451#issuecomment-772311968 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39399/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31449: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

AmplabJenkins commented on pull request #31449: URL: https://github.com/apache/spark/pull/31449#issuecomment-772311967 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134807/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31447: [SPARK-33726][SQL][FOLLOW-UP] Add assertion to FixedLengthRowBasedKeyValueBatch

AmplabJenkins commented on pull request #31447: URL: https://github.com/apache/spark/pull/31447#issuecomment-772311973 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134803/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

AmplabJenkins commented on pull request #31450: URL: https://github.com/apache/spark/pull/31450#issuecomment-772311962 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39400/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

AmplabJenkins commented on pull request #31378: URL: https://github.com/apache/spark/pull/31378#issuecomment-772311963 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134804/ -

[GitHub] [spark] dongjoon-hyun commented on pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

dongjoon-hyun commented on pull request #31453: URL: https://github.com/apache/spark/pull/31453#issuecomment-772311032 Thank you, @HyukjinKwon and @maropu . According to the review comment, I renamed it from `spark.io.compression.zstd.enableBufferPool` to `spark.io.compression.zstd.bufferP

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

HyukjinKwon commented on a change in pull request #31453: URL: https://github.com/apache/spark/pull/31453#discussion_r569195872 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1680,6 +1680,13 @@ package object config { .bytesConf(B

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

dongjoon-hyun commented on a change in pull request #31453: URL: https://github.com/apache/spark/pull/31453#discussion_r569195640 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1680,6 +1680,13 @@ package object config { .bytesConf

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

AngersZh commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r569193648 ## File path: docs/sql-migration-guide.md ## @@ -56,6 +56,8 @@ license: | In Spark 3.1 and earlier, table refreshing leaves dependents uncached.

[GitHub] [spark] maropu commented on pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

maropu commented on pull request #31453: URL: https://github.com/apache/spark/pull/31453#issuecomment-772307357 Looks fine if the tests pass. This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] maropu commented on a change in pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

maropu commented on a change in pull request #31453: URL: https://github.com/apache/spark/pull/31453#discussion_r569192651 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1680,6 +1680,13 @@ package object config { .bytesConf(ByteUn

[GitHub] [spark] SparkQA removed a comment on pull request #31449: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

SparkQA removed a comment on pull request #31449: URL: https://github.com/apache/spark/pull/31449#issuecomment-772233390 **[Test build #134807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134807/testReport)** for PR 31449 at commit [`5f37003`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31449: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

SparkQA commented on pull request #31449: URL: https://github.com/apache/spark/pull/31449#issuecomment-772306025 **[Test build #134807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134807/testReport)** for PR 31449 at commit [`5f37003`](https://github.co

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

HyukjinKwon commented on a change in pull request #31453: URL: https://github.com/apache/spark/pull/31453#discussion_r569190239 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1680,6 +1680,13 @@ package object config { .bytesConf(B

[GitHub] [spark] viirya commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

viirya commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r569188211 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [spark] maropu commented on a change in pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

maropu commented on a change in pull request #31378: URL: https://github.com/apache/spark/pull/31378#discussion_r569188018 ## File path: docs/sql-migration-guide.md ## @@ -56,6 +56,8 @@ license: | In Spark 3.1 and earlier, table refreshing leaves dependents uncached. -

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

HyukjinKwon commented on a change in pull request #31453: URL: https://github.com/apache/spark/pull/31453#discussion_r569187049 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1680,6 +1680,13 @@ package object config { .bytesConf(B

[GitHub] [spark] cloud-fan commented on a change in pull request #30902: [SPARK-33888][SQL] JDBC SQL TIME type represents incorrectly as TimestampType, it should be physical Int in millis

cloud-fan commented on a change in pull request #30902: URL: https://github.com/apache/spark/pull/30902#discussion_r569186823 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala ## @@ -408,6 +421,23 @@ object JdbcUtils extends

[GitHub] [spark] maropu commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

maropu commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r569184928 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [spark] SparkQA removed a comment on pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

SparkQA removed a comment on pull request #31378: URL: https://github.com/apache/spark/pull/31378#issuecomment-772180730 **[Test build #134804 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134804/testReport)** for PR 31378 at commit [`348ab27`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31378: [SPARK-34240][SQL] Unify output of SHOW TBLPROPERTIES pass output attribute properly

SparkQA commented on pull request #31378: URL: https://github.com/apache/spark/pull/31378#issuecomment-772298181 **[Test build #134804 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134804/testReport)** for PR 31378 at commit [`348ab27`](https://github.co

[GitHub] [spark] Yikun opened a new pull request #31454: Skip zinc start and shutdown in aarch64 os.

Yikun opened a new pull request #31454: URL: https://github.com/apache/spark/pull/31454 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was t

[GitHub] [spark] SparkQA commented on pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

SparkQA commented on pull request #31453: URL: https://github.com/apache/spark/pull/31453#issuecomment-772297148 **[Test build #134817 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134817/testReport)** for PR 31453 at commit [`6e88103`](https://github.com

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

dongjoon-hyun commented on a change in pull request #31453: URL: https://github.com/apache/spark/pull/31453#discussion_r569183105 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1680,6 +1680,13 @@ package object config { .bytesConf

[GitHub] [spark] MaxGekk commented on a change in pull request #31423: [SPARK-34314][SQL] Create new file index after partition schema inferring w/ the schema

MaxGekk commented on a change in pull request #31423: URL: https://github.com/apache/spark/pull/31423#discussion_r568362026 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/PartitionedTablePerfStatsSuite.scala ## @@ -370,7 +370,7 @@ class PartitionedTablePerfSta

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

HyukjinKwon commented on a change in pull request #31453: URL: https://github.com/apache/spark/pull/31453#discussion_r569182485 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1680,6 +1680,13 @@ package object config { .bytesConf(B

[GitHub] [spark] SparkQA commented on pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

SparkQA commented on pull request #31450: URL: https://github.com/apache/spark/pull/31450#issuecomment-772295859 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39400/ ---

[GitHub] [spark] MaxGekk commented on a change in pull request #31405: [SPARK-34304][SQL] Remove view checks in v1 alter table commands

MaxGekk commented on a change in pull request #31405: URL: https://github.com/apache/spark/pull/31405#discussion_r569180642 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/AlterTableDropPartitionSuiteBase.scala ## @@ -222,7 +222,7 @@ trait AlterTab

[GitHub] [spark] saikocat commented on a change in pull request #30902: [SPARK-33888][SQL] JDBC SQL TIME type represents incorrectly as TimestampType, it should be physical Int in millis

saikocat commented on a change in pull request #30902: URL: https://github.com/apache/spark/pull/30902#discussion_r569180539 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala ## @@ -408,6 +421,23 @@ object JdbcUtils extends L

[GitHub] [spark] SparkQA commented on pull request #31451: [WIP][SPARK-34338][SQL] Report metrics from Datasource v2 scan

SparkQA commented on pull request #31451: URL: https://github.com/apache/spark/pull/31451#issuecomment-772292807 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39399/ ---

[GitHub] [spark] HyukjinKwon commented on a change in pull request #30363: [SPARK-33438][SQL] Eagerly init all SQLConf objects for command `set -v`

HyukjinKwon commented on a change in pull request #30363: URL: https://github.com/apache/spark/pull/30363#discussion_r569178700 ## File path: core/src/main/scala/org/apache/spark/util/SparkConfRegisterLoader.scala ## @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Fou

[GitHub] [spark] SparkQA commented on pull request #31452: [SPARK-34317][SQL][FOLLOW-UP] Use relationTypeMismatchHint when UnresolvedTable is resolved to a temp view

SparkQA commented on pull request #31452: URL: https://github.com/apache/spark/pull/31452#issuecomment-772291116 **[Test build #134813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134813/testReport)** for PR 31452 at commit [`63ed2c3`](https://github.com

[GitHub] [spark] cloud-fan commented on a change in pull request #30902: [SPARK-33888][SQL] JDBC SQL TIME type represents incorrectly as TimestampType, it should be physical Int in millis

cloud-fan commented on a change in pull request #30902: URL: https://github.com/apache/spark/pull/30902#discussion_r569177848 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala ## @@ -408,6 +421,23 @@ object JdbcUtils extends

[GitHub] [spark] SparkQA removed a comment on pull request #31447: [SPARK-33726][SQL][FOLLOW-UP] Add assertion to FixedLengthRowBasedKeyValueBatch

SparkQA removed a comment on pull request #31447: URL: https://github.com/apache/spark/pull/31447#issuecomment-772180681 **[Test build #134803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134803/testReport)** for PR 31447 at commit [`03cb772`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31447: [SPARK-33726][SQL][FOLLOW-UP] Add assertion to FixedLengthRowBasedKeyValueBatch

SparkQA commented on pull request #31447: URL: https://github.com/apache/spark/pull/31447#issuecomment-772289508 **[Test build #134803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134803/testReport)** for PR 31447 at commit [`03cb772`](https://github.co

[GitHub] [spark] dongjoon-hyun opened a new pull request #31453: [SPARK-34340][CORE] Support ZSTD JNI BufferPool

dongjoon-hyun opened a new pull request #31453: URL: https://github.com/apache/spark/pull/31453 ### What changes were proposed in this pull request? This PR aims to support ZSTD JNI BufferPool feature by adding a new configuration, `spark.io.compression.zstd.enableBufferPool`.

[GitHub] [spark] maropu commented on pull request #31409: [SPARK-34307][SQL] TakeOrderedAndProjectExec avoid shuffle if input rdd has single partition

maropu commented on pull request #31409: URL: https://github.com/apache/spark/pull/31409#issuecomment-772287363 late lgtm This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [spark] SparkQA commented on pull request #30483: [SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

SparkQA commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-772284582 **[Test build #134816 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134816/testReport)** for PR 30483 at commit [`120678d`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #30841: [SPARK-28191][SS] New data source - state - reader part

SparkQA commented on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-772284355 **[Test build #134815 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134815/testReport)** for PR 30841 at commit [`47f12ac`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #31179: [SPARK-34113][SQL] Use metric data update metadata statistic's size and rowCount

SparkQA commented on pull request #31179: URL: https://github.com/apache/spark/pull/31179#issuecomment-772284221 **[Test build #134814 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134814/testReport)** for PR 31179 at commit [`68e0256`](https://github.com

[GitHub] [spark] imback82 commented on a change in pull request #31452: [SPARK-34317][SQL][FOLLOW-UP] Use relationTypeMismatchHint when UnresolvedTable is resolved to a temp view

imback82 commented on a change in pull request #31452: URL: https://github.com/apache/spark/pull/31452#discussion_r569171350 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala ## @@ -192,11 +192,6 @@ private[spark] object QueryCo

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31258: [SPARK-34168] [SQL] Support DPP in AQE when the join is Broadcast hash join at the beginning

AmplabJenkins removed a comment on pull request #31258: URL: https://github.com/apache/spark/pull/31258#issuecomment-772281833 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39398/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31349: [SPARK-34246][SQL] New type coercion syntax rules in ANSI mode

AmplabJenkins removed a comment on pull request #31349: URL: https://github.com/apache/spark/pull/31349#issuecomment-772281832 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39397/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31448: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`.

AmplabJenkins removed a comment on pull request #31448: URL: https://github.com/apache/spark/pull/31448#issuecomment-772281830 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31179: [SPARK-34113][SQL] Use metric data update metadata statistic's size and rowCount

AmplabJenkins removed a comment on pull request #31179: URL: https://github.com/apache/spark/pull/31179#issuecomment-772281834 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39394/

[GitHub] [spark] imback82 opened a new pull request #31452: [SPARK-34317][SQL][FOLLOW-UP] Use relationTypeMismatchHint when UnresolvedTable is resolved to a temp view

imback82 opened a new pull request #31452: URL: https://github.com/apache/spark/pull/31452 ### What changes were proposed in this pull request? This is a follow up to #31424, and proposes to use `UnresolvedTable.relationTypeMismatchHint` when `UnresolvedTable` is resolved

[GitHub] [spark] AmplabJenkins commented on pull request #31179: [SPARK-34113][SQL] Use metric data update metadata statistic's size and rowCount

AmplabJenkins commented on pull request #31179: URL: https://github.com/apache/spark/pull/31179#issuecomment-772281834 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39394/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31258: [SPARK-34168] [SQL] Support DPP in AQE when the join is Broadcast hash join at the beginning

AmplabJenkins commented on pull request #31258: URL: https://github.com/apache/spark/pull/31258#issuecomment-772281833 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39398/ -

[GitHub] [spark] AmplabJenkins commented on pull request #31448: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`.

AmplabJenkins commented on pull request #31448: URL: https://github.com/apache/spark/pull/31448#issuecomment-772281830 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #31349: [SPARK-34246][SQL] New type coercion syntax rules in ANSI mode

AmplabJenkins commented on pull request #31349: URL: https://github.com/apache/spark/pull/31349#issuecomment-772281832 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39397/ -

[GitHub] [spark] allisonwang-db commented on pull request #31444: [SPARK-34335][SQL] Support referencing subquery with column aliases by table alias

allisonwang-db commented on pull request #31444: URL: https://github.com/apache/spark/pull/31444#issuecomment-772280943 cc @maropu This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [spark] SparkQA commented on pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

SparkQA commented on pull request #31450: URL: https://github.com/apache/spark/pull/31450#issuecomment-772279654 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39400/ -

[GitHub] [spark] SparkQA commented on pull request #31451: [WIP][SPARK-34338][SQL] Report metrics from Datasource v2 scan

SparkQA commented on pull request #31451: URL: https://github.com/apache/spark/pull/31451#issuecomment-772277403 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39399/ -

[GitHub] [spark] SparkQA removed a comment on pull request #31448: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`.

SparkQA removed a comment on pull request #31448: URL: https://github.com/apache/spark/pull/31448#issuecomment-772232798 **[Test build #134808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134808/testReport)** for PR 31448 at commit [`2eac9ec`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31448: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`.

SparkQA commented on pull request #31448: URL: https://github.com/apache/spark/pull/31448#issuecomment-772275970 **[Test build #134808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134808/testReport)** for PR 31448 at commit [`2eac9ec`](https://github.co

[GitHub] [spark] LuciferYang edited a comment on pull request #30483: [WIP][SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

LuciferYang edited a comment on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-772273452 Simple test: ``` val df = spark.read.parquet(or orc)("file:/home/work/xxx/data") df.createOrReplaceTempView("test_table") spark.sql("select sum(a),

[GitHub] [spark] LuciferYang edited a comment on pull request #30483: [WIP][SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

LuciferYang edited a comment on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-772273452 Simple test: ``` val df = spark.read.parquet(or orc)("file:/home/work/xxx/data") df.createOrReplaceTempView("test_table") spark.sql("select sum(a),

[GitHub] [spark] LuciferYang commented on pull request #30483: [WIP][SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

LuciferYang commented on pull request #30483: URL: https://github.com/apache/spark/pull/30483#issuecomment-772273452 Simple test: ``` val df = spark.read.parquet(or orc)("file:/home/work/xxx/data") df.createOrReplaceTempView("test_table") spark.sql("select sum(a), sum(b),

[GitHub] [spark] SparkQA commented on pull request #31179: [SPARK-34113][SQL] Use metric data update metadata statistic's size and rowCount

SparkQA commented on pull request #31179: URL: https://github.com/apache/spark/pull/31179#issuecomment-772270200 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39394/ ---

[GitHub] [spark] HeartSaVioR commented on pull request #30841: [SPARK-28191][SS] New data source - state - reader part

HeartSaVioR commented on pull request #30841: URL: https://github.com/apache/spark/pull/30841#issuecomment-772269114 retest this, please This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #31179: [SPARK-34113][SQL] Use metric data update metadata statistic's size and rowCount

AngersZh commented on a change in pull request #31179: URL: https://github.com/apache/spark/pull/31179#discussion_r569161195 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala ## @@ -2581,6 +2581,59 @@ abstract class SQLQuerySuite

[GitHub] [spark] SparkQA commented on pull request #31258: [SPARK-34168] [SQL] Support DPP in AQE when the join is Broadcast hash join at the beginning

SparkQA commented on pull request #31258: URL: https://github.com/apache/spark/pull/31258#issuecomment-772262379 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39398/ ---

[GitHub] [spark] SparkQA commented on pull request #31349: [SPARK-34246][SQL] New type coercion syntax rules in ANSI mode

SparkQA commented on pull request #31349: URL: https://github.com/apache/spark/pull/31349#issuecomment-772261696 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39397/ ---

[GitHub] [spark] zhengruifeng commented on pull request #31409: [SPARK-34307][SQL] TakeOrderedAndProjectExec avoid shuffle if input rdd has single partition

zhengruifeng commented on pull request #31409: URL: https://github.com/apache/spark/pull/31409#issuecomment-772261328 thanks all for reviewing! > We may have more operators that adding shuffle in the doExecute method instead of the planner I will look for other similar operato

[GitHub] [spark] HyukjinKwon commented on a change in pull request #31179: [SPARK-34113][SQL] Use metric data update metadata statistic's size and rowCount

HyukjinKwon commented on a change in pull request #31179: URL: https://github.com/apache/spark/pull/31179#discussion_r569154926 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala ## @@ -2581,6 +2581,59 @@ abstract class SQLQuerySuiteB

[GitHub] [spark] SparkQA commented on pull request #31448: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`.

SparkQA commented on pull request #31448: URL: https://github.com/apache/spark/pull/31448#issuecomment-772260591 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39396/ ---

[GitHub] [spark] SparkQA commented on pull request #31451: [WIP][SPARK-34338][SQL] Report metrics from Datasource v2 scan

SparkQA commented on pull request #31451: URL: https://github.com/apache/spark/pull/31451#issuecomment-772259785 **[Test build #134811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134811/testReport)** for PR 31451 at commit [`4d941c7`](https://github.com

[GitHub] [spark] imback82 commented on a change in pull request #31424: [SPARK-34317][SQL] Introduce relationTypeMismatchHint to UnresolvedTable for a better error message

imback82 commented on a change in pull request #31424: URL: https://github.com/apache/spark/pull/31424#discussion_r569152899 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1144,18 +1144,20 @@ class Analyzer(override val c

[GitHub] [spark] HyukjinKwon commented on pull request #31436: [SPARK-34327][BUILD] Strip passwords from inlining into build information while releasing.

HyukjinKwon commented on pull request #31436: URL: https://github.com/apache/spark/pull/31436#issuecomment-772256896 Merged to master, branch-3.1, branch-3.0 and branch-2.4. This is an automated message from the Apache Git Se

[GitHub] [spark] HyukjinKwon closed pull request #31436: [SPARK-34327][BUILD] Strip passwords from inlining into build information while releasing.

HyukjinKwon closed pull request #31436: URL: https://github.com/apache/spark/pull/31436 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

AmplabJenkins removed a comment on pull request #31450: URL: https://github.com/apache/spark/pull/31450#issuecomment-772255241 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134812/ -

[GitHub] [spark] SparkQA removed a comment on pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

SparkQA removed a comment on pull request #31450: URL: https://github.com/apache/spark/pull/31450#issuecomment-772254067 **[Test build #134812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134812/testReport)** for PR 31450 at commit [`5560ec6`](https://gi

[GitHub] [spark] SparkQA commented on pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

SparkQA commented on pull request #31450: URL: https://github.com/apache/spark/pull/31450#issuecomment-772255230 **[Test build #134812 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134812/testReport)** for PR 31450 at commit [`5560ec6`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

AmplabJenkins commented on pull request #31450: URL: https://github.com/apache/spark/pull/31450#issuecomment-772255241 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134812/ -

[GitHub] [spark] SparkQA commented on pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

SparkQA commented on pull request #31450: URL: https://github.com/apache/spark/pull/31450#issuecomment-772254067 **[Test build #134812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134812/testReport)** for PR 31450 at commit [`5560ec6`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31449: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

AmplabJenkins removed a comment on pull request #31449: URL: https://github.com/apache/spark/pull/31449#issuecomment-772253544 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39395/

[GitHub] [spark] AmplabJenkins commented on pull request #31449: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

AmplabJenkins commented on pull request #31449: URL: https://github.com/apache/spark/pull/31449#issuecomment-772253544 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39395/ -

[GitHub] [spark] SparkQA commented on pull request #31349: [SPARK-34246][SQL] New type coercion syntax rules in ANSI mode

SparkQA commented on pull request #31349: URL: https://github.com/apache/spark/pull/31349#issuecomment-772253263 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39397/ -

[GitHub] [spark] SparkQA commented on pull request #31449: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

SparkQA commented on pull request #31449: URL: https://github.com/apache/spark/pull/31449#issuecomment-772252809 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39395/ ---

[GitHub] [spark] viirya edited a comment on pull request #31398: [SPARK-34297][SQL][SS] Add metrics for data loss and offset out range for KafkaMicroBatchStream

viirya edited a comment on pull request #31398: URL: https://github.com/apache/spark/pull/31398#issuecomment-772250575 > For continuous execution, it's like an endless batch execution, so we can only use heartbeat events to update metrics. And we update the metrics in the UI in every epoch

[GitHub] [spark] SparkQA commented on pull request #31449: [SPARK-34326][CORE][SQL] Fix UTs added in SPARK-31793 depending on the length of temp path

SparkQA commented on pull request #31449: URL: https://github.com/apache/spark/pull/31449#issuecomment-772250583 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39395/ -

[GitHub] [spark] viirya commented on pull request #31398: [SPARK-34297][SQL][SS] Add metrics for data loss and offset out range for KafkaMicroBatchStream

viirya commented on pull request #31398: URL: https://github.com/apache/spark/pull/31398#issuecomment-772250575 > For continuous execution, it's like an endless batch execution, so we can only use heartbeat events to update metrics. And we update the metrics in the UI in every epoch.

[GitHub] [spark] SparkQA commented on pull request #31258: [SPARK-34168] [SQL] Support DPP in AQE when the join is Broadcast hash join at the beginning

SparkQA commented on pull request #31258: URL: https://github.com/apache/spark/pull/31258#issuecomment-772250089 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39398/ -

[GitHub] [spark] viirya commented on pull request #31398: [SPARK-34297][SQL][SS] Add metrics for data loss and offset out range for KafkaMicroBatchStream

viirya commented on pull request #31398: URL: https://github.com/apache/spark/pull/31398#issuecomment-772248993 I extract generalized DS v2 change to #31451. This is an automated message from the Apache Git Service. To respon

[GitHub] [spark] cloud-fan closed pull request #31422: [SPARK-34313][SQL] Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to use UnresolvedTable to resolve the identifier

cloud-fan closed pull request #31422: URL: https://github.com/apache/spark/pull/31422 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] viirya opened a new pull request #31451: [WIP][SPARK-34338][SQL] Report metrics from Datasource v2 scan

viirya opened a new pull request #31451: URL: https://github.com/apache/spark/pull/31451 ### What changes were proposed in this pull request? This patch proposes to add a few public API change to DS v2, to make DS v2 scan can report metrics to Spark. One public int

[GitHub] [spark] cloud-fan commented on pull request #31422: [SPARK-34313][SQL] Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to use UnresolvedTable to resolve the identifier

cloud-fan commented on pull request #31422: URL: https://github.com/apache/spark/pull/31422#issuecomment-772248371 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] cloud-fan commented on a change in pull request #31422: [SPARK-34313][SQL] Migrate ALTER TABLE SET/UNSET TBLPROPERTIES commands to use UnresolvedTable to resolve the identifier

cloud-fan commented on a change in pull request #31422: URL: https://github.com/apache/spark/pull/31422#discussion_r569145719 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ## @@ -167,6 +167,12 @@ abstract class SQLViewSuite extends Que

[GitHub] [spark] SparkQA commented on pull request #31448: [SPARK-28137][SQL] Data Type Formatting Functions: `to_number`.

SparkQA commented on pull request #31448: URL: https://github.com/apache/spark/pull/31448#issuecomment-772247585 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39396/ -

[GitHub] [spark] cloud-fan commented on pull request #31398: [SPARK-34297][SQL][SS] Add metrics for data loss and offset out range for KafkaMicroBatchStream

cloud-fan commented on pull request #31398: URL: https://github.com/apache/spark/pull/31398#issuecomment-772246886 Yea creating a separate PR SGTM. Let's have a high-level discussion first (I haven't read this PR yet). From my understanding, metrics in batch execution can be done as: 1.

[GitHub] [spark] attilapiros opened a new pull request #31450: [WIP][SPARK-33763] Add metrics for better tracking of dynamic allocation

attilapiros opened a new pull request #31450: URL: https://github.com/apache/spark/pull/31450 ### What changes were proposed in this pull request? This PR adds the following metrics to track executor remove reasons during dynamic allocation: - `numberExecutorsGracefullyDecomm

[GitHub] [spark] SparkQA commented on pull request #31349: [SPARK-34246][SQL] New type coercion syntax rules in ANSI mode

SparkQA commented on pull request #31349: URL: https://github.com/apache/spark/pull/31349#issuecomment-772234464 **[Test build #134809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134809/testReport)** for PR 31349 at commit [`462eea1`](https://github.com

  1   2   3   4   5   6   7   8   9   10   >