[GitHub] [spark] HyukjinKwon commented on a change in pull request #30130: [SPARK-33408][SPARK-32354][K8S][R] Use R 3.6.3 in K8s R image and re-enable RTestsSuite

HyukjinKwon commented on a change in pull request #30130: URL: https://github.com/apache/spark/pull/30130#discussion_r534859502 ## File path: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/R/Dockerfile ## @@ -25,7 +25,14 @@ USER 0 RUN mkdir

[GitHub] [spark] Ngone51 commented on a change in pull request #30312: [SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion

Ngone51 commented on a change in pull request #30312: URL: https://github.com/apache/spark/pull/30312#discussion_r534644897 ## File path: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala ## @@ -0,0 +1,458 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534807436 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala ## @@ -0,0 +1,142 @@ +/*

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534842510 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala ## @@ -0,0 +1,142 @@ +/*

[GitHub] [spark] yaooqinn opened a new pull request #30586: [SPARK-33641][SQL] Invalidate new char-like type in public APIs that produce incorrect results

yaooqinn opened a new pull request #30586: URL: https://github.com/apache/spark/pull/30586 … ### What changes were proposed in this pull request? In this PR, we suppose to narrow the use cases of the char-like data types, of which are invalid now or later

[GitHub] [spark] SparkQA commented on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

SparkQA commented on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737727470 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36699/

[GitHub] [spark] SparkQA commented on pull request #30350: [SPARK-33428][SQL] Conv UDF use BigInt to avoid Long value overflow

SparkQA commented on pull request #30350: URL: https://github.com/apache/spark/pull/30350#issuecomment-737727021 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36700/

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534831091 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala ## @@ -0,0 +1,142 @@ +/*

[GitHub] [spark] WeichenXu123 commented on pull request #30471: [SPARK-33520][ML][PySpark] make CrossValidator/TrainValidateSplit/OneVsRest Reader/Writer support Python backend estimator/evaluator

WeichenXu123 commented on pull request #30471: URL: https://github.com/apache/spark/pull/30471#issuecomment-737725930 I would like to keep this PR open for one more day to see whether @srowen has some comment. This is an

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30583: [SPARK-33640][TESTS] Extend connection timeout to DB server for DB2IntegrationSuite and its variants

AmplabJenkins removed a comment on pull request #30583: URL: https://github.com/apache/spark/pull/30583#issuecomment-737724285 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30583: [SPARK-33640][TESTS] Extend connection timeout to DB server for DB2IntegrationSuite and its variants

AmplabJenkins commented on pull request #30583: URL: https://github.com/apache/spark/pull/30583#issuecomment-737724285 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30583: [SPARK-33640][TESTS] Extend connection timeout to DB server for DB2IntegrationSuite and its variants

SparkQA commented on pull request #30583: URL: https://github.com/apache/spark/pull/30583#issuecomment-737724269 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36694/

[GitHub] [spark] beliefer commented on pull request #30564: [SPARK-32670][SQL][FOLLOWUP] Group exception messages in Catalyst Analyzer in one file

beliefer commented on pull request #30564: URL: https://github.com/apache/spark/pull/30564#issuecomment-737722164 cc @cloud-fan @HyukjinKwon @maropu @dongjoon-hyun This is an automated message from the Apache Git Service.

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534812004 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala ## @@ -0,0 +1,142 @@ +/*

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30584: [SPARK-33472][SQL][FOLLOW-UP] Update RemoveRedundantSorts comment

AmplabJenkins removed a comment on pull request #30584: URL: https://github.com/apache/spark/pull/30584#issuecomment-737721415 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal

AmplabJenkins removed a comment on pull request #30433: URL: https://github.com/apache/spark/pull/30433#issuecomment-737721310 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30584: [SPARK-33472][SQL][FOLLOW-UP] Update RemoveRedundantSorts comment

AmplabJenkins commented on pull request #30584: URL: https://github.com/apache/spark/pull/30584#issuecomment-737721415 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30584: [SPARK-33472][SQL][FOLLOW-UP] Update RemoveRedundantSorts comment

SparkQA commented on pull request #30584: URL: https://github.com/apache/spark/pull/30584#issuecomment-737721398 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36693/

[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal

AmplabJenkins commented on pull request #30433: URL: https://github.com/apache/spark/pull/30433#issuecomment-737721310 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal

SparkQA commented on pull request #30433: URL: https://github.com/apache/spark/pull/30433#issuecomment-737721294 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36698/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30481: [SPARK-33526][SQL] Add config to control if cancel invoke interrupt task on thriftserver

AmplabJenkins removed a comment on pull request #30481: URL: https://github.com/apache/spark/pull/30481#issuecomment-737721173 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30481: [SPARK-33526][SQL] Add config to control if cancel invoke interrupt task on thriftserver

AmplabJenkins commented on pull request #30481: URL: https://github.com/apache/spark/pull/30481#issuecomment-737721173 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30481: [SPARK-33526][SQL] Add config to control if cancel invoke interrupt task on thriftserver

SparkQA commented on pull request #30481: URL: https://github.com/apache/spark/pull/30481#issuecomment-737721149 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36697/

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534807436 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala ## @@ -0,0 +1,142 @@ +/*

[GitHub] [spark] SparkQA commented on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

SparkQA commented on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737718059 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36699/

[GitHub] [spark] SparkQA commented on pull request #30243: [SPARK-33335][SQL] Support `has_all` func

SparkQA commented on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-737717879 **[Test build #132102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132102/testReport)** for PR 30243 at commit

[GitHub] [spark] SparkQA commented on pull request #30585: [SPARK-26218][SQL][FOLLOW UP] Fix the corner case of codegen when casting float to Integer

SparkQA commented on pull request #30585: URL: https://github.com/apache/spark/pull/30585#issuecomment-737717527 **[Test build #132101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132101/testReport)** for PR 30585 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30543: [SPARK-33597][SQL] Support REGEXP_LIKE for consistent with mainstream databases

AmplabJenkins removed a comment on pull request #30543: URL: https://github.com/apache/spark/pull/30543#issuecomment-737717106 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30543: [SPARK-33597][SQL] Support REGEXP_LIKE for consistent with mainstream databases

AmplabJenkins commented on pull request #30543: URL: https://github.com/apache/spark/pull/30543#issuecomment-737717106 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30350: [SPARK-33428][SQL] Conv UDF use BigInt to avoid Long value overflow

SparkQA commented on pull request #30350: URL: https://github.com/apache/spark/pull/30350#issuecomment-737716383 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36700/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30582: [SPARK-33636][PYTHON][ML][FOLLOWUP] Update since tag of labelsArray in StringIndexer

AmplabJenkins removed a comment on pull request #30582: URL: https://github.com/apache/spark/pull/30582#issuecomment-737715490 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

AmplabJenkins removed a comment on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-737715489 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

AmplabJenkins removed a comment on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737715488 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #30350: [SPARK-33428][SQL] Conv UDF use BigInt to avoid Long value overflow

SparkQA commented on pull request #30350: URL: https://github.com/apache/spark/pull/30350#issuecomment-737715812 **[Test build #132100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132100/testReport)** for PR 30350 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

AmplabJenkins commented on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737715488 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

AmplabJenkins commented on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-737715489 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #30582: [SPARK-33636][PYTHON][ML][FOLLOWUP] Update since tag of labelsArray in StringIndexer

AmplabJenkins commented on pull request #30582: URL: https://github.com/apache/spark/pull/30582#issuecomment-737715490 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

SparkQA removed a comment on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-737672331 **[Test build #132092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132092/testReport)** for PR 24173 at commit

[GitHub] [spark] SparkQA commented on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

SparkQA commented on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-737713538 **[Test build #132092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132092/testReport)** for PR 24173 at commit

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534778233 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala ## @@ -0,0 +1,142 @@ +/*

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534776507 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala ## @@ -150,6 +172,25 @@ private

[GitHub] [spark] AngersZhuuuu commented on pull request #30243: [SPARK-33335][SQL] Support `has_all` func

AngersZh commented on pull request #30243: URL: https://github.com/apache/spark/pull/30243#issuecomment-737712017 ping @kiszk @cloud-fan @zero323 @viirya @HyukjinKwon Now update function name to `has_all` for disambiguation.

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534772017 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ## @@ -391,10 +399,18 @@ object StateStore

[GitHub] [spark] SparkQA commented on pull request #30583: [SPARK-33640][TESTS] Extend connection timeout to DB server for DB2IntegrationSuite and its variants

SparkQA commented on pull request #30583: URL: https://github.com/apache/spark/pull/30583#issuecomment-737711285 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36694/

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534772017 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala ## @@ -391,10 +399,18 @@ object StateStore

[GitHub] [spark] cloud-fan commented on pull request #30585: [SPARK-26218][SQL][FOLLOW UP] Fix the corner case of codegen when casting float to Integer

cloud-fan commented on pull request #30585: URL: https://github.com/apache/spark/pull/30585#issuecomment-737710751 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [spark] SparkQA commented on pull request #30584: [SPARK-33472][SQL][FOLLOW-UP] Update RemoveRedundantSorts comment

SparkQA commented on pull request #30584: URL: https://github.com/apache/spark/pull/30584#issuecomment-737709894 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36693/

[GitHub] [spark] SparkQA commented on pull request #30481: [SPARK-33526][SQL] Add config to control if cancel invoke interrupt task on thriftserver

SparkQA commented on pull request #30481: URL: https://github.com/apache/spark/pull/30481#issuecomment-737709945 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36697/

[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal

SparkQA commented on pull request #30433: URL: https://github.com/apache/spark/pull/30433#issuecomment-737709304 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36698/

[GitHub] [spark] HeartSaVioR commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

HeartSaVioR commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534759253 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala ## @@ -0,0 +1,142 @@ +/*

[GitHub] [spark] SparkQA commented on pull request #30582: [SPARK-33636][PYTHON][ML][FOLLOWUP] Update since tag of labelsArray in StringIndexer

SparkQA commented on pull request #30582: URL: https://github.com/apache/spark/pull/30582#issuecomment-737706776 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36692/

[GitHub] [spark] luluorta commented on pull request #30585: [SPARK-26218][SQL][FOLLOW UP] Fix the corner case of codegen when casting float to Integer

luluorta commented on pull request #30585: URL: https://github.com/apache/spark/pull/30585#issuecomment-737702651 cc @turboFei @cloud-fan This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] luluorta opened a new pull request #30585: [SPARK-26218][SQL][FOLLOW UP] Fix the corner case of codegen when casting float to Integer

luluorta opened a new pull request #30585: URL: https://github.com/apache/spark/pull/30585 ### What changes were proposed in this pull request? This is a followup of [#27151](https://github.com/apache/spark/pull/27151). It fixes the same issue for the codegen path. ### Why

[GitHub] [spark] SparkQA removed a comment on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

SparkQA removed a comment on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737695740 **[Test build #132099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132099/testReport)** for PR 30581 at commit

[GitHub] [spark] SparkQA commented on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

SparkQA commented on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737701167 **[Test build #132099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132099/testReport)** for PR 30581 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #30573: [SPARK-26341][CORE]Expose executor memory metrics at the stage level, in the Stages tab

AmplabJenkins commented on pull request #30573: URL: https://github.com/apache/spark/pull/30573#issuecomment-737700164 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] manbuyun edited a comment on pull request #30572: [SPARK-33628][SQL] Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

manbuyun edited a comment on pull request #30572: URL: https://github.com/apache/spark/pull/30572#issuecomment-737652179 And, Hive.getPartitions will cause MetaStoreClient close connection. Similar issue: https://issues.apache.org/jira/browse/SPARK-29409 The chain of method calls

[GitHub] [spark] SparkQA removed a comment on pull request #30573: [SPARK-26341][CORE]Expose executor memory metrics at the stage level, in the Stages tab

SparkQA removed a comment on pull request #30573: URL: https://github.com/apache/spark/pull/30573#issuecomment-737654876 **[Test build #132085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132085/testReport)** for PR 30573 at commit

[GitHub] [spark] SparkQA commented on pull request #30573: [SPARK-26341][CORE]Expose executor memory metrics at the stage level, in the Stages tab

SparkQA commented on pull request #30573: URL: https://github.com/apache/spark/pull/30573#issuecomment-737699385 **[Test build #132085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132085/testReport)** for PR 30573 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal

AmplabJenkins removed a comment on pull request #30433: URL: https://github.com/apache/spark/pull/30433#issuecomment-737698947 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AngersZhuuuu commented on pull request #30350: [SPARK-33428][SQL] Conv UDF use BigInt to avoid Long value overflow

AngersZh commented on pull request #30350: URL: https://github.com/apache/spark/pull/30350#issuecomment-737699065 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal

AmplabJenkins commented on pull request #30433: URL: https://github.com/apache/spark/pull/30433#issuecomment-737698947 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal

SparkQA commented on pull request #30433: URL: https://github.com/apache/spark/pull/30433#issuecomment-737698932 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36690/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

AmplabJenkins removed a comment on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737698182 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

AmplabJenkins commented on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737698182 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

SparkQA commented on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737698159 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36689/

[GitHub] [spark] manbuyun edited a comment on pull request #30572: [SPARK-33628][SQL] Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

manbuyun edited a comment on pull request #30572: URL: https://github.com/apache/spark/pull/30572#issuecomment-737652179 And, Hive.getPartitions will cause MetaStoreClient close connection. Similar issue: https://issues.apache.org/jira/browse/SPARK-29409 The chain of method calls

[GitHub] [spark] SparkQA commented on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

SparkQA commented on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737695740 **[Test build #132099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132099/testReport)** for PR 30581 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

AmplabJenkins removed a comment on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-737695354 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

SparkQA commented on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-737695345 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36691/

[GitHub] [spark] AmplabJenkins commented on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

AmplabJenkins commented on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-737695354 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] manbuyun edited a comment on pull request #30572: [SPARK-33628][SQL] Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

manbuyun edited a comment on pull request #30572: URL: https://github.com/apache/spark/pull/30572#issuecomment-737652179 And, Hive.getPartitions will cause MetaStoreClient close connection The chain of method calls is as follows: Hive.getPartitions -> Hive.getUserName ->

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #30350: [SPARK-33428][SQL] Conv UDF use BigInt to avoid Long value overflow

AngersZh edited a comment on pull request #30350: URL: https://github.com/apache/spark/pull/30350#issuecomment-737690572 > Fixing overflow should be good. It's not very intuitive to read the results of this function. Is there any database we can use as a reference? Mysql:

[GitHub] [spark] SparkQA commented on pull request #30582: [SPARK-33636][PYTHON][ML][FOLLOWUP] Update since tag of labelsArray in StringIndexer

SparkQA commented on pull request #30582: URL: https://github.com/apache/spark/pull/30582#issuecomment-737694018 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36692/

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #30350: [SPARK-33428][SQL] Conv UDF use BigInt to avoid Long value overflow

AngersZh edited a comment on pull request #30350: URL: https://github.com/apache/spark/pull/30350#issuecomment-737690572 > Fixing overflow should be good. It's not very intuitive to read the results of this function. Is there any database we can use as a reference? Mysql:

[GitHub] [spark] AngersZhuuuu edited a comment on pull request #30350: [SPARK-33428][SQL] Conv UDF use BigInt to avoid Long value overflow

AngersZh edited a comment on pull request #30350: URL: https://github.com/apache/spark/pull/30350#issuecomment-737690572 > Fixing overflow should be good. It's not very intuitive to read the results of this function. Is there any database we can use as a reference? Mysql:

[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal

SparkQA commented on pull request #30433: URL: https://github.com/apache/spark/pull/30433#issuecomment-737692908 **[Test build #132098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132098/testReport)** for PR 30433 at commit

[GitHub] [spark] SparkQA commented on pull request #30567: [SPARK-33142][SQL] Store SQL text for SQL temp view

SparkQA commented on pull request #30567: URL: https://github.com/apache/spark/pull/30567#issuecomment-737692863 **[Test build #132096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132096/testReport)** for PR 30567 at commit

[GitHub] [spark] SparkQA commented on pull request #30584: [SPARK-33472][SQL][FOLLOW-UP] Update RemoveRedundantSorts comment

SparkQA commented on pull request #30584: URL: https://github.com/apache/spark/pull/30584#issuecomment-737692698 **[Test build #132094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132094/testReport)** for PR 30584 at commit

[GitHub] [spark] SparkQA commented on pull request #30543: [SPARK-33597][SQL] Support REGEXP_LIKE for consistent with mainstream databases

SparkQA commented on pull request #30543: URL: https://github.com/apache/spark/pull/30543#issuecomment-737692642 **[Test build #132097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132097/testReport)** for PR 30543 at commit

[GitHub] [spark] SparkQA commented on pull request #30583: [SPARK-33640][TESTS] Extend connection timeout to DB server for DB2IntegrationSuite and its variants

SparkQA commented on pull request #30583: URL: https://github.com/apache/spark/pull/30583#issuecomment-737692603 **[Test build #132095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132095/testReport)** for PR 30583 at commit

[GitHub] [spark] LuciferYang commented on a change in pull request #30577: [SPARK-33621][SQL] Add a way to inject data source rewrite rules

LuciferYang commented on a change in pull request #30577: URL: https://github.com/apache/spark/pull/30577#discussion_r534718894 ## File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala ## @@ -199,6 +199,21 @@ class SparkSessionExtensions {

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30582: [SPARK-33636][PYTHON][ML][FOLLOWUP] Update since tag of labelsArray in StringIndexer

AmplabJenkins removed a comment on pull request #30582: URL: https://github.com/apache/spark/pull/30582#issuecomment-737691321 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30543: [SPARK-33597][SQL] Support REGEXP_LIKE for consistent with mainstream databases

AmplabJenkins removed a comment on pull request #30543: URL: https://github.com/apache/spark/pull/30543#issuecomment-737691325 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30567: [SPARK-33142][SQL] Store SQL text for SQL temp view

AmplabJenkins removed a comment on pull request #30567: URL: https://github.com/apache/spark/pull/30567#issuecomment-737691328 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30582: [SPARK-33636][PYTHON][ML][FOLLOWUP] Update since tag of labelsArray in StringIndexer

AmplabJenkins commented on pull request #30582: URL: https://github.com/apache/spark/pull/30582#issuecomment-737691321 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30517: [DO-NOT-MERGE][test-maven] Test compatibility for Parquet 1.11.1, Avro 1.10.0 and Hive 2.3.8

AmplabJenkins removed a comment on pull request #30517: URL: https://github.com/apache/spark/pull/30517#issuecomment-737641867 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #30543: [SPARK-33597][SQL] Support REGEXP_LIKE for consistent with mainstream databases

AmplabJenkins commented on pull request #30543: URL: https://github.com/apache/spark/pull/30543#issuecomment-737691325 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #30517: [DO-NOT-MERGE][test-maven] Test compatibility for Parquet 1.11.1, Avro 1.10.0 and Hive 2.3.8

AmplabJenkins commented on pull request #30517: URL: https://github.com/apache/spark/pull/30517#issuecomment-737691336 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #30567: [SPARK-33142][SQL] Store SQL text for SQL temp view

AmplabJenkins commented on pull request #30567: URL: https://github.com/apache/spark/pull/30567#issuecomment-737691328 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AngersZhuuuu commented on pull request #30350: [SPARK-33428][SQL] Conv UDF use BigInt to avoid Long value overflow

AngersZh commented on pull request #30350: URL: https://github.com/apache/spark/pull/30350#issuecomment-737690572 > Fixing overflow should be good. It's not very intuitive to read the results of this function. Is there any database we can use as a reference? Mysql:

[GitHub] [spark] allisonwang-db opened a new pull request #30584: [SPARK-33472][SQL][FOLLOW-UP] Update RemoveRedundantSorts comment

allisonwang-db opened a new pull request #30584: URL: https://github.com/apache/spark/pull/30584 ### What changes were proposed in this pull request? This PR is a follow-up for #30373 that updates the comment for RemoveRedundantSorts in QueryExecution. ### Why are the changes

[GitHub] [spark] manbuyun edited a comment on pull request #30572: [SPARK-33628][SQL] Use the Hive.getPartitionsByNames method instead of Hive.getPartitions in the HiveClientImpl

manbuyun edited a comment on pull request #30572: URL: https://github.com/apache/spark/pull/30572#issuecomment-737652179 And, Hive.getPartitions will cause MetaStoreClient close connection

[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal

SparkQA commented on pull request #30433: URL: https://github.com/apache/spark/pull/30433#issuecomment-737687656 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36690/

[GitHub] [spark] sarutak opened a new pull request #30583: [SPARK-33640][TESTS] Extend connection timeout to DB server for DB2IntegrationSuite and its variants

sarutak opened a new pull request #30583: URL: https://github.com/apache/spark/pull/30583 ### What changes were proposed in this pull request? This PR extends the connection timeout to the DB server for DB2IntegrationSuite and its variants. The container image ibmcom/db2

[GitHub] [spark] SparkQA commented on pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

SparkQA commented on pull request #24173: URL: https://github.com/apache/spark/pull/24173#issuecomment-737686530 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36691/

[GitHub] [spark] SparkQA commented on pull request #30581: [WIP][SPARK-33615][K8S][TESTS] Add spark.archive tests in K8S

SparkQA commented on pull request #30581: URL: https://github.com/apache/spark/pull/30581#issuecomment-737685942 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36689/

[GitHub] [spark] SparkQA removed a comment on pull request #30567: [SPARK-33142][SQL] Store SQL text for SQL temp view

SparkQA removed a comment on pull request #30567: URL: https://github.com/apache/spark/pull/30567#issuecomment-737654886 **[Test build #132086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132086/testReport)** for PR 30567 at commit

[GitHub] [spark] SparkQA commented on pull request #30567: [SPARK-33142][SQL] Store SQL text for SQL temp view

SparkQA commented on pull request #30567: URL: https://github.com/apache/spark/pull/30567#issuecomment-737685247 **[Test build #132086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132086/testReport)** for PR 30567 at commit

[GitHub] [spark] zsxwing commented on a change in pull request #24173: [SPARK-27237][SS] Introduce State schema validation among query restart

zsxwing commented on a change in pull request #24173: URL: https://github.com/apache/spark/pull/24173#discussion_r534679600 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateSchemaCompatibilityChecker.scala ## @@ -0,0 +1,142 @@ +/* + *

  1   2   3   4   5   6   7   8   9   >