[GitHub] [spark] HyukjinKwon commented on pull request #29853: [SPARK-SQL][SPARK-32977] Fix JavaDoc on Default Save Mode

HyukjinKwon commented on pull request #29853: URL: https://github.com/apache/spark/pull/29853#issuecomment-698079990 Yeah, that's a known flaky test. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] viirya commented on a change in pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain

viirya commented on a change in pull request #29828: URL: https://github.com/apache/spark/pull/29828#discussion_r494011718 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/JsonSuite.scala ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache

[GitHub] [spark] dongjoon-hyun closed pull request #29859: [SPARK-32971][K8S][FOLLOWUP] Add `.toSeq` for Scala 2.13 compilation

dongjoon-hyun closed pull request #29859: URL: https://github.com/apache/spark/pull/29859 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] dongjoon-hyun commented on pull request #29859: [SPARK-32971][K8S][FOLLOWUP] Add `.toSeq` for Scala 2.13 compilation

dongjoon-hyun commented on pull request #29859: URL: https://github.com/apache/spark/pull/29859#issuecomment-698084404 Scala 2.13 GA job passed. Thank you, @viirya and @HyukjinKwon . Merged to master. This is an automated

[GitHub] [spark] LuciferYang commented on pull request #29861: [SPARK-32971][K8S][FOLLOWUP] Fix k8s core module compile in Scala 2.13

LuciferYang commented on pull request #29861: URL: https://github.com/apache/spark/pull/29861#issuecomment-698100268 cc @dongjoon-hyun The change of SPARK-32971 blocking GitHub 2.13 build Action This is an automated message

[GitHub] [spark] SparkQA commented on pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API

SparkQA commented on pull request #29756: URL: https://github.com/apache/spark/pull/29756#issuecomment-698100892 **[Test build #129062 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129062/testReport)** for PR 29756 at commit

[GitHub] [spark] xuanyuanking commented on a change in pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API

xuanyuanking commented on a change in pull request #29756: URL: https://github.com/apache/spark/pull/29756#discussion_r494027847 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala ## @@ -260,19 +264,47 @@ class

[GitHub] [spark] xuanyuanking commented on a change in pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API

xuanyuanking commented on a change in pull request #29756: URL: https://github.com/apache/spark/pull/29756#discussion_r494027786 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala ## @@ -731,3 +732,11 @@ case class

[GitHub] [spark] xuanyuanking commented on a change in pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API

xuanyuanking commented on a change in pull request #29756: URL: https://github.com/apache/spark/pull/29756#discussion_r494027734 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ## @@ -1020,16 +1039,35 @@ class Analyzer( //

[GitHub] [spark] SparkQA commented on pull request #29861: [SPARK-32971][K8S][FOLLOWUP] Fix k8s core module compile in Scala 2.13

SparkQA commented on pull request #29861: URL: https://github.com/apache/spark/pull/29861#issuecomment-698100859 **[Test build #129061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129061/testReport)** for PR 29861 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29853: [SPARK-SQL][SPARK-32977] Fix JavaDoc on Default Save Mode

AmplabJenkins removed a comment on pull request #29853: URL: https://github.com/apache/spark/pull/29853#issuecomment-697979195 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain

SparkQA commented on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697983565 **[Test build #129038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129038/testReport)** for PR 29828 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain

SparkQA removed a comment on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697669398 **[Test build #129038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129038/testReport)** for PR 29828 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain

AmplabJenkins commented on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697984381 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

AmplabJenkins removed a comment on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-697990499 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

AmplabJenkins commented on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-697990499 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] dongjoon-hyun opened a new pull request #29856: [SPARK-32981][BUILD] Remove hive-1.2/hadoop-2.7 from Apache Spark 3.1 distribution

dongjoon-hyun opened a new pull request #29856: URL: https://github.com/apache/spark/pull/29856 ### What changes were proposed in this pull request? Apache Spark 3.0 switches its Hive execution version from 1.2 to 2.3, but it still provides the unofficial forked Hive 1.2 version

[GitHub] [spark] dongjoon-hyun commented on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

dongjoon-hyun commented on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-698007198 Yes! Finally, it passed. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] holdenk commented on pull request #29533: [SPARK-24266][K8S][3.0] Restart the watcher when we receive a version changed from k8s

holdenk commented on pull request #29533: URL: https://github.com/apache/spark/pull/29533#issuecomment-698013767 Jenkins retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] holdenk commented on pull request #29788: [SPARK-32913][CORE][K8S] Improve ExecutorDecommissionInfo and ExecutorDecommissionState for different use cases

holdenk commented on pull request #29788: URL: https://github.com/apache/spark/pull/29788#issuecomment-698021508 I think it would be good to see your proposal in code @Ngone51 because I'm not 100% sure what you mean. I would really like to see both this and the precursor tested more

[GitHub] [spark] holdenk commented on a change in pull request #29788: [SPARK-32913][CORE][K8S] Improve ExecutorDecommissionInfo and ExecutorDecommissionState for different use cases

holdenk commented on a change in pull request #29788: URL: https://github.com/apache/spark/pull/29788#discussion_r493950073 ## File path: core/src/main/scala/org/apache/spark/scheduler/ExecutorLossReason.scala ## @@ -71,7 +71,8 @@ case class ExecutorProcessLost( * This is

[GitHub] [spark] SparkQA commented on pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors

SparkQA commented on pull request #29846: URL: https://github.com/apache/spark/pull/29846#issuecomment-698021654 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33671/

[GitHub] [spark] huaxingao commented on a change in pull request #29837: [SPARK-32463][DOCS] SQL data type compatibility

huaxingao commented on a change in pull request #29837: URL: https://github.com/apache/spark/pull/29837#discussion_r493965798 ## File path: docs/sql-ref-datatypes.md ## @@ -314,3 +314,33 @@ SELECT COUNT(*), c2 FROM test GROUP BY c2; |3| Infinity|

[GitHub] [spark] github-actions[bot] closed pull request #27096: [SPARK-28148][SQL] Repartition after join is not optimized away

github-actions[bot] closed pull request #27096: URL: https://github.com/apache/spark/pull/27096 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] maropu commented on a change in pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans

maropu commented on a change in pull request #29585: URL: https://github.com/apache/spark/pull/29585#discussion_r493976424 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala ## @@ -458,7 +463,10 @@ object

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans

AmplabJenkins removed a comment on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-698050221 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29856: [SPARK-32981][BUILD] Remove hive-1.2/hadoop-2.7 from Apache Spark 3.1 distribution

AmplabJenkins removed a comment on pull request #29856: URL: https://github.com/apache/spark/pull/29856#issuecomment-698050222 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29856: [SPARK-32981][BUILD] Remove hive-1.2/hadoop-2.7 from Apache Spark 3.1 distribution

AmplabJenkins commented on pull request #29856: URL: https://github.com/apache/spark/pull/29856#issuecomment-698050222 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans

AmplabJenkins commented on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-698050221 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically

AmplabJenkins commented on pull request #29804: URL: https://github.com/apache/spark/pull/29804#issuecomment-698050224 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] wangyum closed pull request #28642: [SPARK-31809][SQL] Infer IsNotNull for non null intolerant child of null intolerant in join condition

wangyum closed pull request #28642: URL: https://github.com/apache/spark/pull/28642 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

AmplabJenkins removed a comment on pull request #29857: URL: https://github.com/apache/spark/pull/29857#issuecomment-698070610 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

AmplabJenkins commented on pull request #29857: URL: https://github.com/apache/spark/pull/29857#issuecomment-698070610 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

SparkQA commented on pull request #29857: URL: https://github.com/apache/spark/pull/29857#issuecomment-698070226 **[Test build #129054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129054/testReport)** for PR 29857 at commit

[GitHub] [spark] SparkQA commented on pull request #29858: [SPARK-32981][BUILD][FOLLOW-UP] Remove hive-1.2 profiles in PIP installation option

SparkQA commented on pull request #29858: URL: https://github.com/apache/spark/pull/29858#issuecomment-698074003 **[Test build #129055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129055/testReport)** for PR 29858 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans

AmplabJenkins removed a comment on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-698073630 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans

AmplabJenkins removed a comment on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-698073627 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans

SparkQA commented on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-698073524 **[Test build #129053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129053/testReport)** for PR 29585 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29585: [SPARK-32741][SQL] Check if the same ExprId refers to the unique attribute in logical plans

AmplabJenkins commented on pull request #29585: URL: https://github.com/apache/spark/pull/29585#issuecomment-698073627 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29859: [SPARK-32971][K8S][FOLLOWUP] Add `.toSeq` for Scala 2.13 compilation

AmplabJenkins removed a comment on pull request #29859: URL: https://github.com/apache/spark/pull/29859#issuecomment-698078601 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA removed a comment on pull request #29859: [SPARK-32971][K8S][FOLLOWUP] Add `.toSeq` for Scala 2.13 compilation

SparkQA removed a comment on pull request #29859: URL: https://github.com/apache/spark/pull/29859#issuecomment-698075792 **[Test build #129056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129056/testReport)** for PR 29859 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29859: [SPARK-32971][K8S][FOLLOWUP] Add `.toSeq` for Scala 2.13 compilation

AmplabJenkins commented on pull request #29859: URL: https://github.com/apache/spark/pull/29859#issuecomment-698078601 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29859: [SPARK-32971][K8S][FOLLOWUP] Add `.toSeq` for Scala 2.13 compilation

SparkQA commented on pull request #29859: URL: https://github.com/apache/spark/pull/29859#issuecomment-698078494 **[Test build #129056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129056/testReport)** for PR 29859 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29858: [SPARK-32982][BUILD] Remove hive-1.2 profiles in PIP installation option

dongjoon-hyun commented on a change in pull request #29858: URL: https://github.com/apache/spark/pull/29858#discussion_r494006706 ## File path: python/pyspark/install.py ## @@ -26,18 +26,13 @@ DEFAULT_HADOOP = "hadoop3.2" DEFAULT_HIVE = "hive2.3" SUPPORTED_HADOOP_VERSIONS =

[GitHub] [spark] maropu commented on a change in pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain

maropu commented on a change in pull request #29828: URL: https://github.com/apache/spark/pull/29828#discussion_r494010395 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/JsonSuite.scala ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache

[GitHub] [spark] dongjoon-hyun closed pull request #29853: [SPARK-32977][SQL][DOCS] Fix JavaDoc on Default Save Mode

dongjoon-hyun closed pull request #29853: URL: https://github.com/apache/spark/pull/29853 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

SparkQA commented on pull request #29857: URL: https://github.com/apache/spark/pull/29857#issuecomment-698087474 **[Test build #129054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129054/testReport)** for PR 29857 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

AmplabJenkins commented on pull request #29800: URL: https://github.com/apache/spark/pull/29800#issuecomment-698087113 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #29859: [SPARK-32971][K8S][FOLLOWUP] Add `.toSeq` for Scala 2.13 compilation

SparkQA commented on pull request #29859: URL: https://github.com/apache/spark/pull/29859#issuecomment-698087229 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33677/

[GitHub] [spark] SparkQA commented on pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

SparkQA commented on pull request #29800: URL: https://github.com/apache/spark/pull/29800#issuecomment-698086807 **[Test build #129058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129058/testReport)** for PR 29800 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29800: [SPARK-32934][SQL] Improve the performance for NTH_VALUE and reactor the OffsetWindowFunction

AmplabJenkins removed a comment on pull request #29800: URL: https://github.com/apache/spark/pull/29800#issuecomment-698087113 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] Ngone51 commented on pull request #29860: [SPARK-32984][TESTS][SQL] Improve showing the differences between approved and actual plans of PlanStabilitySuite

Ngone51 commented on pull request #29860: URL: https://github.com/apache/spark/pull/29860#issuecomment-698090163 cc @cloud-fan @maropu Please take a look, thanks! This is an automated message from the Apache Git Service. To

[GitHub] [spark] Ngone51 opened a new pull request #29860: [SPARK-32984][TESTS][SQL] Improve showing the differences between approved and actual plans of PlanStabilitySuite

Ngone51 opened a new pull request #29860: URL: https://github.com/apache/spark/pull/29860 ### What changes were proposed in this pull request? This PR proposes to add the caret hints, e.g., `^`, to the approved and actual plans where they first become different.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain

AmplabJenkins removed a comment on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-698089064 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API

AmplabJenkins removed a comment on pull request #29756: URL: https://github.com/apache/spark/pull/29756#issuecomment-698101187 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29756: [SPARK-32885][SS] Add DataStreamReader.table API

AmplabJenkins commented on pull request #29756: URL: https://github.com/apache/spark/pull/29756#issuecomment-698101187 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #29855: SPARK-32915 Network-layer and shuffle RPC layer changes to support push shuffle blocks

AmplabJenkins commented on pull request #29855: URL: https://github.com/apache/spark/pull/29855#issuecomment-697980595 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] Victsm commented on pull request #29855: SPARK-32915 Network-layer and shuffle RPC layer changes to support push shuffle blocks

Victsm commented on pull request #29855: URL: https://github.com/apache/spark/pull/29855#issuecomment-697980575 Fixed the Java style issue and the 1 UT failure. Test build should be clean now. This is an automated message

[GitHub] [spark] SparkQA commented on pull request #29855: SPARK-32915 Network-layer and shuffle RPC layer changes to support push shuffle blocks

SparkQA commented on pull request #29855: URL: https://github.com/apache/spark/pull/29855#issuecomment-697980048 **[Test build #129046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129046/testReport)** for PR 29855 at commit

[GitHub] [spark] zero323 commented on pull request #29591: [SPARK-32714][PYTHON] Initial pyspark-stubs port.

zero323 commented on pull request #29591: URL: https://github.com/apache/spark/pull/29591#issuecomment-697980912 Jenkins, retest this please. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29855: SPARK-32915 Network-layer and shuffle RPC layer changes to support push shuffle blocks

AmplabJenkins removed a comment on pull request #29855: URL: https://github.com/apache/spark/pull/29855#issuecomment-697980595 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain

AmplabJenkins removed a comment on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697984381 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

SparkQA commented on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-697989432 **[Test build #129042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129042/testReport)** for PR 29854 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

dongjoon-hyun commented on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-697989257 Thank you, @holdenk ! Let's see K8s IT result. This is an automated message from the Apache Git Service.

[GitHub] [spark] SparkQA removed a comment on pull request #29824: [SPARK-32954][YARN][TEST][test-hadoop2.7][test-maven] Add jakarta.servlet-api test dependency to yarn module to avoid UTs badcase

SparkQA removed a comment on pull request #29824: URL: https://github.com/apache/spark/pull/29824#issuecomment-697376701 **[Test build #129032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129032/testReport)** for PR 29824 at commit

[GitHub] [spark] SparkQA commented on pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain

SparkQA commented on pull request #29828: URL: https://github.com/apache/spark/pull/29828#issuecomment-697994187 **[Test build #129039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129039/testReport)** for PR 29828 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29824: [SPARK-32954][YARN][TEST][test-hadoop2.7][test-maven] Add jakarta.servlet-api test dependency to yarn module to avoid UTs badca

AmplabJenkins removed a comment on pull request #29824: URL: https://github.com/apache/spark/pull/29824#issuecomment-697993651 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29856: [SPARK-32981][BUILD] Remove hive-1.2/hadoop-2.7 from Apache Spark 3.1 distribution

AmplabJenkins commented on pull request #29856: URL: https://github.com/apache/spark/pull/29856#issuecomment-698004577 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29856: [SPARK-32981][BUILD] Remove hive-1.2/hadoop-2.7 from Apache Spark 3.1 distribution

AmplabJenkins removed a comment on pull request #29856: URL: https://github.com/apache/spark/pull/29856#issuecomment-698004577 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29856: [SPARK-32981][BUILD] Remove hive-1.2/hadoop-2.7 from Apache Spark 3.1 distribution

SparkQA commented on pull request #29856: URL: https://github.com/apache/spark/pull/29856#issuecomment-698004283 **[Test build #129049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129049/testReport)** for PR 29856 at commit

[GitHub] [spark] SparkQA commented on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

SparkQA commented on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-698006612 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33669/

[GitHub] [spark] dongjoon-hyun closed pull request #29856: [SPARK-32981][BUILD] Remove hive-1.2/hadoop-2.7 from Apache Spark 3.1 distribution

dongjoon-hyun closed pull request #29856: URL: https://github.com/apache/spark/pull/29856 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins commented on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

AmplabJenkins commented on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-698006632 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

AmplabJenkins removed a comment on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-698006632 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] dongjoon-hyun closed pull request #29854: [SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

dongjoon-hyun closed pull request #29854: URL: https://github.com/apache/spark/pull/29854 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors

dongjoon-hyun commented on a change in pull request #29846: URL: https://github.com/apache/spark/pull/29846#discussion_r493952674 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesVolumeSpec.scala ## @@ -21,7 +21,10 @@

[GitHub] [spark] SparkQA commented on pull request #29533: [SPARK-24266][K8S][3.0] Restart the watcher when we receive a version changed from k8s

SparkQA commented on pull request #29533: URL: https://github.com/apache/spark/pull/29533#issuecomment-698036245 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33672/

[GitHub] [spark] maropu commented on a change in pull request #29804: [SPARK-32859][SQL] Introduce physical rule to decide bucketing dynamically

maropu commented on a change in pull request #29804: URL: https://github.com/apache/spark/pull/29804#discussion_r493991276 ## File path: sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala ## @@ -1012,4 +1014,43 @@ abstract class BucketedReadSuite

[GitHub] [spark] HyukjinKwon commented on pull request #29817: [SPARK-32850][CORE][K8S] Simplify the RPC message flow of decommission

HyukjinKwon commented on pull request #29817: URL: https://github.com/apache/spark/pull/29817#issuecomment-698062962 > but that's only half true, this PR just broke the test some more. @holdenk, you're kidding right? There was only one test failure that was not caused by this PR in

[GitHub] [spark] holdenk commented on pull request #29817: [SPARK-32850][CORE][K8S] Simplify the RPC message flow of decommission

holdenk commented on pull request #29817: URL: https://github.com/apache/spark/pull/29817#issuecomment-698064374 If it helps I called out the part of the PR I believe most likely responsible for that during my code review already.

[GitHub] [spark] dongjoon-hyun commented on pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors

dongjoon-hyun commented on pull request #29846: URL: https://github.com/apache/spark/pull/29846#issuecomment-698069211 Oh, thank you for reporting! Sure! This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] LuciferYang commented on pull request #29824: [SPARK-32954][YARN][TEST][test-hadoop2.7][test-maven] Add jakarta.servlet-api test dependency to yarn module to avoid UTs badcase

LuciferYang commented on pull request #29824: URL: https://github.com/apache/spark/pull/29824#issuecomment-698069141 Jenkins test finally passed. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] HyukjinKwon opened a new pull request #29858: [SPARK-32981][BUILD][FOLLOW-UP] Remove hive-1.2 profiles in PIP installation option

HyukjinKwon opened a new pull request #29858: URL: https://github.com/apache/spark/pull/29858 ### What changes were proposed in this pull request? This PR removes Hive 1.2 option (and therefore `HIVE_VERSION` environment variable as well). ### Why are the changes needed?

[GitHub] [spark] Ngone51 commented on a change in pull request #29817: [SPARK-32850][CORE][K8S] Simplify the RPC message flow of decommission

Ngone51 commented on a change in pull request #29817: URL: https://github.com/apache/spark/pull/29817#discussion_r494005398 ## File path: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala ## @@ -70,7 +70,10 @@ private[deploy] class Worker( if

[GitHub] [spark] HyukjinKwon removed a comment on pull request #29858: [SPARK-32981][BUILD][FOLLOW-UP] Remove hive-1.2 profiles in PIP installation option

HyukjinKwon removed a comment on pull request #29858: URL: https://github.com/apache/spark/pull/29858#issuecomment-698075993 @dongjoon-hyun, this affects end users, not the dev people who will use `hive-1.2` profile. I can file a separate JIRA but whenever the release distributions are

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29806: [SPARK-32187][PYTHON][DOCS] Doc on Python packaging

HyukjinKwon commented on a change in pull request #29806: URL: https://github.com/apache/spark/pull/29806#discussion_r494009107 ## File path: python/docs/source/user_guide/python_packaging.rst ## @@ -0,0 +1,220 @@ +.. Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [spark] Ngone51 commented on pull request #29817: [SPARK-32850][CORE][K8S] Simplify the RPC message flow of decommission

Ngone51 commented on pull request #29817: URL: https://github.com/apache/spark/pull/29817#issuecomment-698080541 > That being said I still have concerns this PR is not sufficiently tested, can you add some more tests for the new flows you've introduced? There's only one new flow

[GitHub] [spark] viirya commented on a change in pull request #29828: [SPARK-32948][SQL] Optimize to_json and from_json expression chain

viirya commented on a change in pull request #29828: URL: https://github.com/apache/spark/pull/29828#discussion_r494015049 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/JsonSuite.scala ## @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

AmplabJenkins commented on pull request #29857: URL: https://github.com/apache/spark/pull/29857#issuecomment-698087732 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

SparkQA removed a comment on pull request #29857: URL: https://github.com/apache/spark/pull/29857#issuecomment-698070226 **[Test build #129054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129054/testReport)** for PR 29857 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

AmplabJenkins removed a comment on pull request #29857: URL: https://github.com/apache/spark/pull/29857#issuecomment-698087732 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] LuciferYang commented on pull request #29857: [SPARK-32972][ML] Fix UTs of `mllib` module in Scala 2.13 except RandomForestRegressorSuite

LuciferYang commented on pull request #29857: URL: https://github.com/apache/spark/pull/29857#issuecomment-698100080 cc @dongjoon-hyun https://github.com/apache/spark/pull/29861 fix GitHub 2.13 build Action, related to k8s module, I will rebase this pr after it merged.

[GitHub] [spark] LuciferYang opened a new pull request #29861: [SPARK-32971][K8S][FOLLOWUP] Fix k8s core module compile in Scala 2.13

LuciferYang opened a new pull request #29861: URL: https://github.com/apache/spark/pull/29861 ### What changes were proposed in this pull request? Manual call `toSeq` of `MountVolumesFeatureStep.getAdditionalKubernetesResources` method because `ArrayBuffer` not a `Seq` in Scala 2.13

[GitHub] [spark] holdenk commented on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

holdenk commented on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-697982902 cc @Ngone51 @HyukjinKwon @ScrapCodes @dongjoon-hyun This is an automated message from the Apache Git Service.

[GitHub] [spark] SparkQA removed a comment on pull request #29854: [WIP][SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

SparkQA removed a comment on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-697883527 **[Test build #129042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129042/testReport)** for PR 29854 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #29854: [SPARK-32937][SPARK-32980][K8S] Fix decom & launcher tests and add some comments to reduce chance of breakage

dongjoon-hyun commented on pull request #29854: URL: https://github.com/apache/spark/pull/29854#issuecomment-698008172 Merged to master because it recovers K8s IT and unblocks K8s module. This is an automated message from

[GitHub] [spark] dongjoon-hyun commented on pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors

dongjoon-hyun commented on pull request #29846: URL: https://github.com/apache/spark/pull/29846#issuecomment-698009764 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] holdenk commented on pull request #29846: [SPARK-32971][K8S] Support dynamic PVC creation/deletion for K8s executors

holdenk commented on pull request #29846: URL: https://github.com/apache/spark/pull/29846#issuecomment-698009613 Thanks for the clarity @dongjoon-hyun LGTM pending jenkins This is an automated message from the Apache Git

[GitHub] [spark] SparkQA removed a comment on pull request #29533: [SPARK-24266][K8S][3.0] Restart the watcher when we receive a version changed from k8s

SparkQA removed a comment on pull request #29533: URL: https://github.com/apache/spark/pull/29533#issuecomment-698015366 **[Test build #129051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129051/testReport)** for PR 29533 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29533: [SPARK-24266][K8S][3.0] Restart the watcher when we receive a version changed from k8s

AmplabJenkins removed a comment on pull request #29533: URL: https://github.com/apache/spark/pull/29533#issuecomment-698020667 This is an automated message from the Apache Git Service. To respond to the message, please log on

<    1   2   3   4   5   6   7   8   >