[GitHub] [spark] SparkQA commented on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow

SparkQA commented on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675890659 **[Test build #127626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127626/testReport)** for PR 29474 at commit [`dd38f6d`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

SparkQA commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675890700 **[Test build #127627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127627/testReport)** for PR 29452 at commit [`9222f05`](https://github.com

[GitHub] [spark] jiangxb1987 commented on pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow

jiangxb1987 commented on pull request #29474: URL: https://github.com/apache/spark/pull/29474#issuecomment-675889531 cc @mccheah @cloud-fan @Ngone51 @zhengruifeng This is an automated message from the Apache Git Service. To

[GitHub] [spark] jiangxb1987 opened a new pull request #29474: [SPARK-32658][CORE] Fix `PartitionWriterStream` partition length overflow

jiangxb1987 opened a new pull request #29474: URL: https://github.com/apache/spark/pull/29474 # What changes were proposed in this pull request? The `count` in `PartitionWriterStream` should be a long value, instead of int. The issue is introduced by apache/spark@abef84a . When the o

[GitHub] [spark] cloud-fan closed pull request #29466: [SPARK-32651][CORE] Decommission switch configuration should have the highest hierarchy

cloud-fan closed pull request #29466: URL: https://github.com/apache/spark/pull/29466 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [spark] cloud-fan commented on pull request #29466: [SPARK-32651][CORE] Decommission switch configuration should have the highest hierarchy

cloud-fan commented on pull request #29466: URL: https://github.com/apache/spark/pull/29466#issuecomment-675888359 thanks, merging to master! This is an automated message from the Apache Git Service. To respond to the message

[GitHub] [spark] agrawaldevesh commented on a change in pull request #29455: [SPARK-32644][SQL] NAAJ support for ShuffleHashJoin when AQE is on

agrawaldevesh commented on a change in pull request #29455: URL: https://github.com/apache/spark/pull/29455#discussion_r472772355 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala ## @@ -83,15 +83,18 @@ trait ShuffleExchang

[GitHub] [spark] cloud-fan commented on a change in pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

cloud-fan commented on a change in pull request #29452: URL: https://github.com/apache/spark/pull/29452#discussion_r472769129 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ## @@ -1051,15 +1049,21 @@ private[spark] class TaskSetManager(

[GitHub] [spark] cloud-fan commented on a change in pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

cloud-fan commented on a change in pull request #29452: URL: https://github.com/apache/spark/pull/29452#discussion_r472767373 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ## @@ -926,18 +926,21 @@ private[spark] class TaskSchedulerImpl(

[GitHub] [spark] agrawaldevesh commented on pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl

agrawaldevesh commented on pull request #29468: URL: https://github.com/apache/spark/pull/29468#issuecomment-675883231 Thanks for updating the PR description. Now all we need is a real test to validate that the PR is actually fixing the delay scheduling issue you pointed out. The current t

[GitHub] [spark] agrawaldevesh commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl

agrawaldevesh commented on a change in pull request #29468: URL: https://github.com/apache/spark/pull/29468#discussion_r472762189 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ## @@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerI

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

AmplabJenkins removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675881350 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127

[GitHub] [spark] SparkQA removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

SparkQA removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675880439 **[Test build #127625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127625/testReport)** for PR 29452 at commit [`6a5be83`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

AmplabJenkins removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675881340 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To r

[GitHub] [spark] agrawaldevesh commented on a change in pull request #29468: [SPARK-32653][CORE] Decommissioned host/executor should be considered as inactive in TaskSchedulerImpl

agrawaldevesh commented on a change in pull request #29468: URL: https://github.com/apache/spark/pull/29468#discussion_r472760587 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ## @@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerI

[GitHub] [spark] SparkQA commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

SparkQA commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675881324 **[Test build #127625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127625/testReport)** for PR 29452 at commit [`6a5be83`](https://github.co

[GitHub] [spark] AmplabJenkins commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

AmplabJenkins commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675881340 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

AmplabJenkins removed a comment on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675880787 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

AmplabJenkins commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675880787 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] cloud-fan commented on a change in pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters

cloud-fan commented on a change in pull request #29469: URL: https://github.com/apache/spark/pull/29469#discussion_r472758250 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/AlreadyPlanned.scala ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software F

[GitHub] [spark] cloud-fan commented on a change in pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters

cloud-fan commented on a change in pull request #29469: URL: https://github.com/apache/spark/pull/29469#discussion_r472758250 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/AlreadyPlanned.scala ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software F

[GitHub] [spark] SparkQA commented on pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

SparkQA commented on pull request #29452: URL: https://github.com/apache/spark/pull/29452#issuecomment-675880439 **[Test build #127625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127625/testReport)** for PR 29452 at commit [`6a5be83`](https://github.com

[GitHub] [spark] agrawaldevesh commented on a change in pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

agrawaldevesh commented on a change in pull request #29452: URL: https://github.com/apache/spark/pull/29452#discussion_r472756046 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ## @@ -1051,15 +1049,19 @@ private[spark] class TaskSetManager(

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29473: [SPARK-32656][SQL] Repartition bucketed tables for sort merge join / shuffled hash join if applicable

AmplabJenkins removed a comment on pull request #29473: URL: https://github.com/apache/spark/pull/29473#issuecomment-675876095 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29473: [SPARK-32656][SQL] Repartition bucketed tables for sort merge join / shuffled hash join if applicable

AmplabJenkins commented on pull request #29473: URL: https://github.com/apache/spark/pull/29473#issuecomment-675876095 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #29473: [SPARK-32656][SQL] Repartition bucketed tables for sort merge join / shuffled hash join if applicable

SparkQA removed a comment on pull request #29473: URL: https://github.com/apache/spark/pull/29473#issuecomment-675797404 **[Test build #127609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127609/testReport)** for PR 29473 at commit [`21882ab`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29473: [SPARK-32656][SQL] Repartition bucketed tables for sort merge join / shuffled hash join if applicable

SparkQA commented on pull request #29473: URL: https://github.com/apache/spark/pull/29473#issuecomment-675875114 **[Test build #127609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127609/testReport)** for PR 29473 at commit [`21882ab`](https://github.co

[GitHub] [spark] HyukjinKwon edited a comment on pull request #29456: [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter

HyukjinKwon edited a comment on pull request #29456: URL: https://github.com/apache/spark/pull/29456#issuecomment-675874159 I merged into branch-3.0 and branch-2.4 as well. cc @zhengruifeng and @ScrapCodes . this is test-only so should be fine.

[GitHub] [spark] HyukjinKwon commented on pull request #29456: [SPARK-32647][INFRA] Report SparkR test results with JUnit reporter

HyukjinKwon commented on pull request #29456: URL: https://github.com/apache/spark/pull/29456#issuecomment-675874159 I merged into branch-3.0 as well. cc @zhengruifeng. this is test-only so should be fine. This is an automat

[GitHub] [spark] HyukjinKwon removed a comment on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0

HyukjinKwon removed a comment on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675390620 All tests passed (I am re-running to verify flakiness) except two tests: - `ThriftServerQueryTestSuite`: ``` [info] org.apache.spark.sql.hive.thrifts

[GitHub] [spark] HyukjinKwon edited a comment on pull request #29460: [SPARK-32249][INFRA][3.0] Run Github Actions builds in branch-3.0

HyukjinKwon edited a comment on pull request #29460: URL: https://github.com/apache/spark/pull/29460#issuecomment-675831293 Should be ready to be reviewed or go merging. Arrow related SparkR tests which will be fixed at #29462. ---

[GitHub] [spark] LuciferYang commented on a change in pull request #29000: [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode

LuciferYang commented on a change in pull request #29000: URL: https://github.com/apache/spark/pull/29000#discussion_r472737943 ## File path: sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala ## @@ -164,4 +165,48 @@ class PartitionedWriteSuite ex

[GitHub] [spark] HyukjinKwon closed pull request #29366: [SPARK-32550][SQL] Make SpecificInternalRow constructors faster by using while loops instead of maps

HyukjinKwon closed pull request #29366: URL: https://github.com/apache/spark/pull/29366 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on pull request #29366: [SPARK-32550][SQL] Make SpecificInternalRow constructors faster by using while loops instead of maps

HyukjinKwon commented on pull request #29366: URL: https://github.com/apache/spark/pull/29366#issuecomment-675868616 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [spark] cloud-fan commented on a change in pull request #29455: [SPARK-32644][SQL] NAAJ support for ShuffleHashJoin when AQE is on

cloud-fan commented on a change in pull request #29455: URL: https://github.com/apache/spark/pull/29455#discussion_r472731516 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala ## @@ -83,15 +83,18 @@ trait ShuffleExchangeLik

[GitHub] [spark] LuciferYang commented on a change in pull request #29000: [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode

LuciferYang commented on a change in pull request #29000: URL: https://github.com/apache/spark/pull/29000#discussion_r472729801 ## File path: sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala ## @@ -164,4 +165,48 @@ class PartitionedWriteSuite ex

[GitHub] [spark] LuciferYang commented on a change in pull request #29000: [SPARK-27194][SPARK-29302][SQL] Fix commit collision in dynamic partition overwrite mode

LuciferYang commented on a change in pull request #29000: URL: https://github.com/apache/spark/pull/29000#discussion_r472729801 ## File path: sql/core/src/test/scala/org/apache/spark/sql/sources/PartitionedWriteSuite.scala ## @@ -164,4 +165,48 @@ class PartitionedWriteSuite ex

[GitHub] [spark] cloud-fan commented on pull request #29455: [SPARK-32644][SQL] NAAJ support for ShuffleHashJoin when AQE is on

cloud-fan commented on pull request #29455: URL: https://github.com/apache/spark/pull/29455#issuecomment-675866532 I think the patent is about a specific way to optimize null aware anti join, not the general idea. We are definitely different from Oracle as Spark is distributed. -

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources

AmplabJenkins removed a comment on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675862619 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29470: [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation

AmplabJenkins removed a comment on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675862630 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters

AmplabJenkins removed a comment on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675862636 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters

AmplabJenkins commented on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675862636 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources

AmplabJenkins commented on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675862619 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #29470: [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation

AmplabJenkins commented on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675862630 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA removed a comment on pull request #29470: [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation

SparkQA removed a comment on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675852755 **[Test build #127621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127621/testReport)** for PR 29470 at commit [`d9933b7`](https://gi

[GitHub] [spark] SparkQA commented on pull request #29470: [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation

SparkQA commented on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675862353 **[Test build #127621 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127621/testReport)** for PR 29470 at commit [`d9933b7`](https://github.co

[GitHub] [spark] SparkQA commented on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters

SparkQA commented on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675862342 **[Test build #127623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127623/testReport)** for PR 29469 at commit [`dd3e558`](https://github.com

[GitHub] [spark] SparkQA commented on pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources

SparkQA commented on pull request #29437: URL: https://github.com/apache/spark/pull/29437#issuecomment-675862392 **[Test build #127624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127624/testReport)** for PR 29437 at commit [`6f147ee`](https://github.com

[GitHub] [spark] imback82 commented on a change in pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources

imback82 commented on a change in pull request #29437: URL: https://github.com/apache/spark/pull/29437#discussion_r472718669 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextTable.scala ## @@ -28,11 +28,11 @@ import org.apache.spark.

[GitHub] [spark] brkyvz commented on pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters

brkyvz commented on pull request #29469: URL: https://github.com/apache/spark/pull/29469#issuecomment-675861260 @cloud-fan addressed your comments This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [spark] HyukjinKwon commented on pull request #29470: [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation

HyukjinKwon commented on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675860612 BTW your new profile pic looks nice @srowen! This is an automated message from the Apache Git Service. To re

[GitHub] [spark] HyukjinKwon closed pull request #29470: [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation

HyukjinKwon closed pull request #29470: URL: https://github.com/apache/spark/pull/29470 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on pull request #29470: [MINOR][DOCS] Add KMeansSummary and InheritableThread to documentation

HyukjinKwon commented on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675859877 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675858657 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

AmplabJenkins commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675858657 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675857879 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127

[GitHub] [spark] viirya commented on a change in pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources

viirya commented on a change in pull request #29437: URL: https://github.com/apache/spark/pull/29437#discussion_r472711665 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextTable.scala ## @@ -28,11 +28,11 @@ import org.apache.spark.sq

[GitHub] [spark] SparkQA commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

SparkQA commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675858327 **[Test build #127622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127622/testReport)** for PR 29465 at commit [`84846a8`](https://github.com

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675857822 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

HyukjinKwon commented on a change in pull request #29465: URL: https://github.com/apache/spark/pull/29465#discussion_r472710508 ## File path: .github/workflows/build_and_test.yml ## @@ -0,0 +1,241 @@ +name: Build and test + +on: + push: +branches: +- branch-2.4 + pull

[GitHub] [spark] AmplabJenkins commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

AmplabJenkins commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675857822 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

HyukjinKwon commented on a change in pull request #29465: URL: https://github.com/apache/spark/pull/29465#discussion_r472709812 ## File path: .github/workflows/build_and_test.yml ## @@ -0,0 +1,241 @@ +name: Build and test + +on: + push: +branches: +- branch-2.4 + pull

[GitHub] [spark] HyukjinKwon commented on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

HyukjinKwon commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675857451 It should be ready to be reviewed or merged now. This is an automated message from the Apache Git Service. T

[GitHub] [spark] HyukjinKwon removed a comment on pull request #29465: [SPARK-32249][INFRA][2.4] Run Github Actions builds in branch-2.4

HyukjinKwon removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675837138 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] c21 commented on a change in pull request #29455: [SPARK-32644][SQL] NAAJ support for ShuffleHashJoin when AQE is on

c21 commented on a change in pull request #29455: URL: https://github.com/apache/spark/pull/29455#discussion_r472706333 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/EliminateNullAwareAntiJoin.scala ## @@ -20,22 +20,50 @@ package org.apache.spar

[GitHub] [spark] AmplabJenkins commented on pull request #29470: [MINOR][DOCS] Add KMeansSummary to pyspark clustering.py exports

AmplabJenkins commented on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675853113 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29470: [MINOR][DOCS] Add KMeansSummary to pyspark clustering.py exports

AmplabJenkins removed a comment on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675853113 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #29470: [MINOR][DOCS] Add KMeansSummary to pyspark clustering.py exports

SparkQA commented on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675852755 **[Test build #127621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127621/testReport)** for PR 29470 at commit [`d9933b7`](https://github.com

[GitHub] [spark] cloud-fan commented on a change in pull request #29467: [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder

cloud-fan commented on a change in pull request #29467: URL: https://github.com/apache/spark/pull/29467#discussion_r472697970 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala ## @@ -107,4 +108,34 @@ class Obje

[GitHub] [spark] cloud-fan commented on a change in pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources

cloud-fan commented on a change in pull request #29437: URL: https://github.com/apache/spark/pull/29437#discussion_r472696179 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala ## @@ -56,6 +57,13 @@ trait FileDataSourceV2

[GitHub] [spark] cloud-fan commented on a change in pull request #29437: [SPARK-32621][SQL] 'path' option can cause issues while inferring schema in CSV/JSON datasources

cloud-fan commented on a change in pull request #29437: URL: https://github.com/apache/spark/pull/29437#discussion_r472694528 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextTable.scala ## @@ -28,11 +28,11 @@ import org.apache.spark

[GitHub] [spark] HyukjinKwon commented on pull request #29470: [MINOR][DOCS] Add KMeansSummary to pyspark clustering.py exports

HyukjinKwon commented on pull request #29470: URL: https://github.com/apache/spark/pull/29470#issuecomment-675850875 Oh actually we should add it into `python/docs/source/reference/pyspark.ml.rst` now. I know another missing one. Let me push it to here and just merge. ---

[GitHub] [spark] cloud-fan commented on a change in pull request #29104: [SPARK-32290][SQL] SingleColumn Null Aware Anti Join Optimize

cloud-fan commented on a change in pull request #29104: URL: https://github.com/apache/spark/pull/29104#discussion_r472691173 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala ## @@ -903,15 +910,61 @@ private[joins] object LongHash

[GitHub] [spark] HyukjinKwon commented on pull request #29467: [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder

HyukjinKwon commented on pull request #29467: URL: https://github.com/apache/spark/pull/29467#issuecomment-675849620 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] HyukjinKwon closed pull request #29467: [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder

HyukjinKwon closed pull request #29467: URL: https://github.com/apache/spark/pull/29467 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29467: [SPARK-32652][SQL] ObjectSerializerPruning fails for RowEncoder

HyukjinKwon commented on a change in pull request #29467: URL: https://github.com/apache/spark/pull/29467#discussion_r472689757 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala ## @@ -107,4 +108,34 @@ class Ob

[GitHub] [spark] c21 commented on a change in pull request #29455: [SPARK-32644][SQL] NAAJ support for ShuffleHashJoin when AQE is on

c21 commented on a change in pull request #29455: URL: https://github.com/apache/spark/pull/29455#discussion_r472680557 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala ## @@ -83,15 +83,18 @@ trait ShuffleExchangeLike exte

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC

AmplabJenkins removed a comment on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675844857 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC

AmplabJenkins commented on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675844857 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #28953: [SPARK-32013][SQL] Support query execution before reading DataFrame and before/after writing DataFrame over JDBC

SparkQA commented on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-675844652 **[Test build #127620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127620/testReport)** for PR 28953 at commit [`13f0dfc`](https://github.com

[GitHub] [spark] brkyvz commented on a change in pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters

brkyvz commented on a change in pull request #29469: URL: https://github.com/apache/spark/pull/29469#discussion_r472671268 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/AlreadyPlannedSuite.scala ## @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] AmplabJenkins commented on pull request #29453: [SPARK-31999][SQL][FOLLOWUP] Adds negative test cases with typos

AmplabJenkins commented on pull request #29453: URL: https://github.com/apache/spark/pull/29453#issuecomment-675841844 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29453: [SPARK-31999][SQL][FOLLOWUP] Adds negative test cases with typos

AmplabJenkins removed a comment on pull request #29453: URL: https://github.com/apache/spark/pull/29453#issuecomment-675841844 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] brkyvz commented on a change in pull request #29469: [SPARK-28863][SQL] Introduce AlreadyPlanned to prevent reanalysis of V1FallbackWriters

brkyvz commented on a change in pull request #29469: URL: https://github.com/apache/spark/pull/29469#discussion_r472668735 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/AlreadyPlanned.scala ## @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foun

[GitHub] [spark] SparkQA commented on pull request #29453: [SPARK-31999][SQL][FOLLOWUP] Adds negative test cases with typos

SparkQA commented on pull request #29453: URL: https://github.com/apache/spark/pull/29453#issuecomment-675841543 **[Test build #127619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127619/testReport)** for PR 29453 at commit [`b99ced4`](https://github.com

[GitHub] [spark] ulysses-you commented on a change in pull request #29453: [SPARK-31999][SQL][FOLLOWUP] Adds negative test cases with typos

ulysses-you commented on a change in pull request #29453: URL: https://github.com/apache/spark/pull/29453#discussion_r472667471 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ## @@ -3035,7 +3035,10 @@ abstract class DDLSuite extends

[GitHub] [spark] Ngone51 commented on a change in pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

Ngone51 commented on a change in pull request #29452: URL: https://github.com/apache/spark/pull/29452#discussion_r472663605 ## File path: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ## @@ -1051,15 +1049,19 @@ private[spark] class TaskSetManager( l

[GitHub] [spark] LuciferYang commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13

LuciferYang commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675838627 Thx~ @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13

AmplabJenkins removed a comment on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675838315 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][2.4] Run Github Actions builds in other branches as well

AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675838241 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13

AmplabJenkins commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675838315 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] AmplabJenkins commented on pull request #29465: [SPARK-32249][2.4] Run Github Actions builds in other branches as well

AmplabJenkins commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675838241 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [spark] SparkQA commented on pull request #29465: [SPARK-32249][2.4] Run Github Actions builds in other branches as well

SparkQA commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675838032 **[Test build #127617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127617/testReport)** for PR 29465 at commit [`1393174`](https://github.com

[GitHub] [spark] leanken commented on a change in pull request #29455: [SPARK-32644][SQL] NAAJ support for ShuffleHashJoin when AQE is on

leanken commented on a change in pull request #29455: URL: https://github.com/apache/spark/pull/29455#discussion_r472660831 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala ## @@ -225,7 +231,8 @@ object ShuffleExchangeExec

[GitHub] [spark] SparkQA commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13

SparkQA commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675838022 **[Test build #127618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127618/testReport)** for PR 29434 at commit [`36a317a`](https://github.com

[GitHub] [spark] HyukjinKwon commented on pull request #29434: [SPARK-32526][SQL] Pass all test of sql/catalyst module in Scala 2.13

HyukjinKwon commented on pull request #29434: URL: https://github.com/apache/spark/pull/29434#issuecomment-675837861 retest this please This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [spark] SparkQA removed a comment on pull request #29465: [SPARK-32249][2.4] Run Github Actions builds in other branches as well

SparkQA removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675831629 **[Test build #127615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127615/testReport)** for PR 29465 at commit [`1393174`](https://gi

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29465: [SPARK-32249][2.4] Run Github Actions builds in other branches as well

AmplabJenkins removed a comment on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675832423 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] HyukjinKwon commented on pull request #29465: [SPARK-32249][2.4] Run Github Actions builds in other branches as well

HyukjinKwon commented on pull request #29465: URL: https://github.com/apache/spark/pull/29465#issuecomment-675837138 retest this please This is an automated message from the Apache Git Service. To respond to the message, plea

[GitHub] [spark] Ngone51 commented on a change in pull request #29452: [SPARK-32643][CORE] Consolidate state decommissioning in the TaskSchedulerImpl realm

Ngone51 commented on a change in pull request #29452: URL: https://github.com/apache/spark/pull/29452#discussion_r472657515 ## File path: core/src/main/scala/org/apache/spark/scheduler/ExecutorDecommissionInfo.scala ## @@ -18,11 +18,22 @@ package org.apache.spark.scheduler

  1   2   3   4   5   6   7   8   9   10   >