[GitHub] [spark] zwangsheng closed pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code

zwangsheng closed pull request #40118: [SPARK-26365][K8S] In kuberentes cluster mode, spark submit should pass driver exit code URL: https://github.com/apache/spark/pull/40118 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] zhengruifeng commented on pull request #42942: [SPARK-45168][PYTHON][FOLLOWUP] `test_missing_data.py` Code Cleanup

zhengruifeng commented on PR #42942: URL: https://github.com/apache/spark/pull/42942#issuecomment-1720747786 thanks @dongjoon-hyun the CI link is https://github.com/zhengruifeng/spark/actions/runs/6194672684 -- This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] zhengruifeng commented on pull request #42942: [SPARK-45168][PYTHON][FOLLOWUP] `test_missing_data.py` Code Cleanup

zhengruifeng commented on PR #42942: URL: https://github.com/apache/spark/pull/42942#issuecomment-1720745593 after this PR, ``` (spark_dev_310) ➜ spark git:(inc_pd_clean_up) ag --py 'pandas\.__version' python python/pyspark/sql/pandas/utils.py 37:if

[GitHub] [spark] zhengruifeng opened a new pull request, #42942: [SPARK-45168][PYTHON][FOLLOWUP] Code Cleanup

zhengruifeng opened a new pull request, #42942: URL: https://github.com/apache/spark/pull/42942 ### What changes were proposed in this pull request? remove unreachable code path ### Why are the changes needed? code cleanup ### Does this PR introduce _any_ user-facing

[GitHub] [spark] dongjoon-hyun commented on pull request #42930: [SPARK-45168][PYTHON] Increase Pandas minimum version to 1.4.4

dongjoon-hyun commented on PR #42930: URL: https://github.com/apache/spark/pull/42930#issuecomment-1720743852 No problem at all~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [spark] itholic commented on pull request #42941: [WIP][SPARK-43874][FOLLOWUP][TESTS] Enable `GroupbyIndexTests.test_groupby_multiindex_columns`

itholic commented on PR #42941: URL: https://github.com/apache/spark/pull/42941#issuecomment-1720740081 Let me find some more test could be enabled while I'm here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [spark] zhengruifeng commented on pull request #42930: [SPARK-45168][PYTHON] Increase Pandas minimum version to 1.4.4

zhengruifeng commented on PR #42930: URL: https://github.com/apache/spark/pull/42930#issuecomment-1720737302 oh, sorry, I find there are still some similar places to clean up, let me create a follow-up PR -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] itholic opened a new pull request, #42941: [SPARK-43874][FOLLOWUP][TESTS] Enable `GroupbyIndexTests.test_groupby_multiindex_columns`

itholic opened a new pull request, #42941: URL: https://github.com/apache/spark/pull/42941 ### What changes were proposed in this pull request? Follow-up for https://github.com/apache/spark/pull/42533. ### Why are the changes needed? To enable test.

[GitHub] [spark] dongjoon-hyun commented on pull request #42936: [SPARK-45174][CORE] Support `spark.deploy.maxDrivers`

dongjoon-hyun commented on PR #42936: URL: https://github.com/apache/spark/pull/42936#issuecomment-1720730875 For the log message, the log message could be very verbose because `schedule` method is invoked at every submission. For example, if we submit 500 jobs with the max limi 10,

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #42936: [SPARK-45174][CORE] Support `spark.deploy.maxDrivers`

dongjoon-hyun commented on code in PR #42936: URL: https://github.com/apache/spark/pull/42936#discussion_r1326839780 ## core/src/main/scala/org/apache/spark/deploy/master/Master.scala: ## @@ -844,8 +845,8 @@ private[deploy] class Master( // We assign workers to each

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #42904: [SPARK-45151][CORE][UI] Task Level Thread Dump Support

dongjoon-hyun commented on code in PR #42904: URL: https://github.com/apache/spark/pull/42904#discussion_r1326838058 ## core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala: ## @@ -172,6 +180,18 @@ private[v1] class AbstractApplicationResource extends

[GitHub] [spark] dongjoon-hyun commented on pull request #42929: [SPARK-45167][CONNECT] Python client must call `release_all`

dongjoon-hyun commented on PR #42929: URL: https://github.com/apache/spark/pull/42929#issuecomment-1720724824 Got it. Thank you for updating. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [spark] grundprinzip commented on pull request #42929: [SPARK-45167][CONNECT] Python client must call `release_all`

grundprinzip commented on PR #42929: URL: https://github.com/apache/spark/pull/42929#issuecomment-1720723378 @HyukjinKwon @juliuszsompolski I fixed the issue that stemmed from the test shutting down the channel without cleaning up the threadpool of the generator used for the release

<    1   2