[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-633161966 This is tested with the latest JDK8 and K8s 1.17.x manually and backported to `branch-3.0`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-633108612 Due to the conflicts on dependency file, I need to make a backporting PR separately. BTW, `branch-3.0` is currently broken by https://github.com/apache/spark/pull/28604. The backporting PR will fail due the above failure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-633108151 Thank you all. Merged to master/branch-3.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632860673 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632694613 Thank you for review, @maver1ck and @ScrapCodes . In this PR, I tried 4.10.1 and 4.10.0 first. However, the dependency change was too huge. And, `kubernetes-model` was split into many jars in 4.10.x. Given the risk and complexity of 4.10.x, 4.9.2 is more safer option which having a minimal change for this issue. Since we need backport this PR to 3.0.0 RC3 and 2.4.6 RC4, minimizing risk is important. We can revisit 4.10.x later for the other issues. - https://github.com/apache/spark/pull/28601/commits/760030bbb2f50569ef1d2c5427b14d53d2996ef1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632467085 `WorkerDecommissionSuite` is irrelevant to this PR. ``` org.apache.spark.scheduler.WorkerDecommissionSuite.verify a task with all workers decommissioned succeeds ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632439127 cc @dbtsai and @holdenk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632420623 The one failure in the last K8s IT is a known flaky one. ``` KubernetesSuite: - Run SparkPi with no resources *** FAILED *** The code passed to eventually never returned normally. Attempted 130 times over 2.001735922216 minutes. Last failure message: false was not true. (KubernetesSuite.scala:370) - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage - Launcher client dependencies - Test basic decommissioning - Run SparkR on simple dataframe.R example Run completed in 14 minutes, 43 seconds. Total number of tests run: 20 Suites: completed 2, aborted 0 Tests: succeeded 19, failed 1, canceled 0, ignored 0, pending 0 *** 1 TEST FAILED *** ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632420252 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28601: [SPARK-31786][K8S][BUILD] Upgrade kubernetes-client to 4.9.2
dongjoon-hyun commented on pull request #28601: URL: https://github.com/apache/spark/pull/28601#issuecomment-632397839 Since there is a revert on master branch (https://github.com/apache/spark/commit/db5e5fce68f7e3a724eb4516908c4ba9786b0db3), I'll rebase once more to make it sure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
