Hi, There is an unmerged PR which can be used against spark 2.4(if you are interested) or master branch(3.0). Spark 2.3 K8S lacks lot of features. I suggest you upgrade to 2.4 which will be released in few days from now.
https://github.com/apache/spark/pull/22722 Regards Surya From: Matt Cheah <mch...@palantir.com> Sent: Saturday, October 27, 2018 6:12 AM To: Li Gao <eyesofho...@gmail.com>; vincent.gromakow...@gmail.com Cc: caolijun1...@gmail.com; user@spark.apache.org Subject: Re: External shuffle service on K8S Hi there, Please see https://issues.apache.org/jira/browse/SPARK-25299 for more discussion around this matter. -Matt Cheah From: Li Gao <eyesofho...@gmail.com<mailto:eyesofho...@gmail.com>> Date: Friday, October 26, 2018 at 9:10 AM To: "vincent.gromakow...@gmail.com<mailto:vincent.gromakow...@gmail.com>" <vincent.gromakow...@gmail.com<mailto:vincent.gromakow...@gmail.com>> Cc: "caolijun1...@gmail.com<mailto:caolijun1...@gmail.com>" <caolijun1...@gmail.com<mailto:caolijun1...@gmail.com>>, "user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>> Subject: Re: External shuffle service on K8S There are existing 2.2 based ext shuffle on the fork: https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html [apache-spark-on-k8s.github.io]<https://urldefense.proofpoint.com/v2/url?u=https-3A__apache-2Dspark-2Don-2Dk8s.github.io_userdocs_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=gB40LFy6y_6gk7_ZrhJYSJx3zyZLmmvfIEQeZvVpLog&s=enaZSDsU2MW_A_HY6l9TvqhhqoKOMWjbPvEOngFz1oA&e=> You can modify it to suit your needs. -Li On Fri, Oct 26, 2018 at 3:22 AM vincent gromakowski <vincent.gromakow...@gmail.com<mailto:vincent.gromakow...@gmail.com>> wrote: No it's on the roadmap >2.4 Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <caolijun1...@gmail.com<mailto:caolijun1...@gmail.com>> a écrit : Hi all: Does Spark 2.3.2 supports external shuffle service on Kubernetes? I have looked up the documentation(https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_latest_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=gB40LFy6y_6gk7_ZrhJYSJx3zyZLmmvfIEQeZvVpLog&s=1uVGe6F4jUe4jMJnoKyMPlbr4qanZg4AL4vVWuwMjaY&e=>), but couldn't find related suggestions. If suppports, how can I enable it? Best Regards Lijun Cao
