Hi Yikun, GKE <https://cloud.google.com/kubernetes-engine> is Google's Kubernetes engine first in the market and pretty stable.
The cluster deployed is a 3 node GKE with 4 Vcores and 16GB of RAM each. Autoscaling is on to take nodes from 3 to 6. So it is pretty robust. I did 15 sequences of tests with the following results [image: image.png] Now again the readings from the standard spark-submit are pretty stable with the standard deviation of 3., compared to volcano with the standard deviation of 13.6. What is the latency for FEATURES=”org.apache.spark.deploy.k8s.features.VolcanoFeatureStep”, could that be one reason? Regards, Mich view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Fri, 25 Feb 2022 at 02:51, Yikun Jiang <yikunk...@gmail.com> wrote: > @dongjoon-hyun @yangwwei Thanks! > > @Mich Thanks for testing it, I'm not very professional with GKE, > > I'm also not quite sure if it is different in configurations, internal > network, scheduler implementations > itself VS upstream K8S. As far as I know, different K8S vendors also > maintain their own optimizations > in their downstream product. > > But you can see some basic integration test results based on upstream K8S > on x86/arm64: > - x86: https://github.com/apache/spark/pull/35422#issuecomment-1035901775 > - Arm64: > https://github.com/apache/spark/pull/35422#issuecomment-1037039764 > > As can be seen from the results, for a single job, there is no big > difference between default scheduler > and volcano. > > Also custom schedulers such as Volcano, Yunikorn are more for the overall > situation for multiple jobs > and the utilization of the entire K8S cluster. >
