@Santosh We tried to add this in v3.3.0. [1] The main reason for not adding it at that time was: 1. Volcano multi-arch not supported before v1.7.0. (already upgraded to 1.7.0 since Spark 3.4.0) 2. Spark on K8s + Volcano is experimental. (We have removed the experimental [2])
Consider spark volcano integrations already stable to run on spark community (since spark 3.4.0) [3] and volcano community (since spark 3.3.0) [4] for a long time. I think it's stable enough. So I believe we have the capability to enable the volcano module in Apache Spark now (master / maybe Apache Spark 4.0?). [1] https://github.com/apache/spark/pull/35922 [2] https://github.com/apache/spark/pull/40152 [3] https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L1090 [4] https://github.com/volcano-sh/volcano/blob/master/.github/workflows/e2e_spark.yaml#L12 Regards, Yikun On Tue, Aug 22, 2023 at 8:14 PM Santosh Pingale <santosh.ping...@adyen.com.invalid> wrote: > Hey all > > It would useful to support volcano in spark distro itself just like > yunikorn. So I am wondering what is the reason behind this decision of not > packaging it already. > Running Spark on Kubernetes - Spark 3.4.1 Documentation > <https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes> > spark.apache.org > <https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes> > [image: apple-touch-icon.png] > <https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes> > <https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-volcano-as-customized-scheduler-for-spark-on-kubernetes> > > Can we package it to make it easily available and hence usable? > > Kind regards > Santosh >
