Hi Ofir.
Right now, the push based shuffle within Spark is only supported for Spark
on YARN, with external shuffle service running as auxiliary service in
NodeManager, but not natively on K8s.
As far as I know, there are no recent plans to add the support for Spark on
K8s natively.
For question 2,
Hi Ofir,
I can provide some information about use cases for Apache Celeborn.
Apache Celeborn can be deployed on K8s and standalone, both are widely
used in production environment by users. The largest cluster I know contains
more than 1,000 Celeborn workers.
Celeborn is specially beneficial for
Hi,
Regarding the external shuffle service on K8S and especially the push-based
variant that was merged in 3.2:
1.
Are there plans to make it supported and work out-of-the-box in 4.0?
2.
Did anyone make it work for themselves in 3.5 or earlier? If so, can you share
your experience and what w
