Hi Wilson,
As Yinan well said, for batch jobs with dynamic scaling requirements and
communication between driver and executor, it does not fit into the service
oriented Deployment paradigm of k8s. Thus we have the need to abstract
these spark specific differences to k8s CRD and CRD controller to
Hi Wilson,
The behavior of a Deployment doesn't fit with the way Spark executor pods
are run and managed. For example, executor pods are created and deleted per
the requests from the driver dynamically and normally they run to
completion. A Deployment assumes uniformity and statelessness of the
Hi,
I've been playing around with Spark Kubernetes deployments over the past week
and I'm curious to know why Spark deploys as a driver pod that creates more
worker pods.
I've read that it's normal to use Kubernetes Deployments to create a
distributed service, so I am wondering why Spark just
