surahman edited a comment on pull request #3710: URL: https://github.com/apache/incubator-heron/pull/3710#issuecomment-920551592
I think I have an idea of what needs to happen... time to iterate :wink:. @nicknezis You are correct, Spark's architecture has a driver/coordinator with a fleet of executors. Is there already a `loadPodFromTemplate` in Heron or do we need to put one together? I could not find anything in the `V1Controller` or K8S scheduler codebase in Heron. The [`getContainer`](https://github.com/apache/incubator-heron/blob/7322335a5f4e824a8af2797a3b0dc82e315e3dfb/heron/schedulers/src/java/org/apache/heron/scheduler/kubernetes/V1Controller.java#L510-L574) routine is configuring the Docker container for executor deployment, is this the routine you are referring to? There is also [this](https://github.com/apache/spark/blob/bbb33af2e4c90d679542298f56073a71de001fb7/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala#L105) which indicates there is a built-in function in the K8S client library (scala and this potentially java too) that should handle the parse and assembly of the Pod Template. Currently `createStatefulSet` relies on the `V1` K8S API to put together a default Pod Template in which most of the fields are simply set to `null`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
