imtzer commented on PR #45911:
URL: https://github.com/apache/spark/pull/45911#issuecomment-2129285181
> > same problem when using spark operator, it's weird why the code does not
throw anything when configmap is not created
>
> When using spark-submit, there is no error output in the console, and the
client will show that the driver pod is always in the ContainerCreating state
and will never end.
Maybe I met another issuse, I use spark-submit in spark operator pod with
k8s mode, but sometimes driver pod keeps getting stuck in ContainerCreating
state due to missing ConfigMap and the console output shows 'Killed'. I added
some log in KubernetesClientApplication.scala like this:
```
logInfo("before pod create, " + driverPodName)
var watch: Watch = null
var createdDriverPod: Pod = null
try {
createdDriverPod =
kubernetesClient.pods().inNamespace(conf.namespace).resource(resolvedDriverPod).create()
} catch {...}
logInfo("before pre resource refresh, " + driverPodName)
// Refresh all pre-resources' owner references
try {
addOwnerReference(createdDriverPod, preKubernetesResources)
kubernetesClient.resourceList(preKubernetesResources:
_*).forceConflicts().serverSideApply()
} catch {...}
logInfo("before other resource, " + driverPodName)
// setup resources after pod creation, and refresh all resources' owner
references
try {
val otherKubernetesResources =
resolvedDriverSpec.driverKubernetesResources ++ Seq(configMap)
addOwnerReference(createdDriverPod, otherKubernetesResources)
kubernetesClient.resourceList(otherKubernetesResources:
_*).createOrReplace()
logInfo("after other resource, " + driverPodName)
} catch {...}
```
and the log did not show `after other resource` when spark-submit was
'Killed', the configmap that driver pod need had not been successfully created!
Also, I checked oom using `dmesg`, and some process in spark operator pod
was killed because of oom
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
