[
https://issues.apache.org/jira/browse/FLINK-41375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41375:
-----------------------------------
Labels: pull-request-available (was: )
> JobManager crash loops are detected only in the back-off phase and reported
> without the termination reason
> ----------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41375
> URL: https://issues.apache.org/jira/browse/FLINK-41375
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
> Reporter: Dennis-Mircea Ciupitu
> Priority: Major
> Labels: pull-request-available
>
> h1. Problem
> When the JobManager container keeps dying (for example, OOMKilled reason),
> what the operator reports depends on the moment it happens to look at the
> pod, and it never says why the container died.
> The observer only recognises a crash loop while the kubelet is backing off
> ({{{}state.waiting.reason=CrashLoopBackOff{}}}). Between 2 kills, while the
> container is {{running}} again, it reports {{DEPLOYING}} as if everything
> were fine, even though the pod already says
> {{{}lastState.terminated.reason=OOMKilled{}}}, {{exitCode=137}} and counts
> the restarts.
> The actual problem is that the {{OOMKilled}} entry in
> {{CONTAINER_ERROR_REASONS}} never matches on the current observation logic.
> Kubernetes puts that reason under {{{}terminated{}}}, not under
> {{{}waiting{}}}. The longer the JobManager runs before each kill, the more
> often the operator says {{{}DEPLOYING{}}}.
> When it does detect the loop, the error and the event are a copy of the
> kubelet's back-off sentence with the pod name. The reason and the exit code
> are nowhere in the resource. And because that sentence changes with every
> back-off step and every pod, each detection produces a new event instead of
> counting up the existing one.
> On a deployment that was already stable, the {{DEPLOYING}} sample also wipes
> {{{}status.error{}}}, so the resource alternates between an error and no
> error at all.
> h1. Reproduction
> {{examples/basic.yaml}} with the below extra line + "1024m" as JM spec
> resource memory:
> {code:yaml}
> flinkConfiguration:
> env.java.opts.jobmanager: "-Xms1200m -Xmx1200m -XX:+AlwaysPreTouch"
> {code}
> {code:java}
> kubectl get pods
> NAME READY STATUS RESTARTS
> AGE
> basic-example-65774988d7-tzv74 0/1 OOMKilled 0 3s
> kubectl get pods
> NAME READY STATUS
> RESTARTS AGE
> basic-example-65774988d7-tzv74 0/1 CrashLoopBackOff 2 (3s
> ago) 20s
> {code}
> Operator log:
> {code:java}
> 2026-11-07 09:59:52,330 INFO ecutor-flinkdeploymentcontroller-350
> .r.d.AbstractFlinkResourceReconciler Deploying for the first time
> 2026-11-07 09:59:52,335 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.operator.listener.AuditUtils >>> Status[Job] | Info |
> UPGRADING | The resource is being upgraded
> 2026-11-07 09:59:52,336 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.o.r.d.ApplicationReconciler Assigning JobId override to
> 5efdcfec84cb50f1fca3c5aecd224f0b
> 2026-11-07 09:59:52,343 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.operator.listener.AuditUtils >>> Event[Job] | Info | SUBMIT
> | Starting deployment
> 2026-11-07 09:59:52,343 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.o.s.AbstractFlinkService Deploying application cluster
> 2026-11-07 09:59:52,343 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.o.service.NativeFlinkService Deploying application cluster
> 2026-11-07 09:59:52,343 INFO ecutor-flinkdeploymentcontroller-350
> f.c.d.a.c.ApplicationClusterDeployer Submitting application in 'Application
> Mode'.
> 2026-11-07 09:59:52,346 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.r.u.c.m.ProcessMemoryUtils The derived from fraction jvm overhead
> memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb
> (201326592 bytes), min value will be used instead
> 2026-11-07 09:59:52,392 INFO ecutor-flinkdeploymentcontroller-350
> .a.DefaultKubernetesArtifactUploader Local artifact uploading is disabled.
> Set 'kubernetes.artifacts.local-upload-enabled' to enable.
> 2026-11-07 09:59:52,392 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.utils.KubernetesUtils Kubernetes deployment requires a fixed port.
> Configuration blob.server.port will be set to 6124
> 2026-11-07 09:59:52,393 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.utils.KubernetesUtils Kubernetes deployment requires a fixed port.
> Configuration taskmanager.rpc.port will be set to 6122
> 2026-11-07 09:59:52,401 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.configuration.Configuration Config uses fallback configuration key
> 'kubernetes.service-account' instead of key
> 'kubernetes.jobmanager.service-account'
> 2026-11-07 09:59:52,479 WARN ecutor-flinkdeploymentcontroller-350
> o.a.f.k.KubernetesClusterDescriptor Please note that Flink client
> operations(e.g. cancel, list, stop, savepoint, etc.) won't work from outside
> the Kubernetes cluster since 'kubernetes.rest-service.exposed.type' has been
> set to ClusterIP.
> 2026-11-07 09:59:52,480 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.KubernetesClusterDescriptor Create flink application cluster
> basic-example successfully, JobManager Web Interface:
> http://basic-example-rest.default:8081
> 2026-11-07 09:59:52,480 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.o.service.NativeFlinkService Application cluster successfully deployed
> 2026-11-07 09:59:52,486 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.operator.listener.AuditUtils >>> Status[Job] | Info |
> DEPLOYED | The resource is deployed/submitted to Kubernetes, but it’s
> not yet considered to be stable and might be rolled back in the future
> 2026-11-07 09:59:52,486 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
> status: DEPLOYING
> 2026-11-07 09:59:52,489 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.o.o.d.ApplicationObserver JobManager is being deployed
> 2026-11-07 09:59:52,491 INFO ecutor-flinkdeploymentcontroller-350
> o.a.f.k.o.a.state.ConfigMapView Creating ConfigMap autoscaler-basic-example
> 2026-11-07 09:59:52,492 INFO ecutor-flinkdeploymentcontroller-350
> .r.d.AbstractFlinkResourceReconciler Resource fully reconciled, nothing to
> do...
> 2026-11-07 10:00:02,495 INFO ecutor-flinkdeploymentcontroller-369
> o.a.f.k.o.o.d.ApplicationObserver Observing JobManager deployment. Previous
> status: DEPLOYING
> 2026-11-07 10:00:02,508 INFO ecutor-flinkdeploymentcontroller-369
> o.a.f.k.operator.listener.AuditUtils >>> Event[Job] | Warning |
> CRASHLOOPBACKOFF | [flink-main-container] back-off 10s restarting failed
> container=flink-main-container
> pod=basic-example-65774988d7-tzv74_default(b57eb99c-a3cf-4fad-8db0-d7b144669a0d)
> {code}
> This is the error the resource keeps reporting. Nothing in it says why the
> container died:
> {code:java}
> kubectl get flinkdep basic-example -o jsonpath='{.status.error}'
> {"type":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException","message":"[flink-main-container]
> back-off 10s restarting failed container=flink-main-container
> pod=basic-example-65774988d7-q8x7d_default(deccff18-576e-4a16-aeb4-6efaf894eb9a)","stackTrace":"org.apache.flink.kubernetes.operator.exception.DeploymentFailedException:
> [flink-main-container] back-off 10s restarting failed
> container=flink-main-container
> pod=basic-example-65774988d7-q8x7d_default(deccff18-576e-4a16-aeb4-6efaf894eb9a)\n\tat
>
> org.apache.flink.kubernetes.operator.exception.DeploymentFailedException.forContainerStatus(DeploymentFailedException.java:74)\n\tat
>
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.checkContainerError(AbstractFlinkDeploymentObserver.java:251)\n\tat
> java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown
> Source)\n\tat
> java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Unknown
> Source)\n\tat
> java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown
> Source)\n\tat
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.checkContainerErrors(AbstractFlinkDeploymentObserver.java:238)\n\tat
>
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeJmDeployment(AbstractFlinkDeploymentObserver.java:193)\n\tat
>
> org.apache.flink.kubernetes.operator.observer.deployment.AbstractFlinkDeploymentObserver.observeInternal(AbstractFlinkDeploymentObserver.java:89)\n\tat
>
> org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:49)\n\tat
>
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:145)\n\tat
>
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:59)\n\tat
>
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:191)\n\tat
>
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:147)\n\tat
>
> org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:82)\n\tat
>
> io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:146)\n\tat
>
> io.javaoperatorsdk.operator.processing.event.Rec","additionalMetadata":{"reason":"CrashLoopBackOff"},"throwableList":[]}
> {code}
> Sampled every 5 s over two runs (first deployment, and a stable deployment
> whose JobManager starts crash-looping): every observe that lands in the
> {{running}} window reports {{DEPLOYING}} with
> {{{}lastState.terminated.reason=OOMKilled{}}}; on the stable deployment it
> also wipes {{status.error}} for one progress-check interval; two
> {{CrashLoopBackOff}} events in six minutes per run, one per back-off step.
> h1. Expected
> * A JobManager container that has already died with a non-zero exit code and
> been restarted is reported as crash-looping whenever the operator looks at
> it, whether the kubelet is currently backing off or the container is up again
> between two kills.
> * {{status.error}} and the event name the termination reason and exit code,
> and the text does not change with the back-off timer or the pod name, so one
> event is counted up instead of a new one per step.
> * The error is not cleared until the JobManager is actually {{{}READY{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
