rmetzger commented on PR #29379: URL: https://github.com/apache/flink/pull/29379#issuecomment-6152728421
Dumping some AI feedback, which I have briefly skimmed, for you to evaluate Missing in the new entry point from ClusterEntrypoint and TaskManagerRunner: <img width="1846" height="476" alt="image" src="https://github.com/user-attachments/assets/32b5ae5a-f704-4fba-ba24-da8ab54c582c"; /> Sounds like the Changelog storage plugins and failure enrichters are worth adding? maybe more stuff? I am not sure if this is really an issue? ``` 1. Bug (reproduced): SIGTERM can permanently fail the job When the MiniCluster closes, it shuts down the TaskManagers and the JobManager components at the same time (MiniCluster.java:679). The JobManager side waits for the REST endpoint to close before it suspends the job. So the TaskManager disconnects first, and JobMaster.disconnectTaskManager triggers a failover. If the restart strategy doesn't allow a restart, the job becomes globally FAILED. The Dispatcher then stores that result and deletes the job's HA data. A distributed cluster can't hit this, because the TaskManagers live in other processes. "No restart" is the default when checkpointing is disabled. I wrote a temporary ITCase: an unbounded job with restart-strategy.type: none, then SIGTERM, then a restart. After the restart, the job was FAILED instead of RUNNING. The restarted process logged: StandaloneDispatcher - Job abbdef8ee844c0ade36df39ef9080650 reached terminal state FAILED. org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy ... at org.apache.flink.runtime.jobmaster.JobMaster.disconnectTaskManager(JobMaster.java:598) Caused by: org.apache.flink.util.FlinkExpectedException: The TaskExecutor is shutting down. at org.apache.flink.runtime.taskexecutor.TaskExecutor.onStop(TaskExecutor.java:527) It also logged "Successfully recovered 0 persisted job graphs" and ran main() a second time. Suggested fix: when closing without HA cleanup, close the dispatcher first, so the JobMaster is suspended before the TaskManagers stop. Add this scenario to the ITCase. The same problem applies to the fatal-error close path``` This sounds right: ``` 6. Make exit codes and cleanup match ClusterEntrypoint: - exit code 2 for runtime failures, 1 only for startup failures - log the exit code before System.exit - call program.close() on every shutdown path 7. RPC thread pool. Override createLocalRpcService in the subclass so it uses the configured pekko.fork-join-executor.* settings instead of the test pool (2–4 threads). This pool is shared by the JobManager, ResourceManager and TaskManager, so it is a real risk for long-running jobs. Also decide whether the 5 min pekko.ask.timeout default should apply here. 10. JMX server and FileSystem.attachMetrics. JMX is a one-liner, so the author could include it. 11. A fixed working directory, so local recovery works after a restart. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
