inventertom opened a new issue, #14709: URL: https://github.com/apache/dolphinscheduler/issues/14709
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues. ### What happened background：2 masters, 4 workers, high availability of master settings A few days ago, a master down machine, after the reboot the master log grew rapidly, and the disk quickly filled up log : [ERROR] 2023-08-07 13:19:48.670 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[278] - [WorkflowInstance-4053240][TaskInstance-9596384] - State event handle error, will retry this event: StateEvent(key=4053244-0-4053240-9596384, type=TASK_STATE_CHANGE, executionStatus=RUNNING_EXECUTION, taskInstanceId=9596384, taskCode=0, processInstanceId=4053240, context=null, channel=null) org.apache.dolphinscheduler.server.master.event.StateEventHandleException: Task state event handle error, due to the task is not in activeTaskProcessorMaps at org.apache.dolphinscheduler.server.master.event.TaskStateEventHandler.handleStateEvent(TaskStateEventHandler.java:84) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable.handleEvents(WorkflowExecuteRunnable.java:270) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [ERROR] 2023-08-07 13:19:48.670 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[278] - [WorkflowInstance-4053601][TaskInstance-9600083] - State event handle error, will retry this event: StateEvent(key=4053603-0-4053601-9600083, type=TASK_STATE_CHANGE, executionStatus=RUNNING_EXECUTION, taskInstanceId=9600083, taskCode=0, processInstanceId=4053601, context=null, channel=null) org.apache.dolphinscheduler.server.master.event.StateEventHandleException: Task state event handle error, due to the task is not in activeTaskProcessorMaps at org.apache.dolphinscheduler.server.master.event.TaskStateEventHandler.handleStateEvent(TaskStateEventHandler.java:84) at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable.handleEvents(WorkflowExecuteRunnable.java:270) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) We tried the method 1. Modify the ID status of the task instance to Failed，Restart the master of the node, and the log is still full quickly 2. reference #11918 , Delete the task instance that corresponds to the id of the task instance , Restart the master of the node, and the log is still full quickly ### What you expected to happen The log is printed normally, do not grow too fast, causing the disk to fill up ### How to reproduce Set the highly available master, pre-task workflow, and then let the master down machine resume ### Anything else _No response_ ### Version 3.1.x ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
