dibrother opened a new issue, #7472: URL: https://github.com/apache/seatunnel/issues/7472
### Search before asking - [X] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description Simulate power outage,when I kill all seatunnel processes, I am unable to recover jobs based on the checkpoint. When the cluster node is restarted, all previously run tasks are not automatically restored. 1.Deploy SeaTunnel Engine In Separated Cluster Mode, all on the same server ```shell seatunnel: engine: history-job-expire-minutes: 1440 backup-count: 1 queue-type: blockingqueue print-execution-info-interval: 60 print-job-metrics-info-interval: 60 slot-service: dynamic-slot: true checkpoint: interval: 10000 timeout: 60000 storage: type: hdfs max-retained: 3 plugin-config: namespace: /app/seatunnel/checkpoint_snapshot storage.type: hdfs fs.defaultFS: file:///app/ # Ensure that the directory has written permission map: engine*: map-store: enabled: true initial-mode: EAGER factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory properties: type: hdfs namespace: /app/seatunnel/imap clusterName: seatunnel-cluster storage.type: hdfs fs.defaultFS: file:///app/ ``` 2.Starting the SeaTunnel Engine Master Node 3.Starting The SeaTunnel Engine Worker Node 4.Submitting Jobs ```shell ./bin/seatunnel.sh --config config/mysql-to-mysql.conf -n test01 ``` 5.Full data extraction and normal synchronization 6. Kill all seatunnel processes ```shell ps -ef|grep seatunnel kill -9 xxx xxx xxx xxx ``` recovery： 1.Starting the SeaTunnel Engine Master Node 2.Starting The SeaTunnel Engine Worker Node 3.Resuming Jobs ```shell # View job_id ls /app/seatunnel/checkpoint_snapshot/ 879243812122132482 # Resuming Jobs ./bin/seatunnel.sh -r 879243812122132482 -c config/mysql-to-mysql.conf ``` report errors: ```shell 2024-08-23 14:14:11,767 INFO [p.MultipleTableJobConfigParser] [main] - start generating all transforms. 2024-08-23 14:14:11,767 INFO [p.MultipleTableJobConfigParser] [main] - start generating all sinks. 2024-08-23 14:14:11,781 INFO [.s.p.d.AbstractPluginDiscovery] [main] - Load SeaTunnelSink Plugin from /root/seatunnel/apache-seatunnel-2.3.7/connectors 2024-08-23 14:14:11,868 INFO [o.a.s.a.t.f.FactoryUtil ] [main] - Create sink 'jdbc' with upstream input catalog-table[database: sbtest, schema: null, table: sbtest1] 2024-08-23 14:14:12,525 INFO [o.a.s.e.c.j.ClientJobProxy ] [main] - Start submit job, job id: 879243812122132482, with plugin jar [] 2024-08-23 14:14:16,883 INFO [o.a.s.e.c.j.ClientJobProxy ] [main] - Submit job finished, job id: 879243812122132482, job name: SeaTunnel_Job 2024-08-23 14:14:17,674 WARN [o.a.s.e.c.j.JobMetricsRunner ] [job-metrics-runner-879243812122132482] - Failed to get job metrics summary, it maybe first-run 2024-08-23 14:14:36,700 INFO [o.a.s.e.c.j.ClientJobProxy ] [main] - Job (879243812122132482) end with state FAILED 2024-08-23 14:14:36,703 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN 2024-08-23 14:14:36,718 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5802:9312e0d1-46f4-4093-8fa2-125ecd53a0d8, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/127.0.0.1:34341->localhost/127.0.0.1:5802}, remoteAddress=[localhost]:5802, lastReadTime=2024-08-23 14:14:35.055, lastWriteTime=2024-08-23 14:14:35.053, closedTime=2024-08-23 14:14:36.713, connected server version=5.1} 2024-08-23 14:14:36,720 INFO [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [localhost]:5801:c9a9fb0a-5d6b-4f2a-a8d1-cf59831dc43b, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:49359->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-08-23 14:14:36.668, lastWriteTime=2024-08-23 14:14:30.060, closedTime=2024-08-23 14:14:36.719, connected server version=5.1} 2024-08-23 14:14:36,720 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED 2024-08-23 14:14:36,733 INFO [c.h.c.LifecycleService ] [main] - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN 2024-08-23 14:14:36,734 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client...... 2024-08-23 14:14:36,735 INFO [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ...... 2024-08-23 14:14:36,735 ERROR [o.a.s.c.s.SeaTunnel ] [main] - =============================================================================== 2024-08-23 14:14:36,736 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Fatal Error, 2024-08-23 14:14:36,736 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues 2024-08-23 14:14:36,737 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Reason:SeaTunnel job executed failed 2024-08-23 14:14:36,741 ERROR [o.a.s.c.s.SeaTunnel ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:211) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 4 at org.apache.seatunnel.api.table.type.SeaTunnelRowType.getFieldType(SeaTunnelRowType.java:73) at org.apache.seatunnel.api.table.type.SeaTunnelRow.getBytesSize(SeaTunnelRow.java:105) at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.collect(SeaTunnelSourceCollector.java:125) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter$OutputCollector.collect(IncrementalSourceRecordEmitter.java:206) at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializeSchema.deserializeDataChangeRecord(SeaTunnelRowDebeziumDeserializeSchema.java:191) at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializeSchema.deserialize(SeaTunnelRowDebeziumDeserializeSchema.java:111) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.emitElement(IncrementalSourceRecordEmitter.java:198) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.processElement(IncrementalSourceRecordEmitter.java:150) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.emitRecord(IncrementalSourceRecordEmitter.java:101) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.emitRecord(IncrementalSourceRecordEmitter.java:61) at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:110) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:119) at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156) at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127) at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132) at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:717) at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1039) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:203) ... 2 more 2024-08-23 14:14:36,744 ERROR [o.a.s.c.s.SeaTunnel ] [main] - =============================================================================== Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:211) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.ArrayIndexOutOfBoundsException: Index 4 out of bounds for length 4 at org.apache.seatunnel.api.table.type.SeaTunnelRowType.getFieldType(SeaTunnelRowType.java:73) at org.apache.seatunnel.api.table.type.SeaTunnelRow.getBytesSize(SeaTunnelRow.java:105) at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.collect(SeaTunnelSourceCollector.java:125) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter$OutputCollector.collect(IncrementalSourceRecordEmitter.java:206) at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializeSchema.deserializeDataChangeRecord(SeaTunnelRowDebeziumDeserializeSchema.java:191) at org.apache.seatunnel.connectors.cdc.debezium.row.SeaTunnelRowDebeziumDeserializeSchema.deserialize(SeaTunnelRowDebeziumDeserializeSchema.java:111) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.emitElement(IncrementalSourceRecordEmitter.java:198) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.processElement(IncrementalSourceRecordEmitter.java:150) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.emitRecord(IncrementalSourceRecordEmitter.java:101) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceRecordEmitter.emitRecord(IncrementalSourceRecordEmitter.java:61) at org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:110) at org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader.pollNext(IncrementalSourceReader.java:119) at org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.collect(SourceFlowLifeCycle.java:156) at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.collect(SourceSeaTunnelTask.java:127) at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168) at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.call(SourceSeaTunnelTask.java:132) at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:717) at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1039) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:203) ... 2 more 2024-08-23 14:14:36,755 INFO [s.c.s.s.c.ClientExecuteCommand] [Thread-6] - run shutdown hook because get close signal ``` ### Usage Scenario _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
