john8628 opened a new issue, #4202: URL: https://github.com/apache/incubator-seatunnel/issues/4202
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened firstly i setup three nodes of seatunnel-cluster (10.*.*109;10.*.*.110,10.*.*.111) ; then i submit the etl task to the cluster ; then failed to run ; ### SeaTunnel Version dev ### SeaTunnel Config ```conf env { # You can set spark configuration here execution.parallelism = 8 job.mode = "BATCH" } source { OSSJindoFile { path = "/adm/test/dt=20230112/label=country" bucket = "oss://test" type = "parquet" access_key="*" access_secret="*" endpoint="oss-eu-central-1.aliyuncs.com" result_table_name="fake" parallelism=3 } } sink{ ClickhouseFile { source_table_name="fake" host = "10.*.*.*:8123" database = "test" table = "table1" username = "default" password = "" clickhouse_local_path = "/etc/clickhouse local" node_free_password = true node_pass=[] } } ``` ### Running Command ```shell $SEATUNNEL_HOME/bin/seatunnel.sh --config $SEATUNNEL_HOME/config/oss2clickhousefile.template > /tmp/client0217_4.log ``` ### Error Exception ```log 2023-02-24 09:28:32,684 INFO org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 0 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 0 completedTaskCount : 0 taskCount : 0 *********************************************** 2023-02-24 09:30:22,472 INFO com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Received auth from Connection[id=1, /10.8.4.111:5801->/10.8.4.111:42967, qualifier=EndpointQualifier{type='CLIENT'}, endpoint=[10.8.4.111]:42967, remoteUuid=e92a3cf5-1db6-4961-8d87-b6e7408c7469, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: e92a3cf5-1db6-4961-8d87-b6e7408c7469, client name: hz.client_1, client version: 5.1 2023-02-24 09:30:24,379 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681311931096629254, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=1, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:30:24,386 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681311931096629254, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=2, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:30:24,393 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681311931096629254, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=3, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:30:24,493 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=1} 2023-02-24 09:30:24,508 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: OssJindoFile 2023-02-24 09:30:24,533 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002} 2023-02-24 09:30:24,533 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000} 2023-02-24 09:30:24,543 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 0 2023-02-24 09:30:24,543 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 2 2023-02-24 09:30:24,563 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 2 2023-02-24 09:30:24,563 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 0 2023-02-24 09:30:32,683 INFO org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 0 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 0 completedTaskCount : 0 taskCount : 0 *********************************************** 2023-02-24 09:30:48,285 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connection[id=1, /10.8.4.111:5801->/10.8.4.111:42967, qualifier=EndpointQualifier{type='CLIENT'}, endpoint=[10.8.4.111]:42967, remoteUuid=e92a3cf5-1db6-4961-8d87-b6e7408c7469, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side 2023-02-24 09:30:48,286 INFO com.hazelcast.client.impl.ClientEndpointManager - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /10.8.4.111:5801->/10.8.4.111:42967, qualifier=EndpointQualifier{type='CLIENT'}, endpoint=[10.8.4.111]:42967, remoteUuid=e92a3cf5-1db6-4961-8d87-b6e7408c7469, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=e92a3cf5-1db6-4961-8d87-b6e7408c7469, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1677202222468, latest clientAttributes=lastStatisticsCollectionTime=1677202237517,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1677202222463,clientAddress=10.8.4.111,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=14666620928,os.freePhysicalMemorySize=4834521088,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=255350,os.openFileDescriptorCount=53,os.processCpuTime=7580000000,os.systemLoadAverage=0.97,os.total PhysicalMemorySize=33019609088,os.totalSwapSpaceSize=0,runtime.availableProcessors=8,runtime.freeMemory=483194304,runtime.maxMemory=7339507712,runtime.totalMemory=581435392,runtime.uptime=16390,runtime.usedMemory=98241088, labels=[]} 2023-02-24 09:31:24,525 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000}) need cancel. 2023-02-24 09:31:24,525 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=1}) need cancel. 2023-02-24 09:31:24,525 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=1} 2023-02-24 09:31:24,525 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60000, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000} 2023-02-24 09:31:24,525 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70000, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000} 2023-02-24 09:31:24,525 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=1} complete with state CANCELED 2023-02-24 09:31:24,526 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50000} complete with state CANCELED 2023-02-24 09:31:24,526 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002}) need cancel. 2023-02-24 09:31:24,526 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60002, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002} 2023-02-24 09:31:24,526 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70002, taskGroup = TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002} 2023-02-24 09:31:24,526 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681311931096629254, pipelineId=1, taskGroupId=50002} complete with state CANCELED 2023-02-24 09:31:24,609 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 681311931096629254, slot: SlotProfile{worker=[10.8.4.111]:6800, slotID=2, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:31:24,610 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 681311931096629254, slot: SlotProfile{worker=[10.8.4.111]:6800, slotID=1, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:31:24,610 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 681311931096629254, slot: SlotProfile{worker=[10.8.4.111]:6800, slotID=3, ownerJobID=681311931096629254, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:32:21,815 INFO com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Received auth from Connection[id=2, /10.8.4.111:5801->/10.8.4.111:59371, qualifier=EndpointQualifier{type='CLIENT'}, endpoint=[10.8.4.111]:59371, remoteUuid=6277892b-a6b9-4315-bd9f-73533697d2c8, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 6277892b-a6b9-4315-bd9f-73533697d2c8, client name: hz.client_1, client version: 5.1 2023-02-24 09:32:23,037 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=4, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:32:23,051 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=5, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:32:23,073 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1} 2023-02-24 09:32:23,075 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: OssJindoFile 2023-02-24 09:32:23,123 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} 2023-02-24 09:32:23,126 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 2 2023-02-24 09:32:23,132 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 2 2023-02-24 09:32:32,683 INFO org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 0 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 0 completedTaskCount : 0 taskCount : 0 *********************************************** 2023-02-24 09:33:23,110 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}) need cancel. 2023-02-24 09:33:23,110 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1} 2023-02-24 09:33:23,110 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1} complete with state CANCELED 2023-02-24 09:33:23,113 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}) need cancel. 2023-02-24 09:33:23,113 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60002, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} 2023-02-24 09:33:23,113 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70002, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} 2023-02-24 09:33:23,113 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} complete with state CANCELED 2023-02-24 09:34:32,683 INFO org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 0 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 0 completedTaskCount : 0 taskCount : 0 *********************************************** 2023-02-24 09:35:03,746 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=6, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:35:03,803 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} 2023-02-24 09:35:03,805 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 1 2023-02-24 09:35:03,823 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 1 2023-02-24 09:36:03,941 WARN com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connection[id=1, /10.8.4.111:44409->/10.8.4.110:6800, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Exception in Connection[id=1, /10.8.4.111:44409->/10.8.4.110:6800, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=true, connectionType=MEMBER, planeIndex=0], thread=hz.main.IO.thread-in-2 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_362] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_362] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_362] at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_362] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.8.0_362] at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:114) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] 2023-02-24 09:36:03,942 INFO com.hazelcast.internal.server.tcp.TcpServerConnector - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connecting to /10.8.4.110:6800, timeout: 0, bind-any: true 2023-02-24 09:36:03,943 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:43465 and /10.8.4.110:6800 2023-02-24 09:36:03,944 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:6800 and /10.8.4.110:43925 2023-02-24 09:36:03,947 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}) need cancel. 2023-02-24 09:36:03,947 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70001, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} 2023-02-24 09:36:03,947 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60001, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} 2023-02-24 09:36:03,947 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} complete with state CANCELED 2023-02-24 09:36:32,683 INFO org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 0 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 0 completedTaskCount : 0 taskCount : 0 *********************************************** 2023-02-24 09:38:04,733 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=7, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:38:04,738 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=8, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:38:04,742 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=9, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:38:04,753 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=10, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:38:04,780 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1} 2023-02-24 09:38:04,784 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: OssJindoFile 2023-02-24 09:38:04,806 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=3} 2023-02-24 09:38:04,808 INFO org.apache.seatunnel.engine.server.task.SinkAggregatedCommitterTask - starting seatunnel sink aggregated committer task, sink name: ClickhouseFile 2023-02-24 09:38:04,809 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000} 2023-02-24 09:38:04,813 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} 2023-02-24 09:38:04,815 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 0 2023-02-24 09:38:04,817 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 2 2023-02-24 09:38:04,831 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 2 2023-02-24 09:38:04,831 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 0 2023-02-24 09:38:13,050 INFO com.hazelcast.internal.diagnostics.HealthMonitor - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] processors=8, physical.memory.total=30.8G, physical.memory.free=4.2G, swap.space.total=0, swap.space.free=0, heap.memory.used=1.6G, heap.memory.free=7.5G, heap.memory.total=9.1G, heap.memory.max=14.2G, heap.memory.used/total=17.61%, heap.memory.used/max=11.30%, minor.gc.count=2, minor.gc.time=39ms, major.gc.count=2, major.gc.time=255ms, load.process=50.00%, load.system=75.29%, load.systemAverage=0.66, thread.count=68, thread.peakCount=77, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=15068, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, e xecutor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=13, clientEndpoint.count=1, connection.active.count=0, client.connection.count=0, connection.count=0 2023-02-24 09:38:32,683 INFO org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 0 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 0 completedTaskCount : 0 taskCount : 0 *********************************************** 2023-02-24 09:38:33,057 INFO com.hazelcast.internal.diagnostics.HealthMonitor - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] processors=8, physical.memory.total=30.8G, physical.memory.free=4.2G, swap.space.total=0, swap.space.free=0, heap.memory.used=1.6G, heap.memory.free=7.5G, heap.memory.total=9.1G, heap.memory.max=14.2G, heap.memory.used/total=17.69%, heap.memory.used/max=11.35%, minor.gc.count=2, minor.gc.time=39ms, major.gc.count=2, major.gc.time=255ms, load.process=80.00%, load.system=75.47%, load.systemAverage=2.19, thread.count=67, thread.peakCount=77, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=15406, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, e xecutor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=13, clientEndpoint.count=1, connection.active.count=0, client.connection.count=0, connection.count=0 2023-02-24 09:38:53,059 INFO com.hazelcast.internal.diagnostics.HealthMonitor - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] processors=8, physical.memory.total=30.8G, physical.memory.free=4.1G, swap.space.total=0, swap.space.free=0, heap.memory.used=1.6G, heap.memory.free=7.5G, heap.memory.total=9.1G, heap.memory.max=14.2G, heap.memory.used/total=17.80%, heap.memory.used/max=11.42%, minor.gc.count=2, minor.gc.time=39ms, major.gc.count=2, major.gc.time=255ms, load.process=0.00%, load.system=75.37%, load.systemAverage=3.20, thread.count=67, thread.peakCount=77, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=15709, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, ex ecutor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=13, clientEndpoint.count=1, connection.active.count=0, client.connection.count=0, connection.count=0 2023-02-24 09:39:04,820 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}) need cancel. 2023-02-24 09:39:04,820 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1} 2023-02-24 09:39:04,820 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1} complete with state CANCELED 2023-02-24 09:39:04,820 WARN com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Exception in Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=true, connectionType=MEMBER, planeIndex=0], thread=hz.main.seaTunnel.task.thread-8 java.nio.channels.ClosedByInterruptException: null at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[?:1.8.0_362] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:477) ~[?:1.8.0_362] at com.hazelcast.internal.networking.nio.NioOutboundPipeline.flushToSocket(NioOutboundPipeline.java:439) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioOutboundPipeline.process(NioOutboundPipeline.java:324) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioOutboundPipeline.executePipeline(NioOutboundPipeline.java:240) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioOutboundPipeline.write(NioOutboundPipeline.java:218) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioChannel.write(NioChannel.java:79) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.server.tcp.TcpServerConnection.write(TcpServerConnection.java:222) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.OutboundOperationHandler.send(OutboundOperationHandler.java:59) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeRemote(Invocation.java:612) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:582) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:341) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_362] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_362] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_362] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_362] at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_362] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362] 2023-02-24 09:39:04,823 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000}) need cancel. 2023-02-24 09:39:04,823 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000} 2023-02-24 09:39:04,823 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000} 2023-02-24 09:39:04,823 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50000} complete with state CANCELED 2023-02-24 09:39:04,824 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:6800 and /10.8.4.110:37786 2023-02-24 09:39:04,824 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:36663 and /10.8.4.110:6800 2023-02-24 09:39:04,921 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=3}) need cancel. 2023-02-24 09:39:04,921 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 40000, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=3} 2023-02-24 09:39:04,922 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=3} complete with state CANCELED 2023-02-24 09:39:04,923 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002}) need cancel. 2023-02-24 09:39:04,923 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70002, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} 2023-02-24 09:39:04,923 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60002, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} 2023-02-24 09:39:04,923 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50002} complete with state CANCELED 2023-02-24 09:40:32,683 INFO org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] *********************************************** CoordinatorService Thread Pool Status *********************************************** activeCount : 0 corePoolSize : 0 maximumPoolSize : 2147483647 poolSize : 0 completedTaskCount : 0 taskCount : 0 *********************************************** 2023-02-24 09:41:05,704 INFO com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Invocations:1 timeouts:1 backup-timeouts:0 2023-02-24 09:41:05,710 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04.820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]} at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:121) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:100) at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617) at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:343) at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322) at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697) at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04.820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800, pendingResponse={VOID}, back upsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]} at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.newOperationTimeoutException(InvocationFuture.java:194) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:136) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:99) at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617) at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:343) at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322) at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697) at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) at ------ submitted from ------.() at com.hazelcast.internal.util.ExceptionUtil.cloneExceptionWithFixedAsyncStackTrace(ExceptionUtil.java:336) at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:112) ... 16 more 2023-02-24 09:41:05,710 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] notify the job of the task(TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=1}) status failed, retry in 1000 millis 2023-02-24 09:41:05,710 ERROR org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04 .820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800, pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]} java.util.concurrent.ExecutionException: com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04.820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800 , pendingResponse={VOID}, backupsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]} at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:121) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:100) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:343) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_362] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_362] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_362] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_362] at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_362] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362] Caused by: com.hazelcast.core.OperationTimeoutException: NotifyTaskStatusOperation invocation failed to complete due to operation-heartbeat-timeout. Current time: 2023-02-24 09:41:05.704. Start time: 2023-02-24 09:39:04.820. Total elapsed time: 120885 ms. Last operation heartbeat: never. Last operation heartbeat from member: 2023-02-24 09:40:56.516. Invocation{op=org.apache.seatunnel.engine.server.task.operation.NotifyTaskStatusOperation{serviceName='st:impl:seaTunnelServer', identityHash=1206625578, partitionId=-1, replicaIndex=0, callId=-3689, invocationTime=1677202744819 (2023-02-24 09:39:04.819), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, tryCount=20, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1677202744820, firstInvocationTime='2023-02-24 09:39:04.820', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 08:00:00.000', target=[10.8.4.110]:6800, pendingResponse={VOID}, back upsAcksExpected=-1, backupsAcksReceived=0, connection=Connection[id=6, /10.8.4.111:6800->/10.8.4.110:43925, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0]} at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.newOperationTimeoutException(InvocationFuture.java:194) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:136) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:99) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService.notifyTaskStatusToMaster(TaskExecutionService.java:343) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService.lambda$deployLocalTask$3(TaskExecutionService.java:322) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.jet.impl.util.ExceptionUtil.lambda$withTryCatch$0(ExceptionUtil.java:172) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) ~[?:1.8.0_362] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750) ~[?:1.8.0_362] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_362] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_362] at org.apache.seatunnel.engine.server.TaskExecutionService$TaskGroupExecutionTracker.taskDone(TaskExecutionService.java:697) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:502) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_362] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362] at ------ submitted from ------.() ~[?:?] at com.hazelcast.internal.util.ExceptionUtil.cloneExceptionWithFixedAsyncStackTrace(ExceptionUtil.java:336) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:112) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] ... 16 more 2023-02-24 09:41:05,739 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot request, jobID: 681312431611314182, resource profile: ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, return: SlotProfile{worker=[10.8.4.111]:6800, slotID=11, ownerJobID=681312431611314182, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='cee2d45a-0030-4cf6-965b-05366e5b2fb7'} 2023-02-24 09:41:05,785 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] deploying task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} 2023-02-24 09:41:05,786 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 1 2023-02-24 09:41:05,790 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 1 2023-02-24 09:42:05,824 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task (TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001}) need cancel. 2023-02-24 09:42:05,824 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 70001, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} 2023-02-24 09:42:05,824 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] taskDone, taskId = 60001, taskGroup = TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} 2023-02-24 09:42:05,824 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Task TaskGroupLocation{jobId=681312431611314182, pipelineId=1, taskGroupId=50001} complete with state CANCELED 2023-02-24 09:42:06,128 WARN com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connection[id=8, /10.8.4.111:36663->/10.8.4.110:6800, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Exception in Connection[id=8, /10.8.4.111:36663->/10.8.4.110:6800, qualifier=EndpointQualifier{type='MEMBER'}, endpoint=[10.8.4.110]:6800, remoteUuid=278d97cd-37d2-477f-b6b4-32c2b1a0e643, alive=true, connectionType=MEMBER, planeIndex=0], thread=hz.main.IO.thread-in-2 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_362] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_362] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_362] at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_362] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[?:1.8.0_362] at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:114) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.1-SNAPSHOT] 2023-02-24 09:42:06,129 INFO com.hazelcast.internal.server.tcp.TcpServerConnector - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Connecting to /10.8.4.110:6800, timeout: 0, bind-any: true 2023-02-24 09:42:06,130 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] Initialized new cluster connection between /10.8.4.111:39939 and /10.8.4.110:6800 2023-02-24 09:42:32,683 INFO org.apache.seatunnel.engine.server.CoordinatorService - [10.8.4.111]:6800 [seatunnel_default_cluster] [5.1] ``` ### Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
