king-bu commented on issue #4763:
URL:
https://github.com/apache/incubator-seatunnel/issues/4763#issuecomment-1550861866
The error still exists,
details as follows:
Client exception logs are as follows:
-----------------------------------------------------------------------------------------------------------
2023-05-17 14:42:44,626 INFO
org.apache.seatunnel.engine.client.job.ClientJobProxy - Job
(711104356853743617) end with state FAILED
2023-05-17 14:42:44,626 INFO com.hazelcast.core.LifecycleService -
hz.client_1 [seatunnel_bu2] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is
SHUTTING_DOWN
2023-05-17 14:42:44,628 INFO
com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1
[seatunnel_bu2] [5.1] Removed connection to endpoint:
[ecs-fd88-0002]:5801:7b15b077-1912-4a4d-8127-f065495a34c0, connection:
ClientConnection{alive=false, connectionId=1,
channel=NioChannel{/192.168.0.130:44544->ecs-fd88-0002/192.168.0.130:5801},
remoteAddress=[ecs-fd88-0002]:5801, lastReadTime=2023-05-17 14:42:44.618,
lastWriteTime=2023-05-17 14:42:44.525, closedTime=2023-05-17 14:42:44.627,
connected server version=5.1}
2023-05-17 14:42:44,628 INFO com.hazelcast.core.LifecycleService -
hz.client_1 [seatunnel_bu2] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is
CLIENT_DISCONNECTED
2023-05-17 14:42:44,630 INFO com.hazelcast.core.LifecycleService -
hz.client_1 [seatunnel_bu2] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is
SHUTDOWN
2023-05-17 14:42:44,630 INFO
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -
Closed SeaTunnel client......
2023-05-17 14:42:44,630 INFO
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -
Closed metrics executor service ......
2023-05-17 14:42:44,631 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================
2023-05-17 14:42:44,631 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Fatal Error,
2023-05-17 14:42:44,631 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Please submit bug report in https://github.com/apache/incubator-seatunnel/issues
2023-05-17 14:42:44,631 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Reason:SeaTunnel job executed failed
2023-05-17 14:42:44,632 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
Exception
StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException:
SeaTunnel job executed failed
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at
org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by:
org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException:
org.apache.seatunnel.engine.server.checkpoint.CheckpointException:
CheckpointCoordinator inside have error.
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:215)
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:211)
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$2(CheckpointCoordinator.java:201)
at
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
at
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1646)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.CompletionException:
org.apache.seatunnel.common.utils.SeaTunnelException:
java.lang.RuntimeException: java.util.concurrent.ExecutionException
at
com.hazelcast.spi.impl.AbstractInvocationFuture.returnOrThrowWithJoinConventions(AbstractInvocationFuture.java:819)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.resolveAndThrowWithJoinConvention(AbstractInvocationFuture.java:835)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.join(AbstractInvocationFuture.java:545)
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.restoreTaskState(CheckpointCoordinator.java:250)
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$1(CheckpointCoordinator.java:189)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
... 3 more
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException:
java.lang.RuntimeException: java.util.concurrent.ExecutionException
at
org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$run$0(NotifyTaskRestoreOperation.java:100)
at
org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
at
org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.run(NotifyTaskRestoreOperation.java:85)
at
com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
at
com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:411)
at
com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:438)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:601)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:580)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61)
at
org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMemberNode(NodeEngineUtil.java:51)
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.sendOperationToMemberNode(CheckpointManager.java:272)
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.restoreTaskState(CheckpointCoordinator.java:249)
... 5 more
Caused by: java.lang.RuntimeException:
java.util.concurrent.ExecutionException
at
org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.restoreState(SourceFlowLifeCycle.java:262)
at
org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$14(SeaTunnelTask.java:383)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at
org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:380)
at
org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$run$0(NotifyTaskRestoreOperation.java:96)
... 21 more
Caused by: java.util.concurrent.ExecutionException
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:121)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:100)
at
com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:609)
at
org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.restoreState(SourceFlowLifeCycle.java:259)
... 34 more
at
org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:119)
at
org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:174)
Server exception logs are as follows:
------------------------------------------------------------------------------------
2023-05-17 14:42:44,567 INFO
org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting
seatunnel source task, index 0
2023-05-17 14:42:44,575 ERROR
org.apache.seatunnel.engine.server.task.operation.source.RestoredSplitOperation
- [ecs-fd88-0002]:5801 [seatunnel_bu2] [5.1] ErrorCode:[KAFKA-04],
ErrorDescription:[Add a s
plit back to the split enumerator failed,it will only happen when a
SourceReader failed] - java.lang.NullPointerException
at
org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplitEnumerator.listOffsets(KafkaSourceSplitEnumerator.java:331)
at
org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplitEnumerator.convertToNextSplit(KafkaSourceSplitEnumerator.java:186)
at
org.apache.seatunnel.connectors.seatunnel.kafka.source.KafkaSourceSplitEnumerator.addSplitsBack(KafkaSourceSplitEnumerator.java:177)
at
org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask.addSplitsBack(SourceSplitEnumeratorTask.java:187)
at
org.apache.seatunnel.engine.server.task.operation.source.RestoredSplitOperation.lambda$run$1(RestoredSplitOperation.java:98)
at
org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
at
org.apache.seatunnel.engine.server.task.operation.source.RestoredSplitOperation.run(RestoredSplitOperation.java:82)
at
com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
at
com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:411)
at
com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:438)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:601)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:580)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61)
at
org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMemberNode(NodeEngineUtil.java:51)
at
org.apache.seatunnel.engine.server.execution.TaskExecutionContext.sendToMember(TaskExecutionContext.java:47)
at
org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle.restoreState(SourceFlowLifeCycle.java:253)
at
org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$14(SeaTunnelTask.java:383)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at
org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:380)
at
org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$run$0(NotifyTaskRestoreOperation.java:96)
at
org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
at
org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.run(NotifyTaskRestoreOperation.java:85)
at
com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
at
com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
at
com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:411)
at
com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:438)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:601)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:580)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541)
at
com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241)
at
com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61)
at
org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMemberNode(NodeEngineUtil.java:51)
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.sendOperationToMemberNode(CheckpointManager.java:272)
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.restoreTaskState(CheckpointCoordinator.java:249)
at
org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$reportedTask$1(CheckpointCoordinator.java:189)
at
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
......
2023-05-17 14:42:44,580 WARN
org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils -
Principal [null] or keytabPath [null] is empty, it will skip kerberos
authentication
2023-05-17 14:42:44,581 WARN
org.apache.seatunnel.engine.server.task.flow.SourceFlowLifeCycle - source
request split failed {}
java.util.concurrent.ExecutionException: null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
