The error seems to be originating from https://github.com/apache/beam/blob/master/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java#L166
Can you check what is the endpoint for the Data Channel and why was it not set to begin with? Few more questions * Which version of beam are you using? * What is the environment that you are using? If default/Docker then you can check the docker container logs to see if their is more info. * Approximately after how long does this happen? and what is the data size for failure. Thanks, Ankur On Thu, Jun 13, 2019 at 9:54 AM Kyle Weaver <[email protected]> wrote: > I'm not sure the log excerpt you attached contains the root cause of your > issue. Unfortunately, Beam's portable runners print a great deal of > irrelevant error messages like these, even when the SDK harness disconnects > normally. So it is possible the real issue is buried elsewhere in the log? > > Kyle Weaver | Software Engineer | github.com/ibzib | [email protected] > | +16502035555 > > > On Wed, Jun 12, 2019 at 8:59 PM 青雉（祁明良） <[email protected]> wrote: > >> Hi All, >> >> I’m running a simple job decoding tfrecord with python sdk on Flink. It >> works with small input data from hdfs, and when I switch to large input >> data, it fails because of grpc cancelled. The error message makes it >> difficult to debug further. Any suggestions for the next steps? >> >> Best, >> Mingliang >> >> ----------------------- >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: >> CANCELLED: cancelled before receiving half close >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status.asRuntimeException(Status.java:517) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:272) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingServerCallListener.onCancel(PartialForwardingServerCallListener.java:40) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:23) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:40) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Contexts$ContextualizedServerCallListener.onCancel(Contexts.java:96) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:293) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:738) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:748) >> 2019-06-13 03:47:13,430 WARN >> org.apache.beam.runners.fnexecution.logging.GrpcLoggingService - Beam Fn >> Logging client failed to be complete. >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: >> CANCELLED: call already cancelled >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status.asRuntimeException(Status.java:517) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:356) >> at >> org.apache.beam.runners.fnexecution.logging.GrpcLoggingService.completeIfNotNull(GrpcLoggingService.java:78) >> at >> org.apache.beam.runners.fnexecution.logging.GrpcLoggingService.access$400(GrpcLoggingService.java:33) >> at >> org.apache.beam.runners.fnexecution.logging.GrpcLoggingService$InboundObserver.onError(GrpcLoggingService.java:105) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:269) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingServerCallListener.onCancel(PartialForwardingServerCallListener.java:40) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:23) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:40) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Contexts$ContextualizedServerCallListener.onCancel(Contexts.java:96) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:293) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:738) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:748) >> 2019-06-13 03:47:13,430 ERROR >> org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer - Failed to >> handle for unknown endpoint >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: >> CANCELLED: cancelled before receiving half close >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status.asRuntimeException(Status.java:517) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:272) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingServerCallListener.onCancel(PartialForwardingServerCallListener.java:40) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:23) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:40) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Contexts$ContextualizedServerCallListener.onCancel(Contexts.java:96) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:293) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:738) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:748) >> 2019-06-13 03:47:13,432 ERROR >> org.apache.beam.runners.fnexecution.control.FnApiControlClient - >> FnApiControlClient received error >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: >> CANCELLED: cancelled before receiving half close >> 2019-06-13 03:47:13,432 ERROR >> org.apache.beam.runners.fnexecution.control.FnApiControlClient - >> FnApiControlClient closed, clearing outstanding requests >> {4=java.util.concurrent.CompletableFuture@69065918[Not completed, 1 >> dependents], 10=java.util.concurrent.CompletableFuture@6d39d0e0[Not >> completed, 1 dependents]} >> Jun 13, 2019 3:47:13 AM >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor run >> SEVERE: Exception while executing runnable >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@187c2831 >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: >> CANCELLED: call already cancelled >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status.asRuntimeException(Status.java:517) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:356) >> at >> org.apache.beam.runners.fnexecution.state.GrpcStateService$Inbound.onError(GrpcStateService.java:145) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:269) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingServerCallListener.onCancel(PartialForwardingServerCallListener.java:40) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:23) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:40) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Contexts$ContextualizedServerCallListener.onCancel(Contexts.java:96) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:293) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:738) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) >> at >> org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:748) >> >> 本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ >> >> This communication may contain privileged or other confidential >> information of Red. If you have received it in error, please advise the >> sender by reply e-mail and immediately delete the message and any >> attachments without copying or disclosing the contents. Thank you. >> >>
