Hi all, This is Mingliang, I followed the document to setup beam pipeline on local Flink cluster, but when I switch to remote Flink cluster, it doesn’t work directly and I can’t find many materials talking about this.
Firstly, I don’t know if I have to install anything on the Flink jobmanager / taskmanager machine, actually I just setup the Flink cluster itself, let’s say Jobmanager on machine A and two taskmanager on machine B,C. Next I installed apache_beam python package on machine D, and started the beam jobservice end point on machine D with: ./gradlew beam-runners-flink_2.11-job-server:runShadow -PflinkMasterUrl=A:8081 Then I submit the word_count example downloaded from beam to the jobservice end point on machine D with environment_type = LOOPBACK The error message I received was attached below. Any helps will be appreciated, Thanks, Mingliang - Versions Beam: release-2.12.0 Flink: 1.5.6 - Log ---------------- Caused by: java.lang.Exception: The user defined 'open()' method caused an exception: org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498) at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703) ... 1 more Caused by: org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.get(LocalCache.java:4053) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4992) at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory.forStage(DefaultJobBundleFactory.java:186) at org.apache.beam.runners.flink.translation.functions.FlinkDefaultExecutableStageContext.getStageBundleFactory(FlinkDefaultExecutableStageContext.java:49) at org.apache.beam.runners.flink.translation.functions.ReferenceCountingFlinkExecutableStageContextFactory$WrappedContext.getStageBundleFactory(ReferenceCountingFlinkExecutableStageContextFactory.java:203) at org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction.open(FlinkExecutableStageFunction.java:143) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:494) ... 3 more Caused by: org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132) at org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc$BeamFnExternalWorkerPoolBlockingStub.notifyRunnerAvailable(BeamFnExternalWorkerPoolGrpc.java:152) at org.apache.beam.runners.fnexecution.environment.ExternalEnvironmentFactory.createEnvironment(ExternalEnvironmentFactory.java:109) at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:178) at org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:161) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295) at org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208) ... 13 more Caused by: org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:45955 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325) at org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:635) at org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582) at org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) at org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461) at org.apache.beam.vendor.grpc.v1p13p1.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at org.apache.beam.vendor.grpc.v1p13p1.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more Caused by: java.net.ConnectException: Connection refused ... 11 more 本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This communication may contain privileged or other confidential information of Red. If you have received it in error, please advise the sender by reply e-mail and immediately delete the message and any attachments without copying or disclosing the contents. Thank you.
