LOOPBACK is a special testing URN and won't work unless you start a
Python SDK Harness worker. Like Kyle said, I would just use the default.
You will have to have support for Docker on your deployment machines. Or
work around this using a process-based environment.
Thanks,
Max
On 24.05.19 17:39, Kyle Weaver wrote:
Have you tried running without setting the environmentType flag?
On Fri, May 24, 2019 at 8:24 AM 青雉（祁明良） <m...@xiaohongshu.com
<mailto:m...@xiaohongshu.com>> wrote:
Hi all,
This is Mingliang, I followed the document to setup beam pipeline on
local Flink cluster, but when I switch to remote Flink cluster, it
doesn’t work directly and I can’t find many materials talking about
this.
Firstly, I don’t know if I have to install anything on the Flink
jobmanager / taskmanager machine, actually I just setup the Flink
cluster itself, let’s say Jobmanager on machine A and two
taskmanager on machine B,C.
Next I installed apache_beam python package on machine D, and
started the beam jobservice end point on machine D with: ./gradlew
beam-runners-flink_2.11-job-server:runShadow -PflinkMasterUrl=A:8081
Then I submit the word_count example downloaded from beam to the
jobservice end point on machine D with environment_type = LOOPBACK
The error message I received was attached below.
Any helps will be appreciated,
Thanks,
Mingliang
- Versions
Beam: release-2.12.0
Flink: 1.5.6
- Log
----------------
Caused by: java.lang.Exception: The user defined 'open()' method
caused an exception:
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException:
UNAVAILABLE: io exception
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
... 1 more
Caused by:
org.apache.beam.vendor.guava.v20_0.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException:
UNAVAILABLE: io exception
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4992)
at
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory.forStage(DefaultJobBundleFactory.java:186)
at
org.apache.beam.runners.flink.translation.functions.FlinkDefaultExecutableStageContext.getStageBundleFactory(FlinkDefaultExecutableStageContext.java:49)
at
org.apache.beam.runners.flink.translation.functions.ReferenceCountingFlinkExecutableStageContextFactory$WrappedContext.getStageBundleFactory(ReferenceCountingFlinkExecutableStageContextFactory.java:203)
at
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction.open(FlinkExecutableStageFunction.java:143)
at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:494)
... 3 more
Caused by:
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusRuntimeException:
UNAVAILABLE: io exception
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203)
at
org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
at
org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc$BeamFnExternalWorkerPoolBlockingStub.notifyRunnerAvailable(BeamFnExternalWorkerPoolGrpc.java:152)
at
org.apache.beam.runners.fnexecution.environment.ExternalEnvironmentFactory.createEnvironment(ExternalEnvironmentFactory.java:109)
at
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:178)
at
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:161)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
at
org.apache.beam.vendor.guava.v20_0.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
... 13 more
Caused by:
org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: localhost/127.0.0.1:45955 <http://127.0.0.1:45955>
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
at
org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at
org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:635)
at
org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:582)
at
org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)
at
org.apache.beam.vendor.grpc.v1p13p1.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:461)
at
org.apache.beam.vendor.grpc.v1p13p1.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at
org.apache.beam.vendor.grpc.v1p13p1.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
Caused by: java.net.ConnectException: Connection refused
... 11 more
本邮件及其附件含有小红书公司的保密信息，仅限于发送给以上收件人或群
组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复
制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通
知发件人并删除本邮件！
This communication may contain privileged or other confidential
information of Red. If you have received it in error, please advise
the sender by reply e-mail and immediately delete the message and
any attachments without copying or disclosing the contents. Thank you.
--
Kyle Weaver | Software Engineer | github.com/ibzib
<http://github.com/ibzib> | kcwea...@google.com
<mailto:kcwea...@google.com> | +16502035555
