[
https://issues.apache.org/jira/browse/BEAM-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726340#comment-16726340
]
Ahmet Altay commented on BEAM-6249:
-----------------------------------
For the benefit of Beam 2.9.0 users running streaming workflows on Dataflow
using streaming engine, there are 2 options:
- Do not yet upgrade to 2.9. There will be a fix in 2.10 as the vendoring
issues are now resolved.
- If you would like to upgrade to 2.9, also enable Conscrypt by setting the
experimental flag "--experiments=enable_conscrypt_security_provider enables".
However note that, enabling conscrypt could lead to hanging issues as well.
(2.8 version also has conscrypt enabled by default. 2.4 version is the most
recent version that does not have conscrypt on by default other than 2.9.)
> Vendored gRPC doesn't seem to work with dataflow
> ------------------------------------------------
>
> Key: BEAM-6249
> URL: https://issues.apache.org/jira/browse/BEAM-6249
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Affects Versions: 2.9.0
> Reporter: Steve Niemitz
> Assignee: Tyler Akidau
> Priority: Major
>
> I attempted to migrate an existing pipeline (that worked in 2.8.0) to 2.9.0.
> This pipeline is using the experimental streaming engine
> (–experiments=enable_streaming_engine).
> The pipeline fails to start with these logs:
> {code:java}
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64', trying
> other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64 cannot be
> loaded from java.libary.path, now trying export to -Dio.netty.native.workdir:
> /tmp
> D Unable to load the library
> '/tmp/liborg_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_646918605450681921540.so',
> trying other loading mechanism.
> D Unable to load the library 'netty_tcnative_linux_x86_64', trying next
> name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora',
> trying other loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_linux_x86_64_fedora
> cannot be loaded from java.libary.path, now trying export to
> -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_linux_x86_64_fedora', trying
> next name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64', trying other
> loading mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative_x86_64 cannot be loaded
> from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative_x86_64', trying next name...
> D Unable to load the library
> 'org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative', trying other loading
> mechanism.
> D org_apache_beam_vendor_grpc_v1_13_1_netty_tcnative cannot be loaded from
> java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
> D Unable to load the library 'netty_tcnative', trying next name...
> D Failed to load netty-tcnative; OpenSslEngine will be unavailable, unless
> the application has already loaded the symbols by some other means. See
> http://netty.io/wiki/forked-tomcat-native.html for more information.
> D Failed to initialize netty-tcnative; OpenSslEngine will be unavailable.
> See http://netty.io/wiki/forked-tomcat-native.html for more information.
> I netty-tcnative unavailable (this may be normal)
> I Conscrypt not found (this may be normal)
> I Jetty ALPN unavailable (this may be normal)
> E Uncaught exception in main thread. Exiting with status code 1.
> W Please use a logger instead of System.out or System.err.
> Please switch to using org.slf4j.Logger.
> See: https://cloud.google.com/dataflow/pipelines/logging
> E Uncaught exception in main thread. Exiting with status code 1.
> E java.lang.IllegalStateException: Could not find TLS ALPN provider; no
> working netty-tcnative, Conscrypt, or Jetty NPN/ALPN available
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:256)
>
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
>
> E at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120)
>
> E at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.remoteChannel(GrpcWindmillServer.java:343)
>
> E at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.initializeWindmillService(GrpcWindmillServer.java:312)
>
> {code}
>
> The interesting part is in the netty load failure, the stack trace is:
> {code:java}
> exception: "java.lang.UnsatisfiedLinkError at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:276)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:187)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:430)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:97)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:242)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
> at
> org.apache.beam.vendor.grpc.v1_13_1.io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120)
> at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.remoteChannel(GrpcWindmillServer.java:343)
> at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.initializeWindmillService(GrpcWindmillServer.java:312)
> at
> org.apache.beam.runners.dataflow.worker.windmill.GrpcWindmillServer.setWindmillServiceEndpoints(GrpcWindmillServer.java:192)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.getConfigFromDataflowService(StreamingDataflowWorker.java:1528)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.getConfig(StreamingDataflowWorker.java:1583)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.getGlobalConfig(StreamingDataflowWorker.java:1568)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.schedulePeriodicGlobalConfigRequests(StreamingDataflowWorker.java:1543)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.start(StreamingDataflowWorker.java:704)
> at
> org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.main(StreamingDataflowWorker.java:228)
> Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
> at java.security.AccessController.doPrivileged(Native Method) at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
> ... 17 more Caused by: java.lang.NoClassDefFoundError:
> org/apache/beam/vendor/grpc/v1/13/1/io/netty/internal/tcnative/Library at
> java.lang.ClassLoader$NativeLibrary.load(Native Method) at
> java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) at
> java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) at
> java.lang.Runtime.load0(Runtime.java:809) at
> java.lang.System.load(System.java:1086) at
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
> ... 24 more Caused by: java.lang.ClassNotFoundException:
> org.apache.beam.vendor.grpc.v1.13.1.io.netty.internal.tcnative.Library at
> java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 30 more{code}
>
> Notice that the class attempting to be loaded is:
> org.apache.beam.vendor.grpc.v1.13.1.io.netty.internal.tcnative.Library, but
> it's actually defined in the jar as
> org.apache.beam.vendor.grpc.v1_13_1.io.netty.internal.tcnative.Library.
> I traced this back to the jni interop code in tcnative:
> [https://github.com/netty/netty-tcnative/blob/master/openssl-dynamic/src/main/c/jnilib.c#L266]
> Here it replaces all _ in the package prefix with /, which won't work here.
> The fix seems like it would be to repackage the vendored gRPC with a
> different prefix that doesn't contain underscores.
> I'm curious how this ever worked though? Maybe the streaming engine is the
> only thing using this vendored gRPC code?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
