Hi, Actually Flink's inverted class loading feature was designed to mitigate problems with different versions of libraries that are not compatible with each other [1]. You may want to debug why it does not work for you.
You can also try to use the Hadoop free Flink distribution, and export the HADOOP_CLASSPATH variable [2]. Best, Gary [1] https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/debugging_classloading.html#inverted-class-loading-and-classloader-resolution-order [2] https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/deployment/hadoop.html#configuring-flink-with-hadoop-classpaths On Wed, Feb 27, 2019 at 5:23 AM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hi, > > I recently experienced versioning clashes with the okio and okhttp when > trying to deploy a Flink 1.6.0 app to AWS EMR on Hadoop 2.8.4. After > investigating and talking to the okio team (see this issue) > <https://github.com/square/okio/issues/559>, I found that both okio and > okhttp exist in the Flink uber jar with versions 1.4.0 and 2.4.0, > respectively, whereas I'm including versions 2.2.2 and 3.13.1 in my shaded > jar. The okio team suggested that Flink should shade the uber jar to fix > the issue, but I'm wondering if there is something I can do on my end to > have all versions exist simultaneously. > > From the issue, here are the okio contents of the uber jar: > > *jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio* > > META-INF/maven/com.squareup.okio/ > META-INF/maven/com.squareup.okio/okio/ > META-INF/maven/com.squareup.okio/okio/pom.properties > META-INF/maven/com.squareup.okio/okio/pom.xml > okio/ > okio/AsyncTimeout$1.class > okio/AsyncTimeout$2.class > okio/AsyncTimeout$Watchdog.class > okio/AsyncTimeout.class > okio/Base64.class > okio/Buffer$1.class > okio/Buffer$2.class > okio/Buffer.class > okio/BufferedSink.class > okio/BufferedSource.class > okio/ByteString.class > okio/DeflaterSink.class > okio/ForwardingSink.class > okio/ForwardingSource.class > okio/ForwardingTimeout.class > okio/GzipSink.class > okio/GzipSource.class > okio/InflaterSource.class > okio/Okio$1.class > okio/Okio$2.class > okio/Okio$3.class > okio/Okio.class > okio/RealBufferedSink$1.class > okio/RealBufferedSink.class > okio/RealBufferedSource$1.class > okio/RealBufferedSource.class > okio/Segment.class > okio/SegmentPool.class > okio/SegmentedByteString.class > okio/Sink.class > okio/Source.class > okio/Timeout$1.class > okio/Timeout.class > okio/Util.class > > Thank you, > Austin Cawley-Edwards >
