If Guava 18.0 is used to build hbase 1.3, there would be compilation errors such as the following:
[ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/ apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25] error: cannot find symbol [ERROR] symbol: method stopAndWait() [ERROR] location: variable replicationEndpoint of type ReplicationEndpoint [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/ apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[281,47] error: cannot find symbol [ERROR] symbol: method start() [ERROR] location: variable replicationEndpoint of type ReplicationEndpoint Maybe you can shade the guava dependency in hbase 1.3 In the upcoming hbase 2.0 release, third party dependencies such as guava and netty are shaded. Meaning there wouldn't be such conflict with other components you may use. On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio < federico.dambro...@smartlab.ws> wrote: > Hello everyone, I'm new to Flink and am encountering a nasty problem while > trying to submit a streaming Flink Job. I'll try to explain it as > thoroughly as possible. > > Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version > 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as > per documentation > <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions> > and I submit the job using yarn-session.sh and then flink run. > > Getting into more details, the Flink Job is a fat jar built with sbt > assembly and the following dependencies: > > libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % > "provided" > libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % > flinkVersion % "provided" > libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % > flinkVersion % "provided" > libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % > "provided" > libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % > flinkVersion % "provided" > libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0" > libraryDependencies += "org.joda" % "joda-convert" % "1.8.3" > libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2" > > > assemblyOption in assembly := (assemblyOption in > assembly).value.copy(includeScala = false) > > > inside the flink lib folder I have the following Jars: > > flink-connector-filesystem_2.10-1.3.2.jar > flink-python_2.10-1.3.2.jar > flink-connector-kinesis_2.10-1.3.2.jar > flink-shaded-hadoop2-uber-1.3.2.jar > flink-dist_2.10-1.3.2.jar > log4j-1.2.17.jar > flink-hbase_2.10-1.3.2.jar > slf4j-log4j12-1.7.7.jar > > Right now, the issue is that when I flink run the job, I get > NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite it > being inside the fat jar. So, I tried putting both hbase-common.jar and > hbase-client.jars inside the lib folder, getting as a result another > NoClassDefFound, only, this time, for com/google/common/collect/ > ListMultimap. Now, I noticed that flink-connector-kinesis and all the > hbase packages (both hbase-common and flink-hbase deps) have a different > version of guava as dependency: the first one needs version 18.0, while the > second one needs 12.0.1 version. I, then, tried to enable the shading of > guava packages with the following: > > assemblyShadeRules in assembly := Seq( > ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll > ) > > > Only to no avail, getting the exact same error as before. And I really > don't know how to submit the job without pouring every single dependency > inside the lib folder. I thought it could be a compatibility issue with the > YARN cluster, but I tried launching the job locally only to get the same > error. > > As an additional info, I get the following warnings if I don't specify > flink packages as provided: > > [warn] * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over 2.0.1 > [warn] +- org.typelevel:macro-compat_2.10:1.1.1 > (depends on 2.1.0) > [warn] +- org.apache.flink:flink-scala_2.10:1.3.2 > (depends on 2.0.1) > [warn] > [warn] * org.mortbay.jetty:jetty-util:6.1.26 is selected over > 6.1.26.hwx > [warn] +- org.apache.hbase:hbase-common:1.3.0 > (depends on 6.1.26) > [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2 > (depends on 6.1.26.hwx) > [warn] > [warn] * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final > [warn] +- com.data-artisans:flakka-remote_2.10:2.3-custom > (depends on 3.8.0.Final) > [warn] +- org.apache.zookeeper:zookeeper:3.4.6 > (depends on 3.7.0.Final) > [warn] > [warn] * com.google.guava:guava:18.0 is selected over 12.0.1 > [warn] +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2 > (depends on 18.0) > [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 > (depends on 12.0.1) > [warn] +- org.apache.hbase:hbase-procedure:1.3.0 > (depends on 12.0.1) > [warn] +- org.apache.hbase:hbase-client:1.3.0 > (depends on 12.0.1) > [warn] +- org.apache.hbase:hbase-common:1.3.0 > (depends on 12.0.1) > [warn] +- org.apache.hbase:hbase-server:1.3.0 > (depends on 12.0.1) > [warn] > [warn] * io.netty:netty-all:4.0.27.Final is selected over 4.0.23.Final > [warn] +- org.apache.flink:flink-runtime_2.10:1.3.2 > (depends on 4.0.27.Final) > [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2 > (depends on 4.0.27.Final) > [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 > (depends on 4.0.23.Final) > [warn] +- org.apache.hbase:hbase-client:1.3.0 > (depends on 4.0.23.Final) > [warn] +- org.apache.hbase:hbase-server:1.3.0 > (depends on 4.0.23.Final) > [warn] > [warn] * junit:junit:4.12 is selected over 3.8.1 > [warn] +- org.apache.hbase:hbase-protocol:1.3.0 > (depends on 4.12) > [warn] +- org.apache.hbase:hbase-annotations:1.3.0 > (depends on 4.12) > [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 > (depends on 4.12) > [warn] +- org.apache.hbase:hbase-procedure:1.3.0 > (depends on 4.12) > [warn] +- org.apache.hbase:hbase-client:1.3.0 > (depends on 4.12) > [warn] +- org.apache.hbase:hbase-common:1.3.0 > (depends on 4.12) > [warn] +- org.apache.hbase:hbase-server:1.3.0 > (depends on 4.12) > [warn] +- jline:jline:0.9.94 > (depends on 3.8.1) > > And here I am asking for help here. > > Thank you very much for your attention, > Kind regards, > > Federico D'Ambrosio >
