Hi, At the beginning, I was wondering myself that too, and I don't know why hbase-common wasn''t being downloaded and included, so I added it explicitly.
I was in the process to write that maybe I've solved this weird issue: apparently the shading worked and the ClassDefNotFound issue was caused by the missing hbase-client jar inside the fat jar, even though there should have been! So, I added again the explicit dependency for it in the build.sbt and now it's working. 2017-08-25 23:03 GMT+02:00 Robert Metzger <rmetz...@apache.org>: > Hi, > > why do you need to add hbase-common as a separate dependency? Doesn't the > "flink-hbase" dependency transitively pull in hbase? > > On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> If Guava 18.0 is used to build hbase 1.3, there would be compilation >> errors such as the following: >> >> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/had >> oop/hbase/replication/regionserver/ReplicationSource.java:[271,25] >> error: cannot find symbol >> [ERROR] symbol: method stopAndWait() >> [ERROR] location: variable replicationEndpoint of type >> ReplicationEndpoint >> [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/apache/had >> oop/hbase/replication/regionserver/ReplicationSource.java:[281,47] >> error: cannot find symbol >> [ERROR] symbol: method start() >> [ERROR] location: variable replicationEndpoint of type >> ReplicationEndpoint >> >> Maybe you can shade the guava dependency in hbase 1.3 >> >> In the upcoming hbase 2.0 release, third party dependencies such as guava >> and netty are shaded. Meaning there wouldn't be such conflict with other >> components you may use. >> >> On Fri, Aug 25, 2017 at 8:36 AM, Federico D'Ambrosio < >> federico.dambro...@smartlab.ws> wrote: >> >>> Hello everyone, I'm new to Flink and am encountering a nasty problem >>> while trying to submit a streaming Flink Job. I'll try to explain it as >>> thoroughly as possible. >>> >>> Premise: I'm using an HDP 2.6 hadoop cluster, with hadoop version >>> 2.7.3.2.6.1.0-129, Flink compiled from sources accordingly (maven 3.0.5) as >>> per documentation >>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/building.html#vendor-specific-versions> >>> and I submit the job using yarn-session.sh and then flink run. >>> >>> Getting into more details, the Flink Job is a fat jar built with sbt >>> assembly and the following dependencies: >>> >>> libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % >>> "provided" >>> libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % >>> flinkVersion % "provided" >>> libraryDependencies += "org.apache.flink" %% "flink-connector-kinesis" % >>> flinkVersion % "provided" >>> libraryDependencies += "org.apache.flink" %% "flink-hbase" % flinkVersion % >>> "provided" >>> libraryDependencies += "org.apache.flink" %% "flink-connector-filesystem" % >>> flinkVersion % "provided" >>> libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.3.0" >>> libraryDependencies += "org.joda" % "joda-convert" % "1.8.3" >>> libraryDependencies += "com.typesafe.play" %% "play-json" % "2.6.2" >>> >>> >>> assemblyOption in assembly := (assemblyOption in >>> assembly).value.copy(includeScala = false) >>> >>> >>> inside the flink lib folder I have the following Jars: >>> >>> flink-connector-filesystem_2.10-1.3.2.jar >>> flink-python_2.10-1.3.2.jar >>> flink-connector-kinesis_2.10-1.3.2.jar >>> flink-shaded-hadoop2-uber-1.3.2.jar >>> flink-dist_2.10-1.3.2.jar >>> log4j-1.2.17.jar >>> flink-hbase_2.10-1.3.2.jar >>> slf4j-log4j12-1.7.7.jar >>> >>> Right now, the issue is that when I flink run the job, I get >>> NoClassDefFound for org/apache/hadoop/hbase/client/Put class, despite >>> it being inside the fat jar. So, I tried putting both hbase-common.jar and >>> hbase-client.jars inside the lib folder, getting as a result another >>> NoClassDefFound, only, this time, for com/google/common/collect/List >>> Multimap. Now, I noticed that flink-connector-kinesis and all the hbase >>> packages (both hbase-common and flink-hbase deps) have a different version >>> of guava as dependency: the first one needs version 18.0, while the second >>> one needs 12.0.1 version. I, then, tried to enable the shading of guava >>> packages with the following: >>> >>> assemblyShadeRules in assembly := Seq( >>> ShadeRule.rename("com.google.**" -> "shade.com.google.@1").inAll >>> ) >>> >>> >>> Only to no avail, getting the exact same error as before. And I really >>> don't know how to submit the job without pouring every single dependency >>> inside the lib folder. I thought it could be a compatibility issue with the >>> YARN cluster, but I tried launching the job locally only to get the same >>> error. >>> >>> As an additional info, I get the following warnings if I don't specify >>> flink packages as provided: >>> >>> [warn] * org.scalamacros:quasiquotes_2.10:2.1.0 is selected over >>> 2.0.1 >>> [warn] +- org.typelevel:macro-compat_2.10:1.1.1 >>> (depends on 2.1.0) >>> [warn] +- org.apache.flink:flink-scala_2.10:1.3.2 >>> (depends on 2.0.1) >>> [warn] >>> [warn] * org.mortbay.jetty:jetty-util:6.1.26 is selected over >>> 6.1.26.hwx >>> [warn] +- org.apache.hbase:hbase-common:1.3.0 >>> (depends on 6.1.26) >>> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2 >>> (depends on 6.1.26.hwx) >>> [warn] >>> [warn] * io.netty:netty:3.8.0.Final is selected over 3.7.0.Final >>> [warn] +- com.data-artisans:flakka-remote_2.10:2.3-custom >>> (depends on 3.8.0.Final) >>> [warn] +- org.apache.zookeeper:zookeeper:3.4.6 >>> (depends on 3.7.0.Final) >>> [warn] >>> [warn] * com.google.guava:guava:18.0 is selected over 12.0.1 >>> [warn] +- org.apache.flink:flink-connector-kinesis_2.10:1.3.2 >>> (depends on 18.0) >>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 >>> (depends on 12.0.1) >>> [warn] +- org.apache.hbase:hbase-procedure:1.3.0 >>> (depends on 12.0.1) >>> [warn] +- org.apache.hbase:hbase-client:1.3.0 >>> (depends on 12.0.1) >>> [warn] +- org.apache.hbase:hbase-common:1.3.0 >>> (depends on 12.0.1) >>> [warn] +- org.apache.hbase:hbase-server:1.3.0 >>> (depends on 12.0.1) >>> [warn] >>> [warn] * io.netty:netty-all:4.0.27.Final is selected over >>> 4.0.23.Final >>> [warn] +- org.apache.flink:flink-runtime_2.10:1.3.2 >>> (depends on 4.0.27.Final) >>> [warn] +- org.apache.flink:flink-shaded-hadoop2:1.3.2 >>> (depends on 4.0.27.Final) >>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 >>> (depends on 4.0.23.Final) >>> [warn] +- org.apache.hbase:hbase-client:1.3.0 >>> (depends on 4.0.23.Final) >>> [warn] +- org.apache.hbase:hbase-server:1.3.0 >>> (depends on 4.0.23.Final) >>> [warn] >>> [warn] * junit:junit:4.12 is selected over 3.8.1 >>> [warn] +- org.apache.hbase:hbase-protocol:1.3.0 >>> (depends on 4.12) >>> [warn] +- org.apache.hbase:hbase-annotations:1.3.0 >>> (depends on 4.12) >>> [warn] +- org.apache.hbase:hbase-prefix-tree:1.3.0 >>> (depends on 4.12) >>> [warn] +- org.apache.hbase:hbase-procedure:1.3.0 >>> (depends on 4.12) >>> [warn] +- org.apache.hbase:hbase-client:1.3.0 >>> (depends on 4.12) >>> [warn] +- org.apache.hbase:hbase-common:1.3.0 >>> (depends on 4.12) >>> [warn] +- org.apache.hbase:hbase-server:1.3.0 >>> (depends on 4.12) >>> [warn] +- jline:jline:0.9.94 >>> (depends on 3.8.1) >>> >>> And here I am asking for help here. >>> >>> Thank you very much for your attention, >>> Kind regards, >>> >>> Federico D'Ambrosio >>> >> >> >
