Our pom.xml is pulling in Hadoop 3 jars, but HBase is transitively pulling some Hadoop 2 jars, and the combination of both leads to strange runtime errors like ClassNotFoundException. If we manually fix the classpath to only references Hadoop 3 jars, everything works fine, but Maven and Eclipse etc keep on mixing Hadoop 3 and Hadoop 2 jars, which leads to various problems down the road.
We already build HBase with -Dhadoop.profile=3.0 but that's not the question. We want to build *our* MapReduce code and have Maven only pull in Hadoop 3 dependencies. We tried to copy the <profile> from HBase's pom.xml into ours — admittedly mostly just cargo-culting it, since we're not really knowledgeable about Maven in general, hence this question. How is a project dependent on hbase-mapreduce supposed to specify in its pom.xml that it only wants Hadoop 3 dependencies pulled in? On Tue, Jan 4, 2022 at 5:14 AM 张铎(Duo Zhang) <[email protected]> wrote: > I think hadoop 2.10.0 client can communicate with hadoop 3.3.1 cluster? > > If you want to build hbase-2.4.9 by yourself so you can depend on > hadoop 3.3.1, please download the src tarball, try this command > > mvn -DskipTests -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.1 > clean install && mvn -DskipTests -Dhadoop.profile=3.0 > -Dhadoop-three.version=3.3.1 package assembly:single > > Thanks. > > Giuseppe Valente <[email protected]> 于2022年1月4日周二 09:48写道： > > > > Hi, > > > > I have a project that has a dependency on hbase-mapreduce (2.4.9), which > in turn depends on hbase-hadoop2-compat. The latter doesn’t have any > version specified for hadoop-mapreduce-client-core and ends up pulling > 2.10.0, which conflicts with the Hadoop version I’m running against > (3.3.1). What’s the correct way to force hadoop-mapreduce to work with > Hadoop 3.3.1? I tried using profiles but I couldn’t get it to work, I > wonder if it has anything to do with this change < > https://issues.apache.org/jira/browse/HBASE-24309>: > > > > commit dc2146069cd5ef6120f0a9d1918acba0f74ac5c3 > > Author: Duo Zhang <[email protected]> > > Date: Wed May 13 17:59:21 2020 +0800 > > > > HBASE-24309 Avoid introducing log4j and slf4j-log4j dependencies for > … (#1697) > > > > Signed-off-by: stack <[email protected]> > > > > diff --git a/hbase-hadoop2-compat/pom.xml b/hbase-hadoop2-compat/pom.xml > > index 236b52ce65..96472fb8f3 100644 > > --- a/hbase-hadoop2-compat/pom.xml > > +++ b/hbase-hadoop2-compat/pom.xml > > @@ -118,14 +118,12 @@ limitations under the License. > > <dependency> > > <groupId>org.apache.hbase</groupId> > > <artifactId>hbase-hadoop-compat</artifactId> > > - <version>${project.version}</version> > > <type>test-jar</type> > > <scope>test</scope> > > </dependency> > > <dependency> > > <groupId>org.apache.hadoop</groupId> > > <artifactId>hadoop-mapreduce-client-core</artifactId> > > - <version>${hadoop.version}</version> > > <exclusions> > > <exclusion> > > <groupId>com.google.guava</groupId> > > > > Thanks, > > Giuseppe > -- Benoit "tsuna" Sigoure
