[
https://issues.apache.org/jira/browse/YARN-5045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273218#comment-15273218
]
Sangjin Lee commented on YARN-5045:
-----------------------------------
This is basically a *diamond dependency problem*. Timeline service depends on
HBase 1.0.1 which in turn depends on hadoop-common *2.5.1*. However, timeline
service itself depends on hadoop-common *trunk*. In maven, the trunk version is
chosen as the hadoop dependency version is managed.
We knew there is potential risk of this diamond problem when the timeline
service is developed on trunk (where the backward incompatible changes can
happen) while we also depend on HBase which relies on a released version of
hadoop. This is a first manifestation of the issue.
There are several ways we can address this issue, but there are drawbacks on
all of the approaches.
(1) we can try to resurrect the metrics v.1 classes (that were removed in
HADOOP-12504)
We could add back those classes under {{src/test/java}} of the timelineservice
project. While this might be an easy way to get around this specific problem,
there are issues. First, it is in poor taste to resurrect code we're getting
rid of to work around this problem. Also, it does nothing to handle potential
future issues that may arise. What if classes' shapes change instead of being
removed?
(2) isolate the hbase unit test code and (en)force the hadoop version that
HBase relies on
We could try to override the default version of hadoop-common (trunk:
3.0.0-SNAPSHOT) while running the unit tests. It turns out this requires
creating a separate project. Prior to maven 3, there used to be a capability of
choosing a different version for the test scope and test scope only. However,
maven 3 or later seems to have removed that. Thus, for a given project, there
can only be one dependency version regardless of scope.
We could try to isolate the HBase-related unit tests into their own project and
enforce 2.5.1 in that project. This has a decent chance of success.
Still there are caveats. For this to work, the portion of the timeline service
code that exercises HBase has to ensure it works fine against hadoop-common
2.5.1 as well as trunk. I'm not sure how best to enforce this state.
Also, I thought of moving the non-test code that relies on HBase into this new
project as well, but I'm moving away from that idea. The main problem is that
we need to prevent the hadoop-common 2.5.1 jar from being incorporated into the
hadoop distribution tarball. If we have the non-test code in here, the worry is
hadoop-common 2.5.1 may sneak into the tarball.
(3) pull out timeline service entirely out of the hadoop project
This is truly an out-of-the-box idea. Technically it might be feasible to set
up the HBase-related part of the timeline service as a separate project from
hadoop. Then the hadoop-proper part of the timeline service would interact with
the HBase-related part of the timeline service over the wire. That way, hadoop
can still be shielded from the thorny HBase dependency issues.
While there is some possibility this may work technically, I think there are
several major issues, ranging from logistics of setting up a separate project
independent of hadoop, performance implications, etc. I think this is a long
shot at best.
Out of these options, it seems to me (2) is the least bad option. I am working
on making that work. I'd love to hear your thoughts if you have better
suggestions.
> hbase unit tests fail due to dependency issues
> ----------------------------------------------
>
> Key: YARN-5045
> URL: https://issues.apache.org/jira/browse/YARN-5045
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Affects Versions: YARN-2928
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Priority: Blocker
>
> After the 5/4 rebase, the hbase unit tests in the timeline service project
> are failing:
> {noformat}
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
> Time elapsed: 5.103 sec <<< ERROR!
> java.io.IOException: Shutting down
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> at
> org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> at
> org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
> at
> org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:217)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:153)
> at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:213)
> at
> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:978)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:938)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:750)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:87)
> {noformat}
> The root cause is that the hbase mini server depends on hadoop common's
> {{MetricsServlet}} which has been removed in the trunk (HADOOP-12504):
> {noformat}
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/metrics/MetricsServlet
> at
> org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> at
> org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
> at
> org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> ... 26 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
