[
https://issues.apache.org/jira/browse/YARN-5045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273960#comment-15273960
]
Varun Saxena commented on YARN-5045:
------------------------------------
[~sjlee0], thanks for the patch.
The patch looks fine. I ran the test cases and they all passed.
One question though. When I printed dependency tree for timelineservice hbase
tests project, I still see some 3.0.0-SNAPSHOT jars being taken. Few anyways
should be taken, such as hadoop-yarn-api, hadoop-yarn-common, etc.
But some seem to be pulled unnecessarily/wrongly. For instance, hadoop-common
is pulling in hadoop-auth 3.0.0-SNAPSHOT jar. Shouldnt this be 2.5.1 too ? To
avoid later surprises.
We can set it to 2.5.1 and exclude wherever 3.0.0-SNAPSHOT is being pulled in.
However, it is possible hadoop-auth may not be used at runtime.
Also, I see some mapreduce 3.0.0-SNAPSHOT jars being pulled via hbase
dependencies. I am not sure how mapreduce jars will be useful in our test flow
at runtime. Anyways should we enforce 2.5.1 version on them too. Or exclude
them ?
I did try excluding Mapreduce jars and setting hadoop-auth to 2.5.1. Tests
passed even after these changes.
{noformat}
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Hadoop YARN Timeline Service HBase tests 3.0.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.2:tree (default-cli) @
hadoop-yarn-server-timelineservice-hbase-tests ---
[INFO]
org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-tests:jar:3.0.0-SNAPSHOT
[INFO] +- junit:junit:jar:4.11:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +-
org.apache.hadoop:hadoop-yarn-server-timelineservice:jar:3.0.0-SNAPSHOT:test
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:test
[INFO] | | \- jdk.tools:jdk.tools:jar:1.7:system
[INFO] | +- com.google.inject:guice:jar:3.0:test
[INFO] | | +- javax.inject:javax.inject:jar:1:test
[INFO] | | \- aopalliance:aopalliance:jar:1.0:test
[INFO] | +- javax.servlet:servlet-api:jar:2.5:test
[INFO] | +- javax.xml.bind:jaxb-api:jar:2.2.2:test
[INFO] | | +- javax.xml.stream:stax-api:jar:1.0-2:test
[INFO] | | \- javax.activation:activation:jar:1.1:test
[INFO] | +- commons-cli:commons-cli:jar:1.2:test
[INFO] | +- commons-lang:commons-lang:jar:2.6:test
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:test
[INFO] | +- org.apache.commons:commons-csv:jar:1.0:test
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:test
[INFO] | \- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:test
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.5.1:test
[INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:test
[INFO] | +- xmlenc:xmlenc:jar:0.52:test
[INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:test
[INFO] | +- commons-codec:commons-codec:jar:1.4:test
[INFO] | +- commons-net:commons-net:jar:3.1:test
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:test
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:test
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:test
[INFO] | +- com.sun.jersey:jersey-json:jar:1.9:test
[INFO] | | \- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:test
[INFO] | +- com.sun.jersey:jersey-server:jar:1.9:test
[INFO] | | \- asm:asm:jar:3.2:test (version managed from 3.1)
[INFO] | +- tomcat:jasper-compiler:jar:5.5.23:test
[INFO] | +- tomcat:jasper-runtime:jar:5.5.23:test
[INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:test
[INFO] | +- commons-el:commons-el:jar:1.0:test
[INFO] | +- log4j:log4j:jar:1.2.17:test
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:test
[INFO] | | +- org.apache.httpcomponents:httpcore:jar:4.2.5:test
[INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:test
[INFO] | +- commons-configuration:commons-configuration:jar:1.6:test
[INFO] | | +- commons-digester:commons-digester:jar:1.8:test
[INFO] | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:test
[INFO] | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:test
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.10:test
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.10:test
[INFO] | +- org.apache.avro:avro:jar:1.7.4:test
[INFO] | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:test
[INFO] | | \- org.xerial.snappy:snappy-java:jar:1.0.4.1:test
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:test
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:3.0.0-SNAPSHOT:test
[INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:3.9:compile (scope managed
from test)
[INFO] | | | +- net.jcip:jcip-annotations:jar:1.0:compile
[INFO] | | | +- net.minidev:json-smart:jar:1.1.1:compile
[INFO] | | | \- commons-io:commons-io:jar:2.4:compile
[INFO] | | +-
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:test
[INFO] | | | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:test
[INFO] | | | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:test
[INFO] | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:test
[INFO] | | \- org.apache.curator:curator-framework:jar:2.7.1:test
[INFO] | | \- org.apache.curator:curator-client:jar:2.7.1:test
[INFO] | +- com.jcraft:jsch:jar:0.1.51:test (version managed from 0.1.42)
[INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.0:test
[INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:test
[INFO] | \- org.apache.commons:commons-compress:jar:1.4.1:test
[INFO] | \- org.tukaani:xz:jar:1.0:test
[INFO] +- org.apache.hadoop:hadoop-yarn-api:jar:3.0.0-SNAPSHOT:test
[INFO] +- org.apache.hadoop:hadoop-yarn-common:jar:3.0.0-SNAPSHOT:test
[INFO] | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:test (version
managed from 1.8.3)
[INFO] | +- org.codehaus.jackson:jackson-xc:jar:1.9.13:test (version managed
from 1.8.3)
[INFO] | +- com.google.inject.extensions:guice-servlet:jar:3.0:test
[INFO] | \- com.sun.jersey.contribs:jersey-guice:jar:1.9:test
[INFO] +- org.apache.hadoop:hadoop-yarn-server-common:jar:3.0.0-SNAPSHOT:test
[INFO] | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:test
[INFO] +-
org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:3.0.0-SNAPSHOT:test
[INFO] | +- org.codehaus.jettison:jettison:jar:1.1:test
[INFO] | \- de.ruedigermoeller:fst:jar:2.24:test
[INFO] | +- org.javassist:javassist:jar:3.18.1-GA:test
[INFO] | \- org.objenesis:objenesis:jar:2.1:test
[INFO] +- com.google.guava:guava:jar:11.0.2:test
[INFO] +- com.sun.jersey:jersey-core:jar:1.9:test
[INFO] +- com.sun.jersey:jersey-client:jar:1.9:test
[INFO] +- org.apache.hbase:hbase-common:jar:1.0.1:test
[INFO] | +- org.apache.hbase:hbase-annotations:jar:1.0.1:test
[INFO] | +- org.apache.htrace:htrace-core:jar:3.1.0-incubating:test
[INFO] | +-
org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.0.0-SNAPSHOT:test (version
managed from 2.5.1)
[INFO] | \- com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:test
[INFO] +- org.apache.hbase:hbase-client:jar:1.0.1:test
[INFO] | +- org.apache.hbase:hbase-protocol:jar:1.0.1:test
[INFO] | +- io.netty:netty-all:jar:4.1.0.Beta5:test (version managed from
4.0.23.Final)
[INFO] | +- org.jruby.jcodings:jcodings:jar:1.0.8:test
[INFO] | \- org.jruby.joni:joni:jar:2.1.2:test
[INFO] +- org.apache.hbase:hbase-server:jar:1.0.1:test
[INFO] | +- org.apache.hbase:hbase-prefix-tree:jar:1.0.1:test
[INFO] | +- org.apache.hbase:hbase-hadoop-compat:jar:1.0.1:test
[INFO] | +- org.apache.hbase:hbase-hadoop2-compat:jar:1.0.1:test
[INFO] | +- com.yammer.metrics:metrics-core:jar:2.2.0:test
[INFO] | +- org.apache.commons:commons-math:jar:2.2:test
[INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:test
[INFO] | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:test
[INFO] | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:test
[INFO] | +- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:test
[INFO] | +- org.jamon:jamon-runtime:jar:2.3.1:test
[INFO] | +- com.lmax:disruptor:jar:3.3.0:test
[INFO] | \- org.apache.hadoop:hadoop-client:jar:3.0.0-SNAPSHOT:test (version
managed from 2.5.1)
[INFO] | \-
org.apache.hadoop:hadoop-mapreduce-client-app:jar:3.0.0-SNAPSHOT:test
[INFO] +- org.apache.hbase:hbase-common:jar:tests:1.0.1:test
[INFO] +- org.apache.hbase:hbase-server:jar:tests:1.0.1:test
[INFO] +- org.apache.hbase:hbase-it:jar:tests:1.0.1:test
[INFO] | +- org.apache.hbase:hbase-shell:jar:1.0.1:test
[INFO] | | \- org.jruby:jruby-complete:jar:1.6.8:test
[INFO] | \- org.jboss.netty:netty:jar:3.2.4.Final:test
[INFO] +- org.apache.phoenix:phoenix-core:jar:4.5.0-SNAPSHOT:test
[INFO] | +- org.antlr:antlr:jar:3.5:test
[INFO] | | \- org.antlr:ST4:jar:4.0.7:test
[INFO] | +- org.antlr:antlr-runtime:jar:3.5:test
[INFO] | | \- org.antlr:stringtemplate:jar:3.2.1:test
[INFO] | | \- antlr:antlr:jar:2.7.7:test
[INFO] | +- sqlline:sqlline:jar:1.1.8:test
[INFO] | +- net.sourceforge.findbugs:annotations:jar:1.3.2:test
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.2.5:test
[INFO] | +- org.iq80.snappy:snappy:jar:0.3:test
[INFO] | \- org.apache.hbase:hbase-server:test-jar:tests:1.0.1:test
[INFO] +- org.apache.phoenix:phoenix-core:test-jar:tests:4.5.0-SNAPSHOT:test
[INFO] | +- jline:jline:jar:2.11:test
[INFO] | \- joda-time:joda-time:jar:2.7:test
[INFO] +- org.mockito:mockito-all:jar:1.8.5:test
[INFO] +- org.apache.hadoop:hadoop-common:test-jar:tests:2.5.1:test
[INFO] +- org.apache.hadoop:hadoop-hdfs:jar:2.5.1:test
[INFO] | +- commons-daemon:commons-daemon:jar:1.0.13:test
[INFO] | \- io.netty:netty:jar:3.6.2.Final:test
[INFO] +- org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.5.1:test
[INFO] \- org.apache.hbase:hbase-testing-util:jar:1.0.1:test
[INFO] +- org.apache.hbase:hbase-common:test-jar:tests:1.0.1:test
[INFO] +- org.apache.hbase:hbase-annotations:test-jar:tests:1.0.1:test
[INFO] +- org.apache.hbase:hbase-hadoop-compat:test-jar:tests:1.0.1:test
[INFO] +- org.apache.hbase:hbase-hadoop2-compat:test-jar:tests:1.0.1:test
[INFO] \-
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:3.0.0-SNAPSHOT:test
[INFO] +-
org.apache.hadoop:hadoop-mapreduce-client-common:jar:3.0.0-SNAPSHOT:test
[INFO] | \- org.apache.hadoop:hadoop-yarn-client:jar:3.0.0-SNAPSHOT:test
[INFO] \-
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:3.0.0-SNAPSHOT:test
[INFO] \-
org.apache.hadoop:hadoop-yarn-server-nodemanager:jar:3.0.0-SNAPSHOT:test
[INFO] \- com.codahale.metrics:metrics-core:jar:3.0.1:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.132s
[INFO] Finished at: Fri May 06 15:55:11 IST 2016
[INFO] Final Memory: 33M/689M
[INFO] ------------------------------------------------------------------------
{noformat}
> hbase unit tests fail due to dependency issues
> ----------------------------------------------
>
> Key: YARN-5045
> URL: https://issues.apache.org/jira/browse/YARN-5045
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Affects Versions: YARN-2928
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Priority: Blocker
> Attachments: YARN-5045-YARN-2928.01.patch,
> YARN-5045-YARN-2928.poc.patch
>
>
> After the 5/4 rebase, the hbase unit tests in the timeline service project
> are failing:
> {noformat}
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
> Time elapsed: 5.103 sec <<< ERROR!
> java.io.IOException: Shutting down
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
> at
> org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> at
> org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
> at
> org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:217)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.<init>(LocalHBaseCluster.java:153)
> at
> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:213)
> at
> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:978)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:938)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:812)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:750)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:87)
> {noformat}
> The root cause is that the hbase mini server depends on hadoop common's
> {{MetricsServlet}} which has been removed in the trunk (HADOOP-12504):
> {noformat}
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/metrics/MetricsServlet
> at
> org.apache.hadoop.hbase.http.HttpServer.addDefaultServlets(HttpServer.java:677)
> at
> org.apache.hadoop.hbase.http.HttpServer.initializeWebServer(HttpServer.java:546)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:500)
> at org.apache.hadoop.hbase.http.HttpServer.<init>(HttpServer.java:104)
> at
> org.apache.hadoop.hbase.http.HttpServer$Builder.build(HttpServer.java:345)
> at org.apache.hadoop.hbase.http.InfoServer.<init>(InfoServer.java:77)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.putUpWebUI(HRegionServer.java:1697)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:550)
> at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:333)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:139)
> ... 26 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
