Do any map-reduce jobs work on this cluster ? On Fri, Sep 26, 2014 at 5:13 PM, iain wright <[email protected]> wrote:
> Forgot to include the actual copy table command: > [hbase@master2 bin]$ ./hbase org.apache.hadoop.hbase.mapreduce.CopyTable > --starttime=1409128964 --peer.adr=master0.hbasex1.test.cloud.domain.tv, > master1.hbasex1.test.cloud.v.tv,master2.hbasex1.test.cloud.domain.tv:2181 > :/hbase > moderation > > Replication is working well btw for new data. > > Thanks, > > > -- > Iain Wright > > This email message is confidential, intended only for the recipient(s) > named above and may contain information that is privileged, exempt from > disclosure under applicable law. If you are not the intended recipient, do > not disclose or disseminate the message to anyone except the intended > recipient. If you have received this message in error, or are not the named > recipient(s), please immediately notify the sender by return email, and > delete all copies of this message. > > On Fri, Sep 26, 2014 at 5:11 PM, iain wright <[email protected]> wrote: > > > Hi folks, > > > > I'm having trouble using copyTable to seed an existing tables data to a > > replication peer. Surely its an oversight in configuration on our part, > but > > I've scoured the web and doc's for a couple days now. > > > > We have been able to run these jobs with success (perhaps they don't > > require localization of any jars since the jars are already in $PATH?): > > ./hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -fileSize 5GB > > ./hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 5 > > > > When running copyTable it seems like the resources are placed on the yarn > > application manager but not localized to the nodemanagers. > > > > I've tried several settings for yarn.app.mapreduce.am.staging-dir and in > > the below iteration im trying the defaults, the path exists with correct > > permissions on each NM/AM > > > > hadoop.tmp.dir is specified in core-site.xml > > > > Here is the full command we are running: > > > > SLF4J: Class path contains multiple SLF4J bindings. > > SLF4J: Found binding in > > > [jar:file:/usr/local/hbase-0.98.6.1-1a/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: Found binding in > > > [jar:file:/usr/local/hadoop-2.5.0-1a/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > explanation. > > 2014-09-26 23:51:24,141 WARN [main] util.NativeCodeLoader: Unable to > load > > native-hadoop library for your platform... using builtin-java classes > where > > applicable > > 2014-09-26 23:51:24,863 WARN [main] shortcircuit.DomainSocketFactory: > The > > short-circuit local reads feature cannot be used because libhadoop cannot > > be loaded. > > 2014-09-26 23:51:24,910 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.HConstants, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,911 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.client.Put, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.CompatibilityFactory, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,913 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.zookeeper.ZooKeeper, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar > > 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.jboss.netty.channel.ChannelFactory, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar > > 2014-09-26 23:51:24,915 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class com.google.protobuf.Message, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar > > 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class com.google.common.collect.Lists, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar > > 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.cloudera.htrace.Trace, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar > > 2014-09-26 23:51:24,917 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.cliffc.high_scale_lib.Counter, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar > > 2014-09-26 23:51:24,921 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.io.LongWritable, using jar > > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > > 2014-09-26 23:51:24,922 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.io.Text, using jar > > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > > 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.io.LongWritable, using jar > > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > > 2014-09-26 23:51:24,924 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.io.Text, using jar > > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > > 2014-09-26 23:51:24,925 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar > > > /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar > > 2014-09-26 23:51:24,926 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using > jar > > > /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar > > 2014-09-26 23:51:24,952 INFO [main] Configuration.deprecation: > > io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum > > 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.HConstants, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.client.Put, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.CompatibilityFactory, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,965 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.zookeeper.ZooKeeper, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar > > 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.jboss.netty.channel.ChannelFactory, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar > > 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class com.google.protobuf.Message, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar > > 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class com.google.common.collect.Lists, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar > > 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.cloudera.htrace.Trace, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar > > 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.cliffc.high_scale_lib.Counter, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar > > 2014-09-26 23:51:24,971 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.io.Writable, using jar > > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > > 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.io.Writable, using jar > > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > > 2014-09-26 23:51:24,974 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.hbase.mapreduce.TableOutputFormat, using jar > > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > > 2014-09-26 23:51:24,975 DEBUG [main] mapreduce.TableMapReduceUtil: For > > class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using > jar > > > /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar > > 2014-09-26 23:51:25,035 INFO [main] client.RMProxy: Connecting to > > ResourceManager at > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:8032 > > 2014-09-26 23:51:25,191 INFO [main] Configuration.deprecation: > > io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum > > 2014-09-26 23:51:25,247 INFO [main] zookeeper.RecoverableZooKeeper: > > Process identifier=hconnection-0x6bf3ede0 connecting to ZooKeeper > ensemble= > > master1.hbasex1.test.cloud.domain.tv:2181, > > master0.hbasex1.test.cloud.domain.tv:2181, > > master2.hbasex1.test.cloud.domain.tv:2181 > > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 > GMT > > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > > environment:host.name=master2.hbase1a.test.cloud.domain.tv > > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.version=1.7.0_65 > > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.vendor=Oracle Corporation > > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre > > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > > > environment:java.class.path=/usr/local/hbase/bin/../conf:/usr/lib/jvm/java/lib/tools.jar:/usr/local/hbase/bin/..:/usr/local/hbase/bin/../lib/activation-1.1.jar:/usr/local/hbase/bin/../lib/aopalliance-1.0.jar:/usr/local/hbase/bin/../lib/asm-3.1.jar:/usr/local/hbase/bin/../lib/avro-1.7.4.jar:/usr/local/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/bin/../lib/commons-cli-1.2.jar:/usr/local/hbase/bin/../lib/commons-codec-1.7.jar:/usr/local/hbase/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase/bin/../lib/commons-compress-1.4.1.jar:/usr/local/hbase/bin/../lib/commons-configuration-1.6.jar:/usr/local/hbase/bin/../lib/commons-daemon-1.0.13.jar:/usr/local/hbase/bin/../lib/commons-digester-1.8.jar:/usr/local/hbase/bin/../lib/commons-el-1.0.jar:/usr/local/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/local/hbase/bin/../lib/commons-io-2.4.jar:/usr/local/hbase/bin/../lib/commons-lang-2.6.jar:/usr/local/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hbase/bin/../lib/commons-math-2.1.jar:/usr/local/hbase/bin/../lib/commons-net-3.1.jar:/usr/local/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hbase/bin/../lib/gmbal-api-only-3.0.0-b023.jar:/usr/local/hbase/bin/../lib/grizzly-framework-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-server-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-servlet-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-rcm-2.1.2.jar:/usr/local/hbase/bin/../lib/guava-12.0.1.jar:/usr/local/hbase/bin/../lib/guice-3.0.jar:/usr/local/hbase/bin/../lib/guice-servlet-3.0.jar:/usr/local/hbase/bin/../lib/hamcrest-core-1.3.jar:/usr/local/hbase/bin/../lib/hbase-client-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-examples-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop2-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-prefix-tree-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-protocol-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-shell-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-testing-util-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-thrift-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/bin/../lib/htrace-core-2.04.jar:/usr/local/hbase/bin/../lib/httpclient-4.1.3.jar:/usr/local/hbase/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase/bin/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-xc-1.8.8.jar:/usr/local/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/bin/../lib/javax.inject-1.jar:/usr/local/hbase/bin/../lib/javax.servlet-3.1.jar:/usr/local/hbase/bin/../lib/javax.servlet-api-3.0.1.jar:/usr/local/hbase/bin/../lib/jaxb-api-2.2.2.jar:/usr/local/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/bin/../lib/jersey-client-1.9.jar:/usr/local/hbase/bin/../lib/jersey-core-1.8.jar:/usr/local/hbase/bin/../lib/jersey-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jersey-guice-1.9.jar:/usr/local/hbase/bin/../lib/jersey-json-1.8.jar:/usr/local/hbase/bin/../lib/jersey-server-1.8.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-core-1.9.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jets3t-0.6.1.jar:/usr/local/hbase/bin/../lib/jettison-1.3.1.jar:/usr/local/hbase/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase/bin/../lib/jruby-complete-1.6.8.jar:/usr/local/hbase/bin/../lib/jsch-0.1.42.jar:/usr/local/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsr305-1.3.9.jar:/usr/local/hbase/bin/../lib/junit-4.11.jar:/usr/local/hbase/bin/../lib/libthrift-0.9.0.jar:/usr/local/hbase/bin/../lib/log4j-1.2.17.jar:/usr/local/hbase/bin/../lib/management-api-3.0.0-b012.jar:/usr/local/hbase/bin/../lib/metrics-core-2.2.0.jar:/usr/local/hbase/bin/../lib/netty-3.6.6.Final.jar:/usr/local/hbase/bin/../lib/paranamer-2.3.jar:/usr/local/hbase/bin/../lib/protobuf-java-2.5.0.jar:/usr/local/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase/bin/../lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hbase/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase/bin/../lib/xz-1.0.jar:/usr/local/hbase/bin/../lib/zookeeper-3.4.6.jar:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/etc/hadoop/ > > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.library.path=/usr/local/hadoop/lib > > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.io.tmpdir=/tmp > > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > > environment:java.compiler=<NA> > > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > > environment:os.name=Linux > > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > > environment:os.arch=amd64 > > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > > environment:os.version=3.10.42-52.145.amzn1.x86_64 > > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > > environment:user.name=hbase > > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > > environment:user.home=/home/hbase > > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > > environment:user.dir=/usr/local/hbase-0.98.6.1-1a/bin > > 2014-09-26 23:51:25,254 INFO [main] zookeeper.ZooKeeper: Initiating > > client connection, connectString= > master1.hbasex1.test.cloud.domain.tv:2181 > > ,master0.hbasex1.test.cloud.domain.tv:2181, > > master2.hbasex1.test.cloud.domain.tv:2181 sessionTimeout=90000 > > watcher=hconnection-0x6bf3ede0, quorum= > > master1.hbasex1.test.cloud.domain.tv:2181, > > master0.hbasex1.test.cloud.domain.tv:2181, > > master2.hbasex1.test.cloud.domain.tv:2181, baseZNode=/hbase > > 2014-09-26 23:51:25,270 INFO [main-SendThread( > > master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: > Opening > > socket connection to server > > master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181. Will not attempt > > to authenticate using SASL (unknown error) > > 2014-09-26 23:51:25,276 INFO [main-SendThread( > > master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket > > connection established to > > master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, initiating > session > > 2014-09-26 23:51:25,282 INFO [main-SendThread( > > master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: > Session > > establishment complete on server > > master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, sessionid = > > 0x248afb6109b007f, negotiated timeout = 40000 > > 2014-09-26 23:51:25,308 INFO [main] mapreduce.TableOutputFormat: Created > > table instance for moderation > > 2014-09-26 23:51:25,469 INFO [main] zookeeper.RecoverableZooKeeper: > > Process identifier=hconnection-0xa7a371 connecting to ZooKeeper ensemble= > > master1.hbase1a.test.cloud.domain.tv:2181, > > master0.hbase1a.test.cloud.domain.tv:2181, > > master2.hbase1a.test.cloud.domain.tv:2181 > > 2014-09-26 23:51:25,469 INFO [main] zookeeper.ZooKeeper: Initiating > > client connection, connectString= > master1.hbase1a.test.cloud.domain.tv:2181 > > ,master0.hbase1a.test.cloud.domain.tv:2181, > > master2.hbase1a.test.cloud.domain.tv:2181 sessionTimeout=90000 > > watcher=hconnection-0xa7a371, quorum= > > master1.hbase1a.test.cloud.domain.tv:2181, > > master0.hbase1a.test.cloud.domain.tv:2181, > > master2.hbase1a.test.cloud.domain.tv:2181, baseZNode=/hbase > > 2014-09-26 23:51:25,470 INFO [main-SendThread( > > master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: > Opening > > socket connection to server > > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181. Will not attempt > > to authenticate using SASL (unknown error) > > 2014-09-26 23:51:25,470 INFO [main-SendThread( > > master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket > > connection established to > > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, initiating > session > > 2014-09-26 23:51:25,472 INFO [main-SendThread( > > master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: > Session > > establishment complete on server > > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, sessionid = > > 0x34877671747028e, negotiated timeout = 40000 > > 2014-09-26 23:51:25,490 INFO [main] util.RegionSizeCalculator: > > Calculating region sizes for table "moderation". > > 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region > > moderation,,1411774704618.68a1839eacfa355028522286caaf7cd5. has size 0 > > 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region > > sizes calculated > > 2014-09-26 23:51:25,709 DEBUG [main] client.ClientSmallScanner: Finished > > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', > > STARTKEY => '', ENDKEY => ''} > > 2014-09-26 23:51:25,722 DEBUG [main] mapreduce.TableInputFormatBase: > > getSplits: split -> 0 -> ip-172-16-0-247.us-west-1.compute.internal:, > > 2014-09-26 23:51:25,739 INFO [main] mapreduce.JobSubmitter: number of > > splits:1 > > 2014-09-26 23:51:25,748 INFO [main] Configuration.deprecation: > > io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum > > 2014-09-26 23:51:25,895 INFO [main] mapreduce.JobSubmitter: Submitting > > tokens for job: job_1411771844660_0004 > > 2014-09-26 23:51:26,033 INFO [main] impl.YarnClientImpl: Submitted > > application application_1411771844660_0004 > > 2014-09-26 23:51:26,066 INFO [main] mapreduce.Job: The url to track the > > job: > > > http://master2.hbase1a.test.cloud.domain.tv:8088/proxy/application_1411771844660_0004/ > > 2014-09-26 23:51:26,067 INFO [main] mapreduce.Job: Running job: > > job_1411771844660_0004 > > 2014-09-26 23:51:29,082 INFO [main] mapreduce.Job: Job > > job_1411771844660_0004 running in uber mode : false > > 2014-09-26 23:51:29,084 INFO [main] mapreduce.Job: map 0% reduce 0% > > 2014-09-26 23:51:29,094 INFO [main] mapreduce.Job: Job > > job_1411771844660_0004 failed with state FAILED due to: Application > > application_1411771844660_0004 failed 2 times due to AM Container for > > appattempt_1411771844660_0004_000002 exited with exitCode: -1000 due to: > > File > > > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar > > does not exist > > .Failing this attempt.. Failing the application. > > 2014-09-26 23:51:29,111 INFO [main] mapreduce.Job: Counters: 0 > > > > The directory on the AM (master) from which the command is initiated as > > hbase user: > > [hbase@master2 bin]$ ls -lah > > /tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/ > > total 3.7M > > drwx------ 3 hbase hbase 4.0K Sep 26 23:51 . > > drwx------ 10 hbase hadoop 4.0K Sep 26 23:51 .. > > -rw-r--r-- 1 hbase hbase 3.5M Sep 26 23:51 job.jar > > -rw-rw-r-- 1 hbase hbase 28K Sep 26 23:51 .job.jar.crc > > -rw-r--r-- 1 hbase hbase 111 Sep 26 23:51 job.split > > -rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.split.crc > > -rw-r--r-- 1 hbase hbase 56 Sep 26 23:51 job.splitmetainfo > > -rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.splitmetainfo.crc > > -rw-r--r-- 1 hbase hbase 102K Sep 26 23:51 job.xml > > -rw-rw-r-- 1 hbase hbase 824 Sep 26 23:51 .job.xml.crc > > drwx------ 2 hbase hbase 4.0K Sep 26 23:51 libjars > > > > The same directory is empty on each NM. > > > > Heres also the log from a node that got the job/application: > > 2014-09-26 23:51:26,635 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > > Auth successful for appattempt_1411771844660_0004_000001 (auth:SIMPLE) > > 2014-09-26 23:51:26,639 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > > Start request for container_1411771844660_0004_01_000001 by user hbase > > 2014-09-26 23:51:26,639 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > > Creating a new application reference for app > application_1411771844660_0004 > > 2014-09-26 23:51:26,639 INFO > > org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase > > IP=172.16.0.145 OPERATION=Start Container Request > > TARGET=ContainerManageImpl RESULT=SUCCESS > > APPID=application_1411771844660_0004 > > CONTAINERID=container_1411771844660_0004_01_000001 > > 2014-09-26 23:51:26,639 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > > Application application_1411771844660_0004 transitioned from NEW to > INITING > > 2014-09-26 23:51:26,639 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > > Adding container_1411771844660_0004_01_000001 to application > > application_1411771844660_0004 > > 2014-09-26 23:51:26,639 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > > Application application_1411771844660_0004 transitioned from INITING to > > RUNNING > > 2014-09-26 23:51:26,640 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > > Container container_1411771844660_0004_01_000001 transitioned from NEW to > > LOCALIZING > > 2014-09-26 23:51:26,640 INFO > > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: > Got > > event CONTAINER_INIT for appId application_1411771844660_0004 > > 2014-09-26 23:51:26,640 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > > Resource > > > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar > > transitioned from INIT to DOWNLOADING > > 2014-09-26 23:51:26,640 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > > Resource > > > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.splitmetainfo > > transitioned from INIT to DOWNLOADING > > 2014-09-26 23:51:26,640 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > > Resource > > > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.split > > transitioned from INIT to DOWNLOADING > > 2014-09-26 23:51:26,641 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > > Resource > > > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.xml > > transitioned from INIT to DOWNLOADING > > 2014-09-26 23:51:26,641 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > > Created localizer for container_1411771844660_0004_01_000001 > > 2014-09-26 23:51:26,645 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > > Writing credentials to the nmPrivate file > > > /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens. > > Credentials list: > > 2014-09-26 23:51:26,648 INFO > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > > Initializing user hbase > > 2014-09-26 23:51:26,661 INFO > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > Copying > > from > > > /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens > > to > > > /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001.tokens > > 2014-09-26 23:51:26,661 INFO > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD > set > > to > > > /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 > > = > > > file:/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 > > 2014-09-26 23:51:26,696 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > > DEBUG: FAILED { > > > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar, > > 1411775485000, PATTERN, (?:classes/|lib/).* }, File > > > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar > > does not exist > > 2014-09-26 23:51:26,697 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > > Resource > > > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar(->/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/filecache/10/job.jar) > > transitioned from DOWNLOADING to FAILED > > 2014-09-26 23:51:26,697 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > > Container container_1411771844660_0004_01_000001 transitioned from > > LOCALIZING to LOCALIZATION_FAILED > > 2014-09-26 23:51:26,697 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: > > Container container_1411771844660_0004_01_000001 sent RELEASE event on a > > resource request { > > > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar, > > 1411775485000, PATTERN, (?:classes/|lib/).* } not present in cache. > > 2014-09-26 23:51:26,698 INFO > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > > Deleting absolute path : > > > /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001 > > 2014-09-26 23:51:26,698 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > > Unknown localizer with localizerId container_1411771844660_0004_01_000001 > > is sending heartbeat. Ordering it to DIE > > 2014-09-26 23:51:26,698 WARN > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > delete > > returned false for path: > > > [/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001] > > 2014-09-26 23:51:26,698 INFO > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > > Deleting absolute path : > > > /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001 > > 2014-09-26 23:51:26,699 WARN > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > delete > > returned false for path: > > > [/data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001] > > 2014-09-26 23:51:26,699 WARN > > org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase > > OPERATION=Container Finished - Failed TARGET=ContainerImpl > > RESULT=FAILURE DESCRIPTION=Container failed with state: > > LOCALIZATION_FAILED APPID=application_1411771844660_0004 > > CONTAINERID=container_1411771844660_0004_01_000001 > > 2014-09-26 23:51:26,699 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > > Container container_1411771844660_0004_01_000001 transitioned from > > LOCALIZATION_FAILED to DONE > > 2014-09-26 23:51:26,699 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > > Removing container_1411771844660_0004_01_000001 from application > > application_1411771844660_0004 > > 2014-09-26 23:51:26,699 INFO > > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: > Got > > event CONTAINER_STOP for appId application_1411771844660_0004 > > 2014-09-26 23:51:27,629 INFO > > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed > > completed containers from NM context: > > [container_1411771844660_0004_01_000001] > > 2014-09-26 23:51:28,252 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > > Stopping resource-monitoring for container_1411771844660_0004_01_000001 > > 2014-09-26 23:51:29,631 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > > Application application_1411771844660_0004 transitioned from RUNNING to > > APPLICATION_RESOURCES_CLEANINGUP > > 2014-09-26 23:51:29,632 INFO > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > > Deleting absolute path : > > > /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 > > 2014-09-26 23:51:29,632 INFO > > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > > Deleting absolute path : > > > /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 > > 2014-09-26 23:51:29,632 INFO > > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: > Got > > event APPLICATION_STOP for appId application_1411771844660_0004 > > 2014-09-26 23:51:29,632 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > > Application application_1411771844660_0004 transitioned from > > APPLICATION_RESOURCES_CLEANINGUP to FINISHED > > 2014-09-26 23:51:29,632 INFO > > > org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: > > Scheduling Log Deletion for application: application_1411771844660_0004, > > with delay of 10800 seconds > > > > Thank you in advance for any advice > > > > -- > > Iain Wright > > > > This email message is confidential, intended only for the recipient(s) > > named above and may contain information that is privileged, exempt from > > disclosure under applicable law. If you are not the intended recipient, > do > > not disclose or disseminate the message to anyone except the intended > > recipient. If you have received this message in error, or are not the > named > > recipient(s), please immediately notify the sender by return email, and > > delete all copies of this message. > > >
