Forgot to include the actual copy table command: [hbase@master2 bin]$ ./hbase org.apache.hadoop.hbase.mapreduce.CopyTable --starttime=1409128964 --peer.adr=master0.hbasex1.test.cloud.domain.tv, master1.hbasex1.test.cloud.v.tv,master2.hbasex1.test.cloud.domain.tv:2181:/hbase moderation
Replication is working well btw for new data. Thanks, -- Iain Wright This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message. On Fri, Sep 26, 2014 at 5:11 PM, iain wright <iainw...@gmail.com> wrote: > Hi folks, > > I'm having trouble using copyTable to seed an existing tables data to a > replication peer. Surely its an oversight in configuration on our part, but > I've scoured the web and doc's for a couple days now. > > We have been able to run these jobs with success (perhaps they don't > require localization of any jars since the jars are already in $PATH?): > ./hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -fileSize 5GB > ./hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 5 > > When running copyTable it seems like the resources are placed on the yarn > application manager but not localized to the nodemanagers. > > I've tried several settings for yarn.app.mapreduce.am.staging-dir and in > the below iteration im trying the defaults, the path exists with correct > permissions on each NM/AM > > hadoop.tmp.dir is specified in core-site.xml > > Here is the full command we are running: > > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/local/hbase-0.98.6.1-1a/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/local/hadoop-2.5.0-1a/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > 2014-09-26 23:51:24,141 WARN [main] util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > 2014-09-26 23:51:24,863 WARN [main] shortcircuit.DomainSocketFactory: The > short-circuit local reads feature cannot be used because libhadoop cannot > be loaded. > 2014-09-26 23:51:24,910 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.HConstants, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,911 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.client.Put, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.CompatibilityFactory, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,913 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.zookeeper.ZooKeeper, using jar > /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar > 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.jboss.netty.channel.ChannelFactory, using jar > /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar > 2014-09-26 23:51:24,915 DEBUG [main] mapreduce.TableMapReduceUtil: For > class com.google.protobuf.Message, using jar > /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar > 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For > class com.google.common.collect.Lists, using jar > /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar > 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.cloudera.htrace.Trace, using jar > /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar > 2014-09-26 23:51:24,917 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.cliffc.high_scale_lib.Counter, using jar > /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar > 2014-09-26 23:51:24,921 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.io.LongWritable, using jar > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > 2014-09-26 23:51:24,922 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.io.Text, using jar > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.io.LongWritable, using jar > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > 2014-09-26 23:51:24,924 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.io.Text, using jar > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > 2014-09-26 23:51:24,925 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar > /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar > 2014-09-26 23:51:24,926 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar > /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar > 2014-09-26 23:51:24,952 INFO [main] Configuration.deprecation: > io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum > 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.HConstants, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.client.Put, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.CompatibilityFactory, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,965 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.zookeeper.ZooKeeper, using jar > /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar > 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.jboss.netty.channel.ChannelFactory, using jar > /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar > 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For > class com.google.protobuf.Message, using jar > /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar > 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For > class com.google.common.collect.Lists, using jar > /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar > 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.cloudera.htrace.Trace, using jar > /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar > 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.cliffc.high_scale_lib.Counter, using jar > /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar > 2014-09-26 23:51:24,971 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.io.Writable, using jar > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.io.Writable, using jar > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar > 2014-09-26 23:51:24,974 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.hbase.mapreduce.TableOutputFormat, using jar > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar > 2014-09-26 23:51:24,975 DEBUG [main] mapreduce.TableMapReduceUtil: For > class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar > /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar > 2014-09-26 23:51:25,035 INFO [main] client.RMProxy: Connecting to > ResourceManager at master2.hbase1a.test.cloud.domain.tv/172.16.0.145:8032 > 2014-09-26 23:51:25,191 INFO [main] Configuration.deprecation: > io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum > 2014-09-26 23:51:25,247 INFO [main] zookeeper.RecoverableZooKeeper: > Process identifier=hconnection-0x6bf3ede0 connecting to ZooKeeper ensemble= > master1.hbasex1.test.cloud.domain.tv:2181, > master0.hbasex1.test.cloud.domain.tv:2181, > master2.hbasex1.test.cloud.domain.tv:2181 > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > environment:host.name=master2.hbase1a.test.cloud.domain.tv > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > environment:java.version=1.7.0_65 > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > environment:java.vendor=Oracle Corporation > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre > 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client > environment:java.class.path=/usr/local/hbase/bin/../conf:/usr/lib/jvm/java/lib/tools.jar:/usr/local/hbase/bin/..:/usr/local/hbase/bin/../lib/activation-1.1.jar:/usr/local/hbase/bin/../lib/aopalliance-1.0.jar:/usr/local/hbase/bin/../lib/asm-3.1.jar:/usr/local/hbase/bin/../lib/avro-1.7.4.jar:/usr/local/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/bin/../lib/commons-cli-1.2.jar:/usr/local/hbase/bin/../lib/commons-codec-1.7.jar:/usr/local/hbase/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase/bin/../lib/commons-compress-1.4.1.jar:/usr/local/hbase/bin/../lib/commons-configuration-1.6.jar:/usr/local/hbase/bin/../lib/commons-daemon-1.0.13.jar:/usr/local/hbase/bin/../lib/commons-digester-1.8.jar:/usr/local/hbase/bin/../lib/commons-el-1.0.jar:/usr/local/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/local/hbase/bin/../lib/commons-io-2.4.jar:/usr/local/hbase/bin/../lib/commons-lang-2.6.jar:/usr/local/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hbase/bin/../lib/commons-math-2.1.jar:/usr/local/hbase/bin/../lib/commons-net-3.1.jar:/usr/local/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hbase/bin/../lib/gmbal-api-only-3.0.0-b023.jar:/usr/local/hbase/bin/../lib/grizzly-framework-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-server-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-servlet-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-rcm-2.1.2.jar:/usr/local/hbase/bin/../lib/guava-12.0.1.jar:/usr/local/hbase/bin/../lib/guice-3.0.jar:/usr/local/hbase/bin/../lib/guice-servlet-3.0.jar:/usr/local/hbase/bin/../lib/hamcrest-core-1.3.jar:/usr/local/hbase/bin/../lib/hbase-client-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-examples-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop2-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-prefix-tree-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-protocol-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-shell-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-testing-util-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-thrift-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/bin/../lib/htrace-core-2.04.jar:/usr/local/hbase/bin/../lib/httpclient-4.1.3.jar:/usr/local/hbase/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase/bin/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-xc-1.8.8.jar:/usr/local/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/bin/../lib/javax.inject-1.jar:/usr/local/hbase/bin/../lib/javax.servlet-3.1.jar:/usr/local/hbase/bin/../lib/javax.servlet-api-3.0.1.jar:/usr/local/hbase/bin/../lib/jaxb-api-2.2.2.jar:/usr/local/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/bin/../lib/jersey-client-1.9.jar:/usr/local/hbase/bin/../lib/jersey-core-1.8.jar:/usr/local/hbase/bin/../lib/jersey-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jersey-guice-1.9.jar:/usr/local/hbase/bin/../lib/jersey-json-1.8.jar:/usr/local/hbase/bin/../lib/jersey-server-1.8.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-core-1.9.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jets3t-0.6.1.jar:/usr/local/hbase/bin/../lib/jettison-1.3.1.jar:/usr/local/hbase/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase/bin/../lib/jruby-complete-1.6.8.jar:/usr/local/hbase/bin/../lib/jsch-0.1.42.jar:/usr/local/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsr305-1.3.9.jar:/usr/local/hbase/bin/../lib/junit-4.11.jar:/usr/local/hbase/bin/../lib/libthrift-0.9.0.jar:/usr/local/hbase/bin/../lib/log4j-1.2.17.jar:/usr/local/hbase/bin/../lib/management-api-3.0.0-b012.jar:/usr/local/hbase/bin/../lib/metrics-core-2.2.0.jar:/usr/local/hbase/bin/../lib/netty-3.6.6.Final.jar:/usr/local/hbase/bin/../lib/paranamer-2.3.jar:/usr/local/hbase/bin/../lib/protobuf-java-2.5.0.jar:/usr/local/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase/bin/../lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hbase/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase/bin/../lib/xz-1.0.jar:/usr/local/hbase/bin/../lib/zookeeper-3.4.6.jar:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/etc/hadoop/ > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > environment:java.library.path=/usr/local/hadoop/lib > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > environment:java.io.tmpdir=/tmp > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > environment:java.compiler=<NA> > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > environment:os.name=Linux > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > environment:os.arch=amd64 > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > environment:os.version=3.10.42-52.145.amzn1.x86_64 > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > environment:user.name=hbase > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > environment:user.home=/home/hbase > 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client > environment:user.dir=/usr/local/hbase-0.98.6.1-1a/bin > 2014-09-26 23:51:25,254 INFO [main] zookeeper.ZooKeeper: Initiating > client connection, connectString=master1.hbasex1.test.cloud.domain.tv:2181 > ,master0.hbasex1.test.cloud.domain.tv:2181, > master2.hbasex1.test.cloud.domain.tv:2181 sessionTimeout=90000 > watcher=hconnection-0x6bf3ede0, quorum= > master1.hbasex1.test.cloud.domain.tv:2181, > master0.hbasex1.test.cloud.domain.tv:2181, > master2.hbasex1.test.cloud.domain.tv:2181, baseZNode=/hbase > 2014-09-26 23:51:25,270 INFO [main-SendThread( > master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Opening > socket connection to server > master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181. Will not attempt > to authenticate using SASL (unknown error) > 2014-09-26 23:51:25,276 INFO [main-SendThread( > master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket > connection established to > master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, initiating session > 2014-09-26 23:51:25,282 INFO [main-SendThread( > master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Session > establishment complete on server > master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, sessionid = > 0x248afb6109b007f, negotiated timeout = 40000 > 2014-09-26 23:51:25,308 INFO [main] mapreduce.TableOutputFormat: Created > table instance for moderation > 2014-09-26 23:51:25,469 INFO [main] zookeeper.RecoverableZooKeeper: > Process identifier=hconnection-0xa7a371 connecting to ZooKeeper ensemble= > master1.hbase1a.test.cloud.domain.tv:2181, > master0.hbase1a.test.cloud.domain.tv:2181, > master2.hbase1a.test.cloud.domain.tv:2181 > 2014-09-26 23:51:25,469 INFO [main] zookeeper.ZooKeeper: Initiating > client connection, connectString=master1.hbase1a.test.cloud.domain.tv:2181 > ,master0.hbase1a.test.cloud.domain.tv:2181, > master2.hbase1a.test.cloud.domain.tv:2181 sessionTimeout=90000 > watcher=hconnection-0xa7a371, quorum= > master1.hbase1a.test.cloud.domain.tv:2181, > master0.hbase1a.test.cloud.domain.tv:2181, > master2.hbase1a.test.cloud.domain.tv:2181, baseZNode=/hbase > 2014-09-26 23:51:25,470 INFO [main-SendThread( > master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Opening > socket connection to server > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181. Will not attempt > to authenticate using SASL (unknown error) > 2014-09-26 23:51:25,470 INFO [main-SendThread( > master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket > connection established to > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, initiating session > 2014-09-26 23:51:25,472 INFO [main-SendThread( > master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Session > establishment complete on server > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, sessionid = > 0x34877671747028e, negotiated timeout = 40000 > 2014-09-26 23:51:25,490 INFO [main] util.RegionSizeCalculator: > Calculating region sizes for table "moderation". > 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region > moderation,,1411774704618.68a1839eacfa355028522286caaf7cd5. has size 0 > 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region > sizes calculated > 2014-09-26 23:51:25,709 DEBUG [main] client.ClientSmallScanner: Finished > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', > STARTKEY => '', ENDKEY => ''} > 2014-09-26 23:51:25,722 DEBUG [main] mapreduce.TableInputFormatBase: > getSplits: split -> 0 -> ip-172-16-0-247.us-west-1.compute.internal:, > 2014-09-26 23:51:25,739 INFO [main] mapreduce.JobSubmitter: number of > splits:1 > 2014-09-26 23:51:25,748 INFO [main] Configuration.deprecation: > io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum > 2014-09-26 23:51:25,895 INFO [main] mapreduce.JobSubmitter: Submitting > tokens for job: job_1411771844660_0004 > 2014-09-26 23:51:26,033 INFO [main] impl.YarnClientImpl: Submitted > application application_1411771844660_0004 > 2014-09-26 23:51:26,066 INFO [main] mapreduce.Job: The url to track the > job: > http://master2.hbase1a.test.cloud.domain.tv:8088/proxy/application_1411771844660_0004/ > 2014-09-26 23:51:26,067 INFO [main] mapreduce.Job: Running job: > job_1411771844660_0004 > 2014-09-26 23:51:29,082 INFO [main] mapreduce.Job: Job > job_1411771844660_0004 running in uber mode : false > 2014-09-26 23:51:29,084 INFO [main] mapreduce.Job: map 0% reduce 0% > 2014-09-26 23:51:29,094 INFO [main] mapreduce.Job: Job > job_1411771844660_0004 failed with state FAILED due to: Application > application_1411771844660_0004 failed 2 times due to AM Container for > appattempt_1411771844660_0004_000002 exited with exitCode: -1000 due to: > File > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar > does not exist > .Failing this attempt.. Failing the application. > 2014-09-26 23:51:29,111 INFO [main] mapreduce.Job: Counters: 0 > > The directory on the AM (master) from which the command is initiated as > hbase user: > [hbase@master2 bin]$ ls -lah > /tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/ > total 3.7M > drwx------ 3 hbase hbase 4.0K Sep 26 23:51 . > drwx------ 10 hbase hadoop 4.0K Sep 26 23:51 .. > -rw-r--r-- 1 hbase hbase 3.5M Sep 26 23:51 job.jar > -rw-rw-r-- 1 hbase hbase 28K Sep 26 23:51 .job.jar.crc > -rw-r--r-- 1 hbase hbase 111 Sep 26 23:51 job.split > -rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.split.crc > -rw-r--r-- 1 hbase hbase 56 Sep 26 23:51 job.splitmetainfo > -rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.splitmetainfo.crc > -rw-r--r-- 1 hbase hbase 102K Sep 26 23:51 job.xml > -rw-rw-r-- 1 hbase hbase 824 Sep 26 23:51 .job.xml.crc > drwx------ 2 hbase hbase 4.0K Sep 26 23:51 libjars > > The same directory is empty on each NM. > > Heres also the log from a node that got the job/application: > 2014-09-26 23:51:26,635 INFO SecurityLogger.org.apache.hadoop.ipc.Server: > Auth successful for appattempt_1411771844660_0004_000001 (auth:SIMPLE) > 2014-09-26 23:51:26,639 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Start request for container_1411771844660_0004_01_000001 by user hbase > 2014-09-26 23:51:26,639 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Creating a new application reference for app application_1411771844660_0004 > 2014-09-26 23:51:26,639 INFO > org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase > IP=172.16.0.145 OPERATION=Start Container Request > TARGET=ContainerManageImpl RESULT=SUCCESS > APPID=application_1411771844660_0004 > CONTAINERID=container_1411771844660_0004_01_000001 > 2014-09-26 23:51:26,639 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Application application_1411771844660_0004 transitioned from NEW to INITING > 2014-09-26 23:51:26,639 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Adding container_1411771844660_0004_01_000001 to application > application_1411771844660_0004 > 2014-09-26 23:51:26,639 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Application application_1411771844660_0004 transitioned from INITING to > RUNNING > 2014-09-26 23:51:26,640 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > Container container_1411771844660_0004_01_000001 transitioned from NEW to > LOCALIZING > 2014-09-26 23:51:26,640 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got > event CONTAINER_INIT for appId application_1411771844660_0004 > 2014-09-26 23:51:26,640 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > Resource > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar > transitioned from INIT to DOWNLOADING > 2014-09-26 23:51:26,640 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > Resource > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.splitmetainfo > transitioned from INIT to DOWNLOADING > 2014-09-26 23:51:26,640 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > Resource > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.split > transitioned from INIT to DOWNLOADING > 2014-09-26 23:51:26,641 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > Resource > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.xml > transitioned from INIT to DOWNLOADING > 2014-09-26 23:51:26,641 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > Created localizer for container_1411771844660_0004_01_000001 > 2014-09-26 23:51:26,645 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > Writing credentials to the nmPrivate file > /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens. > Credentials list: > 2014-09-26 23:51:26,648 INFO > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > Initializing user hbase > 2014-09-26 23:51:26,661 INFO > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying > from > /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens > to > /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001.tokens > 2014-09-26 23:51:26,661 INFO > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set > to > /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 > = > file:/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 > 2014-09-26 23:51:26,696 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > DEBUG: FAILED { > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar, > 1411775485000, PATTERN, (?:classes/|lib/).* }, File > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar > does not exist > 2014-09-26 23:51:26,697 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: > Resource > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar(->/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/filecache/10/job.jar) > transitioned from DOWNLOADING to FAILED > 2014-09-26 23:51:26,697 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > Container container_1411771844660_0004_01_000001 transitioned from > LOCALIZING to LOCALIZATION_FAILED > 2014-09-26 23:51:26,697 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: > Container container_1411771844660_0004_01_000001 sent RELEASE event on a > resource request { > file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar, > 1411775485000, PATTERN, (?:classes/|lib/).* } not present in cache. > 2014-09-26 23:51:26,698 INFO > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > Deleting absolute path : > /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001 > 2014-09-26 23:51:26,698 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: > Unknown localizer with localizerId container_1411771844660_0004_01_000001 > is sending heartbeat. Ordering it to DIE > 2014-09-26 23:51:26,698 WARN > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: delete > returned false for path: > [/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001] > 2014-09-26 23:51:26,698 INFO > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > Deleting absolute path : > /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001 > 2014-09-26 23:51:26,699 WARN > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: delete > returned false for path: > [/data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001] > 2014-09-26 23:51:26,699 WARN > org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase > OPERATION=Container Finished - Failed TARGET=ContainerImpl > RESULT=FAILURE DESCRIPTION=Container failed with state: > LOCALIZATION_FAILED APPID=application_1411771844660_0004 > CONTAINERID=container_1411771844660_0004_01_000001 > 2014-09-26 23:51:26,699 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: > Container container_1411771844660_0004_01_000001 transitioned from > LOCALIZATION_FAILED to DONE > 2014-09-26 23:51:26,699 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Removing container_1411771844660_0004_01_000001 from application > application_1411771844660_0004 > 2014-09-26 23:51:26,699 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got > event CONTAINER_STOP for appId application_1411771844660_0004 > 2014-09-26 23:51:27,629 INFO > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed > completed containers from NM context: > [container_1411771844660_0004_01_000001] > 2014-09-26 23:51:28,252 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Stopping resource-monitoring for container_1411771844660_0004_01_000001 > 2014-09-26 23:51:29,631 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Application application_1411771844660_0004 transitioned from RUNNING to > APPLICATION_RESOURCES_CLEANINGUP > 2014-09-26 23:51:29,632 INFO > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > Deleting absolute path : > /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 > 2014-09-26 23:51:29,632 INFO > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > Deleting absolute path : > /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 > 2014-09-26 23:51:29,632 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got > event APPLICATION_STOP for appId application_1411771844660_0004 > 2014-09-26 23:51:29,632 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: > Application application_1411771844660_0004 transitioned from > APPLICATION_RESOURCES_CLEANINGUP to FINISHED > 2014-09-26 23:51:29,632 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: > Scheduling Log Deletion for application: application_1411771844660_0004, > with delay of 10800 seconds > > Thank you in advance for any advice > > -- > Iain Wright > > This email message is confidential, intended only for the recipient(s) > named above and may contain information that is privileged, exempt from > disclosure under applicable law. If you are not the intended recipient, do > not disclose or disseminate the message to anyone except the intended > recipient. If you have received this message in error, or are not the named > recipient(s), please immediately notify the sender by return email, and > delete all copies of this message. >
