Hi folks, I'm having trouble using copyTable to seed an existing tables data to a replication peer. Surely its an oversight in configuration on our part, but I've scoured the web and doc's for a couple days now.
We have been able to run these jobs with success (perhaps they don't require localization of any jars since the jars are already in $PATH?): ./hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -fileSize 5GB ./hbase org.apache.hadoop.hbase.PerformanceEvaluation randomWrite 5 When running copyTable it seems like the resources are placed on the yarn application manager but not localized to the nodemanagers. I've tried several settings for yarn.app.mapreduce.am.staging-dir and in the below iteration im trying the defaults, the path exists with correct permissions on each NM/AM hadoop.tmp.dir is specified in core-site.xml Here is the full command we are running: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hbase-0.98.6.1-1a/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop-2.5.0-1a/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2014-09-26 23:51:24,141 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2014-09-26 23:51:24,863 WARN [main] shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2014-09-26 23:51:24,910 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.HConstants, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,911 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.client.Put, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,913 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.zookeeper.ZooKeeper, using jar /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.jboss.netty.channel.ChannelFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar 2014-09-26 23:51:24,915 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.protobuf.Message, using jar /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.common.collect.Lists, using jar /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cloudera.htrace.Trace, using jar /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar 2014-09-26 23:51:24,917 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cliffc.high_scale_lib.Counter, using jar /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar 2014-09-26 23:51:24,921 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.LongWritable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,922 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Text, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.LongWritable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,924 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Text, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,925 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar 2014-09-26 23:51:24,926 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar 2014-09-26 23:51:24,952 INFO [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.HConstants, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.client.Put, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.CompatibilityFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,965 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.zookeeper.ZooKeeper, using jar /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.jboss.netty.channel.ChannelFactory, using jar /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.protobuf.Message, using jar /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For class com.google.common.collect.Lists, using jar /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cloudera.htrace.Trace, using jar /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.cliffc.high_scale_lib.Counter, using jar /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar 2014-09-26 23:51:24,971 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Writable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.io.Writable, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar 2014-09-26 23:51:24,974 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.hbase.mapreduce.TableOutputFormat, using jar /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar 2014-09-26 23:51:24,975 DEBUG [main] mapreduce.TableMapReduceUtil: For class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using jar /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar 2014-09-26 23:51:25,035 INFO [main] client.RMProxy: Connecting to ResourceManager at master2.hbase1a.test.cloud.domain.tv/172.16.0.145:8032 2014-09-26 23:51:25,191 INFO [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2014-09-26 23:51:25,247 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6bf3ede0 connecting to ZooKeeper ensemble= master1.hbasex1.test.cloud.domain.tv:2181, master0.hbasex1.test.cloud.domain.tv:2181, master2.hbasex1.test.cloud.domain.tv:2181 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=master2.hbase1a.test.cloud.domain.tv 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.7.0_65 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre 2014-09-26 23:51:25,252 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/usr/local/hbase/bin/../conf:/usr/lib/jvm/java/lib/tools.jar:/usr/local/hbase/bin/..:/usr/local/hbase/bin/../lib/activation-1.1.jar:/usr/local/hbase/bin/../lib/aopalliance-1.0.jar:/usr/local/hbase/bin/../lib/asm-3.1.jar:/usr/local/hbase/bin/../lib/avro-1.7.4.jar:/usr/local/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/bin/../lib/commons-cli-1.2.jar:/usr/local/hbase/bin/../lib/commons-codec-1.7.jar:/usr/local/hbase/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase/bin/../lib/commons-compress-1.4.1.jar:/usr/local/hbase/bin/../lib/commons-configuration-1.6.jar:/usr/local/hbase/bin/../lib/commons-daemon-1.0.13.jar:/usr/local/hbase/bin/../lib/commons-digester-1.8.jar:/usr/local/hbase/bin/../lib/commons-el-1.0.jar:/usr/local/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/local/hbase/bin/../lib/commons-io-2.4.jar:/usr/local/hbase/bin/../lib/commons-lang-2.6.jar:/usr/local/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hbase/bin/../lib/commons-math-2.1.jar:/usr/local/hbase/bin/../lib/commons-net-3.1.jar:/usr/local/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hbase/bin/../lib/gmbal-api-only-3.0.0-b023.jar:/usr/local/hbase/bin/../lib/grizzly-framework-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-server-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-servlet-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-rcm-2.1.2.jar:/usr/local/hbase/bin/../lib/guava-12.0.1.jar:/usr/local/hbase/bin/../lib/guice-3.0.jar:/usr/local/hbase/bin/../lib/guice-servlet-3.0.jar:/usr/local/hbase/bin/../lib/hamcrest-core-1.3.jar:/usr/local/hbase/bin/../lib/hbase-client-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-examples-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop2-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-prefix-tree-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-protocol-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-shell-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-testing-util-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-thrift-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/bin/../lib/htrace-core-2.04.jar:/usr/local/hbase/bin/../lib/httpclient-4.1.3.jar:/usr/local/hbase/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase/bin/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-xc-1.8.8.jar:/usr/local/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/bin/../lib/javax.inject-1.jar:/usr/local/hbase/bin/../lib/javax.servlet-3.1.jar:/usr/local/hbase/bin/../lib/javax.servlet-api-3.0.1.jar:/usr/local/hbase/bin/../lib/jaxb-api-2.2.2.jar:/usr/local/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/bin/../lib/jersey-client-1.9.jar:/usr/local/hbase/bin/../lib/jersey-core-1.8.jar:/usr/local/hbase/bin/../lib/jersey-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jersey-guice-1.9.jar:/usr/local/hbase/bin/../lib/jersey-json-1.8.jar:/usr/local/hbase/bin/../lib/jersey-server-1.8.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-core-1.9.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jets3t-0.6.1.jar:/usr/local/hbase/bin/../lib/jettison-1.3.1.jar:/usr/local/hbase/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase/bin/../lib/jruby-complete-1.6.8.jar:/usr/local/hbase/bin/../lib/jsch-0.1.42.jar:/usr/local/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsr305-1.3.9.jar:/usr/local/hbase/bin/../lib/junit-4.11.jar:/usr/local/hbase/bin/../lib/libthrift-0.9.0.jar:/usr/local/hbase/bin/../lib/log4j-1.2.17.jar:/usr/local/hbase/bin/../lib/management-api-3.0.0-b012.jar:/usr/local/hbase/bin/../lib/metrics-core-2.2.0.jar:/usr/local/hbase/bin/../lib/netty-3.6.6.Final.jar:/usr/local/hbase/bin/../lib/paranamer-2.3.jar:/usr/local/hbase/bin/../lib/protobuf-java-2.5.0.jar:/usr/local/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase/bin/../lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hbase/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase/bin/../lib/xz-1.0.jar:/usr/local/hbase/bin/../lib/zookeeper-3.4.6.jar:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/etc/hadoop/ 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop/lib 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.42-52.145.amzn1.x86_64 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hbase 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hbase 2014-09-26 23:51:25,253 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/usr/local/hbase-0.98.6.1-1a/bin 2014-09-26 23:51:25,254 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=master1.hbasex1.test.cloud.domain.tv:2181, master0.hbasex1.test.cloud.domain.tv:2181, master2.hbasex1.test.cloud.domain.tv:2181 sessionTimeout=90000 watcher=hconnection-0x6bf3ede0, quorum= master1.hbasex1.test.cloud.domain.tv:2181, master0.hbasex1.test.cloud.domain.tv:2181, master2.hbasex1.test.cloud.domain.tv:2181, baseZNode=/hbase 2014-09-26 23:51:25,270 INFO [main-SendThread( master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Opening socket connection to server master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181. Will not attempt to authenticate using SASL (unknown error) 2014-09-26 23:51:25,276 INFO [main-SendThread( master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket connection established to master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, initiating session 2014-09-26 23:51:25,282 INFO [main-SendThread( master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Session establishment complete on server master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, sessionid = 0x248afb6109b007f, negotiated timeout = 40000 2014-09-26 23:51:25,308 INFO [main] mapreduce.TableOutputFormat: Created table instance for moderation 2014-09-26 23:51:25,469 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0xa7a371 connecting to ZooKeeper ensemble= master1.hbase1a.test.cloud.domain.tv:2181, master0.hbase1a.test.cloud.domain.tv:2181, master2.hbase1a.test.cloud.domain.tv:2181 2014-09-26 23:51:25,469 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=master1.hbase1a.test.cloud.domain.tv:2181, master0.hbase1a.test.cloud.domain.tv:2181, master2.hbase1a.test.cloud.domain.tv:2181 sessionTimeout=90000 watcher=hconnection-0xa7a371, quorum= master1.hbase1a.test.cloud.domain.tv:2181, master0.hbase1a.test.cloud.domain.tv:2181, master2.hbase1a.test.cloud.domain.tv:2181, baseZNode=/hbase 2014-09-26 23:51:25,470 INFO [main-SendThread( master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Opening socket connection to server master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181. Will not attempt to authenticate using SASL (unknown error) 2014-09-26 23:51:25,470 INFO [main-SendThread( master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket connection established to master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, initiating session 2014-09-26 23:51:25,472 INFO [main-SendThread( master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Session establishment complete on server master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, sessionid = 0x34877671747028e, negotiated timeout = 40000 2014-09-26 23:51:25,490 INFO [main] util.RegionSizeCalculator: Calculating region sizes for table "moderation". 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region moderation,,1411774704618.68a1839eacfa355028522286caaf7cd5. has size 0 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region sizes calculated 2014-09-26 23:51:25,709 DEBUG [main] client.ClientSmallScanner: Finished with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2014-09-26 23:51:25,722 DEBUG [main] mapreduce.TableInputFormatBase: getSplits: split -> 0 -> ip-172-16-0-247.us-west-1.compute.internal:, 2014-09-26 23:51:25,739 INFO [main] mapreduce.JobSubmitter: number of splits:1 2014-09-26 23:51:25,748 INFO [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 2014-09-26 23:51:25,895 INFO [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1411771844660_0004 2014-09-26 23:51:26,033 INFO [main] impl.YarnClientImpl: Submitted application application_1411771844660_0004 2014-09-26 23:51:26,066 INFO [main] mapreduce.Job: The url to track the job: http://master2.hbase1a.test.cloud.domain.tv:8088/proxy/application_1411771844660_0004/ 2014-09-26 23:51:26,067 INFO [main] mapreduce.Job: Running job: job_1411771844660_0004 2014-09-26 23:51:29,082 INFO [main] mapreduce.Job: Job job_1411771844660_0004 running in uber mode : false 2014-09-26 23:51:29,084 INFO [main] mapreduce.Job: map 0% reduce 0% 2014-09-26 23:51:29,094 INFO [main] mapreduce.Job: Job job_1411771844660_0004 failed with state FAILED due to: Application application_1411771844660_0004 failed 2 times due to AM Container for appattempt_1411771844660_0004_000002 exited with exitCode: -1000 due to: File file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar does not exist .Failing this attempt.. Failing the application. 2014-09-26 23:51:29,111 INFO [main] mapreduce.Job: Counters: 0 The directory on the AM (master) from which the command is initiated as hbase user: [hbase@master2 bin]$ ls -lah /tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/ total 3.7M drwx------ 3 hbase hbase 4.0K Sep 26 23:51 . drwx------ 10 hbase hadoop 4.0K Sep 26 23:51 .. -rw-r--r-- 1 hbase hbase 3.5M Sep 26 23:51 job.jar -rw-rw-r-- 1 hbase hbase 28K Sep 26 23:51 .job.jar.crc -rw-r--r-- 1 hbase hbase 111 Sep 26 23:51 job.split -rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.split.crc -rw-r--r-- 1 hbase hbase 56 Sep 26 23:51 job.splitmetainfo -rw-rw-r-- 1 hbase hbase 12 Sep 26 23:51 .job.splitmetainfo.crc -rw-r--r-- 1 hbase hbase 102K Sep 26 23:51 job.xml -rw-rw-r-- 1 hbase hbase 824 Sep 26 23:51 .job.xml.crc drwx------ 2 hbase hbase 4.0K Sep 26 23:51 libjars The same directory is empty on each NM. Heres also the log from a node that got the job/application: 2014-09-26 23:51:26,635 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1411771844660_0004_000001 (auth:SIMPLE) 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1411771844660_0004_01_000001 by user hbase 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1411771844660_0004 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase IP=172.16.0.145 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1411771844660_0004 CONTAINERID=container_1411771844660_0004_01_000001 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1411771844660_0004 transitioned from NEW to INITING 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1411771844660_0004_01_000001 to application application_1411771844660_0004 2014-09-26 23:51:26,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1411771844660_0004 transitioned from INITING to RUNNING 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1411771844660_0004_01_000001 transitioned from NEW to LOCALIZING 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1411771844660_0004 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar transitioned from INIT to DOWNLOADING 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.splitmetainfo transitioned from INIT to DOWNLOADING 2014-09-26 23:51:26,640 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.split transitioned from INIT to DOWNLOADING 2014-09-26 23:51:26,641 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.xml transitioned from INIT to DOWNLOADING 2014-09-26 23:51:26,641 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_1411771844660_0004_01_000001 2014-09-26 23:51:26,645 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens. Credentials list: 2014-09-26 23:51:26,648 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user hbase 2014-09-26 23:51:26,661 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens to /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001.tokens 2014-09-26 23:51:26,661 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD set to /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 = file:/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 2014-09-26 23:51:26,696 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: DEBUG: FAILED { file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar, 1411775485000, PATTERN, (?:classes/|lib/).* }, File file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar does not exist 2014-09-26 23:51:26,697 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar(->/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/filecache/10/job.jar) transitioned from DOWNLOADING to FAILED 2014-09-26 23:51:26,697 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1411771844660_0004_01_000001 transitioned from LOCALIZING to LOCALIZATION_FAILED 2014-09-26 23:51:26,697 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl: Container container_1411771844660_0004_01_000001 sent RELEASE event on a resource request { file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar, 1411775485000, PATTERN, (?:classes/|lib/).* } not present in cache. 2014-09-26 23:51:26,698 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001 2014-09-26 23:51:26,698 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Unknown localizer with localizerId container_1411771844660_0004_01_000001 is sending heartbeat. Ordering it to DIE 2014-09-26 23:51:26,698 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: delete returned false for path: [/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001] 2014-09-26 23:51:26,698 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001 2014-09-26 23:51:26,699 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: delete returned false for path: [/data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001] 2014-09-26 23:51:26,699 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: LOCALIZATION_FAILED APPID=application_1411771844660_0004 CONTAINERID=container_1411771844660_0004_01_000001 2014-09-26 23:51:26,699 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1411771844660_0004_01_000001 transitioned from LOCALIZATION_FAILED to DONE 2014-09-26 23:51:26,699 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1411771844660_0004_01_000001 from application application_1411771844660_0004 2014-09-26 23:51:26,699 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1411771844660_0004 2014-09-26 23:51:27,629 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1411771844660_0004_01_000001] 2014-09-26 23:51:28,252 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1411771844660_0004_01_000001 2014-09-26 23:51:29,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1411771844660_0004 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1411771844660_0004 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1411771844660_0004 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED 2014-09-26 23:51:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler: Scheduling Log Deletion for application: application_1411771844660_0004, with delay of 10800 seconds Thank you in advance for any advice -- Iain Wright This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message.
